mirror of
https://github.com/magnus919/agent-skills.git
synced 2026-09-14 13:06:30 +03:00
Magnus HedemarkandGitHub
dde5fa1020
Merge authorized after exact-head validation and paired evaluation checks passed. The follow-up Droid review run on head df31b25 stalled in the model step and ended with an automation error; its actionable findings from the prior review were fixed and independently verified.
1.5 KiB
1.5 KiB
Data-cleaning tool selection
Choose the smallest tool that proves the needed contract.
| Tool/family | Best fit | Boundary |
|---|---|---|
| pandas | moderate in-memory tabular work | memory-bound; declare types |
| Polars | fast local/lazy tabular work | semantics and dtypes differ from pandas |
| pyjanitor | readable pandas cleaning verbs | convenience, not a quality contract |
| Pandera | Python DataFrame schemas/checks | rules must encode domain meaning |
| Great Expectations | named suites, checkpoints, reports | pin current API/version |
| ydata-profiling | exploratory HTML/JSON profiles | discovery, not validation |
| ftfy | Unicode/mojibake repair | inspect potentially changed text |
| Cerberus/Pydantic | nested JSON records | not relational/distribution checks |
| OpenRefine | interactive exports/entity review | human judgment is required |
| Frictionless | tabular/package validation | add custom business checks |
| dbt data tests | SQL models and warehouse boundaries | SQL/warehouse-oriented |
| Deequ/PyDeequ | Spark-scale metrics/constraints | JVM/Spark compatibility cost |
Selection: classify boundary and scale; choose transformation engine separately from validator; use declarative contracts at stable boundaries; keep a dependency-free triage fallback; pin versions; pilot human reconciliation and retain decisions. The orientation article names pyjanitor, Great Expectations, ftfy, ydata-profiling, and Cerberus; primary documentation governs behavior.