# data-cleaning

A practical, evidence-first workflow for turning messy data into trustworthy, reviewable datasets.

## Why Install This Skill

Messy data is not solved by a handful of `dropna()` calls. This skill helps an agent discover what is wrong, decide what may safely change, preserve what was observed, and demonstrate that the result still matches the intended grain and meaning.

It works across CSV, JSON, text, DataFrames, SQL extracts, and pipeline boundaries. It combines a repeatable methodology with tool-selection guidance, reusable plans and reports, and a dependency-free profiling script that produces machine-readable evidence before mutation.

## What You Get

| Path | Purpose |
|---|---|
| `SKILL.md` | Core workflow and completion gate |
| `references/` | Methodology, operations, validation, tools, sources |
| `templates/` | Cleaning plan, decision log, transformation log, exception register, schema contract, quality report |
| `scripts/profile_dataset.py` | Read-only CSV/TSV/JSONL profiler |
| `scripts/reconcile_dataset.py` | Read-only key, row-count, and aggregate reconciliation |
| `scripts/test_*` | Deterministic tests for bundled scripts |
| `evals/evals.json` | Output-quality evaluation cases |

## Quick Start

```bash
python3 scripts/profile_dataset.py input.csv --output profile.json
python3 scripts/profile_dataset.py input.csv --max-rows 10000
```

The profiler does not edit the input. Use its report to fill `templates/cleaning-plan.md`, then validate the transformed output against `templates/schema-contract.yml` or a project-specific contract.

For AI-assisted data work, the [boundary workflow](references/ai-repair-review.md) helps keep uncertain
proposals out of trusted datasets. Use the [companion record](templates/ai-repair-ledger.csv) to retain
validation and review evidence.

## Triggers

- Clean or standardize CSV, JSON, text, spreadsheet exports, or DataFrames
- Diagnose missing values, duplicates, malformed types, dates, encodings, or categories
- Design a reusable cleaning pipeline or data-quality contract
- Choose among pandas, Polars, pyjanitor, Pandera, Great Expectations, OpenRefine, Frictionless, dbt tests, or Spark-scale tools
- Review whether cleaning is reproducible, safe, or leakage-free

## Requirements

- Python 3.9+ for the bundled script
- No external dependency for first-pass profiling
- Optional ecosystem tools require their own installations