mirror of
https://github.com/magnus919/agent-skills.git
synced 2026-09-22 00:56:35 +03:00
Magnus HedemarkandGitHub
9917fb455b
* feat(data): validate AI transformation and repair boundaries * docs(data): expose AI transformation and repair triggers * docs(data-engineering): preserve the full reference index
49 lines
2.4 KiB
Markdown
49 lines
2.4 KiB
Markdown
# data-cleaning
|
|
|
|
A practical, evidence-first workflow for turning messy data into trustworthy, reviewable datasets.
|
|
|
|
## Why Install This Skill
|
|
|
|
Messy data is not solved by a handful of `dropna()` calls. This skill helps an agent discover what is wrong, decide what may safely change, preserve what was observed, and demonstrate that the result still matches the intended grain and meaning.
|
|
|
|
It works across CSV, JSON, text, DataFrames, SQL extracts, and pipeline boundaries. It combines a repeatable methodology with tool-selection guidance, reusable plans and reports, and a dependency-free profiling script that produces machine-readable evidence before mutation.
|
|
|
|
## What You Get
|
|
|
|
| Path | Purpose |
|
|
|---|---|
|
|
| `SKILL.md` | Core workflow and completion gate |
|
|
| `references/` | Methodology, operations, validation, tools, sources |
|
|
| `templates/` | Cleaning plan, decision log, transformation log, exception register, schema contract, quality report |
|
|
| `scripts/profile_dataset.py` | Read-only CSV/TSV/JSONL profiler |
|
|
| `scripts/reconcile_dataset.py` | Read-only key, row-count, and aggregate reconciliation |
|
|
| `scripts/test_*` | Deterministic tests for bundled scripts |
|
|
| `evals/evals.json` | Output-quality evaluation cases |
|
|
|
|
## Quick Start
|
|
|
|
```bash
|
|
python3 scripts/profile_dataset.py input.csv --output profile.json
|
|
python3 scripts/profile_dataset.py input.csv --max-rows 10000
|
|
```
|
|
|
|
The profiler does not edit the input. Use its report to fill `templates/cleaning-plan.md`, then validate the transformed output against `templates/schema-contract.yml` or a project-specific contract.
|
|
|
|
For AI-assisted data work, the [boundary workflow](references/ai-repair-review.md) helps keep uncertain
|
|
proposals out of trusted datasets. Use the [companion record](templates/ai-repair-ledger.csv) to retain
|
|
validation and review evidence.
|
|
|
|
## Triggers
|
|
|
|
- Clean or standardize CSV, JSON, text, spreadsheet exports, or DataFrames
|
|
- Diagnose missing values, duplicates, malformed types, dates, encodings, or categories
|
|
- Design a reusable cleaning pipeline or data-quality contract
|
|
- Choose among pandas, Polars, pyjanitor, Pandera, Great Expectations, OpenRefine, Frictionless, dbt tests, or Spark-scale tools
|
|
- Review whether cleaning is reproducible, safe, or leakage-free
|
|
|
|
## Requirements
|
|
|
|
- Python 3.9+ for the bundled script
|
|
- No external dependency for first-pass profiling
|
|
- Optional ecosystem tools require their own installations
|