mirror of
https://github.com/magnus919/agent-skills.git
synced 2026-09-17 14:36:29 +03:00
Magnus HedemarkandGitHub
dde5fa1020
Merge authorized after exact-head validation and paired evaluation checks passed. The follow-up Droid review run on head df31b25 stalled in the model step and ended with an automation error; its actionable findings from the prior review were fixed and independently verified.
30 lines
2.0 KiB
Markdown
30 lines
2.0 KiB
Markdown
# Data-cleaning methodology
|
|
|
|
## Definition
|
|
|
|
There is no universal clean dataset. Define quality relative to a use case, data contract, and unit of observation. Record intended grain, keys, units, temporal coverage, acceptable error rate, and the decision the data supports.
|
|
|
|
## Lifecycle
|
|
|
|
**Scope → acquire → preserve → profile → diagnose → decide → transform → validate → review → publish → monitor.**
|
|
|
|
### Scope and preservation
|
|
|
|
Write a brief naming consumer, source, time boundary, privacy constraints, and failure criteria. Capture source identity, retrieval time, content hash, encoding, delimiter, schema, and tool versions. Keep immutable raw input, a staged copy, and rejects/quarantine output.
|
|
|
|
### Profile and diagnose
|
|
|
|
Inspect structure, field-level nullness/cardinality/types/ranges, record duplicates, relational keys, temporal freshness, units, text, and distributions. Compare with a known-good baseline. A profile is a hypothesis generator, not permission to auto-fix: rare, new, or extreme values may be real.
|
|
|
|
### Decide and transform
|
|
|
|
Prefer: preserve valid observations; standardize representation; repair only when the corruption mechanism is defensible; impute only with a stated missingness method and fit boundary; quarantine unsafe values; escalate semantic ambiguity. Keep original and normalized values when change is lossy. Make transformations deterministic and idempotent.
|
|
|
|
### Validate and release
|
|
|
|
Reconcile counts, sums, key coverage, category counts, and time ranges. Inspect every class of changed or rejected record. Use holdout/time-split checks when rules are learned. Domain approval is required for deletion, imputation, entity matching, unit conversion, and business-rule changes.
|
|
|
|
## Audit record
|
|
|
|
For each rule retain: ID, input columns, predicate, action, before/after counts, affected identifiers or privacy-safe sample, rationale, confidence, owner, timestamp, code/version, and rollback path. A report that only says “cleaned successfully” is not auditable.
|