mirror of
https://github.com/magnus919/agent-skills.git
synced 2026-09-19 07:27:04 +03:00
Magnus HedemarkandGitHub
dde5fa1020
Merge authorized after exact-head validation and paired evaluation checks passed. The follow-up Droid review run on head df31b25 stalled in the model step and ended with an automation error; its actionable findings from the prior review were fixed and independently verified.
29 lines
2.0 KiB
Markdown
29 lines
2.0 KiB
Markdown
# Cleaning operations and failure modes
|
|
|
|
## Missing values
|
|
Distinguish unknown, not applicable, not collected, refused, suppressed, and structurally absent. Map sentinels only with source evidence and counts. Dropping requires an explicit bias and count-loss rationale. Imputation requires a method, fit boundary, retained indicator, and sensitivity check.
|
|
|
|
## Duplicates and entities
|
|
Define duplicates at the intended grain. Check exact duplicates, duplicate keys, and near-duplicates separately. For conflicts, document survivorship or quarantine. Fuzzy matching generates candidates, not truth: retain fields, scores, thresholds, decisions, and review.
|
|
|
|
## Types, dates, units
|
|
Declare number locale, date format, timezone, and error policy. Quarantine parse failures. Convert units only when both units are known; preserve original unit and conversion. Check cross-field rules such as start ≤ end and subtotal = components.
|
|
|
|
## Text and categories
|
|
Normalize Unicode and whitespace conservatively, preserving source text. Use versioned code lists with unknown/unmapped states. Do not collapse rare categories merely because they are rare.
|
|
|
|
## Outliers, joins, reshape
|
|
Investigate outliers before clipping or deletion. Before joins, assert key uniqueness and expected cardinality, measure unmatched keys and row multiplication, and reconcile totals. For pivot/melt, state the unique key and aggregation rule.
|
|
|
|
## Failure modes
|
|
- Silent coercion turns bad values into nulls.
|
|
- Leakage learns imputation, encodings, or deduplication from holdout/future data.
|
|
- Over-cleaning deletes valid rare events.
|
|
- Many-to-many joins masquerade as new observations.
|
|
- Weak identifiers create false duplicate/entity matches.
|
|
- Locale and timezone assumptions corrupt values.
|
|
- Replacement characters or mojibake are repaired without evidence.
|
|
- Schema drift passes parsing but violates downstream assumptions.
|
|
- Non-idempotent steps change already-clean data on rerun.
|
|
- Unbounded profiling exhausts resources before a plan exists.
|