mirror of
https://github.com/magnus919/agent-skills.git
synced 2026-09-22 00:56:35 +03:00
Magnus HedemarkandGitHub
347c1d3ae7
* feat(skill): add training data annotation methodology * docs(training-data-annotation): register skill and clarify stopping boundary
20 lines
1.0 KiB
Markdown
20 lines
1.0 KiB
Markdown
# Annotator quality and disagreement
|
|
|
|
Qualify people against the actual task, language/domain needs, accessibility,
|
|
and confidentiality requirements. Provide practice items with feedback before
|
|
production work; measure fatigue, throughput, abstention, and error patterns, not
|
|
just speed.
|
|
|
|
Use blinded calibration items and periodic blind repeats. Choose agreement or
|
|
reliability statistics appropriate to nominal, ordinal, continuous, span, or
|
|
ranking labels; route the choice and uncertainty calculation to `data-scientist`.
|
|
Agreement is evidence about reproducibility under the guide, not proof that the
|
|
majority is correct.
|
|
|
|
Classify disagreement as ambiguity, missing evidence, guide defect, annotator
|
|
mistake, or model-induced anchoring. Adjudication records the decision rule,
|
|
authority, evidence, and whether already labeled items need re-review. Preserve
|
|
the original labels and disagreement for audit. If preannotations are visible,
|
|
compare with a blinded control to estimate anchoring and treat the result as a
|
|
workflow effect.
|