mirror of https://github.com/magnus919/agent-skills.git synced 2026-09-14 21:16:36 +03:00

Files

T

92299e1238 feat(skill): beef up ml-engineering with scripts/templates/evals (#257 )

Add a schema-valid eval manifest (6 cases: fine-tuning plan review, eval-set
design, quantization decision, deployment plan, regression triage, training-run
reproducibility), three fillable templates (training-run record, eval regression
table, quantization decision record), a stdlib eval-set overlap/leakage checker
with a unittest suite, routing to the llama-cpp tool skill, and a README Quick
Start documenting the script. Closes #240.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>

2026-08-03 15:25:06 -04:00

1.7 KiB

Raw Blame History

Eval Regression Table

Use this table to track model quality across runs and to make regression triage auditable. One row per eval case or capability subset, one table per model change being compared. Every number must name the eval set version that produced it.

Change Under Evaluation

Change / run IDs compared: [fill: e.g. ft-support-lora-013 vs ft-support-lora-014]
Date: [fill: YYYY-MM-DD]
Evaluator: [fill: name or handle]
Eval set version: [fill: eval set revision/hash — do not compare across versions]
Leakage check status: [fill: overlap-check result between train and eval corpora]

Results

Eval case / capability subset	Baseline score	New score	Delta	Noise estimate	Verdict	Notes
`[fill: case or subset name]`	`[fill: baseline value]`	`[fill: new value]`	`[fill: signed delta]`	`[fill: e.g. +/-0.02 from repeat runs]`	`[fill: regression / improvement / no change]`	`[fill: what changed and why]`
`[fill: ...]`	`[fill: ...]`	`[fill: ...]`	`[fill: ...]`	`[fill: ...]`	`[fill: ...]`	`[fill: ...]`

Analysis

Aggregate verdict: [fill: e.g. "overall +3%, but multi-turn subset regressed"]
Is the delta beyond noise? [fill: yes/no and the evidence]
Per-example inspection of any regression: [fill: link or notes for the regressed cases]
Root-cause hypothesis: [fill: data mix drift, overfitting, trade-off, eval artifact, other]
Contamination check: [fill: re-run the overlap checker if a subset dropped suspiciously]

Decision

Decision: [fill: ship / retrain / tune / roll back]
Rollback or follow-up action: [fill: adapter version to revert to, retrain ticket, etc.]
Sign-off: [fill: who approved and when]

1.7 KiB Raw Blame History