mirror of
https://github.com/magnus919/agent-skills.git
synced 2026-09-18 15:06:28 +03:00
0a80bda800
Add release evaluation layer on top of deterministic paired execution: - schemas/release-eval-v1.schema.json: release report schema with freeze snapshot, per-case trial aggregation, rubric graders, blinded pairwise comparison, calibration tracking, and PASS/CONDITIONAL/HOLD/BLOCK outcomes - eval_runner/release.py: core module for multi-trial aggregation, versioned rubric graders with abstain/insufficient-evidence, pairwise planning with position randomization and order-reversal testing, calibration records, and release decision computation - eval_runner/tests/test_release.py: 34 tests covering all acceptance criteria - schemas/evals-v1.schema.json: optional case_set field (dev/regression/release) - eval_runner/models.py: case_set on EvalCase - eval_runner/runner.py: load case_set from manifest - .github/workflows/skill-eval.yml: run release tests in CI Gate semantics: hard invariants (privacy, auth, destructive) tolerate zero violations and cannot be averaged away. Missing evidence produces HOLD, not PASS. Uncalibrated judge results are advisory only. Closes #106 Co-authored-by: magnus919 <magnus919>