Add release evaluation layer on top of deterministic paired execution:
- schemas/release-eval-v1.schema.json: release report schema with freeze
snapshot, per-case trial aggregation, rubric graders, blinded pairwise
comparison, calibration tracking, and PASS/CONDITIONAL/HOLD/BLOCK outcomes
- eval_runner/release.py: core module for multi-trial aggregation, versioned
rubric graders with abstain/insufficient-evidence, pairwise planning with
position randomization and order-reversal testing, calibration records,
and release decision computation
- eval_runner/tests/test_release.py: 34 tests covering all acceptance criteria
- schemas/evals-v1.schema.json: optional case_set field (dev/regression/release)
- eval_runner/models.py: case_set on EvalCase
- eval_runner/runner.py: load case_set from manifest
- .github/workflows/skill-eval.yml: run release tests in CI
Gate semantics: hard invariants (privacy, auth, destructive) tolerate zero
violations and cannot be averaged away. Missing evidence produces HOLD, not
PASS. Uncalibrated judge results are advisory only.
Closes#106
Co-authored-by: magnus919 <magnus919>
magnus919andMagnus Hedemark
