4 Commits

Include renames

Author	SHA1	Message	Date
magnus919andMagnus Hedemark	cbe1b6927a	fix: reject duplicate eval fixture paths	2026-07-25 01:11:02 -04:00
magnus919andMagnus Hedemark	cb1579f23d	fix: harden paired evaluation isolation	2026-07-25 01:11:02 -04:00
3 people	0a80bda800	feat: add release-grade repeated and rubric-based skill evaluation (#136) ... Add release evaluation layer on top of deterministic paired execution: - schemas/release-eval-v1.schema.json: release report schema with freeze snapshot, per-case trial aggregation, rubric graders, blinded pairwise comparison, calibration tracking, and PASS/CONDITIONAL/HOLD/BLOCK outcomes - eval_runner/release.py: core module for multi-trial aggregation, versioned rubric graders with abstain/insufficient-evidence, pairwise planning with position randomization and order-reversal testing, calibration records, and release decision computation - eval_runner/tests/test_release.py: 34 tests covering all acceptance criteria - schemas/evals-v1.schema.json: optional case_set field (dev/regression/release) - eval_runner/models.py: case_set on EvalCase - eval_runner/runner.py: load case_set from manifest - .github/workflows/skill-eval.yml: run release tests in CI Gate semantics: hard invariants (privacy, auth, destructive) tolerate zero violations and cannot be averaged away. Missing evidence produces HOLD, not PASS. Uncalibrated judge results are advisory only. Closes #106 Co-authored-by: magnus919 <magnus919>	2026-07-24 23:26:07 -04:00
3 people	e83558a6e3	feat: add harness adapter contract and reproducible eval run artifacts (#130) ... Implements #104. Adds a repository-level evaluation runner with: - Typed HarnessAdapter Protocol (adapter.py) - Dataclass models for AdapterInput/AdapterOutput (models.py) - FakeAdapter for deterministic CI without credentials (fake_adapter.py) - CliSubprocessAdapter for non-interactive CLI harnesses (cli_adapter.py) - Run manifest builder with schema validation (manifest.py) - Runner CLI entry point (runner.py, __main__.py) - JSON Schema for trial manifests (schemas/run-manifest-v1.schema.json) - Unit tests including schema validation (tests/test_runner.py) Co-authored-by: magnus919 <magnus919>	2026-07-24 18:46:18 -04:00