2 Commits

Include renames

Author	SHA1	Message	Date
3 people	390f3e3417	feat: run isolated paired candidate and baseline skill evaluations (#133) ... * feat: run isolated paired candidate and baseline skill evaluations Build the first complete paired skill-evaluation path: stage an immutable candidate, run matched candidate and baseline trials in clean environments, execute deterministic outcome graders, and produce a case-level comparison report. - eval_runner/sandbox.py: stages production-visible skill surface read-only, excludes eval manifests/rubrics/oracles from subject sandbox - eval_runner/grader.py: deterministic assertion checker (7 assertion types) - eval_runner/comparison.py: paired comparison report generation - eval_runner/paired.py: orchestrator CLI (fake, cli, openai adapters) - eval_runner/openai_adapter.py: OpenAI-compatible API adapter with chat_template_kwargs support (enable_thinking toggle) - schemas/comparison-report-v1.schema.json: report schema - .github/workflows/skill-eval.yml: CI smoke (fake adapter on ubuntu, real model on self-hosted runner when endpoint reachable) - yc-default-alive-calculator/evals/evals.json: initial 5-case eval manifest Verified against google_gemma-4-26B-A4B-it-IQ4_XS.gguf: 5/5 candidate improvements, 0 regressions. Closes #105 * ci: make paired-eval-model job non-blocking The self-hosted runner may not always be online. Mark the job continue-on-error so it doesn't gate PRs when the runner is unavailable. * ci: isolate model evals from pull requests * fix(raleigh): test arrivals against a daily route, not weekday-only The fixture only had a WEEK (Mon-Fri) service, so test_get_arrivals_for_stop returned 0 arrivals on weekends when _today_date() fell on Saturday/Sunday. Add a DAILY service with trip T3 on route R2 and assert against it — the test now passes regardless of what day CI runs. * ci: trigger checks on amended commit --------- Co-authored-by: magnus919 <magnus919>	2026-07-24 23:12:44 -04:00
3 people	e83558a6e3	feat: add harness adapter contract and reproducible eval run artifacts (#130) ... Implements #104. Adds a repository-level evaluation runner with: - Typed HarnessAdapter Protocol (adapter.py) - Dataclass models for AdapterInput/AdapterOutput (models.py) - FakeAdapter for deterministic CI without credentials (fake_adapter.py) - CliSubprocessAdapter for non-interactive CLI harnesses (cli_adapter.py) - Run manifest builder with schema validation (manifest.py) - Runner CLI entry point (runner.py, __main__.py) - JSON Schema for trial manifests (schemas/run-manifest-v1.schema.json) - Unit tests including schema validation (tests/test_runner.py) Co-authored-by: magnus919 <magnus919>	2026-07-24 18:46:18 -04:00