mirror of
https://github.com/magnus919/agent-skills.git
synced 2026-09-11 19:47:12 +03:00
390f3e3417
* feat: run isolated paired candidate and baseline skill evaluations Build the first complete paired skill-evaluation path: stage an immutable candidate, run matched candidate and baseline trials in clean environments, execute deterministic outcome graders, and produce a case-level comparison report. - eval_runner/sandbox.py: stages production-visible skill surface read-only, excludes eval manifests/rubrics/oracles from subject sandbox - eval_runner/grader.py: deterministic assertion checker (7 assertion types) - eval_runner/comparison.py: paired comparison report generation - eval_runner/paired.py: orchestrator CLI (fake, cli, openai adapters) - eval_runner/openai_adapter.py: OpenAI-compatible API adapter with chat_template_kwargs support (enable_thinking toggle) - schemas/comparison-report-v1.schema.json: report schema - .github/workflows/skill-eval.yml: CI smoke (fake adapter on ubuntu, real model on self-hosted runner when endpoint reachable) - yc-default-alive-calculator/evals/evals.json: initial 5-case eval manifest Verified against google_gemma-4-26B-A4B-it-IQ4_XS.gguf: 5/5 candidate improvements, 0 regressions. Closes #105 * ci: make paired-eval-model job non-blocking The self-hosted runner may not always be online. Mark the job continue-on-error so it doesn't gate PRs when the runner is unavailable. * ci: isolate model evals from pull requests * fix(raleigh): test arrivals against a daily route, not weekday-only The fixture only had a WEEK (Mon-Fri) service, so test_get_arrivals_for_stop returned 0 arrivals on weekends when _today_date() fell on Saturday/Sunday. Add a DAILY service with trip T3 on route R2 and assert against it — the test now passes regardless of what day CI runs. * ci: trigger checks on amended commit --------- Co-authored-by: magnus919 <magnus919>