2 Commits

Include renames

Author	SHA1	Message	Date
usernameandfactory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>	476d7e11b0	feat(ci): add linting, formatting, coverage, and security configs ... Add ruff linter/formatter with pre-commit hooks, pytest-cov with 60% coverage threshold, CODEOWNERS, Dependabot for pip/GHA updates, and .env.example. Auto-fix existing ruff violations across eval_runner/ and scripts/. 10 agent-readiness criteria resolved: lint_config, formatter, pre_commit_hooks, naming_consistency, dead_code_detection, test_coverage_thresholds, test_performance_tracking, codeowners, dependency_update_automation, env_template. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>	2026-07-29 17:43:41 -04:00
3 people	390f3e3417	feat: run isolated paired candidate and baseline skill evaluations (#133) ... * feat: run isolated paired candidate and baseline skill evaluations Build the first complete paired skill-evaluation path: stage an immutable candidate, run matched candidate and baseline trials in clean environments, execute deterministic outcome graders, and produce a case-level comparison report. - eval_runner/sandbox.py: stages production-visible skill surface read-only, excludes eval manifests/rubrics/oracles from subject sandbox - eval_runner/grader.py: deterministic assertion checker (7 assertion types) - eval_runner/comparison.py: paired comparison report generation - eval_runner/paired.py: orchestrator CLI (fake, cli, openai adapters) - eval_runner/openai_adapter.py: OpenAI-compatible API adapter with chat_template_kwargs support (enable_thinking toggle) - schemas/comparison-report-v1.schema.json: report schema - .github/workflows/skill-eval.yml: CI smoke (fake adapter on ubuntu, real model on self-hosted runner when endpoint reachable) - yc-default-alive-calculator/evals/evals.json: initial 5-case eval manifest Verified against google_gemma-4-26B-A4B-it-IQ4_XS.gguf: 5/5 candidate improvements, 0 regressions. Closes #105 * ci: make paired-eval-model job non-blocking The self-hosted runner may not always be online. Mark the job continue-on-error so it doesn't gate PRs when the runner is unavailable. * ci: isolate model evals from pull requests * fix(raleigh): test arrivals against a daily route, not weekday-only The fixture only had a WEEK (Mon-Fri) service, so test_get_arrivals_for_stop returned 0 arrivals on weekends when _today_date() fell on Saturday/Sunday. Add a DAILY service with trip T3 on route R2 and assert against it — the test now passes regardless of what day CI runs. * ci: trigger checks on amended commit --------- Co-authored-by: magnus919 <magnus919>	2026-07-24 23:12:44 -04:00