feat: eval coverage ratchet — gate new skills, track coverage, ratchet thresholds (#99)

Phase 1: New skills (not in grandfathered-skills.txt) must have
evals/evals.json with at least 5 test cases. All 107 existing skills
are grandfathered.

Phase 2: scripts/eval-coverage.py reports coverage (skills with/without
evals, case counts, reference-priority sorting). Added as informational
CI step.

Phase 3: Ratchet thresholds — at 25% coverage, modified skills without
evals get a warning; at 50%, they fail CI. Enforced via
--modified-from flag for PR-scoped checks.

Closes #90
This commit is contained in:
Magnus Hedemark
2026-07-21 21:03:21 -04:00
committed by GitHub
parent dd457f1170
commit be0c8df5e2
4 changed files with 340 additions and 0 deletions
+2
View File
@@ -34,6 +34,8 @@ jobs:
run: ruby scripts/gen-claude-marketplace.rb
- name: Validate Codex plugin packaging
run: ruby scripts/gen-codex-plugin.rb
- name: Report eval coverage
run: python3 scripts/eval-coverage.py
- name: Test llms.txt generator
run: ruby scripts/test-gen-llms-txt.rb
- name: Validate llms.txt catalog