mirror of
https://github.com/magnus919/agent-skills.git
synced 2026-09-21 08:36:33 +03:00
feat: eval coverage ratchet — gate new skills, track coverage, ratchet thresholds (#99)
Phase 1: New skills (not in grandfathered-skills.txt) must have evals/evals.json with at least 5 test cases. All 107 existing skills are grandfathered. Phase 2: scripts/eval-coverage.py reports coverage (skills with/without evals, case counts, reference-priority sorting). Added as informational CI step. Phase 3: Ratchet thresholds — at 25% coverage, modified skills without evals get a warning; at 50%, they fail CI. Enforced via --modified-from flag for PR-scoped checks. Closes #90
This commit is contained in:
@@ -34,6 +34,8 @@ jobs:
|
||||
run: ruby scripts/gen-claude-marketplace.rb
|
||||
- name: Validate Codex plugin packaging
|
||||
run: ruby scripts/gen-codex-plugin.rb
|
||||
- name: Report eval coverage
|
||||
run: python3 scripts/eval-coverage.py
|
||||
- name: Test llms.txt generator
|
||||
run: ruby scripts/test-gen-llms-txt.rb
|
||||
- name: Validate llms.txt catalog
|
||||
|
||||
Reference in New Issue
Block a user