fix: enforce eval coverage ratchet for complete changed skill directories (#107)

Wire the Phase 3 ratchet into CI by passing the PR base SHA to
eval-coverage.py --modified-from. Expand changed-skill detection from
SKILL.md-only diffs to the entire skill directory so that references,
scripts, fixtures, README, and eval manifest edits all count as
modifications. Add a monotonic coverage floor that fails CI when
coverage decreases between the base and candidate revisions.

Add script tests for ratchet-mode detection and coverage-decrease
behaviour. Update AGENTS.md and CONTRIBUTING.md to describe the
behaviour CI now enforces.

Closes #102

Signed-off-by: Magnus Hedemark <magnus919@pm.me>
This commit is contained in:
Magnus Hedemark
2026-07-22 01:51:57 -04:00
committed by GitHub
parent 9b235621d0
commit fc8a8952ee
5 changed files with 298 additions and 9 deletions
+4
View File
@@ -36,6 +36,10 @@ jobs:
run: ruby scripts/gen-codex-plugin.rb
- name: Report eval coverage
run: python3 scripts/eval-coverage.py
- name: Enforce eval coverage ratchet
env:
EVAL_RATCHET_BASE: ${{ github.event.pull_request.base.sha || github.event.before }}
run: python3 scripts/eval-coverage.py --modified-from "$EVAL_RATCHET_BASE"
- name: Test llms.txt generator
run: ruby scripts/test-gen-llms-txt.rb
- name: Validate llms.txt catalog