magnus919_agent-skills/lifecycle-evals/references/regression-detection.md

Regression Detection — Lifecycle Evaluation Corpus

This document defines how to detect and interpret a regression in the lifecycle evaluation corpus across revisions, and how to tell a real regression from a benign content change.

What counts as a regression

A corpus regression is any change that silently reduces the corpus's ability to exercise its required coverage or that invalidates durable evidence references. Concretely:

1. Case removal or renaming. Eval case IDs are durable evidence references (VAL-EVL-005, VAL-CRP-024). Removing a case, or renaming an ID, breaks the mapping in references/coverage-index.json, any committed run artifacts that reference the ID, and any downstream evidence that cites the ID. Never rename an ID; add a new case with a new ID instead.
2. Behavioral-category or integrated-scenario coverage loss. Removing the last case tagged for a behavioral category or integrated scenario makes the corpus fail its mandatory coverage (VAL-CRP-003..008, VAL-CRP-010..015). The machine-checkable gate is validate-corpus-coverage.py, which fails when any of the 5 categories or 6 scenarios has no tagged case.
3. Assertion-set drift on a tagged case. If a case's assertions no longer verify the category's required handling (e.g., a stop/retire case stops asserting the accountable owner, or a handoff assertion is dropped from an integrated case), the corpus silently loses the guarantee that the category/scenario is actually exercised. The committed run artifacts pin the assertion sets at snapshot time; a re-run that changes assertion sets is a signal to review.
4. Fixture-hash changes. Every per-trial manifest records case.prompt_hash and case.fixture_hashes. A change to a prompt or to a referenced fixture changes those hashes. For self-contained cases (inputs inline in prompts) a prompt change is a deliberate content change that should be reviewed against the case's tag; a fixture change on a files-referencing case changes the fixture hash and must be reconciled with references/sources.md and the repository fixture-resolution validator (validate-evals.py).

Comparison procedure (re-run with the fake adapter)

The corpus is deterministic under the fake adapter (no model, no network, no randomness in execution — only timestamps and trial UUIDs vary). To compare two revisions:

# On the old revision (e.g., the merged baseline):
git worktree add /tmp/corpus-old <old-sha>
cd /tmp/corpus-old && bash lifecycle-evals/scripts/run-corpus.sh   # CORPUS_OUT_DIR=/tmp/corpus-runs-old

# On the new revision (the candidate):
cd /Volumes/tank01/magnus/git/agent-skills-issue-204
CORPUS_OUT_DIR=/tmp/corpus-runs-new bash lifecycle-evals/scripts/run-corpus.sh

Then compare per case:

1. Per-case status: every trial must be status == "completed" in both runs (a trial that becomes error/timeout/stopped between revisions is a regression).
2. Per-case identity: the set of case_ids per manifest must be equal between revisions (no removals, no renames).
3. Per-case content: compare case.prompt_hash and the case.fixture_hashes fields in the per-trial manifests. Changed hashes indicate the case content or fixture changed and must be reviewed (see interpretation below).
4. Coverage: run validate-corpus-coverage.py on the candidate; it must exit 0 (all 5 categories, all 6 scenarios covered, every referenced ID present, index current).

A simple diff-oriented check across the two output trees:

diff <(cd /tmp/corpus-runs-old && find . -name '*.manifest.json' | sort) \
     <(cd /tmp/corpus-runs-new && find . -name '*.manifest.json' | sort)

Note that file names embed the trial UUID prefix (<case_id>--<trial_id[:8]>.manifest.json), so compare by case_id sets and by hashes rather than by file name.

Case-ID stability rule

Never rename an eval case ID. IDs are referenced by the coverage index, the coverage matrix, committed run artifacts, and (potentially) external evidence ledgers. Renaming an ID is a regression even when the content is unchanged. To evolve a case: keep the ID, update content, regenerate the index, re-run the corpus, and refresh the run-artifact snapshot at merge time. To add coverage: add a new case with a new lowercase-hyphen ID (≤ 64 chars, unique within its manifest).

Ratchet command

The repository's eval-coverage ratchet must hold on every corpus change:

.venv/bin/python scripts/eval-coverage.py --modified-from origin/main

This exits 0 only when no modified skill lacks a schema-valid manifest and coverage does not decrease versus the base. All 17 corpus manifests are schema-valid, so corpus changes never trip the modified-skill ratchet; the check still runs in CI on every PR.

Fixture-resolution is enforced by the repository validator:

.venv/bin/python scripts/validate-evals.py

Interpreting a change: regression vs. benign content change

Observation	Classification	Required action
A case ID disappears from a manifest	Regression	Restore the case or (if truly obsolete) re-scope: add a replacement case, update the index and matrix, re-run, and record the replacement in the PR body; never silently drop the ID.
A case ID is renamed	Regression	Revert the rename; change content only, or add a new ID.
The last case for a category/scenario is removed or untagged	Regression	validate-corpus-coverage.py fails; restore coverage before merging.
A prompt/assertion is edited to tighten wording without changing the scenario or the category-required handling	Benign content change	Update the run-artifact snapshot at merge time (one snapshot per merge, VAL-CRP-021 ambiguity C); no re-review of category coverage needed beyond validate-corpus-coverage.py.
A prompt is changed so the case now exercises a different scenario, or a category-required assertion is dropped	Material content change	Re-tag the case in the coverage index, regenerate the matrix and index, re-run the corpus, and re-verify the case still satisfies its category's required handling.
A files fixture changes	Material content change	Update references/sources.md (provenance), re-run, and confirm validate-evals.py still resolves the fixture.
Timestamps/UUIDs differ between fake runs	Benign	Expected; timestamps are the scoping/date evidence and are excluded from content comparison.

In all cases, the decision is recorded in the change's PR body or evidence ledger so a future reviewer can see why the corpus changed.

Keeping the index and matrix current

After any case content, ID, or tag change:

.venv/bin/python lifecycle-evals/scripts/validate-corpus-coverage.py --write-index
# then regenerate the human-readable matrix from the index (see coverage-matrix.md header),
# re-run the corpus, and commit the run-artifact snapshot at merge time.