mirror of
https://github.com/magnus919/agent-skills.git
synced 2026-09-11 19:47:12 +03:00
9d6bddad61
* test(evals): scope claims to harness model fixtures and revision Append the neckbeard claims-scoping sentence to one representative expected_output per per-skill manifest so every corpus member states VAL-EVL-032 scope (harness, model, fixtures, revision under test). Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> * test(product-lifecycle): upgrade integrated launch trajectory Add an explicit launch-decision assertion to the new-product lifecycle case so the integrated product-launch scenario terminates in a launch decision recorded as a lifecycle evidence-ledger entry (VAL-CRP-010), and scope its expected_output claims per VAL-EVL-032. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> * test(production-excellence): add integrated migration reconciliation failure case Add integrated-migration-reconciliation-failure: the production-excellence gate model returns No-go on a reconciliation mismatch, records the failure evidence, produces a rollback/roll-forward decision with an accountable owner, and does not proceed to launch (VAL-CRP-012). Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> * test(agent-production-operations): add privacy boundary escalation case Add integrated-privacy-boundary-escalation (VAL-CRP-015): the runtime control plan halts a cross-boundary EU PII trace export before any data processing, names the privacy boundary, and escalates to jurisdiction- specific legal review and a human operator. Also add a tool-authority- health handoff assertion to the read-only contract case (VAL-CRP-016). Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> * test(lifecycle-evals): add lifecycle evaluation corpus Add the #204 corpus home: run tooling (run-corpus.sh, fake adapter only), programmatic coverage validator (validate-corpus-coverage.py), machine- readable coverage index + human-readable coverage matrix, regression- detection and fixture/source notes, the bounded discovery brief, and a one-snapshot committed set of fake-adapter per-trial run artifacts with harness/model/date scoping fields. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> --------- Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
123 lines
6.9 KiB
Markdown
123 lines
6.9 KiB
Markdown
# Regression Detection — Lifecycle Evaluation Corpus
|
|
|
|
This document defines how to detect and interpret a regression in the lifecycle evaluation
|
|
corpus across revisions, and how to tell a real regression from a benign content change.
|
|
|
|
## What counts as a regression
|
|
|
|
A corpus regression is any change that silently reduces the corpus's ability to exercise
|
|
its required coverage or that invalidates durable evidence references. Concretely:
|
|
|
|
1. **Case removal or renaming.** Eval case IDs are **durable evidence references**
|
|
(VAL-EVL-005, VAL-CRP-024). Removing a case, or renaming an ID, breaks the mapping in
|
|
`references/coverage-index.json`, any committed run artifacts that reference the ID, and
|
|
any downstream evidence that cites the ID. Never rename an ID; add a new case with a new
|
|
ID instead.
|
|
2. **Behavioral-category or integrated-scenario coverage loss.** Removing the last case
|
|
tagged for a behavioral category or integrated scenario makes the corpus fail its
|
|
mandatory coverage (VAL-CRP-003..008, VAL-CRP-010..015). The machine-checkable gate is
|
|
`validate-corpus-coverage.py`, which fails when any of the 5 categories or 6 scenarios
|
|
has no tagged case.
|
|
3. **Assertion-set drift on a tagged case.** If a case's assertions no longer verify the
|
|
category's required handling (e.g., a stop/retire case stops asserting the accountable
|
|
owner, or a handoff assertion is dropped from an integrated case), the corpus silently
|
|
loses the guarantee that the category/scenario is *actually* exercised. The committed
|
|
run artifacts pin the assertion sets at snapshot time; a re-run that changes assertion
|
|
sets is a signal to review.
|
|
4. **Fixture-hash changes.** Every per-trial manifest records `case.prompt_hash` and
|
|
`case.fixture_hashes`. A change to a prompt or to a referenced fixture changes those
|
|
hashes. For self-contained cases (inputs inline in prompts) a prompt change is a
|
|
deliberate content change that should be reviewed against the case's tag; a fixture
|
|
change on a `files`-referencing case changes the fixture hash and must be reconciled
|
|
with `references/sources.md` and the repository fixture-resolution validator
|
|
(`validate-evals.py`).
|
|
|
|
## Comparison procedure (re-run with the fake adapter)
|
|
|
|
The corpus is deterministic under the fake adapter (no model, no network, no randomness in
|
|
execution — only timestamps and trial UUIDs vary). To compare two revisions:
|
|
|
|
```sh
|
|
# On the old revision (e.g., the merged baseline):
|
|
git worktree add /tmp/corpus-old <old-sha>
|
|
cd /tmp/corpus-old && bash lifecycle-evals/scripts/run-corpus.sh # CORPUS_OUT_DIR=/tmp/corpus-runs-old
|
|
|
|
# On the new revision (the candidate):
|
|
cd /Volumes/tank01/magnus/git/agent-skills-issue-204
|
|
CORPUS_OUT_DIR=/tmp/corpus-runs-new bash lifecycle-evals/scripts/run-corpus.sh
|
|
```
|
|
|
|
Then compare per case:
|
|
|
|
1. **Per-case status**: every trial must be `status == "completed"` in both runs (a trial
|
|
that becomes `error`/`timeout`/`stopped` between revisions is a regression).
|
|
2. **Per-case identity**: the set of `case_id`s per manifest must be equal between
|
|
revisions (no removals, no renames).
|
|
3. **Per-case content**: compare `case.prompt_hash` and the `case.fixture_hashes` fields in
|
|
the per-trial manifests. Changed hashes indicate the case content or fixture changed and
|
|
must be reviewed (see interpretation below).
|
|
4. **Coverage**: run `validate-corpus-coverage.py` on the candidate; it must exit 0 (all 5
|
|
categories, all 6 scenarios covered, every referenced ID present, index current).
|
|
|
|
A simple diff-oriented check across the two output trees:
|
|
|
|
```sh
|
|
diff <(cd /tmp/corpus-runs-old && find . -name '*.manifest.json' | sort) \
|
|
<(cd /tmp/corpus-runs-new && find . -name '*.manifest.json' | sort)
|
|
```
|
|
|
|
Note that file names embed the trial UUID prefix (`<case_id>--<trial_id[:8]>.manifest.json`),
|
|
so compare by `case_id` sets and by hashes rather than by file name.
|
|
|
|
## Case-ID stability rule
|
|
|
|
**Never rename an eval case ID.** IDs are referenced by the coverage index, the coverage
|
|
matrix, committed run artifacts, and (potentially) external evidence ledgers. Renaming an
|
|
ID is a regression even when the content is unchanged. To evolve a case: keep the ID, update
|
|
content, regenerate the index, re-run the corpus, and refresh the run-artifact snapshot at
|
|
merge time. To add coverage: add a new case with a new lowercase-hyphen ID (≤ 64 chars,
|
|
unique within its manifest).
|
|
|
|
## Ratchet command
|
|
|
|
The repository's eval-coverage ratchet must hold on every corpus change:
|
|
|
|
```sh
|
|
.venv/bin/python scripts/eval-coverage.py --modified-from origin/main
|
|
```
|
|
|
|
This exits 0 only when no modified skill lacks a schema-valid manifest and coverage does
|
|
not decrease versus the base. All 17 corpus manifests are schema-valid, so corpus changes
|
|
never trip the modified-skill ratchet; the check still runs in CI on every PR.
|
|
|
|
Fixture-resolution is enforced by the repository validator:
|
|
|
|
```sh
|
|
.venv/bin/python scripts/validate-evals.py
|
|
```
|
|
|
|
## Interpreting a change: regression vs. benign content change
|
|
|
|
| Observation | Classification | Required action |
|
|
|---|---|---|
|
|
| A case ID disappears from a manifest | **Regression** | Restore the case or (if truly obsolete) re-scope: add a replacement case, update the index and matrix, re-run, and record the replacement in the PR body; never silently drop the ID. |
|
|
| A case ID is renamed | **Regression** | Revert the rename; change content only, or add a new ID. |
|
|
| The last case for a category/scenario is removed or untagged | **Regression** | `validate-corpus-coverage.py` fails; restore coverage before merging. |
|
|
| A prompt/assertion is edited to tighten wording without changing the scenario or the category-required handling | **Benign content change** | Update the run-artifact snapshot at merge time (one snapshot per merge, VAL-CRP-021 ambiguity C); no re-review of category coverage needed beyond `validate-corpus-coverage.py`. |
|
|
| A prompt is changed so the case now exercises a different scenario, or a category-required assertion is dropped | **Material content change** | Re-tag the case in the coverage index, regenerate the matrix and index, re-run the corpus, and re-verify the case still satisfies its category's required handling. |
|
|
| A `files` fixture changes | **Material content change** | Update `references/sources.md` (provenance), re-run, and confirm `validate-evals.py` still resolves the fixture. |
|
|
| Timestamps/UUIDs differ between fake runs | **Benign** | Expected; timestamps are the scoping/date evidence and are excluded from content comparison. |
|
|
|
|
In all cases, the decision is recorded in the change's PR body or evidence ledger so a
|
|
future reviewer can see why the corpus changed.
|
|
|
|
## Keeping the index and matrix current
|
|
|
|
After any case content, ID, or tag change:
|
|
|
|
```sh
|
|
.venv/bin/python lifecycle-evals/scripts/validate-corpus-coverage.py --write-index
|
|
# then regenerate the human-readable matrix from the index (see coverage-matrix.md header),
|
|
# re-run the corpus, and commit the run-artifact snapshot at merge time.
|
|
```
|