test: add lifecycle evaluation corpus for new product and production skills (#232)

* test(evals): scope claims to harness model fixtures and revision

Append the neckbeard claims-scoping sentence to one representative
expected_output per per-skill manifest so every corpus member states
VAL-EVL-032 scope (harness, model, fixtures, revision under test).

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>

* test(product-lifecycle): upgrade integrated launch trajectory

Add an explicit launch-decision assertion to the new-product lifecycle
case so the integrated product-launch scenario terminates in a launch
decision recorded as a lifecycle evidence-ledger entry (VAL-CRP-010),
and scope its expected_output claims per VAL-EVL-032.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>

* test(production-excellence): add integrated migration reconciliation failure case

Add integrated-migration-reconciliation-failure: the production-excellence
gate model returns No-go on a reconciliation mismatch, records the failure
evidence, produces a rollback/roll-forward decision with an accountable
owner, and does not proceed to launch (VAL-CRP-012).

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>

* test(agent-production-operations): add privacy boundary escalation case

Add integrated-privacy-boundary-escalation (VAL-CRP-015): the runtime
control plan halts a cross-boundary EU PII trace export before any data
processing, names the privacy boundary, and escalates to jurisdiction-
specific legal review and a human operator. Also add a tool-authority-
health handoff assertion to the read-only contract case (VAL-CRP-016).

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>

* test(lifecycle-evals): add lifecycle evaluation corpus

Add the #204 corpus home: run tooling (run-corpus.sh, fake adapter only),
programmatic coverage validator (validate-corpus-coverage.py), machine-
readable coverage index + human-readable coverage matrix, regression-
detection and fixture/source notes, the bounded discovery brief, and a
one-snapshot committed set of fake-adapter per-trial run artifacts with
harness/model/date scoping fields.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>

---------

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
This commit is contained in:
Magnus Hedemark
2026-08-02 20:13:36 -04:00
committed by GitHub
co-authored by factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
parent 36d3fa837e
commit 9d6bddad61
123 changed files with 6594 additions and 18 deletions
@@ -53,7 +53,7 @@
{
"id": "stop-bet-with-evidence",
"prompt": "Bet 'Personalized Dashboard' in Now for 6 months. Hypothesis: +15% DAU. After 2 cycles: A/B test 2% lift (p=0.31, not significant), 8/10 users didn't notice, infrastructure costs 20% of engineering capacity. Bet owner wants one more cycle. Evaluate against kill criteria.",
"expected_output": "Recommendation to KILL based on evidence. References kill criteria, distinguishes quantitative evidence (A/B test, capacity data) from qualitative (interviews) from opinion (bet owner). Records revisit criteria.",
"expected_output": "Recommendation to KILL based on evidence. References kill criteria, distinguishes quantitative evidence (A/B test, capacity data) from qualitative (interviews) from opinion (bet owner). Records revisit criteria. Claims are scoped to the harness, model, fixtures, and revision under test.",
"assertions": [
"Recommendation explicitly KILL, not continue or rescope.",
"A/B test result (p=0.31, 2% lift) cited as quantitative evidence.",