mirror of
https://github.com/magnus919/agent-skills.git
synced 2026-09-11 19:47:12 +03:00
test: add lifecycle evaluation corpus for new product and production skills (#232)
* test(evals): scope claims to harness model fixtures and revision Append the neckbeard claims-scoping sentence to one representative expected_output per per-skill manifest so every corpus member states VAL-EVL-032 scope (harness, model, fixtures, revision under test). Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> * test(product-lifecycle): upgrade integrated launch trajectory Add an explicit launch-decision assertion to the new-product lifecycle case so the integrated product-launch scenario terminates in a launch decision recorded as a lifecycle evidence-ledger entry (VAL-CRP-010), and scope its expected_output claims per VAL-EVL-032. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> * test(production-excellence): add integrated migration reconciliation failure case Add integrated-migration-reconciliation-failure: the production-excellence gate model returns No-go on a reconciliation mismatch, records the failure evidence, produces a rollback/roll-forward decision with an accountable owner, and does not proceed to launch (VAL-CRP-012). Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> * test(agent-production-operations): add privacy boundary escalation case Add integrated-privacy-boundary-escalation (VAL-CRP-015): the runtime control plan halts a cross-boundary EU PII trace export before any data processing, names the privacy boundary, and escalates to jurisdiction- specific legal review and a human operator. Also add a tool-authority- health handoff assertion to the read-only contract case (VAL-CRP-016). Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> * test(lifecycle-evals): add lifecycle evaluation corpus Add the #204 corpus home: run tooling (run-corpus.sh, fake adapter only), programmatic coverage validator (validate-corpus-coverage.py), machine- readable coverage index + human-readable coverage matrix, regression- detection and fixture/source notes, the bounded discovery brief, and a one-snapshot committed set of fake-adapter per-trial run artifacts with harness/model/date scoping fields. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> --------- Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
This commit is contained in:
co-authored by
factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
parent
36d3fa837e
commit
9d6bddad61
@@ -53,7 +53,7 @@
|
||||
{
|
||||
"id": "stop-bet-with-evidence",
|
||||
"prompt": "Bet 'Personalized Dashboard' in Now for 6 months. Hypothesis: +15% DAU. After 2 cycles: A/B test 2% lift (p=0.31, not significant), 8/10 users didn't notice, infrastructure costs 20% of engineering capacity. Bet owner wants one more cycle. Evaluate against kill criteria.",
|
||||
"expected_output": "Recommendation to KILL based on evidence. References kill criteria, distinguishes quantitative evidence (A/B test, capacity data) from qualitative (interviews) from opinion (bet owner). Records revisit criteria.",
|
||||
"expected_output": "Recommendation to KILL based on evidence. References kill criteria, distinguishes quantitative evidence (A/B test, capacity data) from qualitative (interviews) from opinion (bet owner). Records revisit criteria. Claims are scoped to the harness, model, fixtures, and revision under test.",
|
||||
"assertions": [
|
||||
"Recommendation explicitly KILL, not continue or rescope.",
|
||||
"A/B test result (p=0.31, 2% lift) cited as quantitative evidence.",
|
||||
|
||||
Reference in New Issue
Block a user