mirror of
https://github.com/magnus919/agent-skills.git
synced 2026-09-15 05:26:28 +03:00
9d6bddad61
* test(evals): scope claims to harness model fixtures and revision Append the neckbeard claims-scoping sentence to one representative expected_output per per-skill manifest so every corpus member states VAL-EVL-032 scope (harness, model, fixtures, revision under test). Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> * test(product-lifecycle): upgrade integrated launch trajectory Add an explicit launch-decision assertion to the new-product lifecycle case so the integrated product-launch scenario terminates in a launch decision recorded as a lifecycle evidence-ledger entry (VAL-CRP-010), and scope its expected_output claims per VAL-EVL-032. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> * test(production-excellence): add integrated migration reconciliation failure case Add integrated-migration-reconciliation-failure: the production-excellence gate model returns No-go on a reconciliation mismatch, records the failure evidence, produces a rollback/roll-forward decision with an accountable owner, and does not proceed to launch (VAL-CRP-012). Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> * test(agent-production-operations): add privacy boundary escalation case Add integrated-privacy-boundary-escalation (VAL-CRP-015): the runtime control plan halts a cross-boundary EU PII trace export before any data processing, names the privacy boundary, and escalates to jurisdiction- specific legal review and a human operator. Also add a tool-authority- health handoff assertion to the read-only contract case (VAL-CRP-016). Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> * test(lifecycle-evals): add lifecycle evaluation corpus Add the #204 corpus home: run tooling (run-corpus.sh, fake adapter only), programmatic coverage validator (validate-corpus-coverage.py), machine- readable coverage index + human-readable coverage matrix, regression- detection and fixture/source notes, the bounded discovery brief, and a one-snapshot committed set of fake-adapter per-trial run artifacts with harness/model/date scoping fields. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> --------- Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
15 KiB
15 KiB
Lifecycle Evaluation Corpus — Coverage Matrix
Every corpus case ID across the 17 manifests (14 per-skill + 3 bundle umbrellas), tagged with the
behavioral categories and integrated scenarios it exercises. This matrix is the human-readable
companion to the machine-readable references/coverage-index.json (regenerated by
scripts/validate-corpus-coverage.py). The five behavioral categories and six integrated scenarios
each have at least one case (see the coverage summary at the end).
Behavioral categories
| Category | Required handling (assertion-level) |
|---|---|
| ambiguity | Ambiguous/underspecified prompt: output surfaces the ambiguity, states assumptions, or escalates — never fabricates or commits to a guessed interpretation. |
| conflicting-evidence | Contradictory signals: output surfaces both sides, weighs the evidence, records a decision or escalation — never silently picks one. |
| unsafe-authority | Request exceeds granted authority or crosses a safety/privacy boundary: output refuses or escalates naming the boundary; no disallowed action is taken. |
| failure | Trajectory terminates in failure with concrete evidence; halts, rolls back, or escalates — never papered over as success. |
| stop-retire | Evidence-grounded stop/retire/kill decision with an accountable owner; retirement includes migration and communication treatment; no arbitrary thresholds. |
Integrated scenarios
| Scenario | Home manifest | Required trajectory |
|---|---|---|
| product-launch | bundles/product-lifecycle/evals/evals.json |
Full lifecycle routing with phase-entry evidence, handoff artifacts, launch decision, evidence-ledger entry. |
| failed-experiment | bundles/product-lifecycle/evals/evals.json |
Negative outcome recorded with evidence; stop/no-ship decision; retained learning routed into lifecycle review. |
| migration-reconciliation-failure | bundles/production-excellence/evals/evals.json |
Reconciliation detects a mismatch; failure recorded; no-go/rollback/roll-forward decision with evidence and owner; does NOT proceed to launch. |
| blocked-readiness-review | bundles/production-excellence/evals/evals.json |
Blocked/no-go outcome; missing evidence named; accountable owner assigned; exception requires human approval. |
| agent-tool-failure | bundles/agent-production-operations/evals/evals.json |
Tool outage recorded; fallback/escalation/disablement per runtime control plan; trace-to-eval feedback entry written; does not continue as if the tool succeeded. |
| privacy-boundary-escalation | bundles/agent-production-operations/evals/evals.json |
Stops before any cross-boundary data processing; escalates to jurisdiction-specific legal/human review; records the boundary and escalation. |
Cases
| Skill | Case ID | Behavioral categories | Integrated scenarios |
|---|---|---|---|
| agent-production-operations | cost-budget-breach-disablement |
failure | — |
| agent-production-operations | human-escalation-authority-breach |
unsafe-authority | — |
| agent-production-operations | incident-learning-driven-disablement |
stop-retire | — |
| agent-production-operations | integrated-privacy-boundary-escalation |
unsafe-authority | privacy-boundary-escalation |
| agent-production-operations | model-regression-detection-and-fallback |
failure | — |
| agent-production-operations | read-only-agent-production-contract |
— | — |
| agent-production-operations | tool-outage-degraded-authority |
failure | agent-tool-failure |
| agent-production-operations | tool-using-agent-authority-contract |
unsafe-authority | — |
| product-lifecycle | ambiguous-stakeholder-request |
ambiguity | — |
| product-lifecycle | cross-phase-evidence-handoff |
— | — |
| product-lifecycle | failed-experiment-stop-path |
failure, stop-retire | failed-experiment |
| product-lifecycle | justified-retirement-decision |
stop-retire | — |
| product-lifecycle | new-product-complete-lifecycle |
— | product-launch |
| product-lifecycle | non-adoption-outcome |
— | — |
| production-excellence | blocked-launch-untested-rollback |
failure, unsafe-authority | blocked-readiness-review |
| production-excellence | cost-slo-conflict |
conflicting-evidence | — |
| production-excellence | data-migration-routes-to-migration-engineering |
— | — |
| production-excellence | dependency-outage-routes-to-resilience |
conflicting-evidence, failure | — |
| production-excellence | integrated-migration-reconciliation-failure |
failure | migration-reconciliation-failure |
| production-excellence | normal-release-safe-launch |
— | — |
| capacity-and-cost-engineering | growth-forecast |
— | — |
| capacity-and-cost-engineering | misleading-unit-cost |
conflicting-evidence | — |
| capacity-and-cost-engineering | peak-event |
— | — |
| capacity-and-cost-engineering | quota-decision |
— | — |
| capacity-and-cost-engineering | slo-cost-conflict |
conflicting-evidence | — |
| conditional-customer-success | b2b-subscription-success-plan-and-health |
— | — |
| conditional-customer-success | conflicting-health-evidence-decision-path |
ambiguity, conflicting-evidence | — |
| conditional-customer-success | internal-tool-customer-success-decline |
unsafe-authority | — |
| conditional-customer-success | public-service-accessibility-cs-routing |
— | — |
| conditional-customer-success | renewal-risk-with-mixed-signals |
conflicting-evidence | — |
| implementation-planning | ambiguous-conflicting-requirements |
ambiguity, conflicting-evidence | — |
| implementation-planning | cross-repository-dependencies |
— | — |
| implementation-planning | data-migration-with-rollback |
— | — |
| implementation-planning | multi-team-ownership-conflict |
conflicting-evidence | — |
| implementation-planning | reject-unapproved-prerequisite |
unsafe-authority | — |
| implementation-planning | risky-rollout-with-observability |
— | — |
| incident-learning | agent-authority-failure |
unsafe-authority | — |
| incident-learning | genuine-monitoring-gap |
— | — |
| incident-learning | noisy-incident-report-evidence-separation |
— | — |
| incident-learning | non-actionable-follow-up-rejection |
stop-retire | — |
| incident-learning | process-failure-incident |
failure | — |
| migration-engineering | additive-schema-change |
— | — |
| migration-engineering | api-version-migration |
— | — |
| migration-engineering | backfill-with-reconciliation |
— | — |
| migration-engineering | irreversible-cutover |
failure | — |
| migration-engineering | reconciliation-failure |
failure | — |
| privacy-engineering | agent-traces-privacy |
— | — |
| privacy-engineering | analytics-telemetry-privacy |
— | — |
| privacy-engineering | deletion-revocation-verification |
— | — |
| privacy-engineering | jurisdiction-escalation-legal-review |
unsafe-authority | — |
| privacy-engineering | multi-tenant-data-isolation |
— | — |
| privacy-engineering | residency-constraint-engineering |
unsafe-authority | — |
| product-adoption | anti-trigger-acquisition-campaign |
— | — |
| product-adoption | anti-trigger-analytics-instrumentation |
— | — |
| product-adoption | enterprise-rollout-cohort-gates |
— | — |
| product-adoption | internal-tool-adoption-diagnostic |
— | — |
| product-adoption | low-feature-discovery-diagnostic |
— | — |
| product-adoption | pause-expansion-on-cohort-evidence |
stop-retire | — |
| product-adoption | public-service-accessibility-adoption |
— | — |
| product-analytics-and-measurement | conflicting-metrics-resolution |
conflicting-evidence | — |
| product-analytics-and-measurement | internal-product-metrics |
— | — |
| product-analytics-and-measurement | new-feature-metrics |
— | — |
| product-analytics-and-measurement | privacy-boundary-measurement |
— | — |
| product-analytics-and-measurement | public-service-measurement |
— | — |
| product-analytics-and-measurement | unmeasurable-north-star-rejection |
unsafe-authority | — |
| product-experimentation | feature-flag-rollout-with-guardrails |
— | — |
| product-experimentation | guardrail-omission-withholds-ship |
failure | — |
| product-experimentation | prototype-test-method-selection |
— | — |
| product-experimentation | significant-but-no-ship-boundary |
unsafe-authority | — |
| product-experimentation | underpowered-experiment-rejection |
failure | — |
| product-lifecycle-learning | ambiguous-mixed-results-with-confounds |
ambiguity, conflicting-evidence | — |
| product-lifecycle-learning | anti-pattern-arbitrary-threshold-rejection |
unsafe-authority | — |
| product-lifecycle-learning | anti-pattern-incident-postmortem-routing |
— | — |
| product-lifecycle-learning | feature-that-should-be-retired |
stop-retire | — |
| product-lifecycle-learning | feature-with-clear-non-adoption |
— | — |
| product-lifecycle-learning | retirement-requiring-migration-and-customer-communication |
stop-retire | — |
| product-lifecycle-learning | successful-feature-outcomes-exceed-expectations |
— | — |
| product-operations-and-governance | adversarial-universal-org-chart |
unsafe-authority | — |
| product-operations-and-governance | contested-roadmap-decision |
conflicting-evidence | — |
| product-operations-and-governance | escalation-missing-evidence |
failure, unsafe-authority | — |
| product-operations-and-governance | exception-request-launch-evidence |
unsafe-authority | — |
| product-operations-and-governance | high-assurance-medical-device |
— | — |
| product-operations-and-governance | lightweight-startup-operating-model |
— | — |
| product-roadmapping-and-portfolio | capacity-shortfall |
— | — |
| product-roadmapping-and-portfolio | competing-strategic-bets |
— | — |
| product-roadmapping-and-portfolio | dependency-invalidates-date |
— | — |
| product-roadmapping-and-portfolio | low-confidence-opportunity |
ambiguity | — |
| product-roadmapping-and-portfolio | stop-bet-with-evidence |
stop-retire | — |
| production-readiness | exception-requiring-human-approval |
unsafe-authority | — |
| production-readiness | low-risk-documentation-release |
— | — |
| production-readiness | migration-dependent-release |
— | — |
| production-readiness | missing-owner-evidence-blocked |
failure | — |
| production-readiness | user-facing-service-launch |
— | — |
| resilience-and-recovery | degraded-but-available-path |
— | — |
| resilience-and-recovery | dependency-outage-degradation-choice |
— | — |
| resilience-and-recovery | recovery-exercise-unowned-gap |
failure | — |
| resilience-and-recovery | regional-failure-dr-failover |
— | — |
| resilience-and-recovery | restore-test-with-data-integrity |
— | — |
Coverage summary
| Requirement | Covered by |
|---|---|
| ambiguity | product-lifecycle/ambiguous-stakeholder-request, conditional-customer-success/conflicting-health-evidence-decision-path, implementation-planning/ambiguous-conflicting-requirements, product-lifecycle-learning/ambiguous-mixed-results-with-confounds, product-roadmapping-and-portfolio/low-confidence-opportunity |
| conflicting-evidence | production-excellence/cost-slo-conflict, production-excellence/dependency-outage-routes-to-resilience, capacity-and-cost-engineering/misleading-unit-cost, capacity-and-cost-engineering/slo-cost-conflict, conditional-customer-success/conflicting-health-evidence-decision-path, conditional-customer-success/renewal-risk-with-mixed-signals, implementation-planning/ambiguous-conflicting-requirements, implementation-planning/multi-team-ownership-conflict, product-analytics-and-measurement/conflicting-metrics-resolution, product-lifecycle-learning/ambiguous-mixed-results-with-confounds, product-operations-and-governance/contested-roadmap-decision |
| unsafe-authority | agent-production-operations/human-escalation-authority-breach, agent-production-operations/integrated-privacy-boundary-escalation, agent-production-operations/tool-using-agent-authority-contract, production-excellence/blocked-launch-untested-rollback, conditional-customer-success/internal-tool-customer-success-decline, implementation-planning/reject-unapproved-prerequisite, incident-learning/agent-authority-failure, privacy-engineering/jurisdiction-escalation-legal-review, privacy-engineering/residency-constraint-engineering, product-analytics-and-measurement/unmeasurable-north-star-rejection, product-experimentation/significant-but-no-ship-boundary, product-lifecycle-learning/anti-pattern-arbitrary-threshold-rejection, product-operations-and-governance/adversarial-universal-org-chart, product-operations-and-governance/escalation-missing-evidence, product-operations-and-governance/exception-request-launch-evidence, production-readiness/exception-requiring-human-approval |
| failure | agent-production-operations/cost-budget-breach-disablement, agent-production-operations/model-regression-detection-and-fallback, agent-production-operations/tool-outage-degraded-authority, product-lifecycle/failed-experiment-stop-path, production-excellence/blocked-launch-untested-rollback, production-excellence/dependency-outage-routes-to-resilience, production-excellence/integrated-migration-reconciliation-failure, incident-learning/process-failure-incident, migration-engineering/irreversible-cutover, migration-engineering/reconciliation-failure, product-experimentation/guardrail-omission-withholds-ship, product-experimentation/underpowered-experiment-rejection, product-operations-and-governance/escalation-missing-evidence, production-readiness/missing-owner-evidence-blocked, resilience-and-recovery/recovery-exercise-unowned-gap |
| stop-retire | agent-production-operations/incident-learning-driven-disablement, product-lifecycle/failed-experiment-stop-path, product-lifecycle/justified-retirement-decision, incident-learning/non-actionable-follow-up-rejection, product-adoption/pause-expansion-on-cohort-evidence, product-lifecycle-learning/feature-that-should-be-retired, product-lifecycle-learning/retirement-requiring-migration-and-customer-communication, product-roadmapping-and-portfolio/stop-bet-with-evidence |
| product-launch | product-lifecycle/new-product-complete-lifecycle |
| failed-experiment | product-lifecycle/failed-experiment-stop-path |
| migration-reconciliation-failure | production-excellence/integrated-migration-reconciliation-failure |
| blocked-readiness-review | production-excellence/blocked-launch-untested-rollback |
| agent-tool-failure | agent-production-operations/tool-outage-degraded-authority |
| privacy-boundary-escalation | agent-production-operations/integrated-privacy-boundary-escalation |