* test(evals): scope claims to harness model fixtures and revision Append the neckbeard claims-scoping sentence to one representative expected_output per per-skill manifest so every corpus member states VAL-EVL-032 scope (harness, model, fixtures, revision under test). Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> * test(product-lifecycle): upgrade integrated launch trajectory Add an explicit launch-decision assertion to the new-product lifecycle case so the integrated product-launch scenario terminates in a launch decision recorded as a lifecycle evidence-ledger entry (VAL-CRP-010), and scope its expected_output claims per VAL-EVL-032. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> * test(production-excellence): add integrated migration reconciliation failure case Add integrated-migration-reconciliation-failure: the production-excellence gate model returns No-go on a reconciliation mismatch, records the failure evidence, produces a rollback/roll-forward decision with an accountable owner, and does not proceed to launch (VAL-CRP-012). Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> * test(agent-production-operations): add privacy boundary escalation case Add integrated-privacy-boundary-escalation (VAL-CRP-015): the runtime control plan halts a cross-boundary EU PII trace export before any data processing, names the privacy boundary, and escalates to jurisdiction- specific legal review and a human operator. Also add a tool-authority- health handoff assertion to the read-only contract case (VAL-CRP-016). Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> * test(lifecycle-evals): add lifecycle evaluation corpus Add the #204 corpus home: run tooling (run-corpus.sh, fake adapter only), programmatic coverage validator (validate-corpus-coverage.py), machine- readable coverage index + human-readable coverage matrix, regression- detection and fixture/source notes, the bounded discovery brief, and a one-snapshot committed set of fake-adapter per-trial run artifacts with harness/model/date scoping fields. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> --------- Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Product Experimentation
Run end-to-end product experiments — from assumption to decision — with the right method, explicit guardrails, and outcomes that update your roadmap.
Why Install This Skill
Product teams waste months building features nobody wants, or they A/B test changes that a five-minute customer interview would have invalidated. Experimentation is fragmented: opportunity-solution trees help you find the right problem, data science handles the statistics, release engineering manages the flags, but nobody owns the complete experiment workflow from assumption to learning.
This skill gives your agent the ability to run a disciplined experiment end-to-end. It starts where your assumptions live, helps pick the lightest-weight method that can actually answer the question (from a $0 interview through a full A/B test), and ensures you never ship a "statistically significant" result that quietly broke your error budget or crossed an ethical line. Every experiment ends with a readout that changes something concrete — a roadmap item, a backlog priority, or a decision record.
If you already use other product skills (product-methodology, data-scientist, release-engineering), this skill connects them into a coherent workflow. If you are just getting started with experimentation, it gives you a complete, safe starting point.
What You Get
| Directory | Contents |
|---|---|
SKILL.md |
Core workflow: assumption mapping, method selection (interview to A/B), guardrails and ethics, decision criteria, and readout recording |
references/discovery-brief.md |
Where experimentation concepts live in other skills and what this skill owns vs routes |
references/method-selection.md |
Decision framework for choosing qualitative, prototype, operational, and quantitative methods |
references/guardrails-and-ethics.md |
Guardrail design, ethical boundaries, stopping rules, and decision ownership |
references/experiment-readout.md |
How to produce a decision-impact readout that updates the roadmap or decision log |
templates/experiment-brief.md |
Fillable experiment brief: hypothesis, method, metrics, sample, decision criteria |
templates/assumption-map.md |
Structured assumption map with risk, evidence, and testability dimensions |
templates/guardrail-and-decision-rule.md |
Guardrail metrics, stopping rules, and decision-authority record |
templates/readout-learning-entry.md |
Experiment outcome and roadmap/decision-log update template |
evals/evals.json |
Five output-quality eval cases covering prototype, feature-flag, underpowered, guardrail omission, and no-ship scenarios |
Quick Start
- Map your assumptions with templates/assumption-map.md.
- Write an experiment brief for the riskiest assumption using templates/experiment-brief.md.
- Select the lightest method that works — do not default to A/B testing.
- Record guardrails, stopping rules, and decision ownership with templates/guardrail-and-decision-rule.md.
- Run the experiment. Route statistical design to
data-scientistand rollout torelease-engineering. - Decide using multiple criteria (never p-value alone). Record the readout with templates/readout-learning-entry.md.
Triggers
Load this skill when you need to:
- Design and run a product experiment from scratch
- Choose between qualitative interviews, prototypes, concierge tests, fake doors, feature flags, and A/B tests
- Test assumptions before committing engineering effort
- Define guardrails, stopping rules, and decision criteria for an experiment
- Interpret experiment results and make a defensible ship/no-ship decision
- Record experiment outcomes that update your product roadmap or decision log
Requirements
- No API keys, external services, or runtime dependencies required.
- Works with any agent framework supporting the Agent Skills format.
- For statistical design, the
data-scientistskill should be available. For rollout mechanics, therelease-engineeringskill should be available. - For measurement contracts, product-analytics-and-measurement should be available.