mirror of
https://github.com/magnus919/agent-skills.git
synced 2026-09-11 19:47:12 +03:00
Magnus HedemarkGitHubusername <username>factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>Add the product-experimentation skill (issue #190): end-to-end experiment workflow from assumption mapping through method selection, guardrail definition, and decision-readout that updates the roadmap. Includes: - SKILL.md with full 8-step workflow, method ladder (interviews through A/B tests), multi-criteria decision framework, and routing to data-scientist and release-engineering - README.md with 5 required human-facing sections - references/discovery-brief.md mapping existing experimentation guidance across product-methodology, data-scientist, release-engineering, product-design-and-ux, and financial-modeling - references/method-selection.md with decision tree and anti-patterns - references/guardrails-and-ethics.md with guardrail design, ethical boundaries, and stopping rules - references/experiment-readout.md with decision-impact field types - 4 templates: assumption-map, experiment-brief, guardrail-and-decision-rule, readout-learning-entry - evals/evals.json with 5 output-quality cases covering prototype test, feature-flag rollout, underpowered experiment, guardrail omission, and significant-but-no-ship boundary Shared updates: root README catalog entry, skill-triggers.md entry, regenerated marketplace/codex/llms catalogs. Co-authored-by: username <username> Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
2.4 KiB
2.4 KiB
Guardrail and Decision Rule Record
Record the guardrails, stopping rules, and decision authority for an experiment. This document must be completed and reviewed before the experiment starts.
Experiment Reference
- Experiment name:
- Experiment brief: (link)
- Date:
Guardrail Metrics
Every experiment must define at least one guardrail metric. Record the metric, threshold, direction, and the action taken if the threshold is breached.
| # | Guardrail metric | Direction | Threshold | Action on breach | Monitoring frequency |
|---|---|---|---|---|---|
| 1 | Error rate | Increase | +X% over control | Stop experiment, investigate | Continuous |
| 2 | p95 latency | Increase | +Y ms over control | Stop experiment, investigate | Continuous |
| 3 | (domain-specific) |
Guardrail omission check: If any row above is empty and no domain-specific guardrail is defined, the experiment is not ready. At minimum, error rate must be monitored.
Stopping Rules
| Rule | Condition | Action |
|---|---|---|
| Guardrail breach | Any guardrail exceeds threshold | Stop immediately; no ship until resolved |
| Sufficiency | Evidence strong enough to decide (e.g., posterior > 0.95) | Stop and decide |
| Futility | Unlikely to reach decision with remaining budget | Stop and record inconclusive |
| Time cap | Calendar deadline reached | Stop and decide with available evidence |
| External event | Product change or incident makes experiment irrelevant | Stop and record reason |
Decision Authority
- Decision owner: (name of individual, not team)
- Backup decision owner: (if primary is unavailable)
- Decision inputs required:
- Statistical evidence (from data-scientist)
- Guardrail evidence (all guardrails must pass)
- Practical significance assessment
- Qualitative evidence (if applicable)
- Reversibility assessment
- Escalation path: (who to escalate to if decision owner cannot decide)
Ethical Review
- User consent model documented
- Data minimization confirmed
- Vulnerable population check completed
- Institutional alignment confirmed
- Ethical concerns documented (if any):
Sign-Off
- Experiment owner: _____ Date: _____
- Decision owner: _____ Date: _____
- Data/science review (if applicable): _____ Date: _____