magnus919_agent-skills/product-experimentation/evals/evals.json

{"schema_version": 1, "skill_name": "product-experimentation", "evals": [{"id": "prototype-test-method-selection", "prompt": "We are considering building a new feature that lets users collaborate on documents in real time. Our engineering team estimates 3 months of work. Before we commit, I want to test whether users actually need this. What experiment should we run?", "expected_output": "The response recommends a prototype test or qualitative interviews rather than defaulting to an A/B test or feature flag. It explains that the riskiest assumption is user need, which can be tested with a lightweight prototype shown to 5-20 users. It explicitly states that an A/B test would be premature because the feature is not built and the question is about value (should we build it?) not execution (does this button color work better?). The response names the method from the ladder (qualitative or prototype) and justifies why heavier methods are inappropriate.", "assertions": ["The response recommends a qualitative or prototype test rather than A/B testing or feature flags", "The response explains that the question is about user need and value, not execution optimization", "The response notes that the feature has not been built yet, making A/B testing premature", "The response explicitly rejects defaulting to the heaviest available method", "The response's method choice is justified by cost, question type, and build stage"]}, {"id": "feature-flag-rollout-with-guardrails", "prompt": "We have built a new checkout flow and want to roll it out safely. Design the experiment: we need to know if it improves conversion, but we cannot degrade the purchase experience. Our baseline conversion is 3.2% and we have about 50,000 checkouts per week.", "expected_output": "The response designs a feature-flag experiment with: a named hypothesis (new checkout flow increases conversion), a control/treatment split, a primary metric (conversion rate), guardrail metrics (error rate on checkout, p95 latency, cart abandonment rate not increasing, successful payment completion rate), stopping rules (guardrail breach triggers immediate stop), and a decision owner. The response names the statistical analysis as routed to data-scientist and the rollout mechanics (flag creation, percentage ramp) as routed to release-engineering. It recommends starting with a small percentage (1-5%) and ramping after guardrail verification.", "assertions": ["The response selects feature-flag rollout as the method and designs a controlled experiment with control and treatment groups", "The response defines guardrail metrics including error rate, latency, and at least one business-safety metric beyond the primary metric", "The response specifies stopping rules triggered by guardrail breach, not only by statistical conclusion", "The response routes statistical design to data-scientist and rollout mechanics to release-engineering", "The response recommends incremental ramp with guardrail verification between stages rather than an all-at-once switch"]}, {"id": "underpowered-experiment-rejection", "prompt": "Our SaaS product has 200 total users and we want to A/B test a new onboarding flow to see if it improves week-1 retention from 40% to 45%. Our data scientist ran a power analysis that says we need at least 1,200 users per variant to detect a 5-percentage-point change at 80% power. Should we run this A/B test?", "expected_output": "The response identifies the experiment as underpowered and recommends against running it. It explains that with only 200 users, the test cannot detect the smallest effect that matters (5pp), making a null result uninformative and a significant result likely a false positive or exaggerated. It suggests alternative methods: qualitative interviews with new users to understand onboarding friction, a concierge test manually guiding a subset of new users through the ideal flow, or redefining the success criterion to a larger effect. It explicitly states that running an underpowered experiment and making decisions based on it is a statistical validity failure.", "assertions": ["The response identifies the experiment as underpowered (sample too small for the desired effect size)", "The response recommends against running the A/B test in its current form", "The response explains that a null result from an underpowered experiment is uninformative and should not drive decisions", "The response suggests at least one alternative method appropriate for the sample size (qualitative, concierge, or redefined effect)", "The response explicitly names statistical validity or adequate power as a prerequisite for running the test"]}, {"id": "guardrail-omission-withholds-ship", "prompt": "Our team ran an experiment on a new recommendation algorithm. The primary metric was daily active users and it showed a statistically significant 12% increase (p=0.003). The experiment ran for 2 weeks on 50% of users. Here is the readout: 'Result is significant, ship it.' I am reviewing this as the product lead. Is there anything missing?", "expected_output": "The response identifies that the experiment has no guardrail metrics defined. It specifically calls out the absence of: error-rate monitoring, latency/degradation checks, and any domain-specific harm metric. The response withholds the ship decision and states that the experiment must be re-evaluated with proper guardrails. It names the missing guardrail (error rate as the minimum) and explains that a statistically significant result with unmonitored side effects is a no-ship. It recommends defining guardrails, checking whether they were breached during the experiment window, and only then making the decision.", "assertions": ["The response identifies that no guardrail metrics were defined or monitored during the experiment", "The response withholds the ship decision despite the statistically significant primary result", "The response names at minimum an error-rate guardrail as the missing safety check", "The response explains that guardrail absence invalidates the decision regardless of statistical evidence", "The response provides a corrective action: define guardrails, retroactively check if possible, or re-run with proper monitoring"]}, {"id": "significant-but-no-ship-boundary", "prompt": "Our A/B test on a new notification frequency algorithm showed a statistically significant 18% increase in daily active users (p<0.001, adequately powered). However, we noticed that the treatment group's opt-out rate tripled and two users filed complaints about notification spam. Our head of engineering also noted the algorithm causes a 15% increase in push-notification infrastructure cost. The data science team says 'the result is clear, ship it.' As product lead, what should we do?", "expected_output": "The response decides no-ship despite the statistical significance. It weighs multiple criteria beyond the p-value: the tripled opt-out rate is a user-harm signal (practical significance and ethics), the infrastructure cost increase is a business guardrail violation, and the user complaints are qualitative evidence of harm. It explains that statistical significance alone does not authorize a ship decision — guardrail, ethical, and practical considerations can override. It recommends either abandoning the change or redesigning the experiment with proper guardrails (opt-out rate as a blocking metric, cost guardrail) and re-running. The response explicitly names that the decision authority rests with the product lead, not the data science team, and that exceeding the authority boundary of user well-being is a no-ship. Claims are scoped to the harness, model, fixtures, and revision under test.", "assertions": ["The response decides no-ship despite the statistically significant primary outcome", "The response names at least two non-statistical reasons for withholding the decision (opt-out rate, user complaints, or infrastructure cost)", "The response explicitly states that statistical significance is not the sole decision criterion", "The response identifies that user-harm signals and business guardrails override the statistical recommendation", "The response names the product lead as decision owner, not the data science team, and explains that ethical/guardrail boundaries represent an authority boundary that cannot be crossed by statistical evidence alone"]}]}