magnus919_agent-skills/product-experimentation/references/method-selection.md

Method Selection

How to choose the right experiment method for a given hypothesis. The core principle: start at the lightest-weight method that can falsify the hypothesis, and only move down the ladder when the question cannot be answered at the current level.

The Method Ladder

Method	What it tests	Sample size	Statistical rigor	When to use	When NOT to use
Qualitative interviews	Mental models, problem validation, unknown unknowns	5-15	None (descriptive)	You do not understand the problem space yet; the riskiest assumption is about user behavior or motivation	You need a precise effect size estimate; the hypothesis is about a quantitative change
Prototype tests	Interaction flow, usability, concept desirability	5-20	None (observational)	The question is "can users figure this out?" or "does this concept resonate?"	You need causal attribution to a business metric; the prototype cannot simulate the real experience
Concierge tests	Value delivery, willingness to pay, operational feasibility	1-20	None (manual)	You need to test whether anyone will pay for or use the service before building automation	The manual delivery cannot approximate the automated experience closely enough
Fake doors	Demand signals, willingness to click/commit	100+	Low (conversion rate difference)	You need to size demand before building; the cost of the real feature is high	The fake door deceives users and the deception risk is unacceptable; the feature is trivial to build
Feature flags	Operational safety, incremental rollout, kill-switch	Configurable	Medium (controlled rollout)	You have a feature ready and need to validate it safely in production with the ability to turn it off	You have not validated the underlying assumption; a flag controls risk but does not test value
A/B tests	Causal attribution of a specific change to a metric	Statistical minimum (power analysis)	High (randomized controlled)	You need causal evidence that a change moves a metric; the sample size is achievable	The question can be answered with a lighter method; the sample is too small for adequate power; the ethics of randomization are questionable

Selection Decision Tree

Can you learn enough from talking to 5-10 users?
  └─ YES → Qualitative interviews
  └─ NO  → Can you build a clickable prototype in a day and test it with 5-20 people?
              └─ YES → Prototype test
              └─ NO  → Can you manually deliver the value for 1-20 users?
                          └─ YES → Concierge test
                          └─ NO  → Can you measure demand with a button that does not yet work?
                                      └─ YES → Fake door
                                      └─ NO  → Do you have the feature built and need safe rollout?
                                                  └─ YES → Feature flag
                                                  └─ NO  → Do you need causal attribution with statistical rigor?
                                                              └─ YES → A/B test
                                                              └─ NO  → Revisit the hypothesis — is it testable?

Anti-Patterns

Defaulting to A/B testing

The most common experimentation failure. A/B testing is expensive: it requires engineering time to build the variant, statistical design (power analysis, sample-size calculation), enough traffic to detect the effect, and time to run. Before committing to an A/B test, ask: "Could I learn enough from 5 interviews to make this decision?" If yes, do the interviews.

Testing the wrong thing

If the hypothesis is "users want this feature" and you A/B test whether a blue button outperforms a green button, you are testing execution, not value. Execution testing is only useful after value is confirmed.

Ignoring method limitations

Every method has blind spots. Qualitative interviews cannot tell you how many users will convert. A/B tests cannot tell you why users behave differently. Use multiple methods together when the decision is consequential.

Measuring Method Adequacy

Before committing to a method, verify:

1. Falsifiability: Can this method produce evidence that would convince you the assumption is wrong?
2. Timeliness: Can you get the evidence fast enough to affect the decision?
3. Cost proportionality: Is the cost of the method proportionate to the cost of being wrong?
4. Ethical fit: Does the method respect user autonomy, consent, and dignity?