magnus919_agent-skills/qa-methodology/references/risk-based-testing.md

Risk-Based Testing

Prioritize test effort by risk. Load when deciding where to invest limited testing time, building a risk register, or running a risk assessment workshop. For the broader strategy context (pyramid shape, shift-left/right), see test-strategy.md.

Risk = Probability × Impact

Every test decision is a risk decision made implicitly. Make it explicit:

Risk Score = Probability (1–5) × Impact (1–5)

Score Range	Zone	Action
20–25	Critical (P0)	Test exhaustively; every path, every edge case
12–19	High (P1)	Test all happy paths + known failure modes
6–11	Medium (P2)	Test happy paths + common failure modes
1–5	Low (P3)	Smoke test only; defer detailed testing

These priority tiers (P0–P3) align with the coverage tiers in test-strategy.md. A P0 risk item demands P0-level coverage; a P3 risk item justifies smoke-only testing.

5×5 Risk Matrix

P \ I	1 — Negligible	2 — Minor	3 — Moderate	4 — Major	5 — Catastrophic
5 — Almost Certain	5 (P3)	10 (P2)	15 (P1)	20 (P0)	25 (P0)
4 — Likely	4 (P3)	8 (P2)	12 (P1)	16 (P0)	20 (P0)
3 — Possible	3 (P3)	6 (P2)	9 (P2)	12 (P1)	15 (P1)
2 — Unlikely	2 (P3)	4 (P3)	6 (P2)	8 (P2)	10 (P2)
1 — Rare	1 (P3)	2 (P3)	3 (P3)	4 (P3)	5 (P3)

Scoring Guidance

Rating	Probability Anchor	Impact Anchor
5	Will fail in production within a quarter (or has already)	Data loss, security breach, revenue stoppage
4	Expected to fail within a year	Major feature outage, SLA breach
3	Could fail; uncertain	Degraded experience, workaround exists
2	Unlikely given current controls	Cosmetic, minor inconvenience
1	Extremely unlikely; well-understood code	No user-visible impact

Risk Assessment Workshop

Run a structured workshop to score risks collaboratively. Solo scoring introduces individual bias; group calibration produces defensible priorities.

Participants

• QA lead (facilitator)
• Engineering leads for affected areas
• Product manager (impact calibration)
• Operations / SRE representative (production context)

Agenda (90 minutes)

Time	Activity
0–15 min	Identify risk items: what could go wrong? (brainstorm from change log, incident history, architecture)
15–50 min	Score each item: probability (group vote, median), impact (product calibrates)
50–65 min	Rank and assign priority tiers (P0–P3) from the matrix
65–80 min	Define mitigations: what tests, who owns them, by when
80–90 min	Agree reassessment triggers and next review date

Calibration Rule

If probability votes span > 2 points, the facilitator asks the highest and lowest voter to state their evidence. Re-vote once. Record dissent in the register.

Risk Register Structure

Column	Description	Example
ID	Unique identifier	RISK-012
Risk Description	What could go wrong	Payment gateway timeout during peak
Component	Affected system area	Checkout service
Probability (1–5)	Likelihood of occurrence	4
Impact (1–5)	Severity if it occurs	5
Score	P × I	20
Priority Tier	P0–P3 (from matrix)	P0
Mitigation / Test Plan	What testing addresses this	Load test at 2× peak; chaos inject timeout
Owner	Who implements the mitigation	QA-2
Status	Open / Mitigating / Closed	Mitigating
Last Reviewed	Date of last reassessment	2025-07-15
Reassessment Trigger	What event re-opens this	Payment provider API change

Reassessment Triggers

Risk is not static. Re-score the register when any trigger fires:

Trigger	Rationale
Production incident in the area	Actual failure updates probability upward
Architecture change (new dependency, refactor)	Changes both probability and impact landscape
New regulatory requirement	May raise impact (compliance penalty)
Major release or migration	New failure modes introduced
Quarterly calendar review	Prevents register staleness (default cadence)
Team change (key engineer leaves)	Knowledge gaps raise probability
Customer escalation	Business impact may have changed

Cost-of-Failure Reasoning

Risk-based testing investment is justified by the cost differential between catching a defect early vs late:

Detection Phase	Relative Fix Cost	Risk-Based Justification
Design / Spec review	1×	Highest-leverage test: risk workshop catches design flaws
Unit / PR testing	5–10×	P0/P1 items justify exhaustive unit coverage
Integration / Staging	20–50×	Contract and integration tests for cross-boundary risks
Production	100×+	Shift-right monitoring for residual P0 risk

Decision rule: Allocate test effort proportional to risk score. A P0 item (score 20–25) receives 3–5× the per-item test design budget of a P3 item (score 1–5).

Test Estimation Heuristic

Estimate test effort from the risk register:

total_test_hours = Σ (risk_items_in_tier × hours_per_tier)

Hours per tier (default):
  P0: 8–16 hours per risk item (exhaustive design + automation)
  P1: 4–8 hours per risk item
  P2: 2–4 hours per risk item
  P3: 0.5–1 hour per risk item (smoke only)

Adjustment factors: multiply by 1.5× for legacy/unfamiliar code, 0.7× for well-automated areas with existing coverage.

Worked Example

A release has 3 P0 risks, 5 P1 risks, 8 P2 risks, and 12 P3 risks:

Tier	Items	Hours/Item	Subtotal
P0	3	12	36
P1	5	6	30
P2	8	3	24
P3	12	0.75	9
Total	28	—	99 hours

With a 2-person QA team (80 hours/sprint), this release requires ~1.25 sprints of test design effort. Negotiate scope or add capacity for P0 items; P3 items can be deferred.

Gotchas

Gotcha — Static register: A risk register written once at project start and never updated is fiction. Reassess on triggers (above) and at minimum quarterly. A stale register misallocates effort toward risks that no longer exist.

Gotcha — Consensus theater: If the workshop rubber-stamps the loudest voice's scores without evidence, the register is political, not analytical. Require evidence anchors for every score. Record dissent.

Gotcha — Full suite on every PR regardless of risk: Running everything on every change wastes CI minutes and trains teams to ignore results. Use risk tiers to gate test selection (P0 always runs; P3 runs nightly).

Exit Condition

You are done applying this reference when: (1) a risk register exists with scored items mapped to P0–P3 tiers, (2) test allocation is proportional to risk scores, (3) reassessment triggers are defined with a calendar backstop, and (4) the estimation heuristic produces a capacity-checked plan.

Composition Links

• Broader test strategy and pyramid shape: test-strategy.md
• Regression suite tiering by risk: regression-testing.md
• Quality gate design (blocking vs advisory per tier): quality-gates-and-metrics.md
• Verification planning and evidence standards: verification-methodology

---

Sources: ISO/IEC 25010 (systems and software quality requirements), ISTQB Foundation Level Syllabus 2023 (risk-based testing chapter), James Bach (context-driven testing, risk heuristics), Kaner/Bach/Pettichord "Lessons Learned in Software Testing" (Wiley, 2002), DORA State of DevOps Reports (cost-of-failure data).