magnus919_agent-skills/qa-methodology/references/regression-testing.md

Regression Testing

Core Principle

Every fixed bug becomes a regression test. When a defect is fixed, the test that reproduces it (or should have caught it) is added to the suite permanently. This is the single highest-ROI regression practice: the suite's ability to catch known failure modes grows monotonically. Track "regression test added for fix" rate; target > 80% of bug fixes.

What Belongs in a Regression Suite

Include	Exclude
Every fixed bug (as a test)	Tests that haven't failed in 6+ months (review for archive)
Critical user paths (P0/P1)	Tests for deprecated features
Known failure patterns	Tests duplicating lower-level coverage
API contract checks	Visual regression on work-in-progress UI
Data integrity assertions	Performance tests (separate suite)

Test Impact Analysis (Shift-Left Selection)

Not every change needs the full suite. Impact analysis selects the subset of tests affected by a change.

Approach	How It Works	Trade-off
Static call-graph	Map changed code → functions → tests that call them	Fast, conservative (over-selects); misses dynamic dispatch
Dynamic trace	Record which tests execute which code (coverage instrumentation), then map changes → tests	Accurate; needs profiling infra; Ekstazi uses this
ML prediction	Train on historical (change, failing-test) pairs	Best precision; needs history; see test-automation.md

Tools: Ekstazi (dynamic, JVM), Launchable (ML, language-agnostic), custom coverage-map scripts.

Selection Math

Score each test for inclusion on a change:

selection_score(test) =
    w1 × changed_code_coverage(test)      # does it touch the diff? (0/1)
  + w2 × historical_failure_rate(test)    # how often has it failed recently?
  + w3 × business_risk(test)              # P0/P1 path? (see test-strategy.md)

Default weights: w1=0.5, w2=0.3, w3=0.2. Run all tests with score > threshold, plus always run the full P0 smoke set regardless of score.

Change Type	Required Tests
Bug fix	Repro test + related unit tests
Feature addition	New feature tests + adjacent smoke
Refactoring	Full unit suite + integration smoke
Dependency update	Full regression suite
Infrastructure change	Integration + E2E suite

Suite Evolution (Tiering)

Tests move between tiers as the system and risk profile change:

Tier	Cadence	Contents	Promotion / Retirement
Tier 0 — Smoke	Every PR (< 5 min)	P0 paths, build sanity	Promote from Tier 1 if a path becomes critical
Tier 1 — Core	Every merge (< 30 min)	P0/P1 regression, contracts	Promote from Tier 2 on rising failure rate
Tier 2 — Extended	Nightly	P2, slower integration, E2E	Retire tests with 0 failures in 6 months + no risk coverage
Tier 3 — Archive	On demand	Retired, legacy	Restore if a related defect escapes

Evolution rule: A test that fails in production-adjacent tiers gets promoted; a test with zero failures over 6 months and no P0/P1 risk coverage is a retirement candidate (review, don't auto-delete).

Shift-Right Feedback Loops

Production signals drive regression suite composition — closing the loop between what escapes and what you test.

Signal	Suite Action
Production error spike (observability/APM)	Add a regression test reproducing the failing condition; promote related tests to Tier 1
Canary analysis failure	The canary check becomes a permanent regression assertion; review why left-side testing missed it
Escaped defect (customer report)	Root-cause: add the missing test, then audit the tier that should have caught it
Feature-flag rollout	Flag-gated code gets targeted regression selection while flagged; full coverage before flag removal

Loop discipline: Every escaped defect gets a post-mortem question — "which tier should have caught this, and why didn't it?" The answer drives suite evolution, not blame.

Flaky Test Discipline: Rerun-Once, Never-Twice

Protocol: In a blocking regression run, a failed test may be rerun exactly once. If it passes on the single rerun, record it as a flake (not a pass) and route to quarantine. If it fails the rerun, it is a real failure — block. Never rerun a second time.

Why never twice: a second rerun turns flake-masking into a habit. If a test needs two retries to pass, it is not reliable enough to gate on, and repeated retries hide a rising flake rate until CI trust collapses.

Outcome	Classification	Action
Pass on first run	Pass	Record green
Fail, then pass on 1st rerun	Flake	Do NOT count as pass; open quarantine issue; see test-automation.md
Fail, fail on 1st rerun	Real failure	Block the pipeline
Needs 2nd rerun to pass	Unreliable	Treat as flake; quarantine immediately

Track flake rate (flaky outcomes / total runs). Sustain at < 2%; above that, invest in stabilization before adding more tests.

Gotcha — Ignoring flakes: A "pass on retry" is not a pass. Flakes are latent failures; a suite with 5% flakiness on 10,000 daily tests produces ~500 false signals/day and erodes trust until teams bypass automation entirely.

Suite Hygiene

Condition	Action
Test hasn't failed in 6 months + no risk coverage	Review for archive
Test flakes > 2% over 50 runs	Quarantine and fix
Test takes > 5s (unit) / > 30s (integration)	Optimize or promote to slower tier
Test depends on another test's state	Fix isolation (tests must be independent)
Test runs against production data	Switch to synthetic fixtures (see test-data-management.md)

Composition Links

• Impact-analysis tooling and ML selection: test-automation.md
• Risk-based tier prioritization (P0–P3): test-strategy.md and risk-based-testing.md
• Flaky quarantine workflow: test-automation.md
• Quality gates and metrics: quality-gates-and-metrics.md

---

Sources: Ekstazi (Gligoric et al.), Launchable (launchableinc.com), DORA State of DevOps Reports, Predić et al. arXiv:2106.13891 (2021).