mirror of
https://github.com/magnus919/agent-skills.git
synced 2026-09-15 21:46:29 +03:00
usernameandfactory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
966edcf0c7
Refresh test-strategy.md, regression-testing.md; split the 28KB test-automation-gates-metrics.md into focused test-automation.md + quality-gates-and-metrics.md; delete the old monolithic file. - test-strategy.md: pyramid-as-heuristic (Trophy, Quadrants, Crab), shift-left/right, cost-of-failure, coverage-as-diagnostic, estimation, RTM traceability, accessibility dimension - test-automation.md: framework decision matrix (pytest/Playwright/ Vitest/Cypress), parallelism/sharding config, ML/predictive selection (Launchable, arXiv:2106.13891), flaky quarantine workflow, mutation testing (PIT/Stryker/mutmut) - quality-gates-and-metrics.md: blocking-vs-advisory gate design, DORA four keys + reliability, vanity-vs-actionable metrics, defect severity/priority classification with escalation rules - regression-testing.md: impact analysis (static/dynamic/ML), suite evolution tiering, selection math, every-fixed-bug-becomes-a-test, shift-right observability feedback loops, rerun-once-never-twice Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
113 lines
6.4 KiB
Markdown
113 lines
6.4 KiB
Markdown
# Regression Testing
|
||
|
||
## Core Principle
|
||
|
||
> **Every fixed bug becomes a regression test.** When a defect is fixed, the test that reproduces it (or should have caught it) is added to the suite permanently. This is the single highest-ROI regression practice: the suite's ability to catch known failure modes grows monotonically. Track "regression test added for fix" rate; target > 80% of bug fixes.
|
||
|
||
## What Belongs in a Regression Suite
|
||
|
||
| Include | Exclude |
|
||
|---------|---------|
|
||
| Every fixed bug (as a test) | Tests that haven't failed in 6+ months (review for archive) |
|
||
| Critical user paths (P0/P1) | Tests for deprecated features |
|
||
| Known failure patterns | Tests duplicating lower-level coverage |
|
||
| API contract checks | Visual regression on work-in-progress UI |
|
||
| Data integrity assertions | Performance tests (separate suite) |
|
||
|
||
## Test Impact Analysis (Shift-Left Selection)
|
||
|
||
Not every change needs the full suite. Impact analysis selects the subset of tests affected by a change.
|
||
|
||
| Approach | How It Works | Trade-off |
|
||
|----------|-------------|-----------|
|
||
| **Static call-graph** | Map changed code → functions → tests that call them | Fast, conservative (over-selects); misses dynamic dispatch |
|
||
| **Dynamic trace** | Record which tests execute which code (coverage instrumentation), then map changes → tests | Accurate; needs profiling infra; Ekstazi uses this |
|
||
| **ML prediction** | Train on historical (change, failing-test) pairs | Best precision; needs history; see [test-automation.md](./test-automation.md) |
|
||
|
||
**Tools:** Ekstazi (dynamic, JVM), Launchable (ML, language-agnostic), custom coverage-map scripts.
|
||
|
||
### Selection Math
|
||
|
||
Score each test for inclusion on a change:
|
||
|
||
```
|
||
selection_score(test) =
|
||
w1 × changed_code_coverage(test) # does it touch the diff? (0/1)
|
||
+ w2 × historical_failure_rate(test) # how often has it failed recently?
|
||
+ w3 × business_risk(test) # P0/P1 path? (see test-strategy.md)
|
||
```
|
||
|
||
Default weights: `w1=0.5, w2=0.3, w3=0.2`. Run all tests with score > threshold, plus always run the full P0 smoke set regardless of score.
|
||
|
||
| Change Type | Required Tests |
|
||
|-------------|---------------|
|
||
| Bug fix | Repro test + related unit tests |
|
||
| Feature addition | New feature tests + adjacent smoke |
|
||
| Refactoring | Full unit suite + integration smoke |
|
||
| Dependency update | Full regression suite |
|
||
| Infrastructure change | Integration + E2E suite |
|
||
|
||
## Suite Evolution (Tiering)
|
||
|
||
Tests move between tiers as the system and risk profile change:
|
||
|
||
| Tier | Cadence | Contents | Promotion / Retirement |
|
||
|------|---------|----------|----------------------|
|
||
| **Tier 0 — Smoke** | Every PR (< 5 min) | P0 paths, build sanity | Promote from Tier 1 if a path becomes critical |
|
||
| **Tier 1 — Core** | Every merge (< 30 min) | P0/P1 regression, contracts | Promote from Tier 2 on rising failure rate |
|
||
| **Tier 2 — Extended** | Nightly | P2, slower integration, E2E | Retire tests with 0 failures in 6 months + no risk coverage |
|
||
| **Tier 3 — Archive** | On demand | Retired, legacy | Restore if a related defect escapes |
|
||
|
||
**Evolution rule:** A test that fails in production-adjacent tiers gets promoted; a test with zero failures over 6 months and no P0/P1 risk coverage is a retirement candidate (review, don't auto-delete).
|
||
|
||
## Shift-Right Feedback Loops
|
||
|
||
Production signals drive regression suite composition — closing the loop between what escapes and what you test.
|
||
|
||
| Signal | Suite Action |
|
||
|--------|-------------|
|
||
| **Production error spike** (observability/APM) | Add a regression test reproducing the failing condition; promote related tests to Tier 1 |
|
||
| **Canary analysis failure** | The canary check becomes a permanent regression assertion; review why left-side testing missed it |
|
||
| **Escaped defect (customer report)** | Root-cause: add the missing test, then audit the tier that should have caught it |
|
||
| **Feature-flag rollout** | Flag-gated code gets targeted regression selection while flagged; full coverage before flag removal |
|
||
|
||
**Loop discipline:** Every escaped defect gets a post-mortem question — "which tier should have caught this, and why didn't it?" The answer drives suite evolution, not blame.
|
||
|
||
## Flaky Test Discipline: Rerun-Once, Never-Twice
|
||
|
||
> **Protocol:** In a blocking regression run, a failed test may be rerun **exactly once**. If it passes on the single rerun, record it as a **flake** (not a pass) and route to quarantine. If it fails the rerun, it is a **real failure** — block. **Never rerun a second time.**
|
||
|
||
Why never twice: a second rerun turns flake-masking into a habit. If a test needs two retries to pass, it is not reliable enough to gate on, and repeated retries hide a rising flake rate until CI trust collapses.
|
||
|
||
| Outcome | Classification | Action |
|
||
|---------|---------------|--------|
|
||
| Pass on first run | Pass | Record green |
|
||
| Fail, then pass on 1st rerun | **Flake** | Do NOT count as pass; open quarantine issue; see [test-automation.md](./test-automation.md) |
|
||
| Fail, fail on 1st rerun | **Real failure** | Block the pipeline |
|
||
| Needs 2nd rerun to pass | Unreliable | Treat as flake; quarantine immediately |
|
||
|
||
Track flake rate (flaky outcomes / total runs). Sustain at < 2%; above that, invest in stabilization before adding more tests.
|
||
|
||
> **Gotcha — Ignoring flakes:** A "pass on retry" is not a pass. Flakes are latent failures; a suite with 5% flakiness on 10,000 daily tests produces ~500 false signals/day and erodes trust until teams bypass automation entirely.
|
||
|
||
## Suite Hygiene
|
||
|
||
| Condition | Action |
|
||
|-----------|--------|
|
||
| Test hasn't failed in 6 months + no risk coverage | Review for archive |
|
||
| Test flakes > 2% over 50 runs | Quarantine and fix |
|
||
| Test takes > 5s (unit) / > 30s (integration) | Optimize or promote to slower tier |
|
||
| Test depends on another test's state | Fix isolation (tests must be independent) |
|
||
| Test runs against production data | Switch to synthetic fixtures (see [test-data-management.md](./test-data-management.md)) |
|
||
|
||
## Composition Links
|
||
|
||
- Impact-analysis tooling and ML selection: [test-automation.md](./test-automation.md)
|
||
- Risk-based tier prioritization (P0–P3): [test-strategy.md](./test-strategy.md) and [risk-based-testing.md](./risk-based-testing.md)
|
||
- Flaky quarantine workflow: [test-automation.md](./test-automation.md)
|
||
- Quality gates and metrics: [quality-gates-and-metrics.md](./quality-gates-and-metrics.md)
|
||
|
||
---
|
||
|
||
*Sources: Ekstazi (Gligoric et al.), Launchable (launchableinc.com), DORA State of DevOps Reports, Predić et al. arXiv:2106.13891 (2021).*
|