mirror of
https://github.com/magnus919/agent-skills.git
synced 2026-09-11 19:47:12 +03:00
feat(qa-methodology): add 5 fillable templates and 3 assets
Templates: test-strategy.md, risk-register.md, exploratory-charter.md, bug-report.md, verification-plan.md — all with placeholder fields. Assets: risk-matrix-grid.md (5x5 P×I grid consistent with risk-based-testing.md and risk-prioritize.py), test-design-techniques-checklist.md, qa-definition-of-done.md. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
This commit is contained in:
username
co-authored by
factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
parent
8b0220b75b
commit
332578766d
@@ -0,0 +1,68 @@
|
|||||||
|
# QA Definition of Done
|
||||||
|
|
||||||
|
> QA contribution to the team's definition of done. Adapt to your project's risk profile and release cadence. Check each item that applies before declaring a feature or release complete.
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
| Field | Value |
|
||||||
|
|-------|-------|
|
||||||
|
| Feature / Release | <name> |
|
||||||
|
| Team | <team> |
|
||||||
|
| Date | <YYYY-MM-DD> |
|
||||||
|
|
||||||
|
## Functional Verification
|
||||||
|
|
||||||
|
- [ ] All acceptance criteria verified with evidence attached
|
||||||
|
- [ ] AC→verification-method traceability is 100% (no unmapped ACs)
|
||||||
|
- [ ] Happy path tested at each applicable level (unit / integration / E2E)
|
||||||
|
- [ ] Known failure modes tested (error handling, edge cases)
|
||||||
|
- [ ] Regression suite passes (no new failures introduced)
|
||||||
|
|
||||||
|
## Risk-Based Coverage
|
||||||
|
|
||||||
|
- [ ] Risk register reviewed; all P0/P1 risks have corresponding tests
|
||||||
|
- [ ] Test allocation proportional to risk scores (P0 items receive exhaustive coverage)
|
||||||
|
- [ ] Residual risk documented and accepted by stakeholders
|
||||||
|
|
||||||
|
## Test Quality
|
||||||
|
|
||||||
|
- [ ] No flaky tests in the suite (quarantined or fixed before release)
|
||||||
|
- [ ] Test data uses synthetic or masked data (no production PII)
|
||||||
|
- [ ] Tests are deterministic (non-deterministic code verified with N-run sampling)
|
||||||
|
- [ ] Mutation testing run on critical paths (if applicable)
|
||||||
|
|
||||||
|
## Non-Functional Requirements
|
||||||
|
|
||||||
|
- [ ] Performance thresholds met (p95 latency, throughput under load)
|
||||||
|
- [ ] Security scan clean (zero critical/high findings from SAST/SCA)
|
||||||
|
- [ ] Accessibility checked for user-facing changes (WCAG 2.1/2.2 AA)
|
||||||
|
- [ ] Contract tests pass for all API consumers (if applicable)
|
||||||
|
|
||||||
|
## Automation and CI
|
||||||
|
|
||||||
|
- [ ] New tests automated and committed to the repository
|
||||||
|
- [ ] Tests integrated into CI pipeline at the correct stage
|
||||||
|
- [ ] Every fixed bug has a regression test ("every fixed bug becomes a regression test")
|
||||||
|
- [ ] CI pipeline green on the release branch
|
||||||
|
|
||||||
|
## Documentation and Handoff
|
||||||
|
|
||||||
|
- [ ] Test strategy document up to date (if changes affect scope)
|
||||||
|
- [ ] Exploratory session debriefs recorded (if sessions were run)
|
||||||
|
- [ ] Known issues and workarounds documented
|
||||||
|
- [ ] Risk register updated with new or changed risks
|
||||||
|
|
||||||
|
## Release Gate
|
||||||
|
|
||||||
|
| Gate Item | Owner | Status |
|
||||||
|
|-----------|-------|--------|
|
||||||
|
| All P0 exit criteria met | <QA lead> | Pending / Met / Waived |
|
||||||
|
| All P1 exit criteria met | <QA lead> | Pending / Met / Waived |
|
||||||
|
| Stakeholder sign-off on residual risk | <product / eng lead> | Pending / Met / Waived |
|
||||||
|
| Release decision | <release manager> | Pending / Approved / Blocked |
|
||||||
|
|
||||||
|
## Exceptions Log
|
||||||
|
|
||||||
|
| Item | Reason for Exception | Approved By | Risk Accepted |
|
||||||
|
|------|---------------------|-------------|:-------------:|
|
||||||
|
| <which DoD item> | <why it was skipped> | <name> | Yes / No |
|
||||||
@@ -0,0 +1,81 @@
|
|||||||
|
# Risk Matrix Grid (5×5 Probability × Impact)
|
||||||
|
|
||||||
|
> Use this grid during risk assessment workshops to score and zone risks. This grid is consistent with [risk-based-testing.md](../references/risk-based-testing.md) and the `risk-prioritize.py` script.
|
||||||
|
|
||||||
|
## Scoring Formula
|
||||||
|
|
||||||
|
```
|
||||||
|
Risk Score = Probability (1–5) × Impact (1–5)
|
||||||
|
```
|
||||||
|
|
||||||
|
## 5×5 Grid
|
||||||
|
|
||||||
|
| P \ I | 1 — Negligible | 2 — Minor | 3 — Moderate | 4 — Major | 5 — Catastrophic |
|
||||||
|
|-------|:--------------:|:---------:|:------------:|:---------:|:----------------:|
|
||||||
|
| **5 — Almost Certain** | 5 (P3) | 10 (P2) | 15 (P1) | 20 (P0) | 25 (P0) |
|
||||||
|
| **4 — Likely** | 4 (P3) | 8 (P2) | 12 (P1) | 16 (P0) | 20 (P0) |
|
||||||
|
| **3 — Possible** | 3 (P3) | 6 (P2) | 9 (P2) | 12 (P1) | 15 (P1) |
|
||||||
|
| **2 — Unlikely** | 2 (P3) | 4 (P3) | 6 (P2) | 8 (P2) | 10 (P2) |
|
||||||
|
| **1 — Rare** | 1 (P3) | 2 (P3) | 3 (P3) | 4 (P3) | 5 (P3) |
|
||||||
|
|
||||||
|
## Zone Thresholds
|
||||||
|
|
||||||
|
| Zone | Score Range | Priority Tier | Action |
|
||||||
|
|------|-------------|:-------------:|--------|
|
||||||
|
| **Critical** | 20–25 | P0 | Test exhaustively; every path, every edge case |
|
||||||
|
| **High** | 12–19 | P1 | Test all happy paths + known failure modes |
|
||||||
|
| **Medium** | 6–11 | P2 | Test happy paths + common failure modes |
|
||||||
|
| **Low** | 1–5 | P3 | Smoke test only; defer detailed testing |
|
||||||
|
|
||||||
|
## Probability Anchors
|
||||||
|
|
||||||
|
| Rating | Label | Anchor |
|
||||||
|
|:------:|-------|--------|
|
||||||
|
| 5 | Almost Certain | Will fail in production within a quarter (or has already) |
|
||||||
|
| 4 | Likely | Expected to fail within a year |
|
||||||
|
| 3 | Possible | Could fail; uncertain |
|
||||||
|
| 2 | Unlikely | Unlikely given current controls |
|
||||||
|
| 1 | Rare | Extremely unlikely; well-understood code |
|
||||||
|
|
||||||
|
## Impact Anchors
|
||||||
|
|
||||||
|
| Rating | Label | Anchor |
|
||||||
|
|:------:|-------|--------|
|
||||||
|
| 5 | Catastrophic | Data loss, security breach, revenue stoppage |
|
||||||
|
| 4 | Major | Major feature outage, SLA breach |
|
||||||
|
| 3 | Moderate | Degraded experience, workaround exists |
|
||||||
|
| 2 | Minor | Cosmetic, minor inconvenience |
|
||||||
|
| 1 | Negligible | No user-visible impact |
|
||||||
|
|
||||||
|
## Workshop Scoring Sheet
|
||||||
|
|
||||||
|
Record scores during the workshop, then transfer to [templates/risk-register.md](../templates/risk-register.md).
|
||||||
|
|
||||||
|
| # | Risk (short label) | P (1–5) | I (1–5) | Score | Zone | Tier |
|
||||||
|
|---|---------------------|:-------:|:-------:|:-----:|------|:----:|
|
||||||
|
| 1 | | | | | | |
|
||||||
|
| 2 | | | | | | |
|
||||||
|
| 3 | | | | | | |
|
||||||
|
| 4 | | | | | | |
|
||||||
|
| 5 | | | | | | |
|
||||||
|
| 6 | | | | | | |
|
||||||
|
| 7 | | | | | | |
|
||||||
|
| 8 | | | | | | |
|
||||||
|
|
||||||
|
## Calibration Rule
|
||||||
|
|
||||||
|
If probability votes span more than 2 points:
|
||||||
|
1. The facilitator asks the highest and lowest voter to state their evidence.
|
||||||
|
2. Re-vote once.
|
||||||
|
3. Record dissent in the register.
|
||||||
|
|
||||||
|
## Test Allocation by Tier
|
||||||
|
|
||||||
|
| Tier | Hours per Risk Item (default) | Strategy |
|
||||||
|
|:----:|------------------------------:|----------|
|
||||||
|
| P0 | 8–16 | Exhaustive design + automation |
|
||||||
|
| P1 | 4–8 | Happy paths + failure modes |
|
||||||
|
| P2 | 2–4 | Happy paths + common failures |
|
||||||
|
| P3 | 0.5–1 | Smoke only |
|
||||||
|
|
||||||
|
Adjustment: multiply by 1.5× for legacy/unfamiliar code, 0.7× for well-automated areas.
|
||||||
@@ -0,0 +1,88 @@
|
|||||||
|
# Test Design Techniques Checklist
|
||||||
|
|
||||||
|
> Quick-reference for selecting test design techniques. Check the techniques that apply to your scenario, then derive test cases. See [test-design-techniques.md](../references/test-design-techniques.md) for detailed guidance and worked examples.
|
||||||
|
|
||||||
|
## Feature Under Test
|
||||||
|
|
||||||
|
| Field | Value |
|
||||||
|
|-------|-------|
|
||||||
|
| Feature / Component | <name> |
|
||||||
|
| Tester | <name> |
|
||||||
|
| Date | <YYYY-MM-DD> |
|
||||||
|
|
||||||
|
## Technique Selection
|
||||||
|
|
||||||
|
| Technique | Applies? | Rationale |
|
||||||
|
|-----------|:--------:|-----------|
|
||||||
|
| Equivalence Partitioning (EP) | [ ] | <Why: input domains exist with distinct behavior classes> |
|
||||||
|
| Boundary Value Analysis (BVA) | [ ] | <Why: numeric or ordered ranges with edge behavior> |
|
||||||
|
| Decision Tables | [ ] | <Why: business rules with multiple interacting conditions> |
|
||||||
|
| State Transition | [ ] | <Why: stateful workflow with valid/invalid transitions> |
|
||||||
|
| Pairwise / Combinatorial | [ ] | <Why: multi-parameter configuration space> |
|
||||||
|
| Error Guessing | [ ] | <Why: known defect patterns, historical failure areas> |
|
||||||
|
|
||||||
|
## When to Use Which
|
||||||
|
|
||||||
|
| Scenario Type | Primary Technique | Supporting Technique |
|
||||||
|
|---------------|-------------------|---------------------|
|
||||||
|
| Input validation (ranges, formats) | EP + BVA | Error Guessing |
|
||||||
|
| Business logic with conditions | Decision Tables | EP |
|
||||||
|
| Workflow with states and transitions | State Transition | Decision Tables |
|
||||||
|
| Configuration with many parameters | Pairwise | EP |
|
||||||
|
| Integration with external systems | Error Guessing | EP (interface partitions) |
|
||||||
|
| UI with form fields | EP + BVA | Error Guessing |
|
||||||
|
| Permission / role-based access | Decision Tables | State Transition |
|
||||||
|
|
||||||
|
## Technique Application Notes
|
||||||
|
|
||||||
|
### Equivalence Partitioning
|
||||||
|
|
||||||
|
- [ ] Identified all input domains
|
||||||
|
- [ ] Defined valid and invalid partitions
|
||||||
|
- [ ] Selected one representative per partition
|
||||||
|
- [ ] Included type/format partitions (non-integer, empty, null)
|
||||||
|
|
||||||
|
### Boundary Value Analysis
|
||||||
|
|
||||||
|
- [ ] Used 2-value (min, max) for quick coverage
|
||||||
|
- [ ] Used 3-value (min-1, min, max, max+1) where off-by-one is likely
|
||||||
|
- [ ] Tested empty/null boundaries
|
||||||
|
|
||||||
|
### Decision Tables
|
||||||
|
|
||||||
|
- [ ] Listed all conditions and actions
|
||||||
|
- [ ] Generated complete rule combinations
|
||||||
|
- [ ] Reduced redundant rules where outcomes are identical
|
||||||
|
- [ ] Covered impossible/contradictory combinations explicitly
|
||||||
|
|
||||||
|
### State Transition
|
||||||
|
|
||||||
|
- [ ] Drew or referenced the state model
|
||||||
|
- [ ] Covered all valid transitions (0-switch)
|
||||||
|
- [ ] Tested at least one invalid transition per state
|
||||||
|
- [ ] Checked for dead states and unreachable states
|
||||||
|
|
||||||
|
### Pairwise / Combinatorial
|
||||||
|
|
||||||
|
- [ ] Listed parameters and their values
|
||||||
|
- [ ] Generated pairwise combinations (tool: PICT / allpairs)
|
||||||
|
- [ ] Added known-bad combinations from defect history
|
||||||
|
|
||||||
|
### Error Guessing
|
||||||
|
|
||||||
|
- [ ] Reviewed defect history for this component
|
||||||
|
- [ ] Checked common failure patterns (null, overflow, concurrency, timeout)
|
||||||
|
- [ ] Tested integration boundaries with invalid/malformed data
|
||||||
|
|
||||||
|
## Coverage Summary
|
||||||
|
|
||||||
|
| Dimension | Covered | Notes |
|
||||||
|
|-----------|:-------:|-------|
|
||||||
|
| All valid partitions tested | [ ] | |
|
||||||
|
| All boundaries tested | [ ] | |
|
||||||
|
| All decision rules covered | [ ] | |
|
||||||
|
| All state transitions covered | [ ] | |
|
||||||
|
| Pairwise combinations generated | [ ] | |
|
||||||
|
| Known error patterns tested | [ ] | |
|
||||||
|
|
||||||
|
> The pyramid level for each technique is a starting point, not a rule. Adapt to your system's risk profile. See [test-strategy.md](../references/test-strategy.md).
|
||||||
@@ -0,0 +1,78 @@
|
|||||||
|
# Bug Report
|
||||||
|
|
||||||
|
> File one report per defect. Provide enough detail for any engineer to reproduce without asking questions.
|
||||||
|
|
||||||
|
## Title
|
||||||
|
|
||||||
|
<One-line summary: [Component] Observed behavior under specific condition>
|
||||||
|
|
||||||
|
## Classification
|
||||||
|
|
||||||
|
| Field | Value |
|
||||||
|
|-------|-------|
|
||||||
|
| Severity | Critical / High / Medium / Low |
|
||||||
|
| Priority | P1 (Immediate) / P2 (This sprint) / P3 (Backlog) / P4 (Won't fix) |
|
||||||
|
| Component | <affected module or service> |
|
||||||
|
| Discovered In | <test type: unit / integration / E2E / exploratory / production> |
|
||||||
|
| Charter / Test ID | <link to exploratory charter or test case, if applicable> |
|
||||||
|
|
||||||
|
### Severity Definitions
|
||||||
|
|
||||||
|
| Severity | Meaning |
|
||||||
|
|----------|---------|
|
||||||
|
| Critical | Data loss, security breach, system down, revenue stoppage |
|
||||||
|
| High | Major feature broken, no workaround, SLA breach |
|
||||||
|
| Medium | Feature impaired but workaround exists |
|
||||||
|
| Low | Cosmetic, minor inconvenience, documentation error |
|
||||||
|
|
||||||
|
## Environment
|
||||||
|
|
||||||
|
| Field | Value |
|
||||||
|
|-------|-------|
|
||||||
|
| OS / Platform | <e.g., macOS 15, Ubuntu 24.04, iOS 18> |
|
||||||
|
| Browser / Client | <e.g., Chrome 131, API client v2.3> |
|
||||||
|
| Application Version | <commit SHA, release tag, or build number> |
|
||||||
|
| Environment | <local / CI / staging / production> |
|
||||||
|
| Relevant Config | <feature flags, env vars, tenant settings> |
|
||||||
|
|
||||||
|
## Reproduction Steps
|
||||||
|
|
||||||
|
1. <Step 1 — starting state or navigation>
|
||||||
|
2. <Step 2 — action taken>
|
||||||
|
3. <Step 3 — action taken>
|
||||||
|
4. <...add steps as needed>
|
||||||
|
|
||||||
|
**Reproducibility:** Always / Sometimes (___ in ___ attempts) / Once (not yet reproduced)
|
||||||
|
|
||||||
|
## Expected vs Actual
|
||||||
|
|
||||||
|
| | Description |
|
||||||
|
|---|-------------|
|
||||||
|
| **Expected** | <What should happen according to spec, docs, or reasonable behavior> |
|
||||||
|
| **Actual** | <What actually happens — be specific about the observed behavior> |
|
||||||
|
|
||||||
|
## Evidence
|
||||||
|
|
||||||
|
Attach or link:
|
||||||
|
|
||||||
|
- [ ] Screenshot / screen recording: <path or URL>
|
||||||
|
- [ ] Error message / stack trace: <paste below or link>
|
||||||
|
- [ ] Logs: <path or link>
|
||||||
|
- [ ] Network capture: <path or link>
|
||||||
|
- [ ] Test output: <CI link or local path>
|
||||||
|
|
||||||
|
```
|
||||||
|
<paste error output or stack trace here>
|
||||||
|
```
|
||||||
|
|
||||||
|
## Additional Context
|
||||||
|
|
||||||
|
<Anything that helps triage: recent changes, related issues, suspected root cause, whether it blocks release.>
|
||||||
|
|
||||||
|
## Escalation
|
||||||
|
|
||||||
|
| Condition | Action |
|
||||||
|
|-----------|--------|
|
||||||
|
| Severity = Critical | Blocks release; escalate to engineering leadership within 1 hour |
|
||||||
|
| Severity = High + Priority = P1 | Escalate to team lead within 4 hours |
|
||||||
|
| Unreproducible after 3 attempts | Add to investigation backlog with environment capture |
|
||||||
@@ -0,0 +1,83 @@
|
|||||||
|
# Exploratory Testing Charter
|
||||||
|
|
||||||
|
> Fill in for each SBTM session. One charter per session. See [exploratory-testing.md](../references/exploratory-testing.md) for charter quality guidance and oracle heuristics.
|
||||||
|
|
||||||
|
## Charter
|
||||||
|
|
||||||
|
**Explore** <target area / feature / component>
|
||||||
|
**with** <resources, constraints, or test conditions>
|
||||||
|
**to discover** <information or risks sought>.
|
||||||
|
|
||||||
|
## Session Setup
|
||||||
|
|
||||||
|
| Field | Value |
|
||||||
|
|-------|-------|
|
||||||
|
| Tester(s) | <name(s)> |
|
||||||
|
| Date | <YYYY-MM-DD> |
|
||||||
|
| Timebox | <60 / 90 / 120> minutes |
|
||||||
|
| Environment | <local / staging / specific config> |
|
||||||
|
| Test Data / Tools | <specific data sets, proxies, throttling, accounts> |
|
||||||
|
|
||||||
|
## Charter Quality Checklist
|
||||||
|
|
||||||
|
Before starting, confirm:
|
||||||
|
|
||||||
|
- [ ] Target is specific (not "the app")
|
||||||
|
- [ ] Resources or constraints are named (test data, tools, conditions)
|
||||||
|
- [ ] Information goal is stated (not just "find bugs")
|
||||||
|
- [ ] Scope fits within the timebox
|
||||||
|
|
||||||
|
## Heuristics Applied
|
||||||
|
|
||||||
|
Select oracles and coverage heuristics to guide exploration (see [exploratory-testing.md](../references/exploratory-testing.md)):
|
||||||
|
|
||||||
|
- [ ] SFDIPOT coverage (Structure, Function, Data, Interfaces, Platform, Operations, Time)
|
||||||
|
- [ ] HICCUPPS oracle (History, Image, Comparable, Claims, Users, Product, Purpose, Standards)
|
||||||
|
- [ ] Tours (e.g., Guidebook, Money, Supermodel, Saboteur, Back-Alley)
|
||||||
|
- [ ] Other: <specify>
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
<Record observations, questions, areas explored, anomalies, and hunches during the session. Append chronologically.>
|
||||||
|
|
||||||
|
-
|
||||||
|
|
||||||
|
## T/B/B Metrics
|
||||||
|
|
||||||
|
Track time allocation at session end:
|
||||||
|
|
||||||
|
| Metric | Minutes | Percentage |
|
||||||
|
|--------|--------:|-----------:|
|
||||||
|
| **T** — Test time (designing + executing) | | ___% |
|
||||||
|
| **B** — Bug investigation | | ___% |
|
||||||
|
| **B** — Setup / Interruption | | ___% |
|
||||||
|
| **Total** | | 100% |
|
||||||
|
|
||||||
|
> Target: T ≥ 70%, Setup ≤ 10%. If T < 60%, fix environmental blockers before scheduling more sessions.
|
||||||
|
|
||||||
|
## Debrief
|
||||||
|
|
||||||
|
### What did you test?
|
||||||
|
|
||||||
|
<Areas covered, charters fulfilled, techniques used.>
|
||||||
|
|
||||||
|
### What did you find?
|
||||||
|
|
||||||
|
<Bugs filed (IDs), risks identified, questions raised.>
|
||||||
|
|
||||||
|
| Finding | Type (Bug / Risk / Question) | Severity / Priority | Bug ID |
|
||||||
|
|---------|------------------------------|---------------------|--------|
|
||||||
|
| <description> | | | |
|
||||||
|
| <description> | | | |
|
||||||
|
|
||||||
|
### What is left untested?
|
||||||
|
|
||||||
|
<Scope not reached, new areas discovered, follow-up needed.>
|
||||||
|
|
||||||
|
### Follow-Up Actions
|
||||||
|
|
||||||
|
- [ ] <New charter needed: ___>
|
||||||
|
- [ ] <Automation candidate: ___>
|
||||||
|
- [ ] <Risk register update: ___>
|
||||||
|
- [ ] <Spec clarification needed: ___>
|
||||||
|
- [ ] <Other: ___>
|
||||||
@@ -0,0 +1,70 @@
|
|||||||
|
# Risk Register
|
||||||
|
|
||||||
|
> Record risk assessment results. Score each risk using the 5×5 P×I grid in [assets/risk-matrix-grid.md](../assets/risk-matrix-grid.md). See [risk-based-testing.md](../references/risk-based-testing.md) for scoring guidance and reassessment triggers.
|
||||||
|
|
||||||
|
## Metadata
|
||||||
|
|
||||||
|
| Field | Value |
|
||||||
|
|-------|-------|
|
||||||
|
| Project / Release | <name> |
|
||||||
|
| Workshop Date | <YYYY-MM-DD> |
|
||||||
|
| Facilitator | <name> |
|
||||||
|
| Participants | <names / roles> |
|
||||||
|
| Next Review Date | <YYYY-MM-DD> |
|
||||||
|
|
||||||
|
## Risk Items
|
||||||
|
|
||||||
|
| ID | Risk | Component | Probability (1–5) | Impact (1–5) | Score | Tier | Mitigation / Test Plan | Owner | Status | Last Reviewed | Reassessment Trigger |
|
||||||
|
|----|------|-----------|--------------------:|-------------:|------:|------|------------------------|-------|--------|---------------|----------------------|
|
||||||
|
| RISK-001 | <what could go wrong> | <affected area> | | | | P0 / P1 / P2 / P3 | <tests or controls> | <who> | Open / Mitigating / Closed | <YYYY-MM-DD> | <event that re-opens> |
|
||||||
|
| RISK-002 | <what could go wrong> | <affected area> | | | | P0 / P1 / P2 / P3 | <tests or controls> | <who> | Open / Mitigating / Closed | <YYYY-MM-DD> | <event that re-opens> |
|
||||||
|
| RISK-003 | <what could go wrong> | <affected area> | | | | P0 / P1 / P2 / P3 | <tests or controls> | <who> | Open / Mitigating / Closed | <YYYY-MM-DD> | <event that re-opens> |
|
||||||
|
|
||||||
|
### Column Guide
|
||||||
|
|
||||||
|
| Column | How to Fill |
|
||||||
|
|--------|-------------|
|
||||||
|
| ID | Sequential unique identifier (RISK-001, RISK-002, ...) |
|
||||||
|
| Risk | Plain-language description of the failure scenario |
|
||||||
|
| Component | System area or service affected |
|
||||||
|
| Probability (1–5) | 1=Rare, 2=Unlikely, 3=Possible, 4=Likely, 5=Almost Certain |
|
||||||
|
| Impact (1–5) | 1=Negligible, 2=Minor, 3=Moderate, 4=Major, 5=Catastrophic |
|
||||||
|
| Score | Probability × Impact (range 1–25) |
|
||||||
|
| Tier | From score: 20–25=P0, 12–19=P1, 6–11=P2, 1–5=P3 |
|
||||||
|
| Mitigation / Test Plan | Specific tests or controls that reduce this risk |
|
||||||
|
| Owner | Person accountable for implementing the mitigation |
|
||||||
|
| Status | Open (identified, no action) / Mitigating (in progress) / Closed (mitigated or accepted) |
|
||||||
|
| Last Reviewed | Date of most recent reassessment |
|
||||||
|
| Reassessment Trigger | Event that forces a re-score (incident, architecture change, release) |
|
||||||
|
|
||||||
|
## Scoring Anchors
|
||||||
|
|
||||||
|
| Rating | Probability | Impact |
|
||||||
|
|--------|-------------|--------|
|
||||||
|
| 5 | Will fail in production within a quarter | Data loss, security breach, revenue stoppage |
|
||||||
|
| 4 | Expected to fail within a year | Major feature outage, SLA breach |
|
||||||
|
| 3 | Could fail; uncertain | Degraded experience, workaround exists |
|
||||||
|
| 2 | Unlikely given current controls | Cosmetic, minor inconvenience |
|
||||||
|
| 1 | Extremely unlikely; well-understood code | No user-visible impact |
|
||||||
|
|
||||||
|
## Reassessment Triggers
|
||||||
|
|
||||||
|
Re-score the register when any of these events occur:
|
||||||
|
|
||||||
|
- [ ] Production incident in a registered area
|
||||||
|
- [ ] Architecture change (new dependency, refactor)
|
||||||
|
- [ ] New regulatory requirement
|
||||||
|
- [ ] Major release or migration
|
||||||
|
- [ ] Quarterly calendar review (default cadence)
|
||||||
|
- [ ] Team change (key engineer leaves)
|
||||||
|
- [ ] Customer escalation
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
| Tier | Count | Total Test Hours (estimate) |
|
||||||
|
|------|------:|----------------------------:|
|
||||||
|
| P0 (20–25) | | |
|
||||||
|
| P1 (12–19) | | |
|
||||||
|
| P2 (6–11) | | |
|
||||||
|
| P3 (1–5) | | |
|
||||||
|
| **Total** | | |
|
||||||
@@ -0,0 +1,107 @@
|
|||||||
|
# Test Strategy
|
||||||
|
|
||||||
|
> Fill in each section for the target project or feature. Replace `<...>` placeholders with concrete values. Delete guidance notes once populated.
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
| Field | Value |
|
||||||
|
|-------|-------|
|
||||||
|
| Project / Feature | <name> |
|
||||||
|
| Version / Release | <version or release identifier> |
|
||||||
|
| Author | <author> |
|
||||||
|
| Date | <YYYY-MM-DD> |
|
||||||
|
| Status | Draft / In Review / Approved |
|
||||||
|
|
||||||
|
### Scope
|
||||||
|
|
||||||
|
<What is in scope for this strategy? List the components, features, or services under test.>
|
||||||
|
|
||||||
|
### Out of Scope
|
||||||
|
|
||||||
|
<What is explicitly excluded and why?>
|
||||||
|
|
||||||
|
## Risk Tiers
|
||||||
|
|
||||||
|
Assign each area a risk tier (P0–P3) using probability × impact scoring. See [risk-based-testing.md](../references/risk-based-testing.md) and [assets/risk-matrix-grid.md](../assets/risk-matrix-grid.md) for the scoring grid.
|
||||||
|
|
||||||
|
| Area / Component | Probability (1–5) | Impact (1–5) | Score | Tier |
|
||||||
|
|------------------|--------------------:|-------------:|------:|------|
|
||||||
|
| <area 1> | | | | P0 / P1 / P2 / P3 |
|
||||||
|
| <area 2> | | | | P0 / P1 / P2 / P3 |
|
||||||
|
| <area 3> | | | | P0 / P1 / P2 / P3 |
|
||||||
|
|
||||||
|
### Tier Definitions
|
||||||
|
|
||||||
|
| Tier | Score Range | Coverage Approach |
|
||||||
|
|------|-------------|-------------------|
|
||||||
|
| P0 — Critical | 20–25 | Exhaustive: every path, every edge case |
|
||||||
|
| P1 — High | 12–19 | All happy paths + known failure modes |
|
||||||
|
| P2 — Medium | 6–11 | Happy paths + common failure modes |
|
||||||
|
| P3 — Low | 1–5 | Smoke test only; defer detailed testing |
|
||||||
|
|
||||||
|
## Test Levels
|
||||||
|
|
||||||
|
| Level | What It Covers | Framework / Tool | Target Coverage by Tier |
|
||||||
|
|-------|---------------|-----------------|------------------------|
|
||||||
|
| Unit | <e.g., business logic, validators> | <framework> | P0: ___% / P1: ___% / P2: ___% / P3: ___% |
|
||||||
|
| Integration | <e.g., API contracts, service boundaries> | <framework> | P0: ___% / P1: ___% / P2: ___% / P3: ___% |
|
||||||
|
| End-to-End | <e.g., critical user journeys> | <framework> | P0: ___% / P1: ___% / P2: ___% / P3: ___% |
|
||||||
|
| Exploratory | <e.g., new features, unknown-risk areas> | SBTM charter | Sessions per sprint: ___ |
|
||||||
|
|
||||||
|
> The pyramid shape is a heuristic, not dogma. Adjust level allocation to match your system's risk profile. See [test-strategy.md](../references/test-strategy.md).
|
||||||
|
|
||||||
|
## Coverage Approach
|
||||||
|
|
||||||
|
| Dimension | How It Is Measured | Diagnostic Target | Notes |
|
||||||
|
|-----------|--------------------|-------------------|-------|
|
||||||
|
| Code coverage | <tool> | <e.g., 80% line on P0 modules> | Coverage is diagnostic, not a goal |
|
||||||
|
| Requirement coverage | <traceability matrix / RTM> | Every AC has ≥ 1 test | |
|
||||||
|
| Risk coverage | <risk register mapping> | Every P0/P1 risk has tests | |
|
||||||
|
| Mutation score | <tool, e.g., PIT/Stryker/mutmut> | <target % on critical paths> | Optional; measures test effectiveness |
|
||||||
|
|
||||||
|
## Environment
|
||||||
|
|
||||||
|
| Environment | Purpose | Data Strategy | Access |
|
||||||
|
|-------------|---------|---------------|--------|
|
||||||
|
| <e.g., local / CI> | <unit + integration> | <fixtures / factories> | <who> |
|
||||||
|
| <e.g., staging> | <integration + E2E> | <masked subset / synthetic> | <who> |
|
||||||
|
| <e.g., perf> | <load / soak> | <generated at scale> | <who> |
|
||||||
|
|
||||||
|
> Never use production PII in test environments. See [test-data-management.md](../references/test-data-management.md).
|
||||||
|
|
||||||
|
## Execution Plan
|
||||||
|
|
||||||
|
| Phase | What Runs | Trigger | Cadence |
|
||||||
|
|-------|-----------|---------|---------|
|
||||||
|
| Pre-merge | <unit + lint + fast integration> | PR opened / updated | Every PR |
|
||||||
|
| Post-merge | <full integration + E2E> | Merge to main | Every merge |
|
||||||
|
| Nightly | <full regression + perf smoke> | Scheduled | Daily |
|
||||||
|
| Release | <full suite + exploratory + security scan> | Release candidate | Per release |
|
||||||
|
|
||||||
|
## Automation Targets
|
||||||
|
|
||||||
|
| Area | Current State | Target State | Priority |
|
||||||
|
|------|--------------|--------------|----------|
|
||||||
|
| <area 1> | <manual / partial / automated> | <target> | P0 / P1 / P2 / P3 |
|
||||||
|
| <area 2> | <manual / partial / automated> | <target> | P0 / P1 / P2 / P3 |
|
||||||
|
|
||||||
|
## Exit Criteria
|
||||||
|
|
||||||
|
The test strategy is complete when:
|
||||||
|
|
||||||
|
- [ ] Every in-scope area has a risk tier assigned (P0–P3)
|
||||||
|
- [ ] Test level allocation is defined per tier
|
||||||
|
- [ ] Coverage dimensions and diagnostic targets are set
|
||||||
|
- [ ] Environment and data strategy are specified
|
||||||
|
- [ ] Execution cadence is agreed with the team
|
||||||
|
- [ ] Automation targets have owners and timelines
|
||||||
|
- [ ] Exit criteria for each tier are defined below
|
||||||
|
|
||||||
|
### Per-Tier Exit Criteria
|
||||||
|
|
||||||
|
| Tier | Exit Criteria |
|
||||||
|
|------|--------------|
|
||||||
|
| P0 | <e.g., 100% path coverage, zero open critical defects, mutation score ≥ ___%> |
|
||||||
|
| P1 | <e.g., all happy paths + failure modes pass, zero open high defects> |
|
||||||
|
| P2 | <e.g., happy paths pass, known issues documented> |
|
||||||
|
| P3 | <e.g., smoke suite green> |
|
||||||
@@ -0,0 +1,88 @@
|
|||||||
|
# Verification Plan
|
||||||
|
|
||||||
|
> Plan independent verification for a spec, feature, or AI-generated implementation. Fill in the traceability table, assign verifiers, define evidence, and set exit criteria. See [ai-code-quality-gates.md](../references/ai-code-quality-gates.md) for gate context and the independent-verification principle.
|
||||||
|
|
||||||
|
## Metadata
|
||||||
|
|
||||||
|
| Field | Value |
|
||||||
|
|-------|-------|
|
||||||
|
| Feature / Spec | <name or spec document reference> |
|
||||||
|
| Author | <who wrote this plan> |
|
||||||
|
| Date | <YYYY-MM-DD> |
|
||||||
|
| Gate Entry | Gate 1 (spec review) / Gate 3 (implementation) / Gate 4 (acceptance) |
|
||||||
|
| Status | Draft / In Review / Active / Complete |
|
||||||
|
|
||||||
|
## AC → Verification-Method Traceability
|
||||||
|
|
||||||
|
Every acceptance criterion must map to at least one verification method. No AC may be unmapped.
|
||||||
|
|
||||||
|
| AC ID | Acceptance Criterion | Verification Method | Verifier | Evidence Format | Status |
|
||||||
|
|-------|---------------------|--------------------:|----------|-----------------|--------|
|
||||||
|
| AC-1 | <criterion text> | Test / Inspection / Analysis / Demonstration | <who or what> | <artifact that proves it> | Pending / Pass / Fail |
|
||||||
|
| AC-2 | <criterion text> | Test / Inspection / Analysis / Demonstration | <who or what> | <artifact that proves it> | Pending / Pass / Fail |
|
||||||
|
| AC-3 | <criterion text> | Test / Inspection / Analysis / Demonstration | <who or what> | <artifact that proves it> | Pending / Pass / Fail |
|
||||||
|
|
||||||
|
### Verification Method Definitions
|
||||||
|
|
||||||
|
| Method | When to Use |
|
||||||
|
|--------|-------------|
|
||||||
|
| **Test** | Executable check: automated script, unit/integration/E2E test |
|
||||||
|
| **Inspection** | Static review: code review, spec walkthrough, checklist audit |
|
||||||
|
| **Analysis** | Computed or derived evidence: metrics, logs, statistical sampling |
|
||||||
|
| **Demonstration** | Live or recorded walkthrough showing behavior under controlled conditions |
|
||||||
|
|
||||||
|
## Verifier Assignment
|
||||||
|
|
||||||
|
> The implementing agent (or developer) MUST NOT self-verify. Assign verification to a separate agent session, a different human, or an independent automated process.
|
||||||
|
|
||||||
|
| AC ID(s) | Verifier | Independence Mechanism |
|
||||||
|
|-----------|----------|----------------------|
|
||||||
|
| <AC-1, AC-2> | <separate agent session / human reviewer / CI pipeline> | <fresh context, no shared priors / different team member> |
|
||||||
|
| <AC-3> | <separate agent session / human reviewer / CI pipeline> | <fresh context, no shared priors / different team member> |
|
||||||
|
|
||||||
|
## Evidence Format
|
||||||
|
|
||||||
|
| Evidence Type | Format | Storage / Link |
|
||||||
|
|---------------|--------|----------------|
|
||||||
|
| Test output | <log file, JSON report, CI run URL> | <path or URL> |
|
||||||
|
| Inspection record | <review comments, checklist sign-off> | <path or URL> |
|
||||||
|
| Metrics snapshot | <dashboard screenshot, exported CSV> | <path or URL> |
|
||||||
|
| Demonstration | <screen recording, live session recording> | <path or URL> |
|
||||||
|
|
||||||
|
## NFR Verification
|
||||||
|
|
||||||
|
| NFR | Category | Measurement Approach | Threshold | Evidence |
|
||||||
|
|-----|----------|---------------------|-----------|----------|
|
||||||
|
| <e.g., API latency> | Performance | <p95 under load test> | <e.g., p95 < 200ms> | <perf report link> |
|
||||||
|
| <e.g., error rate> | Reliability | <soak test + monitoring> | <e.g., < 0.1%> | <monitoring snapshot> |
|
||||||
|
| <e.g., auth bypass> | Security | <SAST scan + manual review> | <zero critical findings> | <scan report link> |
|
||||||
|
| <e.g., response shape> | Contract | <contract test (Pact)> | <all consumers pass> | <contract test output> |
|
||||||
|
|
||||||
|
## Non-Determinism Handling
|
||||||
|
|
||||||
|
| Condition | Verification Approach |
|
||||||
|
|-----------|----------------------|
|
||||||
|
| Output is deterministic (same input → same output) | Single-run equality check |
|
||||||
|
| Output varies across N runs but has invariants | Property-based assertions + N-run sampling (N ≥ 5) |
|
||||||
|
| Output is probabilistic with known distribution | Statistical tolerance bands (e.g., p95 < X) |
|
||||||
|
| Output depends on external state | Pin external state; verify under controlled conditions |
|
||||||
|
|
||||||
|
**Decision rule:** Run the implementation 5 times with identical inputs. If any output differs, use property-based or statistical verification.
|
||||||
|
|
||||||
|
## Exit Criteria
|
||||||
|
|
||||||
|
Verification is complete when:
|
||||||
|
|
||||||
|
- [ ] 100% of ACs have a mapped verification method (no unmapped ACs)
|
||||||
|
- [ ] All verifications executed with evidence attached
|
||||||
|
- [ ] Zero critical or high severity findings remain open
|
||||||
|
- [ ] NFR thresholds met with evidence
|
||||||
|
- [ ] Verifier independence confirmed (implementer ≠ verifier)
|
||||||
|
- [ ] <Additional project-specific criteria: ___>
|
||||||
|
|
||||||
|
## Findings Log
|
||||||
|
|
||||||
|
| Finding ID | AC | Description | Severity | Status | Resolution |
|
||||||
|
|------------|-----|-------------|----------|--------|------------|
|
||||||
|
| VF-001 | <AC-ID> | <what was found> | Critical / High / Medium / Low | Open / Resolved / Accepted | <how resolved> |
|
||||||
|
| VF-002 | <AC-ID> | <what was found> | Critical / High / Medium / Low | Open / Resolved / Accepted | <how resolved> |
|
||||||
Reference in New Issue
Block a user