magnus919_agent-skills/qa-methodology/references/test-debugging.md

Test Debugging

Diagnosing broken tests. Load when a test that should pass is failing, a mock isn't intercepting, a fixture is producing wrong state, or a test behaves differently in CI than locally. Distinct from test design (that's test-strategy.md) and CI infrastructure triage (that's ci-failure-triage.md).

Diagnostic Order

1. Read the actual failure output. Not the summary line — the full traceback, assertion values, and any captured stdout/stderr.
2. Reproduce locally with the same command CI runs (including paths, markers, and filters).
3. Check collection. pytest --collect-only <path> — is the test even being collected? Zero items means the test is invisible.
4. Check environment completeness. Install the project's declared dev/test dependencies using its own manifest.
5. Isolate the variable. Run the single failing test, then the file, then the directory. Narrow until the failure appears and disappears.

CI-vs-Local Divergence Checklist

When a test passes locally but fails in CI (or vice versa), work through these causes systematically:

#	Divergence Cause	Symptom	Diagnosis	Fix
1	Environment variables	Test reads os.environ differently	`diff <(env	sort) <(ci_env
2	Timing and concurrency	Race conditions surface under CI load	Run failing test 50× locally with pytest-repeat; add --count=50	Fix the race (proper synchronization), not the timing
3	Test ordering / shared state	Passes alone, fails in suite	Run with pytest-randomly or reverse order	Eliminate shared mutable state; each test constructs its own fixtures
4	Filesystem differences	Path separators, symlinks, case sensitivity (Linux CI vs macOS local)	Check for hardcoded paths; find . -name "Test_*" vs "test_*"	Use pathlib.Path; never hardcode separators
5	Network access	CI has no internet or restricted egress	Check for real HTTP calls; CI logs show ConnectionRefused	Mock external calls; use recorded fixtures (VCR.py)
6	Dependency versions	CI resolves different versions than local	Compare pip freeze / npm ls outputs	Commit lockfiles; use exact pins in CI
7	Timezone and locale	Date formatting, string collation differ	echo $TZ $LANG locally vs CI	Set TZ=UTC and LC_ALL=C explicitly in tests

Gotcha: "Works on My Machine" Is a Bug Report

A CI-only failure is not "CI being flaky" until you have ruled out all seven causes above. The most common root causes are ordering (3) and environment variables (1).

Test Ordering and Shared State

Tests that pass individually but fail as a suite have ordering dependencies. This is a test design defect, not an infrastructure problem.

Detection

# Install pytest-randomly — it randomizes order on every run
pip install pytest-randomly

# Run with a specific seed to reproduce
pytest --randomly-seed=12345 tests/

# Reverse execution order
pip install pytest-reverse
pytest --reverse tests/

Common Shared-State Patterns

Pattern	Symptom	Fix
Module-level mutable global	Test A sets state; Test B reads it	Reset in fixture setup/teardown; prefer function-scoped fixtures
Class-level setUpClass mutation	Later tests depend on earlier test's writes	Move to per-test setup; use setUp not setUpClass
Database rows from prior test	Query returns unexpected count	Transaction rollback per test (see test-data-management.md)
File artifacts on disk	Test reads a file another test created	Use tmp_path fixture; never write to shared directories
Environment variable mutation	os.environ["X"] = "y" leaks across tests	Use monkeypatch.setenv() (auto-reverts)
Monkeypatched module attribute	module.GLOBAL = value without cleanup	Use monkeypatch.setattr() (auto-reverts)

pytest-randomly as a Design Tool

Run pytest-randomly in CI on every run. Tests that fail under random ordering have latent shared-state bugs. Fix the isolation defect rather than pinning the order — pinning hides the problem until the next refactor breaks the assumed order.

Mock Path Binding at the Usage Point

When you patch() a name in Python, you must patch it where it is looked up (the usage point), not where it is defined.

Root Cause

Python imports bind names into the importing module's namespace at import time:

# myapp/service.py
from myapp.clients import HttpClient  # binds 'HttpClient' in service.py's namespace

def fetch_data():
    client = HttpClient()  # looks up 'HttpClient' in service.py's globals
    return client.get("/api/data")

# WRONG: patches the definition point — service.py still has the original reference
@patch("myapp.clients.HttpClient")

# RIGHT: patches where service.py looks it up
@patch("myapp.service.HttpClient")
def test_fetch_data(mock_client):
    mock_client.return_value.get.return_value = {"result": "ok"}
    assert fetch_data() == {"result": "ok"}

Signals This Is the Problem

Signal	Meaning
AttributeError: module X does not have the attribute Y	Patching at a module that doesn't import Y directly
Real HTTP calls despite patch() with return_value	Patched the wrong namespace; real client still bound
Mock works in one test file but not another	Each importing module has its own binding; patch each
Refactor moved code to a subpackage	All patch() paths targeting the old module are now wrong

Rule of Thumb

Import Style	Patch Target
from module import Class	patch("consumer_module.Class")
import module; module.Class()	patch("module.Class")
from module import func used in 3 files	Patch in all 3 consumer modules

FastAPI Startup Race

When a FastAPI app's @app.on_event("startup") handler re-assigns module-level state, any mock state set before with TestClient(app) as tc: is silently overwritten.

# Fix: set mock state AFTER context entry
@pytest.fixture
def client():
    with TestClient(app) as tc:  # startup runs here
        server_mod._active_engines = {"mock": MockEngine()}  # set AFTER
        yield tc

Test Execution Integrity

A passing command is not necessarily an executed test suite.

1. Read the collection summary. 0 items, N skipped, or exit code 5 means the intended behavior was not exercised.
2. For a module-level target, require a nonzero collected count and a passing test relevant to the change.
3. If a test is skipped because its fixture or path is wrong, repair that harness defect before opening the PR.
4. Re-run after the repair and record the actual result (e.g., 62 passed), not only the exit status.

CI Collection-Path Gate

A new test can pass locally and provide zero CI protection when it lives outside the directories selected by the workflow.

1. Read the exact CI test command, including explicit paths, -k filters, markers, and ignore flags.
2. Confirm the new test's path is included by that command.
3. Put the test under an already-collected directory when that matches its scope.
4. Inspect CI logs for the test/module after pushing.

Gotchas

• Do not mask a test design failure with retries or xfail. If a test fails under random ordering, fix the isolation — don't pin the order.
• Do not treat a green exit code as evidence. Read the collection count. Zero collected tests with exit 0 is not a passing suite.
• Mock at usage, not source. After any module→package refactor, audit every patch() path against the new import structure.
• Set mocks after startup, not before. Any framework lifecycle hook that re-assigns module state will overwrite pre-context mock setup.
• Patching time.sleep instead of freezing time creates fragile tests. Use freezegun or timecop (see test-data-management.md).

Composition

• Systematic debugging methodology (hypothesis-driven, evidence-first): systematic-debugging
• CI infrastructure triage (exit codes, bisect, runner issues): ci-failure-triage.md
• Test data isolation and time freezing: test-data-management.md

Sources: pytest-randomly (GitHub, adamchainz), pytest docs on monkeypatch (docs.pytest.org), Python unittest.mock docs (docs.python.org), freezegun (GitHub, spulec).