magnus919_agent-skills/qa-methodology/references/test-debugging.md

Test Debugging

Diagnosing broken tests. Load when a test that should pass is failing, a mock isn't intercepting, a fixture is producing wrong state, or a test behaves differently in CI than locally. This is distinct from test design (that's test-strategy.md) and test quality (that's the contribution-test-quality skill's scorecard).

Diagnostic Order

1. Read the actual failure output. Not the summary line — the full traceback, assertion values, and any captured stdout/stderr.
2. Reproduce locally with the same command CI runs (including paths, markers, and filters).
3. Check collection. pytest --collect-only <path> — is the test even being collected? Zero items means the test is invisible.
4. Check environment completeness. Install the project's declared dev/test dependencies using its own manifest. A bare editable install plus pytest can omit plugins the project deliberately keeps out of runtime dependencies.
5. Isolate the variable. Run the single failing test, then the file, then the directory. Narrow until the failure appears and disappears.

Mock Path Binding After Package Refactors

When a Python module is refactored into a package (research.py → research/__init__.py + submodules), all test mock paths targeting that module break silently.

Root Cause

Relative imports bind local references at import time, before patch() runs:

# agent/research/loop.py
from ..llm import LLMClient  # binds LLMClient in loop.py's namespace NOW

Mocking at the source module (agent.llm.LLMClient) replaces the class object, but loop.py's local reference still points to the original. The mock is invisible.

Fix: Mock at the Usage Point

Wrong (source-level)	Right (usage-level)
patch("agent.llm.LLMClient")	patch("agent.research.loop.LLMClient")
patch("agent.clients.HttpClient")	patch("agent.research.loop.HttpClient")

Each submodule has its own namespace. Mock at each one exercised by the test.

Signals This Is the Problem

• AttributeError: module X does not have the attribute Y for a class that IS imported at the top of the source file
• Real API calls (401 errors, timeouts) leaking through despite a source-level patch() return_value
• The refactor commit message mentions "split X.py into X/ package"

FastAPI Startup Race

When a FastAPI app's @app.on_event("startup") handler re-assigns module-level state, any mock state set before with TestClient(app) as tc: is silently overwritten.

# server.py
_active_engines: dict = {}

@app.on_event("startup")
async def startup():
    global _active_engines
    _active_engines = discover_engines()  # overrides anything test set

Fix: Set Mock State After Context Entry

@pytest.fixture
def client():
    with TestClient(app) as tc:  # startup runs here
        server_mod._active_engines = {"mock": MockEngine()}  # set AFTER
        yield tc

Alternative: Guard the Startup Handler

@app.on_event("startup")
async def startup():
    global _active_engines
    if not _active_engines:  # allow test pre-seeding
        _active_engines = discover_engines()

httpx Mock Testing

When testing async httpx-based adapters that create their own httpx.AsyncClient inline, patch the constructor with a mock transport:

class MockHTTP:
    def __init__(self, handler):
        self.transport = httpx.MockTransport(handler)

    async def __aenter__(self):
        self.mock_client = httpx.AsyncClient(transport=self.transport)
        self.patcher = patch("httpx.AsyncClient")
        mock_class = self.patcher.start()
        mock_class.return_value.__aenter__.return_value = self.mock_client
        return self

    async def __aexit__(self, *args):
        self.patcher.stop()

# Usage:
async with MockHTTP(lambda r: httpx.Response(200, json={"results": []})):
    result = await adapter.search("query")

Test Execution Integrity

A passing command is not necessarily an executed test suite.

1. Read the collection summary. 0 items, N skipped, or exit code 5 means the intended behavior was not exercised.
2. For a module-level target, require a nonzero collected count and a passing test relevant to the change.
3. If a test is skipped because its fixture, path, or module lookup is wrong, repair that harness defect before opening the PR. Do not describe the run as passed.
4. Re-run after the repair and record the actual result (e.g., 62 passed), not only the command exit status.

CI Collection-Path Gate

A new test can pass locally and provide zero CI protection when it lives outside the directories selected by the workflow.

1. Read the exact CI test command, including explicit paths, -k filters, markers, ignore flags, and project addopts.
2. Confirm the new test's path is included by that command. Do not infer collection from a test_*.py filename alone.
3. Put the test under an already-collected directory when that matches its scope.
4. Inspect CI logs for the test/module after pushing. A green broad suite that never collected the new test is not evidence for the new contract.

Deterministic Integration Seeds

When an integration test proves a narrow behavior (ownership, authorization, ordering, isolation) but creates its precondition through a live or variable dependency (web search, third-party content, provider ranking):

1. State the behavior the test is actually proving.
2. Seed the minimal prerequisite through an existing deterministic fixture or a stable, already-tested service path.
3. Assert the seed succeeded with a small contract check.
4. Preserve the original behavior assertion unchanged.

An HTTP-successful search with zero results can be valid. If a test fails at assert refs, it has not exercised the ownership/isolation condition it claims to cover. Retrying or tuning runner resources does not repair that test design.

API Signature Changes — Fixture Recovery

When a function gains a new parameter, unit tests that mock that function silently fail. Every fixture that creates a mock of the changed function needs the new method wired.

Batch Fix Method

Do NOT use read_file/write_file for bulk fixture updates — they add line-number prefixes. Instead:

from pathlib import Path
path = Path("tests/test_file.py")
content = path.read_text()
old = "scraper = MagicMock()\n        scraper.scrape = AsyncMock()"
new = """scraper = MagicMock()
        scraper.scrape = AsyncMock()
        scraper.scrape_with_fallback = AsyncMock()"""
path.write_text(content.replace(old, new))

Lockfile Hygiene During Verification

uv run can rewrite a tracked uv.lock while resolving a local workspace package. Before testing, inspect status. If testing alone created unrelated lockfile drift, restore only that exact generated file after confirming it was not intentional. Use uv run --no-sync when the existing environment is sufficient. Never restore a lockfile containing intentional dependency changes.

Pitfalls

• Do not mask a test design failure with retries or xfail. If a test for isolation seeds itself through a live search, the fix is deterministic seeding — not retrying until the search returns results.
• Do not treat a green exit code as evidence. Read the collection count. Zero collected tests with exit 0 is not a passing suite.
• Mock at usage, not source. After any module→package refactor, audit every patch() path in the test suite against the new import structure.
• Set mocks after startup, not before. Any framework lifecycle hook that re-assigns module state will overwrite pre-context mock setup.