magnus919_agent-skills/opensource-contributions/references/ci-debugging-loop.md

CI Debugging Loop

A systematic approach to fixing CI failures on a Python project. Use this anytime CI turns red after a push.

Step 1: Read the CI Logs

# Get the failed run ID
gh run list --limit 3 --json conclusion,displayTitle,status,databaseId

# View failed steps
gh run view <RUN_ID> --log-failed

# View full output for a specific job
gh run view <RUN_ID> --log

What you're looking for: The exact test name that failed, the assertion error, and any clues about why (env var mismatch, missing method, import error, etc.).

Step 2: Reproduce Locally

# Reproduce just the failing test
pytest -xvs tests/test_file.py::test_name

# If it passes locally but fails in CI, check for environment differences:
# - Python version (CI may use 3.11 vs your 3.13)
# - Installed dependencies (CI installs fresh; you may have stale packages)
# - Environment variables (CI sets HF_HUB_OFFLINE=1, etc.)
# - Entry-point metadata (`pip install -e .` may cache old entry points)

Step 3: Isolate Pre-existing vs New Failures

Before assuming your change broke something, check if the test was already failing:

# Stash your changes, run the failing test on clean main
git stash
pytest -xvs tests/test_file.py::test_name
git stash pop

Result	Diagnosis
Fails on clean main too	Pre-existing — your change didn't cause it. Fix separately or skip.
Passes on clean main, fails with your changes	Regression — your change caused it. Debug the interaction.
Different test fails on clean main	Test ordering / flaky — env sensitivity or timing race. Run again.

Step 4: Root Cause Categories

Env Var Leakage

The user's development environment may set env vars that leak into tests. Common culprits:

• CASHEW_* vars from a running Hermes session override config defaults in tests
• OPENAI_API_KEY, ANTHROPIC_API_KEY, etc. causing unintended API calls
• Path vars that influence import resolution

Fix pattern: Add an autouse fixture in conftest.py that strips the leaking vars:

@pytest.fixture(autouse=True)
def _clear_leaking_env_vars(monkeypatch: pytest.MonkeyPatch):
    for key in list(os.environ):
        if key.startswith("CASHEW_"):
            monkeypatch.delenv(key, raising=False)

Mirror the existing pattern already used by the project (e.g., HF_HUB_OFFLINE=1 in conftest).

Stale Package Metadata

pip install -e . installs entry point metadata that persists across source changes. If you change pyproject.toml's entry point but don't reinstall, the old metadata is used:

# Check what's actually registered
python3 -c "import importlib.metadata as im; eps=list(im.entry_points(group='hermes_agent.plugins')); [print(f'{ep.name} = {ep.value}') for ep in eps]"

Fix: Update the installed entry point metadata directly, or reinstall:

# Option A: Patch the installed entry_points.txt
# Find the dist-info directory:
python3 -c "import importlib.metadata as im; d=im.distribution('your-package'); print(d._path)"
# Edit entry_points.txt in that directory

# Option B: Reinstall
pip install -e ".[dev]" --force-reinstall --no-deps

Specific gotcha with entry point suffixed targets: An entry point declared as module.path (without :suffix) resolves to a module via ep.load(). If declared as module.path:function, it resolves to the function. If you switch between these, any tests that call ep.load() and expect a particular type will break. On macOS, stale dist-info may retain the old format — always check with python3 -c "import importlib.metadata as im; ..." to see what's actually registered.

Removed/Refactored Methods

After a major refactor, tests that reference old method names or monkeypatch old internals will fail:

# Check if the method still exists
python3 -c "from your_module import YourClass; print(hasattr(YourClass, 'old_method_name'))"

Fix pattern: Update the test to use the new API surface. If mocking internal methods, mock at the boundary instead (the public API or the dependency's interface).

Immutable Types (Python 3.11+)

On modern Python, some C-level types (notably sqlite3.Connection) are immutable. monkeypatch.setattr appears to succeed but the teardown raises TypeError:

# Detection: try to set a dummy attribute
try:
    sqlite3.Connection._test_mutability = lambda: None
    del sqlite3.Connection._test_mutability
except TypeError:
    # Immutable — can't use monkeypatch on this type
    pass

Fix: Detect immutability and adapt the test — either skip the mock and test the fallback path directly, or mock at a different level.

Direct Dependency in PyPI Package

If pyproject.toml has a git+SHA pinned dependency (e.g., package @ git+https://github.com/user/repo.git@abc123), the PyPI publish step will reject the package with 400 Can't have direct dependency. The build and wheel-smoke steps pass — only the upload fails.

Fix: Check if the dependency is available on PyPI. If yes, switch to a version specifier:

# Before (blocks PyPI):
"package @ git+https://github.com/user/repo.git@abc123"

# After:
"package>=1.0.0,<2.0.0"

Step 5: Fix and Verify

1. Make the minimal fix
2. Run the failing test locally: pytest -xvs tests/test_file.py::test_name
3. Run the full test suite: pytest
4. Commit with conventional commit + DCO sign-off: git commit -s -m "fix: description"
5. Push: git push

Step 6: Verify CI Passes

After pushing, check CI status proactively — don't wait for the user to tell you it's red:

# Wait for CI to start and complete
gh run watch <RUN_ID> --exit-status

# Or check status periodically
gh run list --limit 1 --json conclusion,status

If still red, go back to Step 1. Repeat until green.

Step 7: Multiple Fixes, Multiple Commits

When fixing a series of related CI issues, use one commit per root cause — not one commit per test file, and definitely not one mega-commit. This keeps the history reviewable and makes it easy to revert individual fixes if needed.

Example from a real session:

fix: switch dependency from git+SHA to PyPI specifier         # unblocks release
fix: handle repr-style list env vars in config                # fixes config parsing
fix: align entry-point test with module-load contract          # fixes test gap
fix: update macos-fallback test for refactored retrieval       # fixes stale mock

Each commit is a single logical change, has its own DCO sign-off, and could stand alone.

Step 8: Structural Release-Workflow Fixes

After fixing the immediate CI failure, consider whether the workflow itself has a structural gap that allowed the bad state to reach production (or PyPI).

Gate Release Workflow Behind Tests

The most common gap: the release workflow runs on tag push but has no dependency on the test workflow. A broken tag can publish to PyPI.

Fix pattern: Add a test job as a prerequisite to build (and thus publish-pypi):

jobs:
  test:
    name: Run tests
    runs-on: ubuntu-latest
    env:
      HF_HUB_OFFLINE: "1"
      # … any env vars your tests need
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
      - run: pip install -e ".[dev]"
      - run: pytest -xvs

  build:
    name: Build distribution
    needs: test          # ← never build if tests fail
    # … rest of build steps

  publish-pypi:
    name: Publish to PyPI
    needs: build         # ← never publish if build fails
    if: contains(github.ref_name, '-') == false  # skip RC tags
    # … publish steps

Add workflow_dispatch for Retries

Without workflow_dispatch, the only way to retry a failed release is to delete and recreate the tag — which rewrites history. Add a manual trigger:

on:
  push:
    tags:
      - "v*"
  workflow_dispatch:     # ← manual retry button in the Actions tab

This lets you retry a failed publish without touching the tag. If the release process has a "publish to PyPI" step that depends on an environment with protection rules, workflow_dispatch will respect those rules on manual runs too.

Tag-Delete-and-Recreate Cycle

If a release does fail and workflow_dispatch wasn't yet added (or the tag itself was pushed before the fix was in):

# Delete the broken tag
git tag -d vX.Y.Z
git push --delete origin vX.Y.Z

# Fix the issue, push the fix
git add …
git commit -s -m "fix: root cause"
git push

# Recreate the tag on the new HEAD
git tag vX.Y.Z
git push origin vX.Y.Z

This re-triggers the release workflow. Only do this for failed releases — never for successful ones (would unpublish the package if downstream consumers already depend on it).

Quick Reference

Step	Command
Read CI logs	gh run view <ID> --log-failed
Run single test	pytest -xvs tests/file.py::test_name
Check pre-existing	git stash && pytest ... && git stash pop
Check installed entry point	python3 -c "import importlib.metadata as im; ..."
Full test suite	pytest
Commit + push	git commit -s -m "fix: ..." && git push
Watch CI	gh run watch <ID> --exit-status