fix(ci): align skill test discovery with coverage guardrail

Discover skill-local test dirs from git ls-files so nested bundle sub-skill
scripts/ dirs are covered, and force python_files=test_*.py so pytest
collection matches the guardrail's covered model everywhere (skills with a
local pytest.ini would otherwise fall back to the default collection).
Also soften the docs' guardrail claims to describe the enforced naming
convention precisely instead of overclaiming.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
This commit is contained in:
Magnus Hedemark
2026-08-14 23:15:21 -04:00
co-authored by factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
parent 98eb5e4c4d
commit 4255d4429f
3 changed files with 6 additions and 8 deletions
+3 -5
View File
@@ -73,12 +73,10 @@ jobs:
- name: Run skill-local script tests
run: |
set -euo pipefail
for dir in */scripts; do
if find "$dir" -name 'test_*.py' | grep -q .; then
while IFS= read -r dir; do
echo "=== $dir ==="
python3 -m pytest "$dir" -o "addopts=-ra --strict-markers --tb=short" -v --durations=10
fi
done
python3 -m pytest "$dir" -o "addopts=-ra --strict-markers --tb=short" -o "python_files=test_*.py" -v --durations=10
done < <(git ls-files | grep -E '/scripts/test_[^/]*\.py$' | sed 's#/[^/]*$##' | sort -u)
- name: Run skill-local shell test scripts
run: python3 scripts/check-skill-tests.py --run
- name: Check skill test coverage
+1 -1
View File
@@ -166,7 +166,7 @@ Skills that perform diagnosis, planning, or multi-step work must state when they
### Skill Script Tests
Skills that ship executable scripts must ship pytest-runnable tests in `scripts/test_*.py`; CI auto-discovers them, so Python test files must use the exact `test_*.py` name. Shell-based tests are the exception: register them in `scripts/check-skill-tests.py` as a `run` or `manual` entry. CI enforces this via `python3 scripts/check-skill-tests.py --check`, which fails on any unregistered skill test file, stale registry entry, or registry inconsistency.
Skills that ship executable scripts must name their script tests `scripts/test_*.py`; CI auto-discovers and runs them, so Python test files must use the exact `test_*.py` name. Shell-based tests are the exception: register them in `scripts/check-skill-tests.py` as a `run` or `manual` entry. CI enforces the naming convention via `python3 scripts/check-skill-tests.py --check`: a test-like file under a skill's `scripts/` directory that is neither a Python `test_*.py` nor registered fails the check.
## Validate Your Output
+1 -1
View File
@@ -79,7 +79,7 @@ If a skill includes executable scripts or a package, run its documented checks a
### Skill script tests
Every skill that ships executable scripts must ship pytest-runnable tests named `scripts/test_*.py`; pytest auto-discovers these in CI for every skill's `scripts/` directory, so Python test files must use the exact `test_*.py` name. Shell-based tests are the exception: register them in `scripts/check-skill-tests.py` as a `run` entry (executed in CI with `bash`) or a `manual` entry (documented only, when the test needs network access, credentials, or external tooling). The guardrail `python3 scripts/check-skill-tests.py --check` fails CI on any unregistered skill test file, on stale registry entries, and on registry inconsistencies, so a new, un-run shell test will not appear silently.
Every skill that ships executable scripts must name its script tests `scripts/test_*.py`; pytest auto-discovers and runs them in CI for every skill's `scripts/` directory, so Python test files must use the exact `test_*.py` name. Shell-based tests are the exception: register them in `scripts/check-skill-tests.py` as a `run` entry (executed in CI with `bash`) or a `manual` entry (documented only, when the test needs network access, credentials, or external tooling). CI enforces the naming convention: a file under any skill `scripts/` directory whose name matches the test conventions (`test_*.py`, `test*.sh`, `*_test.*`, `*-test.*`, or `.bats`) must be a Python `test_*.py` (auto-run) or be registered in `scripts/check-skill-tests.py`; `python3 scripts/check-skill-tests.py --check` fails otherwise.
## Deprecating a skill