skills/magnus919_agent-skills
1
0
Fork 0
mirror of https://github.com/magnus919/agent-skills.git synced 2026-09-18 23:16:38 +03:00
Code Issues Packages Projects Releases Wiki Activity

ci(validate): run skill shell tests and add coverage guardrail

Browse Source 
Add scripts/check-skill-tests.py as the single source of truth for
skill-local shell tests. --run executes the registered shell tests in CI;
--check (default) fails on any unregistered skill test file, stale registry
entry, or registry inconsistency, so new un-run shell tests cannot slip in.
Register the 9 deterministic shell suites as run entries and the 4 suites
that need network, credentials, or third-party libraries not installed by
CI as manual entries.

Wire both modes into validate.yml right after the skill-local pytest step
and run the new guardrail tests in the existing pytest invocations.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
This commit is contained in:
Magnus Hedemark
2026-08-14 22:41:53 -04:00
co-authored by factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
parent 222f431f02
commit 837a329531
3 changed files with 395 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/validate.yml
+6 -2
View File
@@ -79,10 +79,14 @@ jobs:
python3 -m pytest "$dir" -o "addopts=-ra --strict-markers --tb=short" -v --durations=10
fi
done
- name: Run skill-local shell test scripts
run: python3 scripts/check-skill-tests.py --run
- name: Check skill test coverage
run: python3 scripts/check-skill-tests.py --check
- name: Run core test suite with coverage
run: python3 -m pytest scripts/test-eval-validation.py scripts/test-eval-coverage.py eval_runner/tests/ -v --durations=10 --cov=scripts --cov=eval_runner --cov-fail-under=60 --cov-report=term-missing
run: python3 -m pytest scripts/test-eval-validation.py scripts/test-eval-coverage.py scripts/test_check_skill_tests.py eval_runner/tests/ -v --durations=10 --cov=scripts --cov=eval_runner --cov-fail-under=60 --cov-report=term-missing
- name: Run tests in parallel (isolation check)
run: python3 -m pytest scripts/test-eval-validation.py scripts/test-eval-coverage.py eval_runner/tests/ -n auto -v --durations=10
run: python3 -m pytest scripts/test-eval-validation.py scripts/test-eval-coverage.py scripts/test_check_skill_tests.py eval_runner/tests/ -n auto -v --durations=10
- name: Validate AGENTS.md consistency
run: python3 scripts/validate-agents-md.py
- name: Run integration tests