Add NVIDIA SkillEvaluator as an advisory (non-blocking) Tier 1 gate on
PRs and pushes touching skill-relevant paths. Runs the documented
keyless check set (schema,pii,license,quality,unicode,lint) pinned to
skillevaluator v0.1.0 per changed skill, uploads JSON reports as an
artifact, and posts one advisory summary line per skill to the job log.
SkillSpector is deliberately not installed: upstream still ships it
"separately installed and unpinned", so the security scan is excluded
by design until a documented compatible pair exists.
Closes#383
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
2026-08-23 00:06:26 -04:00
Magnus Hedemarkandfactory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Pin the epub skill's test-only dependencies (EbookLib, beautifulsoup4) with
exact versions in requirements-epub-test.txt and install from that file in
CI, instead of unpinned package names. The file is test-only and notes that
EbookLib is AGPL; requirements-dev.txt is untouched.
Also surface the epublib skip in epub/scripts/test_epub_skill.sh: when
epublib is not importable (it requires Python 3.13+), epub-edit and
epub-convert now print an explicit SKIPPED line and count as SKIP in the
summary instead of silently passing.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
2026-08-14 23:42:25 -04:00
Magnus Hedemarkandfactory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Install the epub skill's documented test dependencies (EbookLib and
beautifulsoup4) on the runner just before the shell-test step, and move
epub/scripts/test_epub_skill.sh from the manual registry into the run
registry. EbookLib is AGPL-licensed, so the install step is named and
scoped as test-only and does not touch requirements-dev.txt.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
2026-08-14 23:22:29 -04:00
Magnus Hedemarkandfactory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Discover skill-local test dirs from git ls-files so nested bundle sub-skill
scripts/ dirs are covered, and force python_files=test_*.py so pytest
collection matches the guardrail's covered model everywhere (skills with a
local pytest.ini would otherwise fall back to the default collection).
Also soften the docs' guardrail claims to describe the enforced naming
convention precisely instead of overclaiming.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
2026-08-14 23:15:21 -04:00
Magnus Hedemarkandfactory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Add scripts/check-skill-tests.py as the single source of truth for
skill-local shell tests. --run executes the registered shell tests in CI;
--check (default) fails on any unregistered skill test file, stale registry
entry, or registry inconsistency, so new un-run shell tests cannot slip in.
Register the 9 deterministic shell suites as run entries and the 4 suites
that need network, credentials, or third-party libraries not installed by
CI as manual entries.
Wire both modes into validate.yml right after the skill-local pytest step
and run the new guardrail tests in the existing pytest invocations.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
2026-08-14 22:41:53 -04:00
Magnus Hedemarkandfactory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* feat(validation): enforce 60K-char cap on skill reference files
Implements issue #277:
- validate-references.rb: new oversized_reference_errors check — every
references/*.md must be <= 60,000 characters; error reports path, size,
and the split-and-reindex remediation; wired into validate-skills.rb
- test-validate-skills.rb: 5 fixture tests (under-limit passes, over-limit
fails with path+size, exactly-at-limit passes, remediation message,
non-.md ignored); the suite now runs in validate.yml after the format
check (it was previously untested in CI)
- Docs: agent-skills/SKILL.md, agent-skills/references/best-practices.md,
and the AGENTS.md Format Compliance table document the cap and the
split-and-reindex procedure
- Compliance: split remote-systems-administration/references/ansible.md
and programming-principles/references/refactoring-guru.full.md into an
index + focused parts (content moved verbatim); SKILL.md routing,
README, and source-index references updated; pre-existing stale
refactoring-guru-smells.md reference repointed to the index
- Fix pre-existing quality-gate violations in the programming-principles
and remote-systems-administration descriptions (imperative verb +
negative boundary) so this PR's CI quality step passes; regenerated
llms.txt and marketplace artifacts
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* test(evals): add eval manifests to modified skills for ratchet
The eval-coverage ratchet requires schema-valid eval manifests for any
skill modified once coverage is past 50%. This PR modifies
programming-principles and remote-systems-administration (splitting
their oversized references), so add evals/evals.json to both:
- programming-principles: 6 output-quality cases (task-to-book mapping,
principled code review, refactor-vs-rewrite, no-op detection, rule
distillation, principle conflicts)
- remote-systems-administration: 6 output-quality cases (discovery
before change, smallest control plane, rollback planning, platform
identification, verification evidence, escalation on missing
authority)
Coverage: 87/145 (60.0%) schema-valid; ratchet clean.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
---------
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
2026-08-04 22:39:14 -04:00
Magnus HedemarkGitHubfactory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* feat(bundles): add bundle manifest schema, manifests, and validation (#203)
Introduce a machine-readable composition contract for canonical bundles:
purpose, audience, stages, included skills, prerequisites, outputs,
handoffs, conflicts, and eval suite (schemas/bundle-manifest-v1.schema.json,
following the evals-v1 versioned-schema convention). Ship the bounded design
note (docs/bundle-manifest-design.md), a schema-conformant example, canonical
manifests for the three new milestone bundles, and a stdlib-only validator
(scripts/validate-bundles.rb) that rejects incomplete, contradictory, and
undeclared-overlapping manifests while keeping bundles an optional layer.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* feat(bundles): add lifecycle capability matrix generator and validators (#203)
Add scripts/gen-lifecycle-matrix.rb, which deterministically produces the
human-readable docs/lifecycle-capability-matrix.md (one row per canonical
bundle) and the machine-readable docs/lifecycle-capability-matrix.json (with
per-cell source provenance) reusing the gen-*.rb conventions. Add
scripts/validate-lifecycle-matrix.rb to check bundle coverage, cell
traceability, artifact currency, and catalog-exactness of nested bundle
helpers.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* test(bundles): add bundle manifest validation tests (#203)
Add scripts/test-validate-bundles.rb covering schema conformance of the
committed example, valid-manifest and declared-conflict positives, per-field
incomplete-manifest rejections, contradictory-manifest rejections (missing
skill, undeclared handoff artifact, non-catalog conflict), undeclared-overlap
rejection naming both manifests, and matrix generator/validator
completeness and drift detection.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* ci(bundles): wire bundle manifest validation into the gate (#203)
Add validate-bundles.rb, test-validate-bundles.rb, the lifecycle matrix
generator check, and the matrix validator to .github/workflows/validate.yml
alongside the existing validator steps.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
---------
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Add a live canary that runs the full Hub catalog check (190 endpoints)
and probes all fixed non-Hub adapters (geocode, transit GTFS,
development, civic JSON:API, civic RSS, meetings eSCRIBE, imagery).
Validates source-specific minimum schemas, classifies failures into
transport_outage, auth_regression, arcgis_error, schema_drift,
parser_failure, and empty_but_valid. Retries bounded transient
failures and preserves first-failure evidence.
Runs daily at 10:17 UTC and on workflow_dispatch, separate from PR CI
so upstream outages do not block unrelated changes.
Closes#120
Co-authored-by: magnus919 <magnus919>
Add release evaluation layer on top of deterministic paired execution:
- schemas/release-eval-v1.schema.json: release report schema with freeze
snapshot, per-case trial aggregation, rubric graders, blinded pairwise
comparison, calibration tracking, and PASS/CONDITIONAL/HOLD/BLOCK outcomes
- eval_runner/release.py: core module for multi-trial aggregation, versioned
rubric graders with abstain/insufficient-evidence, pairwise planning with
position randomization and order-reversal testing, calibration records,
and release decision computation
- eval_runner/tests/test_release.py: 34 tests covering all acceptance criteria
- schemas/evals-v1.schema.json: optional case_set field (dev/regression/release)
- eval_runner/models.py: case_set on EvalCase
- eval_runner/runner.py: load case_set from manifest
- .github/workflows/skill-eval.yml: run release tests in CI
Gate semantics: hard invariants (privacy, auth, destructive) tolerate zero
violations and cannot be averaged away. Missing evidence produces HOLD, not
PASS. Uncalibrated judge results are advisory only.
Closes#106
Co-authored-by: magnus919 <magnus919>
* feat: run isolated paired candidate and baseline skill evaluations
Build the first complete paired skill-evaluation path: stage an immutable
candidate, run matched candidate and baseline trials in clean environments,
execute deterministic outcome graders, and produce a case-level comparison
report.
- eval_runner/sandbox.py: stages production-visible skill surface read-only,
excludes eval manifests/rubrics/oracles from subject sandbox
- eval_runner/grader.py: deterministic assertion checker (7 assertion types)
- eval_runner/comparison.py: paired comparison report generation
- eval_runner/paired.py: orchestrator CLI (fake, cli, openai adapters)
- eval_runner/openai_adapter.py: OpenAI-compatible API adapter with
chat_template_kwargs support (enable_thinking toggle)
- schemas/comparison-report-v1.schema.json: report schema
- .github/workflows/skill-eval.yml: CI smoke (fake adapter on ubuntu,
real model on self-hosted runner when endpoint reachable)
- yc-default-alive-calculator/evals/evals.json: initial 5-case eval manifest
Verified against google_gemma-4-26B-A4B-it-IQ4_XS.gguf: 5/5 candidate
improvements, 0 regressions.
Closes#105
* ci: make paired-eval-model job non-blocking
The self-hosted runner may not always be online. Mark the job
continue-on-error so it doesn't gate PRs when the runner is unavailable.
* ci: isolate model evals from pull requests
* fix(raleigh): test arrivals against a daily route, not weekday-only
The fixture only had a WEEK (Mon-Fri) service, so
test_get_arrivals_for_stop returned 0 arrivals on weekends when
_today_date() fell on Saturday/Sunday. Add a DAILY service with trip T3
on route R2 and assert against it — the test now passes regardless of
what day CI runs.
* ci: trigger checks on amended commit
---------
Co-authored-by: magnus919 <magnus919>
Wire the Phase 3 ratchet into CI by passing the PR base SHA to
eval-coverage.py --modified-from. Expand changed-skill detection from
SKILL.md-only diffs to the entire skill directory so that references,
scripts, fixtures, README, and eval manifest edits all count as
modifications. Add a monotonic coverage floor that fails CI when
coverage decreases between the base and candidate revisions.
Add script tests for ratchet-mode detection and coverage-decrease
behaviour. Update AGENTS.md and CONTRIBUTING.md to describe the
behaviour CI now enforces.
Closes#102
Signed-off-by: Magnus Hedemark <magnus919@pm.me>
Phase 1: New skills (not in grandfathered-skills.txt) must have
evals/evals.json with at least 5 test cases. All 107 existing skills
are grandfathered.
Phase 2: scripts/eval-coverage.py reports coverage (skills with/without
evals, case counts, reference-priority sorting). Added as informational
CI step.
Phase 3: Ratchet thresholds — at 25% coverage, modified skills without
evals get a warning; at 50%, they fail CI. Enforced via
--modified-from flag for PR-scoped checks.
Closes#90
Generate a root discovery catalog from public skill frontmatter and fail CI
when the committed index drifts. Add a fixture-based regression test for
bundle paths, nested-helper exclusion, normalized descriptions,
deterministic ordering, and stale-file recovery.
Closes#78
AI-assisted: yes (Jasper/Hermes Agent)
Co-authored-by: magnus919 <magnus919>
* feat: add Codex plugin packaging (single-plugin, metadata-only)
Adds .codex-plugin/plugin.json with a skills array listing all public
skills, plus .agents/plugins/marketplace.json for one-command install:
codex plugin marketplace add magnus919/agent-skills
codex plugin install magnus919
Same pattern as mattpocock/skills — one plugin, explicit skill paths,
no dist/, no curation, no duplication. Bundle-internal helpers excluded
by the shared glob. CI check mode fails if the manifest drifts.
Closes#79
AI-assisted: yes (Jasper/Hermes Agent)
* chore: trigger CI
* chore: regenerate Codex plugin manifest to include neckbeard bundle
Adds .claude-plugin/marketplace.json exposing all 95 public skills as
installable plugins via /plugin marketplace add magnus919/agent-skills.
Metadata-only approach: each entry uses source './' + skills ['./<name>']
+ strict:false, so no per-skill plugin.json or directory restructuring is
needed. Bundle-internal helper skills are excluded; bundle entrypoints are
included.
- scripts/gen-claude-marketplace.rb: generates and validates the manifest
- CI step fails if marketplace.json drifts from the skill tree
- README: Claude Code install instructions
Closes#76
AI-assisted: yes (Jasper/Hermes Agent)
Accumulates conventional commits into a Release PR that bumps the
version and updates CHANGELOG.md. Nothing is tagged until a human
merges the Release PR.
Co-authored-by: Jasper <jasper@magnus919.com>
Magnus Hedemarkandfactory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
dependabot[bot]andGitHub
usernameandfactory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
magnus919andMagnus Hedemark
