9 Commits
Author SHA1 Message Date
Magnus HedemarkandGitHub 24f21c64e3 test(evals): complete canonical coverage
Squash-merge the verified #412 eval coverage implementation. Required validate and paired evaluation checks passed at exact head b43ac564a5919a0f23fdab49ba052d7c514915cb; droid-review BYOK failure had no findings and is advisory.
2026-09-01 19:58:35 -04:00
Magnus Hedemarkandfactory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> 3480b7ff5a fix(prose): refresh top-level skill path references for flat layout
Re-point promise-theory reference links to ../../workflow-architect/SKILL.md
(one level deeper than SKILL.md), drop the bundles/ prefix from the
pace-plan evidence-ledger path, and refresh the stale eval-coverage.py
comment to describe only the flat <root>/<skill-name>/SKILL.md shape.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
2026-08-14 16:50:06 -04:00
usernameandfactory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> bfe05ef18a feat(ci): add mypy, radon, and deptry to CI pipeline
Add mypy strict type checking, radon cyclomatic complexity analysis,
and deptry unused dependency detection. Fix type annotations in
eval-coverage.py and release.py to pass strict mypy checks.

Resolves 4 agent-readiness signals: type_check, strict_typing,
cyclomatic_complexity, unused_dependencies_detection.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
2026-07-29 17:52:28 -04:00
usernameandfactory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> 476d7e11b0 feat(ci): add linting, formatting, coverage, and security configs
Add ruff linter/formatter with pre-commit hooks, pytest-cov with 60%
coverage threshold, CODEOWNERS, Dependabot for pip/GHA updates, and
.env.example. Auto-fix existing ruff violations across eval_runner/
and scripts/.

10 agent-readiness criteria resolved: lint_config, formatter,
pre_commit_hooks, naming_consistency, dead_code_detection,
test_coverage_thresholds, test_performance_tracking, codeowners,
dependency_update_automation, env_template.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
2026-07-29 17:43:41 -04:00
username 687d7d7c33 fix: permit removal of eval-covered skills 2026-07-27 10:04:27 -04:00
Magnus HedemarkGitHubmagnus919 <magnus919>
a617ccaf2d feat: validate eval manifest coverage states (#129)
* feat: validate eval manifest coverage states

* test: create fixture directory explicitly

---------

Co-authored-by: magnus919 <magnus919>
2026-07-24 18:13:32 -04:00
Magnus HedemarkandGitHub 1e10a56199 fix: complete eval ratchet verification (#109)
Closes #102

Authored by Jasper (AI agent on behalf of @magnus919).
2026-07-22 02:16:36 -04:00
Magnus HedemarkandGitHub fc8a8952ee fix: enforce eval coverage ratchet for complete changed skill directories (#107)
Wire the Phase 3 ratchet into CI by passing the PR base SHA to
eval-coverage.py --modified-from. Expand changed-skill detection from
SKILL.md-only diffs to the entire skill directory so that references,
scripts, fixtures, README, and eval manifest edits all count as
modifications. Add a monotonic coverage floor that fails CI when
coverage decreases between the base and candidate revisions.

Add script tests for ratchet-mode detection and coverage-decrease
behaviour. Update AGENTS.md and CONTRIBUTING.md to describe the
behaviour CI now enforces.

Closes #102

Signed-off-by: Magnus Hedemark <magnus919@pm.me>
2026-07-22 01:51:57 -04:00
Magnus HedemarkandGitHub be0c8df5e2 feat: eval coverage ratchet — gate new skills, track coverage, ratchet thresholds (#99)
Phase 1: New skills (not in grandfathered-skills.txt) must have
evals/evals.json with at least 5 test cases. All 107 existing skills
are grandfathered.

Phase 2: scripts/eval-coverage.py reports coverage (skills with/without
evals, case counts, reference-priority sorting). Added as informational
CI step.

Phase 3: Ratchet thresholds — at 25% coverage, modified skills without
evals get a warning; at 50%, they fail CI. Enforced via
--modified-from flag for PR-scoped checks.

Closes #90
2026-07-21 21:03:21 -04:00