docs: update governance docs for description, eval, and generator contracts (#101)

Align AGENTS.md and CONTRIBUTING.md with the description-quality gates (#97, #98), eval coverage ratchet (#99), and trigger-boundary and eval requirements (#100) merged today. Document that CI validates generated artifact freshness but does not regenerate; contributors run generators locally with --write.

Co-authored-by: magnus919 <magnus919>
This commit is contained in:
Magnus Hedemark
2026-07-21 22:33:56 -04:00
committed by GitHub
co-authored by magnus919 <magnus919>
parent 9a6eed2de4
commit 8514c00b20
2 changed files with 36 additions and 4 deletions
+22 -3
View File
@@ -11,7 +11,7 @@ Every skill in this repository conforms to the [Agent Skills specification](http
| Directory | Each skill in its own directory named by the skill |
| Entry point | `SKILL.md` with YAML frontmatter + markdown body |
| `name` field | Lowercase, hyphens only, matches parent directory name |
| `description` field | Trigger-oriented, describes what and when |
| `description` field | Trigger-oriented, starts with an imperative verb, defines both positive and negative trigger boundaries |
| Progressive disclosure | Core instructions in `SKILL.md` (< 500 lines, < 5,000 tokens), supporting material in `references/`, `templates/`, `scripts/` |
| File references | Relative paths from skill root, one level deep |
| **Human-readable README** | **`README.md`** in skill root — required for every skill. See [README Format](#readme-format) below |
@@ -93,7 +93,13 @@ Use each skill's `description` field as the primary routing source. For keyword
## Use-When Sections
Every skill description must identify when to load it. Skills with meaningful overlap should also include a `## When not to use` section naming the nearest alternative or prerequisite. Keep these sections trigger-oriented and concise; implementation details belong in references.
Every skill description must start with an imperative verb and define both when to load it and when not to. Skills with meaningful overlap should also include a `## When not to use` section naming the nearest alternative or prerequisite. Keep these sections trigger-oriented and concise; implementation details belong in references. The repository's quality validator enforces these requirements on changed skills.
## Eval Requirements
Every new skill must include `evals/evals.json` with at least five representative output-quality cases. Each case needs a realistic prompt, an expected outcome, and observable assertions. Trigger-only checks (should-trigger / should-not-trigger probes) are harness-specific and belong in a separate test set, not in `evals/evals.json`.
Existing skills are grandfathered via `scripts/grandfathered-skills.txt`. As overall eval coverage climbs past 25%, modified skills without evals receive a warning; past 50%, they fail CI. The coverage report is available via `python3 scripts/eval-coverage.py`.
## Best Practices
@@ -117,12 +123,25 @@ Skills that perform diagnosis, planning, or multi-step work must state when they
When creating or modifying a skill in this repo, validate against the format:
- `name` matches parent directory name
- `description` is 1-1024 chars, non-empty
- `description` is 1-1024 chars, non-empty, starts with an imperative verb, and defines a negative boundary
- Body under 500 lines and 5,000 tokens
- All file references use relative paths from skill root
- Frontmatter YAML is valid
- **`README.md` exists in the skill root** with all required sections (see [README Format](#readme-format) above)
- **README is written for humans** — no agent instructions, JSON schemas, or progressive disclosure notes in the README. Those belong in `SKILL.md`.
- **`evals/evals.json` exists** with at least five output-quality cases for new skills (see [Eval Requirements](#eval-requirements))
### Generated Artifacts
This repository tracks generated catalog files (`.claude-plugin/marketplace.json`, `.codex-plugin/plugin.json`, `.agents/plugins/marketplace.json`, `llms.txt`). CI validates that these are current; it does not regenerate them. If CI reports a stale artifact, regenerate locally:
```sh
ruby scripts/gen-claude-marketplace.rb --write
ruby scripts/gen-codex-plugin.rb --write
ruby scripts/gen-llms-txt.rb --write
```
Each script also runs in check mode (without `--write`) to verify freshness.
### Respect Attribution
+14 -1
View File
@@ -18,21 +18,34 @@ Each skill must:
- Include a human-facing `README.md` with the required sections described in `AGENTS.md`.
- Keep core instructions concise and put deeper material in `references/`, `templates/`, or `scripts/`.
- Use relative links that work from a fresh clone.
- Use an imperative-verb description that defines both positive and negative trigger boundaries.
- Describe when the skill should be loaded, and identify the nearest alternative when overlap matters.
- Include `evals/evals.json` with at least five representative output-quality cases for every new skill. Existing skills are grandfathered via `scripts/grandfathered-skills.txt`; the ratchet tightens as overall coverage climbs.
## Development
Clone the repository and run the validator from its root:
Clone the repository and run the validators from its root:
```sh
git clone https://github.com/magnus919/agent-skills.git
cd agent-skills
ruby scripts/validate-skills.rb
ruby scripts/validate-skill-quality.rb --base origin/main
python3 scripts/eval-coverage.py
```
The structural validator checks the whole repository. The quality validator checks only added, renamed, modified, or uncommitted `SKILL.md` files relative to the supplied base. Changed descriptions must begin with an imperative verb and define a negative boundary in the description or a `When not to use` section. Generic no-op instructions are reported as warnings. The same validation runs in GitHub Actions for pushes and pull requests.
This repository also tracks generated catalog files. CI validates their freshness but does not regenerate them. If a check reports a stale artifact, regenerate locally:
```sh
ruby scripts/gen-claude-marketplace.rb --write
ruby scripts/gen-codex-plugin.rb --write
ruby scripts/gen-llms-txt.rb --write
```
Each generator also runs in check mode (without `--write`) to verify freshness.
If a skill includes executable scripts or a package, run its documented checks as well and include the commands and results in your pull request.
## Deprecating a skill