Paul BakausandClaude Opus 4.7
9e637a3732
Adds three new LLM-backed scenarios to tests/skill-behavior: - S6: `/impeccable polish` → loads polish.md - S7: `/impeccable audit` → loads audit.md - S8: existing SvelteKit project (PRODUCT.md + DESIGN.md + src/app.css + src/lib/components/*.svelte + src/routes/+page.svelte) → agent reads at least one project code file to understand the existing design system S6/S7 surface a real model-floor: gpt-5.4-mini reads brand.md, reads the target index.html, and just does the polish/audit without ever loading the sub-command reference. Stronger SKILL.md wording didn't move it. Captured in the README baseline as a known weakness. Claude and Gemini honor the load reliably. To fix Gemini on S6/S7, sub-command reference loading is now Setup step 2 (right after context.mjs), not step 4 — placing it before the model gets focused on "doing the work". Step 3 (design-system familiarization) is tightened to require at least one project code read even when a sub-command reference loads in step 2, so Claude doesn't laser-focus on the sub-command flow and skip the broader exploration. Two new fixtures: MINIMAL_LANDING_HTML (a tiny static landing page for S6/S7) and SVELTE_PROJECT_FILES (a minimal SvelteKit scaffold with tokens, components, and a routes/+page.svelte for S8). Both designed to look real enough that agents treat them as production code. Suite is now 24 tests across three providers; baseline is 21-22/24, with the stable failures being gpt-5.4-mini scenarios 6 and 7. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
4.3 KiB
Skill-behavior tests
LLM-backed scenarios that verify how the impeccable skill drives PRODUCT.md / DESIGN.md loading. Each scenario runs against the cheapest tier of each major provider (Anthropic, OpenAI, Google) so a full sweep costs a few cents and finishes in ~2 minutes.
These are the tests you re-run when you refactor anything in SKILL.md's
## Setup section. They fail when the agent stops following the loading
contract.
Run
bun run test:skill-behavior
IMPECCABLE_SKILL_BEHAVIOR_VERBOSE=1 bun run test:skill-behavior # dump per-scenario traces
IMPECCABLE_SKILL_BEHAVIOR_MODELS=claude-haiku-4-5 bun run test:skill-behavior # scope to one model
Requires .env at repo root with at least one of ANTHROPIC_API_KEY,
OPENAI_API_KEY, GOOGLE_CLOUD_API_KEY. Providers without a key are
skipped, not failed.
How it works
Each scenario:
prepareWorkspace()mints a temp dir, symlinks the canonical skill into<workspace>/.claude/skills/impeccable, and optionally writesPRODUCT.md/DESIGN.mdfixtures.runTurn()inlinesSKILL.md(placeholders neutralized) as the system prompt and runs Vercel AI SDKgenerateTextwith four workspace-scoped tools:bash,read,write,list.- The tools record every call into a
tracethat the test asserts on. - For scenario 4, a second
runTurnreuses turn 1'sresponseMessagesso the model sees a real multi-turn conversation.
The trace is the source of truth, not the model's free-form reply.
Scenarios
| # | Setup | Assertion |
|---|---|---|
| 1 | empty workspace | runs context.mjs (which prints a NO_PRODUCT_MD directive); agent then loads reference/teach.md via Read or cat; does not start writing HTML/CSS |
| 2 | PRODUCT.md only (with ## Register: brand) |
runs context.mjs 1-3 times; loads reference/brand.md |
| 3 | PRODUCT.md + DESIGN.md (brand register) | runs context.mjs 1-3 times; loads reference/brand.md; consults the design system (DESIGN.md bundled in output, but CSS / tokens / directory listing also count) |
| 4 | PRODUCT.md + DESIGN.md, context already loaded in turn 1 | turn 2 does not re-run context.mjs; reference/brand.md is loaded across turns 1+2 |
| 5 | PRODUCT.md WITHOUT a ## Register field; task cue says "landing page" |
runs context.mjs (which emits a generic register directive); agent loads reference/brand.md via task-cue cascade |
| 6 | PRODUCT.md + DESIGN.md + a minimal index.html; prompt is /impeccable polish |
loads reference/polish.md |
| 7 | same fixture; prompt is /impeccable audit |
loads reference/audit.md |
| 8 | PRODUCT.md + DESIGN.md + a SvelteKit scaffold (src/app.css, components, +page.svelte); prompt is /impeccable polish src/routes/+page.svelte |
reads at least one project code file (CSS / component / page) — not just the skill's reference files |
Baseline state (2026-05-20)
Captured after moving sub-command reference loading from step 4 to step 2
of Setup (so the agent loads reference/<command>.md right after
context.mjs, before "doing the work" preempts it), and tightening
step 3 to require at least one project code read even when a sub-command
reference loads first. Use this table when comparing pre/post refactor:
a regression is "more failures than baseline", not "any failures at all".
| Scenario | claude-haiku-4-5 | gpt-5.4-mini | gemini-3.1-flash-lite |
|---|---|---|---|
| 1 (no context) | pass (rare flake — agent stops after context.mjs without loading teach.md) |
pass | pass |
| 2 (product only) | pass | pass | pass |
| 3 (product + design) | pass | pass | pass (rare flake — sub-command ref loads but register ref doesn't) |
| 4 (already loaded) | pass | pass | pass |
| 5 (no register field, task-cue cascade) | pass | pass | pass |
6 (polish routing) |
pass | fail | pass |
7 (audit routing) |
pass | fail | pass |
| 8 (existing project, explore design system) | pass | pass | pass |
21-22 / 24 typical. The stable failures are gpt-5.4-mini scenarios 6 and 7:
the model reads index.html (the target file), recognizes "polish" or
"audit" as a familiar action, and proceeds with the work without ever
loading the sub-command reference. Stronger SKILL.md wording (MUST,
"non-optional", reordered earlier) didn't move it; this looks like a
model-floor behavior rather than a skill ambiguity. Claude and Gemini
honor the load.