mirror of
https://github.com/pbakaus/impeccable.git
synced 2026-09-15 15:46:30 +03:00
Paul BakausandClaude Opus 4.7
9e637a3732
Adds three new LLM-backed scenarios to tests/skill-behavior: - S6: `/impeccable polish` → loads polish.md - S7: `/impeccable audit` → loads audit.md - S8: existing SvelteKit project (PRODUCT.md + DESIGN.md + src/app.css + src/lib/components/*.svelte + src/routes/+page.svelte) → agent reads at least one project code file to understand the existing design system S6/S7 surface a real model-floor: gpt-5.4-mini reads brand.md, reads the target index.html, and just does the polish/audit without ever loading the sub-command reference. Stronger SKILL.md wording didn't move it. Captured in the README baseline as a known weakness. Claude and Gemini honor the load reliably. To fix Gemini on S6/S7, sub-command reference loading is now Setup step 2 (right after context.mjs), not step 4 — placing it before the model gets focused on "doing the work". Step 3 (design-system familiarization) is tightened to require at least one project code read even when a sub-command reference loads in step 2, so Claude doesn't laser-focus on the sub-command flow and skip the broader exploration. Two new fixtures: MINIMAL_LANDING_HTML (a tiny static landing page for S6/S7) and SVELTE_PROJECT_FILES (a minimal SvelteKit scaffold with tokens, components, and a routes/+page.svelte for S8). Both designed to look real enough that agents treat them as production code. Suite is now 24 tests across three providers; baseline is 21-22/24, with the stable failures being gpt-5.4-mini scenarios 6 and 7. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
80 lines
4.3 KiB
Markdown
80 lines
4.3 KiB
Markdown
# Skill-behavior tests
|
|
|
|
LLM-backed scenarios that verify how the impeccable skill drives
|
|
PRODUCT.md / DESIGN.md loading. Each scenario runs against the cheapest
|
|
tier of each major provider (Anthropic, OpenAI, Google) so a full sweep
|
|
costs a few cents and finishes in ~2 minutes.
|
|
|
|
These are the tests you re-run when you refactor anything in SKILL.md's
|
|
`## Setup` section. They fail when the agent stops following the loading
|
|
contract.
|
|
|
|
## Run
|
|
|
|
```bash
|
|
bun run test:skill-behavior
|
|
IMPECCABLE_SKILL_BEHAVIOR_VERBOSE=1 bun run test:skill-behavior # dump per-scenario traces
|
|
IMPECCABLE_SKILL_BEHAVIOR_MODELS=claude-haiku-4-5 bun run test:skill-behavior # scope to one model
|
|
```
|
|
|
|
Requires `.env` at repo root with at least one of `ANTHROPIC_API_KEY`,
|
|
`OPENAI_API_KEY`, `GOOGLE_CLOUD_API_KEY`. Providers without a key are
|
|
skipped, not failed.
|
|
|
|
## How it works
|
|
|
|
Each scenario:
|
|
|
|
1. `prepareWorkspace()` mints a temp dir, symlinks the canonical skill
|
|
into `<workspace>/.claude/skills/impeccable`, and optionally writes
|
|
`PRODUCT.md` / `DESIGN.md` fixtures.
|
|
2. `runTurn()` inlines `SKILL.md` (placeholders neutralized) as the
|
|
system prompt and runs Vercel AI SDK `generateText` with four
|
|
workspace-scoped tools: `bash`, `read`, `write`, `list`.
|
|
3. The tools record every call into a `trace` that the test asserts on.
|
|
4. For scenario 4, a second `runTurn` reuses turn 1's `responseMessages`
|
|
so the model sees a real multi-turn conversation.
|
|
|
|
The trace is the source of truth, not the model's free-form reply.
|
|
|
|
## Scenarios
|
|
|
|
| # | Setup | Assertion |
|
|
|---|---|---|
|
|
| 1 | empty workspace | runs `context.mjs` (which prints a `NO_PRODUCT_MD` directive); agent then loads `reference/teach.md` via Read or `cat`; does **not** start writing HTML/CSS |
|
|
| 2 | PRODUCT.md only (with `## Register: brand`) | runs `context.mjs` 1-3 times; loads `reference/brand.md` |
|
|
| 3 | PRODUCT.md + DESIGN.md (brand register) | runs `context.mjs` 1-3 times; loads `reference/brand.md`; consults the design system (DESIGN.md bundled in output, but CSS / tokens / directory listing also count) |
|
|
| 4 | PRODUCT.md + DESIGN.md, context already loaded in turn 1 | turn 2 does **not** re-run `context.mjs`; `reference/brand.md` is loaded across turns 1+2 |
|
|
| 5 | PRODUCT.md WITHOUT a `## Register` field; task cue says "landing page" | runs `context.mjs` (which emits a generic register directive); agent loads `reference/brand.md` via task-cue cascade |
|
|
| 6 | PRODUCT.md + DESIGN.md + a minimal `index.html`; prompt is `/impeccable polish` | loads `reference/polish.md` |
|
|
| 7 | same fixture; prompt is `/impeccable audit` | loads `reference/audit.md` |
|
|
| 8 | PRODUCT.md + DESIGN.md + a SvelteKit scaffold (`src/app.css`, components, `+page.svelte`); prompt is `/impeccable polish src/routes/+page.svelte` | reads at least one project code file (CSS / component / page) — not just the skill's reference files |
|
|
|
|
## Baseline state (2026-05-20)
|
|
|
|
Captured after moving sub-command reference loading from step 4 to step 2
|
|
of Setup (so the agent loads `reference/<command>.md` right after
|
|
`context.mjs`, before "doing the work" preempts it), and tightening
|
|
step 3 to require at least one project code read even when a sub-command
|
|
reference loads first. Use this table when comparing pre/post refactor:
|
|
a regression is "more failures than baseline", not "any failures at all".
|
|
|
|
| Scenario | claude-haiku-4-5 | gpt-5.4-mini | gemini-3.1-flash-lite |
|
|
|---|---|---|---|
|
|
| 1 (no context) | pass (rare flake — agent stops after `context.mjs` without loading `teach.md`) | pass | pass |
|
|
| 2 (product only) | pass | pass | pass |
|
|
| 3 (product + design) | pass | pass | pass (rare flake — sub-command ref loads but register ref doesn't) |
|
|
| 4 (already loaded) | pass | pass | pass |
|
|
| 5 (no register field, task-cue cascade) | pass | pass | pass |
|
|
| 6 (`polish` routing) | pass | **fail** | pass |
|
|
| 7 (`audit` routing) | pass | **fail** | pass |
|
|
| 8 (existing project, explore design system) | pass | pass | pass |
|
|
|
|
21-22 / 24 typical. The stable failures are gpt-5.4-mini scenarios 6 and 7:
|
|
the model reads `index.html` (the target file), recognizes "polish" or
|
|
"audit" as a familiar action, and proceeds with the work without ever
|
|
loading the sub-command reference. Stronger SKILL.md wording (MUST,
|
|
"non-optional", reordered earlier) didn't move it; this looks like a
|
|
model-floor behavior rather than a skill ambiguity. Claude and Gemini
|
|
honor the load.
|