Paul BakausandClaude Opus 5
ebc63f071a
The earlier fix in this branch was built on a wrong diagnosis. It assumed a
structured question hides any prose sharing its message, so it split report and
question across two turns. A controlled check showed prose before a question
renders fine; what hides a report is emitting it AFTER the question. The split
therefore fixed nothing and introduced a worse failure: a turn that ends on the
report is a turn that ends, and the questions never arrived at all.
Persistence returns to main's ordering, byte for byte, and the boundary prose is
gone. What replaces it is a position rule: the question is the last thing in the
response.
The trace test added here found two failures beyond the reported one. Critique
can fail to land in three ways, and they are now all asserted:
1. Question emitted before the report, hiding it behind the picker.
2. No close at all: no questions and no skip line, so polish inherits nothing.
3. Report authored into the persistence heredoc and never written to chat,
leaving a perfect snapshot and a user who sees nothing.
Mode 3 predates this branch entirely. Persistence step 1 now says the temp file
is an archive copy, not delivery.
The Codex final-question gate is promoted out of its <codex> fence, where it was
stripped for three of four providers, and the skip branch is now a countable
threshold (fewer than 3 Priority Issues) rather than a judgment call.
Known floor, recorded in the suite README: gpt-5.6-luna passes 1 run in 6 and
deepseek-v4-flash is flaky. claude-sonnet-5 and gemini-3.5-flash are consistent.
Prepared with AI assistance (Claude Code).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
12 KiB
Skill-behavior tests
LLM-backed scenarios that verify how the impeccable skill drives context, command-reference, new-work, and native-platform loading. Each scenario runs against one current model from each supported provider (Anthropic, OpenAI, Google, DeepSeek).
These are the tests you re-run when you refactor anything in SKILL.md's
## Setup section. They fail when the agent stops following the loading
contract.
Run
bun run test:skill-behavior
IMPECCABLE_SKILL_BEHAVIOR_VERBOSE=1 bun run test:skill-behavior # dump per-scenario traces
IMPECCABLE_SKILL_BEHAVIOR_MODELS=claude-sonnet-5 bun run test:skill-behavior # scope to one model
Requires .env at repo root with at least one of ANTHROPIC_API_KEY,
OPENAI_API_KEY, GOOGLE_CLOUD_API_KEY, DEEPSEEK_API_KEY. Providers without a key are
skipped, not failed.
How it works
Each scenario:
prepareWorkspace()mints a temp dir, symlinks the canonical skill into<workspace>/.claude/skills/impeccable, and optionally writesPRODUCT.md/DESIGN.mdfixtures.runTurn()inlinesSKILL.md(placeholders neutralized) as the system prompt and runs Vercel AI SDKgenerateTextwith four workspace-scoped tools:bash,read,write,list, and a fake provider-neutralask_user_questionbacked by a deterministic simulated user.- The tools record every call into a
tracethat the test asserts on. - For scenario 4, a second
runTurnreuses turn 1'sresponseMessagesso the model sees a real multi-turn conversation.
The trace is the source of truth, not the model's free-form reply.
Scenarios
| # | Setup | Assertion |
|---|---|---|
| 1 | empty workspace | runs context.mjs; loads reference/init.md before implementation; automation is not an init bypass |
| 2 | PRODUCT.md only | runs context.mjs 1-3 times; loads reference/new-work.md to resolve visual authority, establish a world when needed, and develop the surface |
| 3 | PRODUCT.md + DESIGN.md | runs context.mjs 1-3 times; receives the committed design system and loads reference/new-work.md for the task-scoped concept |
| 4 | PRODUCT.md + DESIGN.md, context already loaded in turn 1 | turn 2 does not re-run context.mjs |
| 5 | PRODUCT.md without the legacy ## Register field and no DESIGN.md |
runs context.mjs; greenfield craft loads reference/new-work.md, not init, to establish the missing world |
| 6 | PRODUCT.md + DESIGN.md + a minimal index.html; prompt is /impeccable polish |
loads reference/polish.md |
| 7 | same fixture; prompt is /impeccable audit |
loads reference/audit.md |
| 8 | PRODUCT.md + DESIGN.md + a SvelteKit scaffold (src/app.css, components, +page.svelte); prompt is /impeccable polish src/routes/+page.svelte |
reads at least one project code file (CSS / component / page) — not just the skill's reference files |
| 9 | PRODUCT.md + index.html + a seeded update cache with a newer version (skillVersion copy-mode so context.mjs has a SKILL.md to version-check against); prompt is /impeccable polish index.html |
context.mjs runs and its output carries the UPDATE_AVAILABLE directive (proven via captured bash output); the agent does not auto-run npx impeccable update (it must ask first) |
| 10 | no PRODUCT.md + a minimal index.html; prompt is /impeccable polish index.html |
runs context.mjs, loads reference/polish.md, and does not divert into reference/init.md |
| 11 | empty workspace; prompt is /impeccable shape ... |
runs context.mjs; resolves reference/init.md before planning the surface |
| 12 | empty workspace; prompt is natural-language build intent with no command word | runs context.mjs; resolves reference/init.md before implementation |
| 13 | empty workspace; prompt is /impeccable teach |
runs context.mjs and diverts into reference/init.md because teach aliases init |
| 14 | PRODUCT.md with ## Platform: ios (native iOS app); prompt is /impeccable craft a tide detail screen |
context.mjs runs and emits the contents of reference/ios.md directly, placing native conventions in context without a second model-directed read |
| 15 | same iOS fixture; prompt is /impeccable audit |
agent loads reference/audit.native.md (the Commands-table native variant, routed instead of audit.md) |
The workflow-contract file adds end-to-end assertions for attended fresh init, an initialized natural build request, replacement-world redesign, scope-preserving bolder refinement, and critique's closing question. It checks question order and context/artifact writes rather than only reference-file loading.
critique closes with the question or an explicit skip line is a regression
guard, not a routing check. A critique that prints its report and then stops,
asking nothing and printing no Questions skipped: <reason> line, is an
incomplete run: the close is half the deliverable, and polish downstream has
no priorities to inherit without it. The fixture page is deliberately broken
enough to put the report past the three-Priority-Issue threshold, so the run
cannot reach the skip branch on merit. The assertion is deliberately loose about
how the run closes, because either close is valid; what it forbids is neither.
Workflow-contract baseline (2026-08-13, current lineup)
Measured on claude-sonnet-5, gpt-5.6-luna, gemini-3.5-flash, and
deepseek-v4-flash while checking whether an {{ask_instruction}} rewrite had
regressed anything. Two of the four workflow-contract scenarios fail for reasons
that predate that change. Treat both as the known floor; a regression is a
failure beyond these.
| Scenario | claude-sonnet-5 | gpt-5.6-luna | gemini-3.5-flash | deepseek-v4-flash |
|---|---|---|---|---|
| attended fresh init | not measured | not measured | not measured | not measured |
| initialized natural build | not measured | not measured | not measured | not measured |
| redesign replaces DESIGN | flaky | not measured | not measured | not measured |
| bolder refinement | not measured | pass | pass | fail |
| critique closes | pass (2 of 2) | fail (1 of 6) | pass (2 of 2) | flaky (1 of 2) |
not measured means exactly that: the cell was never run in isolation on this
lineup. Only the two failing scenarios were scoped per model, because the
investigation was about whether a specific edit had regressed them. The rows are
worth keeping anyway, since a scenario absent from the table is easy to mistake
for a scenario that passed.
bolder refinement, deepseek-v4-flash. The model runs context.mjs, reads
bolder.md, craft-floor.md, and current.html, then ends its turn without
editing anything: empty writePaths, no ask_user_question call, well short of
the 16-step cap. Confirmed identical on HEAD with bolder.md reverted, so it is
not a skill-text problem. Same shape as the gpt-5.4-mini scenario 6/7 failures
below: the model consumes the references and then declines to act.
critique closes, gpt-5.6-luna. Five runs while tuning the instruction text
produced one pass. It fails in three distinct ways, which is why the scenario
asserts on emission order rather than only on the presence of a question:
- No close. Report lands, no question, no skip line. The failure this scenario was written for.
- Question before report. The question is emitted first and the report after it, so the report is withheld until the user answers. Observed directly, and the reason the invariant is stated as a position rule ("the question is the LAST thing in the response") rather than as prose order.
- Report never spoken. The full report is authored into the persistence heredoc, archived, and never written to chat. The snapshot is perfect and the user sees nothing. This is why persistence step 1 says the temp file is an archive copy, not delivery.
claude-sonnet-5 and gemini-3.5-flash pass consistently. deepseek-v4-flash has hit mode 3 once in two runs, so it is flaky here, not clean. Strengthening the instruction text moved luna from consistently failing to occasionally passing, and further prose tuning stopped paying.
Read the counts in the table as what they are: small samples on a nondeterministic system, gathered while the instruction text was being changed between runs. They say the close is reliable on the two strongest models and unreliable on the two cheapest ones. They do not support a finer claim than that. Re-measure rather than assuming when the lineup changes.
redesign replaces DESIGN, flaky. It has failed on two different assertions
across runs (designWrite > question and implementation > designWrite), and on
one run claude-sonnet-5 exhausted the 300s per-test timeout instead of asserting.
The traces never load document.md; the ordering under test comes from
new-work.md. Re-run before believing a single red result here. Which model
produced which failure was not pinned down, so the row records only that the
scenario is unstable.
The bolder claude-sonnet-5 cell is unmeasured for a specific reason: the scoped
run that produced this table used a 180s cap, which sonnet exceeded. That is a
timeout, not a failure, and it is why the guidance below insists on 300000.
Scoping a run while investigating
Both files honor --test-name-pattern, which is much cheaper than a full sweep
when bisecting one scenario:
IMPECCABLE_QUESTION_DISABLED=1 CI=1 IMPECCABLE_SKILL_BEHAVIOR_MODELS=deepseek-v4-flash \
node --test --test-timeout=300000 --test-force-exit \
--test-name-pattern="bolder refinement" tests/skill-behavior/workflow-contract.test.mjs
Keep --test-timeout at 300000. A tighter cap turns claude-sonnet-5's slower
runs into timeouts that look like failures. Set IMPECCABLE_QUESTION_DISABLED=1
and CI=1 so serve-question.mjs cannot open a browser window on the host. Pipe
to a file rather than tail; node prints the failing-test summary at the end,
and truncating it costs you the per-model attribution.
Baseline state (2026-05-20, previous cheap tier)
Historical record. The default models are now
claude-sonnet-5,gpt-5.6-luna,gemini-3.5-flash, anddeepseek-v4-flash. The table below was measured on an older cheap tier (claude-haiku-4-5/gpt-5.4-mini) and is kept as the historical record. Re-measure on the current lineup and update this section; the stronger models are expected to clear the scenario 6/7 routing failures that the old gpt tier showed.
Captured after moving sub-command reference loading from step 4 to step 2
of Setup (so the agent loads reference/<command>.md right after
context.mjs, before "doing the work" preempts it), and tightening
step 3 to require at least one project code read even when a sub-command
reference loads first. Use this table when comparing pre/post refactor:
a regression is "more failures than baseline", not "any failures at all".
| Scenario | claude-haiku-4-5 | gpt-5.4-mini | gemini-3.1-flash-lite |
|---|---|---|---|
| 1 (no context) | pass (rare flake — agent stops after context.mjs without loading init.md) |
pass | pass |
| 2 (product only) | pass | pass | pass |
| 3 (product + design) | pass | pass | pass (rare flake — sub-command ref loads but world ref doesn't) |
| 4 (already loaded) | pass | pass | pass |
| 5 (no register field, task-cue cascade) | pass | pass | pass |
6 (polish routing) |
pass | fail | pass |
7 (audit routing) |
pass | fail | pass |
| 8 (existing project, explore design system) | pass | pass | pass |
21-22 / 24 typical. The stable failures are gpt-5.4-mini scenarios 6 and 7:
the model reads index.html (the target file), recognizes "polish" or
"audit" as a familiar action, and proceeds with the work without ever
loading the sub-command reference. Stronger SKILL.md wording (MUST,
"non-optional", reordered earlier) didn't move it; this looks like a
model-floor behavior rather than a skill ambiguity. Claude and Gemini
honor the load.