pbakaus_impeccable/tests/skill-behavior

Skill-behavior tests

LLM-backed scenarios that verify how the impeccable skill drives context, command-reference, new-work, and native-platform loading. Each scenario runs against one current model from each supported provider (Anthropic, OpenAI, Google, DeepSeek).

These are the tests you re-run when you refactor anything in SKILL.md's ## Setup section. They fail when the agent stops following the loading contract.

Run

bun run test:skill-behavior
IMPECCABLE_SKILL_BEHAVIOR_VERBOSE=1 bun run test:skill-behavior   # dump per-scenario traces
IMPECCABLE_SKILL_BEHAVIOR_MODELS=claude-sonnet-5 bun run test:skill-behavior   # scope to one model
IMPECCABLE_SKILL_BEHAVIOR_EFFORT=xhigh bun run test:skill-behavior             # OpenAI reasoning effort (default: high)

Requires .env at repo root with at least one of ANTHROPIC_API_KEY, OPENAI_API_KEY, GOOGLE_CLOUD_API_KEY, DEEPSEEK_API_KEY. Providers without a key are skipped, not failed.

Also requires the engine binary (bun run fetch:engine, or IMPECCABLE_BIN). The staged skill dir ships the launcher (scripts/impeccable); the harness exports IMPECCABLE_BIN into every bash call the agent makes, so the launcher resolves the binary in both symlink and copy mode without a download. Without a binary the suites skip.

To run a single scenario against one model:

IMPECCABLE_SKILL_BEHAVIOR_MODELS=claude-sonnet-5 IMPECCABLE_SKILL_BEHAVIOR_VERBOSE=1 \
  node --test --test-timeout=600000 --test-name-pattern="scenario 6" tests/skill-behavior/scenarios.test.mjs

How it works

Each scenario:

1. prepareWorkspace() mints a temp dir, symlinks the canonical skill into <workspace>/.claude/skills/impeccable (so its launcher is at .claude/skills/impeccable/scripts/impeccable), and optionally writes PRODUCT.md / DESIGN.md fixtures.
2. runTurn() inlines SKILL.md (placeholders neutralized) as the system prompt and runs Vercel AI SDK generateText with four workspace-scoped tools: bash, read, write, list, and a fake provider-neutral ask_user_question backed by a deterministic simulated user.
3. The tools record every call into a trace that the test asserts on.
4. For scenario 4, a second runTurn reuses turn 1's responseMessages so the model sees a real multi-turn conversation.

The trace is the source of truth, not the model's free-form reply.

Scenarios

#	Setup	Assertion
1	empty workspace	runs impeccable context; loads reference/init.md before implementation; automation is not an init bypass
2	PRODUCT.md only	runs impeccable context 1-3 times; loads reference/new-work.md to resolve visual authority, establish a world when needed, and develop the surface
3	PRODUCT.md + DESIGN.md	runs impeccable context 1-3 times; receives the committed design system and loads reference/new-work.md for the task-scoped concept
4	PRODUCT.md + DESIGN.md, context already loaded in turn 1	turn 2 does not re-run impeccable context
5	PRODUCT.md without the legacy ## Register field and no DESIGN.md	runs impeccable context; greenfield craft loads reference/new-work.md, not init, to establish the missing world
6	PRODUCT.md + DESIGN.md + a minimal index.html; prompt is /impeccable polish	loads reference/polish.md
7	same fixture; prompt is /impeccable audit	loads reference/audit.md
8	PRODUCT.md + DESIGN.md + a SvelteKit scaffold (src/app.css, components, +page.svelte); prompt is /impeccable polish src/routes/+page.svelte	reads at least one project code file (CSS / component / page) — not just the skill's reference files
9	PRODUCT.md + index.html + a seeded update cache with a newer version (skillVersion copy-mode so impeccable context has a SKILL.md to version-check against); prompt is /impeccable polish index.html	impeccable context runs and its output carries the UPDATE_AVAILABLE directive (proven via captured bash output); the agent does not auto-run npx impeccable update (it must ask first)
10	no PRODUCT.md + a minimal index.html; prompt is /impeccable polish index.html	runs impeccable context, loads reference/polish.md, and does not divert into reference/init.md
11	empty workspace; prompt is /impeccable shape ...	runs impeccable context; resolves reference/init.md before planning the surface
12	empty workspace; prompt is natural-language build intent with no command word	runs impeccable context; resolves reference/init.md before implementation
13	empty workspace; prompt is /impeccable teach	runs impeccable context and diverts into reference/init.md because teach aliases init
14	PRODUCT.md with ## Platform: ios (native iOS app); prompt is /impeccable craft a tide detail screen	impeccable context runs and emits the contents of reference/ios.md directly, placing native conventions in context without a second model-directed read
15	same iOS fixture; prompt is /impeccable audit	agent loads reference/audit.native.md (the Commands-table native variant, routed instead of audit.md)
16	existing surface, with and without PRODUCT.md; asks where to start	loads routing.md, delivers advice, and does not edit project files, start an interview, archive a critique, or run menu scans
17	existing surface; asks whether critique is required before polish	loads routing.md and both command references, then delivers advice without executing the playbooks
18	existing surface; explicitly requests polish followed by a next-command recommendation	loads polish.md rather than substituting workflow advice for the requested work
19	tiny spacing edit with PRODUCT.md + DESIGN.md; Bash denied, a real-loader success control, and a denied-launcher planning-only case	edits require successful playbook/craft-floor reads and a pre-edit denial warning; planning stays read-only and skips craft-floor

Setup launcher-failure branch (2026-09-06, PR #750)

Scenario 19 injects a host permission error before any shell command executes, including retries and compound commands. File tools remain available; the staged skill is read-only. Assertions require successful reads, an actual UI edit, unchanged context files, and disclosure before the first write in the assistant message sequence. Failed read attempts and shell commands merely mentioning a reference do not count as loading it. The success control allows only the real context-loader command through Bash and requires exit 0.

The suite measures continuation after a tool refusal, not Claude's skill activation or plugin substitution. Those are separate loader/path checks. It also does not establish behavior for every missing-binary or runtime error.

Focused baseline (2026-09-06): both scenario 19 cases passed on claude-sonnet-5, one run each. The original wording also continued after denial in the comparison run, but disclosed the failure only in its final summary. This does not reproduce the reporter's complete reference-loading failure or establish a multi-provider pass. An earlier scenario 6 result used attempt-based reference assertions and is not counted as a success control.

Refusal follow-up (2026-09-06, #744)

The provider-neutral harness models a loaded skill with a known base directory using synthetic workspace-relative host metadata, not each provider's exact generated prompt. The source instructions and <skill-base-dir> resolution remain under test; reference reads still have to succeed. Provider transforms and plugin loading have separate path/loader tests. The harness also sets an explicit 16,384-token response ceiling for DeepSeek: the Anthropic-compatible SDK otherwise treats that model as unknown and caps it at 4,096. Truncation still fails the scenario; this changes the test runner, not the shipped skill.

The unchanged-source baseline with directory metadata passed 5/8 focused cases: Sonnet skipped craft-floor in its successful-launcher control, OpenAI stopped without editing after denial, and Gemini warned only after editing. DeepSeek passed both cases. An initial candidate got OpenAI to edit but still warned late on Sonnet, OpenAI, and Gemini; DeepSeek's denial response truncated. All four successful-launcher controls passed that candidate.

The pre-review candidate separates the fallback from the long first step, says to send the warning first and continue through permitted tools, and clarifies that craft-floor also applies to small refinements. Setup grows by 18 whitespace-separated words; the description is unchanged. Sonnet and OpenAI passed both final cases, as did DeepSeek with the explicit output ceiling. Gemini still warned after the edit; its control passed. The final result is 7/8; the warning-order assertion remains unchanged.

Review follow-up: the fallback now says to follow the applicable steps 2–3, preserving step 3's planning-only exclusion (20 added Setup words overall). A new denied-launcher planning case requires a real plan, no mutations or craft-floor read, and successful context/target/playbook reads. On Sonnet, the editing denial and successful-loader cases both passed again. The planning run stayed read-only and skipped craft-floor, but failed because it did not read polish.md. That assertion remains intact: this is another reference-loading gap under #744, not a green planning result. Other providers were not rerun for this wording-only review clarification.

The planning case also checks the response-message sequence: the launcher must actually be denied, then an assistant warning must precede the first fallback PRODUCT.md or DESIGN.md read. Deterministic tests reject silent continuation, final-only warnings, warnings before denial, and user-authored warnings. This checks disclosure even when no editing occurs; the editing cases retain their existing pre-edit warning assertion.

One focused Sonnet rerun with this guard read the playbook and produced a read-only plan without craft-floor, but omitted the launcher warning entirely. The strengthened assertion correctly failed that run; #744 remains open for the behavior failure rather than treating this coverage fix as a skill fix.

These are single samples per case and candidate, not reliability estimates. This API harness starts with the skill loaded and readable references. It does not measure activation, reproduce Windows command parsing, or establish fallback behavior when the host also denies required file reads or writes. Keep #744 open; evaluate activation separately with #375.

To repeat only these cases (provider keys and an engine binary required):

IMPECCABLE_SKILL_BEHAVIOR_MODELS=claude-sonnet-5,gpt-5.6-terra,gemini-3.7-flash,deepseek-v4-flash \
  node --test --test-name-pattern='scenario 19:' tests/skill-behavior/scenarios.test.mjs

Workflow-advice baseline (2026-09-05, PR #737)

The four cases in scenarios 16-18 are new; prior scenario results do not establish their behavior. The advice-only assertions inspect write-tool calls and file mutations from bash (excluding context's internal .impeccable/ state, but not critique reports), require an actual answer, and reject interviews and menu scans. Scenario 18 checks command-reference precedence; it does not assert completion of a full polish pass. These cases allow only the exact context-loader command through bash; references use read/list, and the write tool remains available for project files so unsolicited edits still fail the test. Writes to the staged skill are rejected. This keeps the routing measurement from running arbitrary shell searches outside its fixture.

Scenario	claude-sonnet-5	gpt-5.6-terra	gemini-3.7-flash
16 (existing project)	pass	pass	pass
16 (missing product context)	flaky (1 of 2)	pass	pass
17 (command comparison)	pass	pass	pass
18 (explicit command)	pass	pass	pass

Rechecked after reducing the skill addition to 59 words. Claude's first missing-context response stayed read-only but skipped routing.md; an unchanged repeat passed on all three providers. Keep that reference-read miss visible as a flake rather than adding instructions for a single observation. All assertions remain unchanged. Scenario 18 uses an eight-step budget and actual read-tool evidence, not rejected bash reads. The initial unrestricted run (stopped after host-wide search attempts), earlier rejected-read results, and broader suite's sandboxed provider DNS errors are excluded from this baseline.

The workflow-contract file adds end-to-end assertions for attended fresh init, an initialized natural build request, replacement-world redesign, scope-preserving bolder refinement, and critique's closing question. It checks question order and context/artifact writes rather than only reference-file loading.

critique closes with the question or an explicit skip line is a regression guard, not a routing check. A critique that prints its report and then stops, asking nothing and printing no Questions skipped: <reason> line, is an incomplete run: the close is half the deliverable, and polish downstream has no priorities to inherit without it. The fixture page is deliberately broken enough to put the report past the three-Priority-Issue threshold, so the run cannot reach the skip branch on merit. The assertion is deliberately loose about how the run closes, because either close is valid; what it forbids is neither.

Workflow-contract baseline (2026-08-13, current lineup)

Measured while checking whether an {{ask_instruction}} rewrite had regressed anything.

The last two columns are no longer in the default lineup. gpt-5.6-luna and deepseek-v4-flash were dropped in 2026-08 for being below the frontier tier: they fail scenarios by stopping mid-run or archiving a report without stating it, which is model-floor behavior rather than a skill-text defect. Their columns stay here because they are the record of what a weaker model does with this text, and that is the useful part. Reproduce with IMPECCABLE_SKILL_BEHAVIOR_MODELS=gpt-5.6-luna,deepseek-v4-flash.

Against the current default lineup, two cells are the known floor: redesign replaces DESIGN is flaky on every model, and critique closes is flaky on gemini-3.6-flash. A regression is a failure beyond those two.

The Google slot in DEFAULT_MODELS moved to gemini-3.7-flash on 2026-08-15. Every Gemini cell in the tables below was measured on 3.6-flash (or 3.5-flash where marked), and per the cross-version rule further down, those results are unmeasured on 3.7, not inherited. Re-run the sweep on the next Setup or routing change and update the tables to the new column.

Read any failure against the clock before calling it behavior. The suite ran at a 300s per-test timeout until 2026-08-13, and for the workflow-contract scenarios that cap was below the runtime of a correct run. initialized natural build on claude-sonnet-5 was measured at 579s when it stopped to put the concept to the user before building, while the runs that skipped that checkpoint and failed the assertion finished in 130-200s. The cap was therefore selecting for the behavior the scenario forbids: thorough runs were killed, hasty ones were graded. The timeout is now 900s (scripts/test-suites.mjs). A duration at or just past the cap is a timeout, not a verdict.

Scenario	claude-sonnet-5	gpt-5.6-terra	gemini-3.6-flash	luna / deepseek (dropped)
attended fresh init	pass	pass	pass	not measured
initialized natural build	flaky (1 of 4, and see the clock note)	pass	pass	not measured
redesign replaces DESIGN	flaky (timeout this run)	fail	fail (timeout)	not measured
bolder refinement	pass	pass	pass (on 3.5)	luna pass, deepseek fail
critique closes	pass (4 of 4)	pass (3 of 3)	flaky (1 of 4)	luna fail (1 of 6), deepseek flaky

Gemini's bolder refinement and critique closes runs in this sweep died on AI_APICallError / ETIMEDOUT before completing a turn. Network failures are not behavior measurements and are excluded from the counts above.

Scenario baseline (2026-08-13, current lineup)

Measured on the same sweep. scenarios.test.mjs passes 15 of 15 on gpt-5.6-terra and gemini-3.6-flash. Only claude-sonnet-5 fails anything, and that asymmetry is the finding: the two cells below fail on the frontier model while two weaker-on-paper lineups route correctly, so read them as a text problem that one model's priors expose rather than as a model floor.

Scenario	claude-sonnet-5	gpt-5.6-terra	gemini-3.6-flash
1-7, 10, 12-15	pass	pass	pass
8 (SvelteKit exploration)	flaky	pass	pass
11 (shape resolves the build gate)	flaky	pass	pass

Scenarios 8 and 11 pass on re-run, so treat a single failure there as flake and confirm with a second run before investigating.

Scenarios 9 and 15 both failed on sonnet when this baseline was first measured, and the two causes are worth keeping because neither was where it looked:

• 9 was a real defect in the directive. UPDATE_AVAILABLE said to ask the user, then "If they agree, run npx impeccable update", then to continue without waiting. With no wait there is no agreement to read, so the command was the only concrete instruction left standing and sonnet ran it. Fixed by removing the command from the turn entirely rather than by strengthening the warning around it.
• 15 was a broken fixture. The iOS workspace held PRODUCT.md and nothing else, so audit the app in this workspace named an app that did not exist. Sonnet spent its whole step budget looking for it and read no reference file at all, which the assertion reported as "loaded audit.md instead of the variant". The fixture now ships one SwiftUI screen, the same courtesy MINIMAL_LANDING_HTML already did for the web scenarios. The scenario passes on unmodified main once the fixture is answerable, which is the proof the routing text was never at fault.

The general lesson is worth more than either fix: an assertion reports the property it checks, not the reason it failed. Both of these read as routing defects and neither was one. Pull the trace before writing the diagnosis, and prefer IMPECCABLE_SKILL_BEHAVIOR_VERBOSE=1 over inference from the message.

Gemini cells marked on 3.5 were measured on the superseded gemini-3.5-flash and have not been re-run on 3.6. That distinction is not pedantic. critique closes passed twice on 3.5-flash, then failed three times in a row on 3.6-flash against identical instruction text, and only passed once the report delivery step was made explicit. A version bump inside one family changed the outcome, so treat cross-version carryover as unmeasured rather than inherited.

not measured means exactly that: the cell was never run in isolation on this lineup. Only the scenarios under investigation were scoped per model. The rows are worth keeping anyway, since a scenario absent from the table is easy to mistake for a scenario that passed.

bolder refinement, deepseek-v4-flash. The model runs impeccable context, reads bolder.md, craft-floor.md, and current.html, then ends its turn without editing anything: empty writePaths, no ask_user_question call, well short of the 16-step cap. Confirmed identical on HEAD with bolder.md reverted, so it is not a skill-text problem. Same shape as the gpt-5.4-mini scenario 6/7 failures below: the model consumes the references and then declines to act.

critique closes: the three ways a critique fails to land. The scenario asserts emission order, not just the presence of a question, because the command fails in three distinct ways and only one of them was the reported bug:

1. No close. Report lands, no question, no skip line. polish downstream inherits nothing.
2. Question before report. The question is emitted first and the report after it, so the report is withheld until the user answers. Observed directly on gpt-5.6-luna, and the reason the invariant is a position rule ("the question is the LAST thing in the response") rather than a statement about prose order.
3. Report never spoken. The report is authored straight into the persistence heredoc, archived, and never written to chat. A perfect snapshot and a user who sees nothing.

Mode 3 is the one worth understanding, because it was structural rather than a model quirk. critique.md described the report's format and then went directly to writing a temp file, with no step that said to output the report. Both gemini-3.6-flash and luna responded by bundling heredoc, snapshot write, trend read, and cleanup into a single bash call and stopping. The Deliver the Report section exists to close that gap, and it worked: gemini-3.6-flash failed three consecutive runs before it, and its failures afterward all show the report reaching chat.

Mode 1 is not fixed on gemini-3.6-flash. It passes 1 run in 3 on the final text. Two structural attempts were made and neither settled it: the close was promoted into Hard Invariants with a printable Questions skipped: <reason> string, then made step 6 of the persistence list so it would sit inside the numbered flow rather than after it (the shape that fixed mode 3). Both moved it from consistently failing to intermittently passing, and further prose tuning was not paying, so it stopped. claude-sonnet-5 and gpt-5.6-terra are clean. Treat this cell as the known floor and re-measure rather than tuning blindly: the next useful move is probably a structural one, such as making the close something the run cannot syntactically finish without, not another paragraph.

Read the counts here as what they are: small samples on a nondeterministic system, several of them gathered while the instruction text was still changing between runs. They support "the close works on the current lineup" and not much finer than that. Re-measure rather than assuming when the lineup changes.

redesign replaces DESIGN, flaky. It has failed on two different assertions across runs (designWrite > question and implementation > designWrite), and on one run claude-sonnet-5 exhausted the 300s per-test timeout instead of asserting. The traces never load document.md; the ordering under test comes from new-work.md. Re-run before believing a single red result here. Which model produced which failure was not pinned down, so the row records only that the scenario is unstable.

The bolder claude-sonnet-5 cell is unmeasured for a specific reason: the scoped run that produced this table used a 180s cap, which sonnet exceeded. That is a timeout, not a failure, and it is why the guidance below insists on 300000.

Scoping a run while investigating

Both files honor --test-name-pattern, which is much cheaper than a full sweep when bisecting one scenario:

IMPECCABLE_QUESTION_DISABLED=1 CI=1 IMPECCABLE_SKILL_BEHAVIOR_MODELS=deepseek-v4-flash \
  node --test --test-timeout=300000 --test-force-exit \
  --test-name-pattern="bolder refinement" tests/skill-behavior/workflow-contract.test.mjs

Keep --test-timeout at 300000. A tighter cap turns claude-sonnet-5's slower runs into timeouts that look like failures. Set IMPECCABLE_QUESTION_DISABLED=1 and CI=1 so impeccable serve-question cannot open a browser window on the host. Pipe to a file rather than tail; node prints the failing-test summary at the end, and truncating it costs you the per-model attribution.

Baseline state (2026-05-20, previous cheap tier)

Historical record. The default models are now claude-sonnet-5 and gemini-3.6-flash. The table below was measured on an older cheap tier (claude-haiku-4-5 / gpt-5.4-mini) and is kept as the historical record. Re-measure on the current lineup and update this section; the stronger models are expected to clear the scenario 6/7 routing failures that the old gpt tier showed.

Captured after moving sub-command reference loading from step 4 to step 2 of Setup (so the agent loads reference/<command>.md right after impeccable context, before "doing the work" preempts it), and tightening step 3 to require at least one project code read even when a sub-command reference loads first. Use this table when comparing pre/post refactor: a regression is "more failures than baseline", not "any failures at all".

Scenario	claude-haiku-4-5	gpt-5.4-mini	gemini-3.1-flash-lite
1 (no context)	pass (rare flake — agent stops after impeccable context without loading init.md)	pass	pass
2 (product only)	pass	pass	pass
3 (product + design)	pass	pass	pass (rare flake — sub-command ref loads but world ref doesn't)
4 (already loaded)	pass	pass	pass
5 (no register field, task-cue cascade)	pass	pass	pass
6 (polish routing)	pass	fail	pass
7 (audit routing)	pass	fail	pass
8 (existing project, explore design system)	pass	pass	pass

21-22 / 24 typical. The stable failures are gpt-5.4-mini scenarios 6 and 7: the model reads index.html (the target file), recognizes "polish" or "audit" as a familiar action, and proceeds with the work without ever loading the sub-command reference. Stronger SKILL.md wording (MUST, "non-optional", reordered earlier) didn't move it; this looks like a model-floor behavior rather than a skill ambiguity. Claude and Gemini honor the load.