mirror of
https://github.com/pbakaus/impeccable.git
synced 2026-09-13 06:36:26 +03:00
c11cc7b58c
* Route native projects to native command variants for audit and adapt Follow-up to #269. The web audit.md and adapt.md carried "translate this yourself" Platform notes, so a native invocation paid for the full web file (~1.8k / ~2.6k tokens, mostly inapplicable) and did error-prone run-time translation. Authored with AI assistance (Claude Code) under maintainer direction. - New reference/audit.native.md and reference/adapt.native.md: authored native content (VoiceOver/TalkBack, platform conformance, adaptivity dimensions; phone-to-tablet, platform-to-platform, web-to-native strategies). One variant per command covers ios, android, and adaptive; per-OS specifics stay in the platform refs Setup loads regardless. - SKILL.src.md: Commands table lists the variants; Setup step 2 reads the variant instead of the web file when the platform is native. - audit.md / adapt.md: Platform sections replaced with a one-line web-only guard pointing at the variant. - animate.md / layout.md: Platform sections deleted; the Motion and Layout sections of the already-loaded platform refs carry that content. Web users now pay zero tokens for the platform axis in these files. - Skill-behavior scenario 15 pins the route-instead behavior (passes live on claude-sonnet-4-6); CLAUDE.md documents the variant convention. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Phrase command-reference routing as one rule, not rule-plus-exception Copilot review catch: step 2 said "MUST read reference/<command>.md" and then carved out the native variant, which invites loading both files. Now a single rule: read the web reference or the table's native variant, one file, not both. Scenario 15 re-verified live. Applied with AI assistance (Claude Code) under maintainer direction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Anchor native runs in animate/layout, drop loaded-refs assumption Review-thread fixes, applied with AI assistance (Claude Code) under maintainer direction: - Greptile: deleting the animate/layout Platform sections left native runs alone with web tooling instructions (CSS keyframes, GSAP, Grid, clamp()). Restore a one-line anchor in each pointing at the loaded platform reference's Motion / Layout section (~20 tokens, not the old restatements). - Bugbot: audit.native.md and adapt.native.md asserted the platform refs were "already loaded in Setup", but the command reference loads at step 2, before step 5. Now they instruct: read the platform reference first if Setup hasn't already. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Carry the native-variant rule into routing rules 2 and 3 Bugbot catch: Setup step 2 routed native projects to the variant, but routing rules 2 and 3 (the operative text at command time) still said to load the generic reference file. Both now reference the same one-file variant rule. Applied with AI assistance (Claude Code) under maintainer direction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Point animate/layout native anchors at the files, not "loaded" refs Bugbot catch, same class as the variant wording fix: the anchor lines said "the loaded platform reference" but command files load at step 2, before the platform refs at step 5. Both anchors now name the files and instruct reading them first if Setup hasn't already. Applied with AI assistance (Claude Code) under maintainer direction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
98 lines
6.5 KiB
Markdown
98 lines
6.5 KiB
Markdown
# Skill-behavior tests
|
|
|
|
LLM-backed scenarios that verify how the impeccable skill drives
|
|
PRODUCT.md / DESIGN.md loading. Each scenario runs against the cheapest
|
|
tier of each major provider (Anthropic, OpenAI, Google) so a full sweep
|
|
costs a few cents and finishes in ~2 minutes.
|
|
|
|
These are the tests you re-run when you refactor anything in SKILL.md's
|
|
`## Setup` section. They fail when the agent stops following the loading
|
|
contract.
|
|
|
|
## Run
|
|
|
|
```bash
|
|
bun run test:skill-behavior
|
|
IMPECCABLE_SKILL_BEHAVIOR_VERBOSE=1 bun run test:skill-behavior # dump per-scenario traces
|
|
IMPECCABLE_SKILL_BEHAVIOR_MODELS=claude-sonnet-4-6 bun run test:skill-behavior # scope to one model
|
|
```
|
|
|
|
Requires `.env` at repo root with at least one of `ANTHROPIC_API_KEY`,
|
|
`OPENAI_API_KEY`, `GOOGLE_CLOUD_API_KEY`. Providers without a key are
|
|
skipped, not failed.
|
|
|
|
## How it works
|
|
|
|
Each scenario:
|
|
|
|
1. `prepareWorkspace()` mints a temp dir, symlinks the canonical skill
|
|
into `<workspace>/.claude/skills/impeccable`, and optionally writes
|
|
`PRODUCT.md` / `DESIGN.md` fixtures.
|
|
2. `runTurn()` inlines `SKILL.md` (placeholders neutralized) as the
|
|
system prompt and runs Vercel AI SDK `generateText` with four
|
|
workspace-scoped tools: `bash`, `read`, `write`, `list`.
|
|
3. The tools record every call into a `trace` that the test asserts on.
|
|
4. For scenario 4, a second `runTurn` reuses turn 1's `responseMessages`
|
|
so the model sees a real multi-turn conversation.
|
|
|
|
The trace is the source of truth, not the model's free-form reply.
|
|
|
|
## Scenarios
|
|
|
|
| # | Setup | Assertion |
|
|
|---|---|---|
|
|
| 1 | empty workspace | runs `context.mjs` (which prints a `NO_PRODUCT_MD` directive); agent then loads `reference/init.md` via Read or `cat`; does **not** start writing HTML/CSS |
|
|
| 2 | PRODUCT.md only (with `## Register: brand`) | runs `context.mjs` 1-3 times; loads `reference/brand.md` |
|
|
| 3 | PRODUCT.md + DESIGN.md (brand register) | runs `context.mjs` 1-3 times; loads `reference/brand.md`; consults the design system (DESIGN.md bundled in output, but CSS / tokens / directory listing also count) |
|
|
| 4 | PRODUCT.md + DESIGN.md, context already loaded in turn 1 | turn 2 does **not** re-run `context.mjs`; `reference/brand.md` is loaded across turns 1+2 |
|
|
| 5 | PRODUCT.md WITHOUT a `## Register` field; task cue says "landing page" | runs `context.mjs` (which emits a generic register directive); agent loads `reference/brand.md` via task-cue cascade |
|
|
| 6 | PRODUCT.md + DESIGN.md + a minimal `index.html`; prompt is `/impeccable polish` | loads `reference/polish.md` |
|
|
| 7 | same fixture; prompt is `/impeccable audit` | loads `reference/audit.md` |
|
|
| 8 | PRODUCT.md + DESIGN.md + a SvelteKit scaffold (`src/app.css`, components, `+page.svelte`); prompt is `/impeccable polish src/routes/+page.svelte` | reads at least one project code file (CSS / component / page) — not just the skill's reference files |
|
|
| 9 | PRODUCT.md + `index.html` + a seeded update cache with a newer version (`skillVersion` copy-mode so `context.mjs` has a `SKILL.md` to version-check against); prompt is `/impeccable polish index.html` | `context.mjs` runs and its output carries the `UPDATE_AVAILABLE` directive (proven via captured bash output); the agent does **not** auto-run `npx impeccable update` (it must ask first) |
|
|
| 10 | no PRODUCT.md + a minimal `index.html`; prompt is `/impeccable polish index.html` | runs `context.mjs`, loads `reference/polish.md`, and does **not** divert into `reference/init.md` |
|
|
| 11 | empty workspace; prompt is `/impeccable shape ...` | runs `context.mjs`, diverts into `reference/init.md`, and does **not** start writing HTML/CSS |
|
|
| 12 | empty workspace; prompt is natural-language build intent with no command word | runs `context.mjs`, diverts into `reference/init.md`, and does **not** start writing HTML/CSS |
|
|
| 13 | empty workspace; prompt is `/impeccable teach` | runs `context.mjs` and diverts into `reference/init.md` because `teach` aliases `init` |
|
|
| 14 | PRODUCT.md with `## Register: product` + `## Platform: ios` (native iOS app); prompt is `/impeccable craft a tide detail screen` | `context.mjs` runs and emits a NEXT STEP pointing at `reference/ios.md` (proven via captured bash output); agent loads `reference/ios.md` (Setup step 5, native conventions on top of the register reference) |
|
|
| 15 | same iOS fixture; prompt is `/impeccable audit` | agent loads `reference/audit.native.md` (the Commands-table native variant, routed instead of `audit.md`) |
|
|
|
|
Scenario 9 passed on all three current-lineup providers (`claude-sonnet-4-6`,
|
|
`gpt-5.5`, `gemini-3.1-flash-lite`) on 2026-05-28.
|
|
|
|
## Baseline state (2026-05-20, previous cheap tier)
|
|
|
|
> **Lineup changed.** The default models are now `claude-sonnet-4-6`,
|
|
> `gpt-5.5`, and `gemini-3.1-flash-lite` (production-tier on Anthropic and
|
|
> OpenAI). The table below was measured on the *old* cheap tier
|
|
> (`claude-haiku-4-5` / `gpt-5.4-mini`) and is kept as the historical record.
|
|
> Re-measure on the current lineup and update this section; the stronger
|
|
> models are expected to clear the scenario 6/7 routing failures that the old
|
|
> gpt tier showed.
|
|
|
|
Captured after moving sub-command reference loading from step 4 to step 2
|
|
of Setup (so the agent loads `reference/<command>.md` right after
|
|
`context.mjs`, before "doing the work" preempts it), and tightening
|
|
step 3 to require at least one project code read even when a sub-command
|
|
reference loads first. Use this table when comparing pre/post refactor:
|
|
a regression is "more failures than baseline", not "any failures at all".
|
|
|
|
| Scenario | claude-haiku-4-5 | gpt-5.4-mini | gemini-3.1-flash-lite |
|
|
|---|---|---|---|
|
|
| 1 (no context) | pass (rare flake — agent stops after `context.mjs` without loading `init.md`) | pass | pass |
|
|
| 2 (product only) | pass | pass | pass |
|
|
| 3 (product + design) | pass | pass | pass (rare flake — sub-command ref loads but register ref doesn't) |
|
|
| 4 (already loaded) | pass | pass | pass |
|
|
| 5 (no register field, task-cue cascade) | pass | pass | pass |
|
|
| 6 (`polish` routing) | pass | **fail** | pass |
|
|
| 7 (`audit` routing) | pass | **fail** | pass |
|
|
| 8 (existing project, explore design system) | pass | pass | pass |
|
|
|
|
21-22 / 24 typical. The stable failures are gpt-5.4-mini scenarios 6 and 7:
|
|
the model reads `index.html` (the target file), recognizes "polish" or
|
|
"audit" as a familiar action, and proceeds with the work without ever
|
|
loading the sub-command reference. Stronger SKILL.md wording (MUST,
|
|
"non-optional", reordered earlier) didn't move it; this looks like a
|
|
model-floor behavior rather than a skill ambiguity. Claude and Gemini
|
|
honor the load.
|