mirror of
https://github.com/pbakaus/impeccable.git
synced 2026-09-15 07:36:50 +03:00
3e38e595c7
* Add a platform axis (web / ios / android / adaptive) to the skill Orthogonal to register: register decides whether design IS or SERVES the product; platform decides the delivery target and which native conventions apply. Set `## Platform` in PRODUCT.md; a missing field defaults to `web`, so legacy projects are unaffected. - extractPlatform() in skill/scripts/context.mjs (mirrors extractRegister); the CLI appends a NEXT STEP directive to read the native reference(s). `adaptive` (Flutter / RN / KMP shipping both iOS and Android) loads both ios.md and android.md. - New reference/ios.md (Apple HIG distilled) and reference/android.md (Material 3 distilled); reference/web.md is a thin pointer. The native refs frame register's role as narrow: platform conformance is the bar, brand lives in the expressive layer the platform gives you, never by breaking the rails. - Setup step 5 loads the native reference(s) when platform is native. Live mode and the detect CLI stay web-only, gated off ios/android/adaptive. - init asks platform right after register; adapt/audit/animate/layout carry short platform divergence notes; all secondary spots thread `adaptive`. - a11y stays in audit.md (loading it at design time makes output timid), so the native refs carry no Accessibility section; audit.md's Platform section owns native a11y. - Tests: extractPlatform unit coverage + skill-behavior scenario 10 (PRODUCT.md platform ios -> agent loads ios.md). Source-first: only skill/, scripts/, tests/, CLAUDE.md, NOTICE.md, the changelog and version are committed; the sync workflow regenerates the provider trees and ./plugin on merge. ios.md / android.md are distilled from the MIT-licensed ehmo/platform-design-skills; attribution in NOTICE.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Address review: gate web tools on native platforms, drop version churn Maintainer-review fixes applied with AI assistance (Claude Code), on top of the rebased platform-axis commit: - Design hook (post-edit and Cursor pre-edit) now resolves the project platform via loadContext + extractPlatform and skips its web rule scan for ios / android / adaptive projects, so React Native / Flutter code never draws web-shaped findings (new hook-lib resolveProjectPlatform / isNativePlatform helpers, covered by unit and subprocess tests). - context.mjs CLI warns on an unrecognized ## Platform value (e.g. a toolchain name like `flutter`) instead of silently defaulting to web; extractRegister / extractPlatform now share extractSectionValue. - Removed reference/web.md: nothing loaded it; CLAUDE.md carries the "web has no extra rulebook" explanation. - init.md: skip live-mode config (Step 6) for native platforms; note the per-app PRODUCT.md pattern for repos shipping web + native. - android.md: Material-everywhere apps that also ship on iPhone still owe iOS OS guarantees (safe areas, Reduce Motion, edge-swipe back). - ios.md: reworded a design-time line that framed Dynamic Type as an accessibility check (a11y stays owned by audit.md). - Renumbered the new skill-behavior scenario to 14 after main's 10-13; updated CLAUDE.md scenario list; added android + unrecognized-value CLI test cases. - No version or changelog changes: versioning happens at release time. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Tighten platform reference prose Editorial pass on the platform-axis text, applied with AI assistance (Claude Code) under maintainer direction: - ios.md / android.md rewritten to house style: single-line paragraphs (no hard wraps), one-sentence scope intro, deduplicated intro/slop-test, register-compression down to two sentences. In-file attribution paragraphs removed (NOTICE.md owns attribution); "read on top of the register reference" cruft removed (SKILL step 5 and the context.mjs directive already say it). Bans sections dropped: they restated the rules above them; the two additive items (tab-bar overload, hover-dependent affordances) folded into rules. ~40% smaller each. - Sub-command Platform sections (adapt, audit, animate, layout), SKILL step 5, init.md platform prose, and the context.mjs directive trimmed the same way. Build (prose validators, counts) and both test runners green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Treat an empty PRODUCT.md section as absent, not the next heading Copilot review catch: extractSectionValue read the next `## ...` heading as the section value when a field was left empty, which made the CLI warn "value `## Product Purpose` is not recognized". Stop at the next heading and return null instead. Regression tests for extractPlatform, extractRegister, and the CLI warning path. Applied with AI assistance (Claude Code) under maintainer direction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Only read a token list of both native targets as adaptive Bugbot catch: after the exact platform tokens failed, any Platform line containing the words ios and android was classified adaptive, so negated or explanatory prose ("web only, not ios or android") silently loaded both native refs and skipped the hook, with no warning. The combo parse now accepts only list separators and the two platform words; anything else falls through to the CLI's unrecognized-value WARNING. Regression tests added. Applied with AI assistance (Claude Code) under maintainer direction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Paul Bakaus <paul.bakaus@gmail.com>
97 lines
6.3 KiB
Markdown
97 lines
6.3 KiB
Markdown
# Skill-behavior tests
|
|
|
|
LLM-backed scenarios that verify how the impeccable skill drives
|
|
PRODUCT.md / DESIGN.md loading. Each scenario runs against the cheapest
|
|
tier of each major provider (Anthropic, OpenAI, Google) so a full sweep
|
|
costs a few cents and finishes in ~2 minutes.
|
|
|
|
These are the tests you re-run when you refactor anything in SKILL.md's
|
|
`## Setup` section. They fail when the agent stops following the loading
|
|
contract.
|
|
|
|
## Run
|
|
|
|
```bash
|
|
bun run test:skill-behavior
|
|
IMPECCABLE_SKILL_BEHAVIOR_VERBOSE=1 bun run test:skill-behavior # dump per-scenario traces
|
|
IMPECCABLE_SKILL_BEHAVIOR_MODELS=claude-sonnet-4-6 bun run test:skill-behavior # scope to one model
|
|
```
|
|
|
|
Requires `.env` at repo root with at least one of `ANTHROPIC_API_KEY`,
|
|
`OPENAI_API_KEY`, `GOOGLE_CLOUD_API_KEY`. Providers without a key are
|
|
skipped, not failed.
|
|
|
|
## How it works
|
|
|
|
Each scenario:
|
|
|
|
1. `prepareWorkspace()` mints a temp dir, symlinks the canonical skill
|
|
into `<workspace>/.claude/skills/impeccable`, and optionally writes
|
|
`PRODUCT.md` / `DESIGN.md` fixtures.
|
|
2. `runTurn()` inlines `SKILL.md` (placeholders neutralized) as the
|
|
system prompt and runs Vercel AI SDK `generateText` with four
|
|
workspace-scoped tools: `bash`, `read`, `write`, `list`.
|
|
3. The tools record every call into a `trace` that the test asserts on.
|
|
4. For scenario 4, a second `runTurn` reuses turn 1's `responseMessages`
|
|
so the model sees a real multi-turn conversation.
|
|
|
|
The trace is the source of truth, not the model's free-form reply.
|
|
|
|
## Scenarios
|
|
|
|
| # | Setup | Assertion |
|
|
|---|---|---|
|
|
| 1 | empty workspace | runs `context.mjs` (which prints a `NO_PRODUCT_MD` directive); agent then loads `reference/init.md` via Read or `cat`; does **not** start writing HTML/CSS |
|
|
| 2 | PRODUCT.md only (with `## Register: brand`) | runs `context.mjs` 1-3 times; loads `reference/brand.md` |
|
|
| 3 | PRODUCT.md + DESIGN.md (brand register) | runs `context.mjs` 1-3 times; loads `reference/brand.md`; consults the design system (DESIGN.md bundled in output, but CSS / tokens / directory listing also count) |
|
|
| 4 | PRODUCT.md + DESIGN.md, context already loaded in turn 1 | turn 2 does **not** re-run `context.mjs`; `reference/brand.md` is loaded across turns 1+2 |
|
|
| 5 | PRODUCT.md WITHOUT a `## Register` field; task cue says "landing page" | runs `context.mjs` (which emits a generic register directive); agent loads `reference/brand.md` via task-cue cascade |
|
|
| 6 | PRODUCT.md + DESIGN.md + a minimal `index.html`; prompt is `/impeccable polish` | loads `reference/polish.md` |
|
|
| 7 | same fixture; prompt is `/impeccable audit` | loads `reference/audit.md` |
|
|
| 8 | PRODUCT.md + DESIGN.md + a SvelteKit scaffold (`src/app.css`, components, `+page.svelte`); prompt is `/impeccable polish src/routes/+page.svelte` | reads at least one project code file (CSS / component / page) — not just the skill's reference files |
|
|
| 9 | PRODUCT.md + `index.html` + a seeded update cache with a newer version (`skillVersion` copy-mode so `context.mjs` has a `SKILL.md` to version-check against); prompt is `/impeccable polish index.html` | `context.mjs` runs and its output carries the `UPDATE_AVAILABLE` directive (proven via captured bash output); the agent does **not** auto-run `npx impeccable update` (it must ask first) |
|
|
| 10 | no PRODUCT.md + a minimal `index.html`; prompt is `/impeccable polish index.html` | runs `context.mjs`, loads `reference/polish.md`, and does **not** divert into `reference/init.md` |
|
|
| 11 | empty workspace; prompt is `/impeccable shape ...` | runs `context.mjs`, diverts into `reference/init.md`, and does **not** start writing HTML/CSS |
|
|
| 12 | empty workspace; prompt is natural-language build intent with no command word | runs `context.mjs`, diverts into `reference/init.md`, and does **not** start writing HTML/CSS |
|
|
| 13 | empty workspace; prompt is `/impeccable teach` | runs `context.mjs` and diverts into `reference/init.md` because `teach` aliases `init` |
|
|
| 14 | PRODUCT.md with `## Register: product` + `## Platform: ios` (native iOS app); prompt is `/impeccable craft a tide detail screen` | `context.mjs` runs and emits a NEXT STEP pointing at `reference/ios.md` (proven via captured bash output); agent loads `reference/ios.md` (Setup step 5, native conventions on top of the register reference) |
|
|
|
|
Scenario 9 passed on all three current-lineup providers (`claude-sonnet-4-6`,
|
|
`gpt-5.5`, `gemini-3.1-flash-lite`) on 2026-05-28.
|
|
|
|
## Baseline state (2026-05-20, previous cheap tier)
|
|
|
|
> **Lineup changed.** The default models are now `claude-sonnet-4-6`,
|
|
> `gpt-5.5`, and `gemini-3.1-flash-lite` (production-tier on Anthropic and
|
|
> OpenAI). The table below was measured on the *old* cheap tier
|
|
> (`claude-haiku-4-5` / `gpt-5.4-mini`) and is kept as the historical record.
|
|
> Re-measure on the current lineup and update this section; the stronger
|
|
> models are expected to clear the scenario 6/7 routing failures that the old
|
|
> gpt tier showed.
|
|
|
|
Captured after moving sub-command reference loading from step 4 to step 2
|
|
of Setup (so the agent loads `reference/<command>.md` right after
|
|
`context.mjs`, before "doing the work" preempts it), and tightening
|
|
step 3 to require at least one project code read even when a sub-command
|
|
reference loads first. Use this table when comparing pre/post refactor:
|
|
a regression is "more failures than baseline", not "any failures at all".
|
|
|
|
| Scenario | claude-haiku-4-5 | gpt-5.4-mini | gemini-3.1-flash-lite |
|
|
|---|---|---|---|
|
|
| 1 (no context) | pass (rare flake — agent stops after `context.mjs` without loading `init.md`) | pass | pass |
|
|
| 2 (product only) | pass | pass | pass |
|
|
| 3 (product + design) | pass | pass | pass (rare flake — sub-command ref loads but register ref doesn't) |
|
|
| 4 (already loaded) | pass | pass | pass |
|
|
| 5 (no register field, task-cue cascade) | pass | pass | pass |
|
|
| 6 (`polish` routing) | pass | **fail** | pass |
|
|
| 7 (`audit` routing) | pass | **fail** | pass |
|
|
| 8 (existing project, explore design system) | pass | pass | pass |
|
|
|
|
21-22 / 24 typical. The stable failures are gpt-5.4-mini scenarios 6 and 7:
|
|
the model reads `index.html` (the target file), recognizes "polish" or
|
|
"audit" as a familiar action, and proceeds with the work without ever
|
|
loading the sub-command reference. Stronger SKILL.md wording (MUST,
|
|
"non-optional", reordered earlier) didn't move it; this looks like a
|
|
model-floor behavior rather than a skill ambiguity. Claude and Gemini
|
|
honor the load.
|