mirror of
https://github.com/pbakaus/impeccable.git
synced 2026-09-12 22:26:38 +03:00
Paul BakausandClaude Code
17dabf4b7e
A ground-up hardening of live mode, driven by a production session in a nested-app monorepo that hit six distinct failure classes. Full design rationale in docs/LIVE-REWRITE-PLAN.md; every Codex-reported failure now has a mechanical fix and a regression test. Roots: live/roots.mjs resolves appRoot/repoRoot/contextRoot once at boot (keyed on dev-server configs, not monorepo brand markers), persists a manifest, and every live CLI re-anchors onto it at startup, so a helper run from the wrong directory can no longer fork session state. Context files are discovered upward to the git root. Render truth: variant_mounted / variant_mount_failed events give the journal per-variant mount state; failures reach the agent's poll queue, raise a persistent error card with Retry (no more localStorage wipe), and an attach probe names root/dev-server mismatches explicitly. The browser rehydrates from the server when localStorage is gone. Svelte: the scaffolder now parses with the app's own svelte 5 compiler. Control flow survives (an each collection crosses the contract as one structured prop), keyed each blocks hydrate synthetic keys, and anything a detached preview cannot support falls back to source-preview instead of shipping a wrong scaffold. Preview modules live in per-publish revision directories, defeating stale transform caches. Accept: CSS is reconciled, not appended. Matching selectors are replaced, params bake from params.json kinds, the compiler's unused-selector pass prunes superseded rules (pre-existing dead rules protected), a selector- loss postcondition refuses any write that would drop hand-written rules, and live-complete refuses to finish while live plumbing remains in source. Also: framework registry (live/frameworks/) with a crash-safe injection journal, session-store snapshot caching with read-only reads, protocol enum consolidation, steer Send button, honest DESIGN-panel empty states. Testing: new unit suites (roots, AST scaffolder, accept CSS, accept pipeline, framework conformance); e2e now fails on preview-tree 404s, proves computed-style mount for every variant, drives the Tune panel through baked params, and injects failures (broken mounts, republish, storage loss). New runtime fixtures: monorepo-nested-vite (repo root != app root) and vite8-sveltekit-stateful (each blocks + state). Nightly full-matrix cron. An independent adversarial review pass preceded this commit; its blocker and major findings are fixed and regression-tested. This work was produced with AI assistance (Claude Code). Co-Authored-By: Claude Code <noreply@anthropic.com>
209 lines
13 KiB
Markdown
209 lines
13 KiB
Markdown
# Framework fixtures
|
|
|
|
Representative project shapes for exercising live mode against different framework conventions. Each fixture is a small directory tree that the test harness copies into a temp git repo, then drives `live-inject.mjs`, `live-wrap.mjs`, `live-accept.mjs`, and `lib/is-generated.mjs` against.
|
|
|
|
Fixtures can also opt into a **runtime E2E** pass that actually installs dependencies, boots the framework dev server, and drives a Playwright browser to verify the live handshake. See the `runtime` block below.
|
|
|
|
## Layout
|
|
|
|
```
|
|
<fixture>/
|
|
files/ project tree the test copies into tmp
|
|
gitignore.txt becomes .gitignore in tmp (so we can commit the real files here)
|
|
fixture.json config + expected results the test consumes
|
|
```
|
|
|
|
`fixture.json` schema:
|
|
|
|
```json
|
|
{
|
|
"name": "human-readable label",
|
|
"config": { ...contents for .impeccable/live/config.json ... },
|
|
"sourceFiles": ["paths that is-generated should classify as source (false)"],
|
|
"generatedFiles": ["paths that is-generated should classify as generated (true)"],
|
|
"wrapCases": [
|
|
{
|
|
"name": "description",
|
|
"args": { "classes": "...", "tag": "...", "elementId": "..." },
|
|
"expectedFile": "where wrap should land (relative to fixture root)",
|
|
"expectsError": "optional error code, e.g. element_not_in_source"
|
|
}
|
|
],
|
|
"csp": {
|
|
"shape": "shared-helper | inline-headers | middleware | meta-tag | null",
|
|
"signals": ["diagnostic hints — paths where CSP was detected"],
|
|
"patchTarget": "which file the agent should modify",
|
|
"expectedAfter": "filename of the reference post-patch output inside this fixture"
|
|
},
|
|
"runtime": {
|
|
"styling": "plain-css | tailwind-v4 | styled-components | ...",
|
|
"appDir": "website",
|
|
"install": ["npm", "install"],
|
|
"devCommand": ["npm", "run", "dev"],
|
|
"scheme": "http",
|
|
"ignoreHTTPSErrors": false,
|
|
"readyPattern": "Local:\\s+https?://[^:]+:(\\d+)",
|
|
"readyTimeoutMs": 120000,
|
|
"pickSelector": "h1.hero-title",
|
|
"pickPosition": { "x": 10, "y": 10 },
|
|
"variantSequence": [3, 1, 2],
|
|
"acceptedSourcePattern": "<ul[^>]*class=\"[^\"]*\\bexpense-list\\b",
|
|
"assertSourceContains": ["{#each expenses as expense, i}"],
|
|
"stateProbe": {
|
|
"textSelector": "[data-testid='open-count']",
|
|
"expectedText": "3 offen",
|
|
"windowProperty": "__impeccableStatefulMounts",
|
|
"expectedWindowValue": 1,
|
|
"expectWindowUnchanged": true
|
|
},
|
|
"paramsScenario": {
|
|
"variant": 2,
|
|
"rangeLabel": "Lead",
|
|
"rangeValue": 1.8,
|
|
"stepsLabel": "Density",
|
|
"stepsOptionLabel": "Snug",
|
|
"expectSourceContains": ["line-height: 1.8"],
|
|
"expectSourceMissing": ["letter-spacing: 0.14em"]
|
|
},
|
|
"componentFailureScenarios": { "variant": 2, "storageLoss": false },
|
|
"mode": "insert",
|
|
"insert": {
|
|
"anchorSelector": "section#features",
|
|
"position": "after",
|
|
"prompt": "Add a testimonial strip below features",
|
|
"expectSelector": ".inserted-strip",
|
|
"assertAnchorContains": "feature-grid"
|
|
},
|
|
"preActions": [
|
|
{ "type": "click", "selector": "[data-testid='open-modal']" },
|
|
{ "type": "goto", "path": "/about" }
|
|
],
|
|
"reloadProbe": {
|
|
"preActions": [{ "type": "click", "selector": "[data-testid='open-modal']" }],
|
|
"expectSelector": "h1.hero-title"
|
|
},
|
|
"steer": {
|
|
"message": "steer-e2e mark hero",
|
|
"expectSelector": "h1.hero-title[data-impeccable-steer=\"e2e\"]"
|
|
},
|
|
"probe": {
|
|
"expectLiveInit": true,
|
|
"expectConsoleClean": true
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
The `expectedAfter` file lives alongside `fixture.json` (not inside `files/`) and is a human/agent-review reference — tests don't auto-apply the patch.
|
|
|
|
The `runtime` block is optional. Fixtures without it only run the static unit checks (is-generated, inject, wrap, csp-detect). Fixtures *with* it additionally run the E2E suite in `tests/live-e2e.test.mjs` (`bun run test:live-e2e`), which:
|
|
|
|
1. Stages the fixture into a tmp repo.
|
|
2. Runs `runtime.install` to install real deps.
|
|
3. Starts `live-server.mjs --background` and runs `live-inject.mjs --port` against it.
|
|
4. Spawns `runtime.devCommand` and scrapes the port from stdout using `runtime.readyPattern` (the first capture group must be the port).
|
|
5. Opens Playwright Chromium at the dev URL and asserts `window.__IMPECCABLE_LIVE_INIT__ === true` (the browser-side handshake oracle) within `runtime.readyTimeoutMs`.
|
|
6. Runs a **Steer smoke** step (unless `runtime.steer === false`): submit a message in the global Steer bar, wait for the fake agent to reply `steer_done`, assert the bar unlocks and a `data-impeccable-steer` marker lands in source + DOM. Then continues with pick → Go → cycle → accept.
|
|
7. Tears everything down (Playwright close, dev server SIGTERM, live-server stop, tmp rm).
|
|
|
|
### `runtime.appDir`
|
|
|
|
Optional, defaults to `.`. Set it when the served app is **not** the repo root, the shape live mode has to resolve on its own (a CLI package at the root with the site in `website/`, for example). With `appDir` set, the harness:
|
|
|
|
- stages `files/` and runs `git init` at the tmp root, as always;
|
|
- writes `.impeccable/live/config.json` under `<tmp>/<appDir>/`, and treats every fixture-relative path in `fixture.json` (`steer.sourceFile`, manual-edit `expectedSourceFile`, and so on) as relative to that app dir;
|
|
- runs `runtime.install` and `runtime.devCommand` with the app dir as cwd;
|
|
- boots through `live.mjs` **from the tmp root** instead of calling `live-server.mjs` and `live-inject.mjs` directly, so the run exercises root resolution (`skill/scripts/live/roots.mjs`) rather than assuming it. The parsed `live.mjs` payload is exposed as `session.liveBoot`, and `tests/live-e2e.test.mjs` asserts on `roots.appRoot`, `roots.contextRoot`, the persisted `roots.json`, and the repo-root pointer.
|
|
|
|
The session object carries both paths: `session.tmp` is the repo root (use it for git and for artifact capture) and `session.appRoot` is the app. They are the same directory for every fixture without `appDir`.
|
|
|
|
### Picking, cycling, and the render proof
|
|
|
|
`pickSelector` names the element the run picks. The picker resolves whatever is
|
|
under the cursor, so a **container whose centre is covered by a child can never
|
|
be picked**: add `pickPosition` (`{x, y}` in px from the element's top-left) to
|
|
aim at a point the container owns, such as its own padding.
|
|
|
|
`variantSequence` (default `[2]`) is the order the run cycles through; the last
|
|
entry is the variant it accepts. Every variant the run lands on gets a computed
|
|
`font-weight` assertion in fake-agent mode, because the fake agent renders each
|
|
variant at a distinct weight (`FAKE_VARIANT_FONT_WEIGHTS` in
|
|
`tests/live-e2e/agent.mjs`: 300 / 900 / 600). That turns "variant N is visible"
|
|
from a bar-label claim into a render fact, so a sequence like `[3, 1, 2]` proves
|
|
all three variants really render.
|
|
|
|
`acceptedSourcePattern` overrides the default post-accept source check (an `<h1
|
|
class="hero-title">`), and `assertSourceContains` lists strings that must
|
|
survive the whole wrap → accept → carbonize cycle. On Svelte component previews
|
|
that is how a fixture proves control flow was not flattened: list the `{#each}`
|
|
header and the per-item expressions.
|
|
|
|
`stateProbe` asserts page state is not lost. `textSelector` / `expectedText`
|
|
check rendered state; `windowProperty` with `expectedWindowValue` and
|
|
`expectWindowUnchanged` check a counter the app bumps on mount, so a scaffold
|
|
that silently remounted the page fails. It runs after preActions, after the
|
|
variants land, and (outside component previews) after accept.
|
|
|
|
### Extra scenarios
|
|
|
|
Beyond the core cycle, a fixture opts into scenarios by declaring their config.
|
|
Each one is also gated by `IMPECCABLE_E2E_SCENARIOS`:
|
|
|
|
| Scenario name | Enabled by | What it proves |
|
|
|---|---|---|
|
|
| `params` | `runtime.paramsScenario` | Dial a `range` and a `steps` knob in the real Tune popover, accept, and assert the chosen values are baked into source as literals with the unchosen branch dropped and no `data-p-*` / `var(--p-*)` left behind. |
|
|
| `mount-failure` | `runtime.componentFailureScenarios` | Corrupt the published `r<N>/v<variant>` revision file, step onto it, and assert the persistent mount-error card appears, the session survives (bar + localStorage intact), and a `variant_mount_failed` event lands in the session journal. Then restore, Retry, and reach the variant again. |
|
|
| `republish` | `runtime.componentFailureScenarios` | Re-author every variant and reply `done` again; the browser must mount the new content, which is what the server's revision-dir bump exists to guarantee. |
|
|
| `storage-loss` | `runtime.componentFailureScenarios` (unless `storageLoss: false`) | Clear localStorage, reload, and assert the comparison comes back from the server's durable session record alone. |
|
|
|
|
`componentFailureScenarios.variant` picks which variant to break or observe.
|
|
Set `storageLoss: false` for fixtures whose picked element only exists after
|
|
preActions: the reload discards that state, and re-creating it races the
|
|
preview mount.
|
|
|
|
These scenarios assert on deterministic content, so they skip under
|
|
`IMPECCABLE_E2E_AGENT=llm`.
|
|
|
|
One gotcha: `tests/framework-fixtures.test.mjs` stages the same fixture flat and writes the live config at the tmp root, so `config.files` has to resolve from both the repo root and the app root. A glob (`"**/index.html"`) satisfies both; a literal `"index.html"` only works from the app root and makes the static sweep report `file_not_found`.
|
|
|
|
Useful runtime E2E filters:
|
|
|
|
- `IMPECCABLE_E2E_ONLY=<fixture>[,<fixture>]` scopes the run to selected fixture names.
|
|
- `IMPECCABLE_E2E_SCENARIOS=core` runs only the main click → Go → cycle → accept path; omit it or use `all` to include manual edit, annotation, exit, params, and the component failure-injection probes. Names: `core`, `manual`, `annotations`, `exit`, `missed-done`, `params`, `mount-failure`, `republish`, `storage-loss`.
|
|
- `IMPECCABLE_E2E_TEST_TIMEOUT_MS`, `IMPECCABLE_E2E_INSTALL_TIMEOUT_MS`, and `IMPECCABLE_E2E_DEV_READY_TIMEOUT_MS` tighten CI smoke timeouts without changing fixture metadata.
|
|
|
|
Optional `runtime.steer` fields:
|
|
|
|
```json
|
|
"steer": {
|
|
"message": "steer-e2e mark hero",
|
|
"sourceFile": "src/routes/About.jsx",
|
|
"expectSelector": "h1.hero-title[data-impeccable-steer=\"e2e\"]",
|
|
"expectSourceContains": "data-impeccable-steer=\"e2e\"",
|
|
"preActions": [{ "type": "click", "selector": "[data-testid='nav-about']" }]
|
|
}
|
|
```
|
|
|
|
When `preActions` is omitted, steer smoke inherits `runtime.preActions` to reveal hidden heroes before the DOM check. Source is asserted first; a reload + retry covers HMR lag. Set `"steer": false` to skip, or `"expectDom": false` for source-only verification.
|
|
|
|
## Current fixtures
|
|
|
|
| Fixture | Shape |
|
|
|---|---|
|
|
| `vite-react/` | Tracked `index.html` shell + `src/App.jsx`. Inject into the shell. |
|
|
| `nextjs-app/` | `app/layout.tsx` as JSX inject target (commentSyntax `jsx`). |
|
|
| `astro/` | `src/layouts/Layout.astro` as inject target. HTML comments. |
|
|
| `sveltekit/` | `src/app.html` shell + `src/routes/+page.svelte`. |
|
|
| `vite8-sveltekit-stateful/` | Svelte 5 route with `$state`, an `{#if}` branch, and an `{#each}` list. Picks the list container, so the component-preview scaffold has to carry the loop across as one `collection` prop and hydrate its items from the live DOM. Also carries the params, failure-injection, and state-preservation probes. |
|
|
| `nuxt-vite7/` | Nuxt 4 `app/` structure + Vue 3 SFC. Live loads through a generated dev-only client plugin. |
|
|
| `tanstack-router-vite/` | Vite + TanStack Router (code-based SPA). Tracked `index.html` shell inject (the baseline Vite path, no adapter). |
|
|
| `tanstack-start/` | Vite + TanStack Start (SSR). No static `index.html`; Live patches the `__root.tsx` document to mount a generated dev-only React component that loads the bundle. |
|
|
| `multipage-with-generator/` | `src/` tracked, `dist/` gitignored. Exercises the is-generated guard and `element_not_in_source` fallback. |
|
|
| `nextjs-turborepo/` | Monorepo with shared CSP helper (`createBaseNextConfig`). CSP shape `append-arrays`. |
|
|
| `nextjs-inline-csp/` | App-level `next.config.js` with a literal CSP string. CSP shape `append-string`. |
|
|
| `sveltekit-csp/` | SvelteKit `kit.csp.directives` in `svelte.config.js`. CSP shape `append-arrays`. |
|
|
| `nuxt-csp/` | Nuxt `routeRules` with literal CSP header in `nuxt.config.ts`. CSP shape `append-string`. |
|
|
| `monorepo-nested-vite/` | Repo root is a CLI package with no dev config and no workspaces; the served Vite + React app lives in `website/`. Exercises `runtime.appDir` and live-mode root resolution. |
|
|
|
|
Add new fixtures by cloning a directory, swapping files, and updating `fixture.json`. A fixture with a `runtime` block also needs its name added to the live-e2e matrices in `.github/workflows/ci.yml`: the `live-e2e-full` group list, and one `live-e2e-smoke` group when it should run on every PR. Keep the groups roughly the same size, since they run in parallel and the job times out at 15 minutes.
|