pbakaus_impeccable/tests/framework-fixtures/README.md

# Framework fixtures

Representative project shapes for exercising live mode against different framework conventions. Each fixture is a small directory tree that the test harness copies into a temp git repo, then drives `live-inject.mjs`, `live-wrap.mjs`, `live-accept.mjs`, and `lib/is-generated.mjs` against.

Fixtures can also opt into a **runtime E2E** pass that actually installs dependencies, boots the framework dev server, and drives a Playwright browser to verify the live handshake. See the `runtime` block below.

## Layout

```
<fixture>/
  files/              project tree the test copies into tmp
  gitignore.txt       becomes .gitignore in tmp (so we can commit the real files here)
  fixture.json        config + expected results the test consumes
```

`fixture.json` schema:

```json
{
  "name": "human-readable label",
  "config": { ...contents for .impeccable/live/config.json ... },
  "sourceFiles": ["paths that is-generated should classify as source (false)"],
  "generatedFiles": ["paths that is-generated should classify as generated (true)"],
  "wrapCases": [
    {
      "name": "description",
      "args": { "classes": "...", "tag": "...", "elementId": "..." },
      "expectedFile": "where wrap should land (relative to fixture root)",
      "expectsError": "optional error code, e.g. element_not_in_source"
    }
  ],
  "csp": {
    "shape": "shared-helper | inline-headers | middleware | meta-tag | null",
    "signals": ["diagnostic hints — paths where CSP was detected"],
    "patchTarget": "which file the agent should modify",
    "expectedAfter": "filename of the reference post-patch output inside this fixture"
  },
  "runtime": {
    "styling": "plain-css | tailwind-v4 | styled-components | ...",
    "install": ["npm", "install"],
    "devCommand": ["npm", "run", "dev"],
    "scheme": "http",
    "ignoreHTTPSErrors": false,
    "readyPattern": "Local:\\s+https?://[^:]+:(\\d+)",
    "readyTimeoutMs": 120000,
    "pickSelector": "h1.hero-title",
    "mode": "insert",
    "insert": {
      "anchorSelector": "section#features",
      "position": "after",
      "prompt": "Add a testimonial strip below features",
      "expectSelector": ".inserted-strip",
      "assertAnchorContains": "feature-grid"
    },
    "preActions": [
      { "type": "click", "selector": "[data-testid='open-modal']" },
      { "type": "goto",  "path": "/about" }
    ],
    "reloadProbe": {
      "preActions": [{ "type": "click", "selector": "[data-testid='open-modal']" }],
      "expectSelector": "h1.hero-title"
    },
    "steer": {
      "message": "steer-e2e mark hero",
      "expectSelector": "h1.hero-title[data-impeccable-steer=\"e2e\"]"
    },
    "probe": {
      "expectLiveInit": true,
      "expectConsoleClean": true
    }
  }
}
```

The `expectedAfter` file lives alongside `fixture.json` (not inside `files/`) and is a human/agent-review reference — tests don't auto-apply the patch.

The `runtime` block is optional. Fixtures without it only run the static unit checks (is-generated, inject, wrap, csp-detect). Fixtures *with* it additionally run the E2E suite in `tests/live-e2e.test.mjs` (`bun run test:live-e2e`), which:

1. Stages the fixture into a tmp repo.
2. Runs `runtime.install` to install real deps.
3. Starts `live-server.mjs --background` and runs `live-inject.mjs --port` against it.
4. Spawns `runtime.devCommand` and scrapes the port from stdout using `runtime.readyPattern` (the first capture group must be the port).
5. Opens Playwright Chromium at the dev URL and asserts `window.__IMPECCABLE_LIVE_INIT__ === true` (the browser-side handshake oracle) within `runtime.readyTimeoutMs`.
6. Runs a **Steer smoke** step (unless `runtime.steer === false`): submit a message in the global Steer bar, wait for the fake agent to reply `steer_done`, assert the bar unlocks and a `data-impeccable-steer` marker lands in source + DOM. Then continues with pick → Go → cycle → accept.
7. Tears everything down (Playwright close, dev server SIGTERM, live-server stop, tmp rm).

Useful runtime E2E filters:

- `IMPECCABLE_E2E_ONLY=<fixture>[,<fixture>]` scopes the run to selected fixture names.
- `IMPECCABLE_E2E_SCENARIOS=core` runs only the main click → Go → cycle → accept path; omit it or use `all` to include manual edit, annotation, and exit probes.
- `IMPECCABLE_E2E_TEST_TIMEOUT_MS`, `IMPECCABLE_E2E_INSTALL_TIMEOUT_MS`, and `IMPECCABLE_E2E_DEV_READY_TIMEOUT_MS` tighten CI smoke timeouts without changing fixture metadata.

Optional `runtime.steer` fields:

```json
"steer": {
  "message": "steer-e2e mark hero",
  "sourceFile": "src/routes/About.jsx",
  "expectSelector": "h1.hero-title[data-impeccable-steer=\"e2e\"]",
  "expectSourceContains": "data-impeccable-steer=\"e2e\"",
  "preActions": [{ "type": "click", "selector": "[data-testid='nav-about']" }]
}
```

When `preActions` is omitted, steer smoke inherits `runtime.preActions` to reveal hidden heroes before the DOM check. Source is asserted first; a reload + retry covers HMR lag. Set `"steer": false` to skip, or `"expectDom": false` for source-only verification.

## Current fixtures

| Fixture | Shape |
|---|---|
| `vite-react/` | Tracked `index.html` shell + `src/App.jsx`. Inject into the shell. |
| `nextjs-app/` | `app/layout.tsx` as JSX inject target (commentSyntax `jsx`). |
| `astro/` | `src/layouts/Layout.astro` as inject target. HTML comments. |
| `sveltekit/` | `src/app.html` shell + `src/routes/+page.svelte`. |
| `nuxt-vite7/` | Nuxt 4 `app/` structure + Vue 3 SFC. Live loads through a generated dev-only client plugin. |
| `multipage-with-generator/` | `src/` tracked, `dist/` gitignored. Exercises the is-generated guard and `element_not_in_source` fallback. |
| `nextjs-turborepo/` | Monorepo with shared CSP helper (`createBaseNextConfig`). CSP shape `append-arrays`. |
| `nextjs-inline-csp/` | App-level `next.config.js` with a literal CSP string. CSP shape `append-string`. |
| `sveltekit-csp/` | SvelteKit `kit.csp.directives` in `svelte.config.js`. CSP shape `append-arrays`. |
| `nuxt-csp/` | Nuxt `routeRules` with literal CSP header in `nuxt.config.ts`. CSP shape `append-string`. |

Add new fixtures by cloning a directory, swapping files, and updating `fixture.json`.

## External quality-eval fixtures

The public Live benchmark can execute a fixture owned by another repository
without copying its task corpus or rubric into Impeccable:

```sh
bun run bench:live -- \
  --fixture-dir=/absolute/path/to/private-fixture \
  --agent=codex \
  --action=bolder \
  --iterations=1 \
  --evidence-bundle=/absolute/path/to/output-bundle
```

An external fixture has the same shape as a directory in this folder:
`fixture.json`, `gitignore.txt`, and `files/`. Use the optional
`evidenceCapture` block in `fixture.json` for rubric-free capture metadata:

```json
{
  "evidenceCapture": {
    "captureSelector": "section.case-study",
    "viewport": { "width": 1440, "height": 1080 },
    "action": "bolder"
  }
}
```

The bundle contains `report.json`, the original capture, each progressively
delivered variant capture, geometry/overflow facts, hashes, and timing data.
It deliberately cannot run `--judge-rendered`; comparative rubrics, private
fixtures, human calibration, and quality decisions belong in the consuming
evaluation harness. The normal public E2E suite remains responsible for Live
protocol, framework, source-commit, cleanup, and recovery correctness.