feat(test): pluggable LLM agent for live-mode E2E suite

tests/live-e2e/agents/llm-agent.mjs: a Claude-backed VariantAgent that
implements the same one-method interface as the fake agent
(generateVariants(event, context) → { scopedCss, variants[] }). Default
model claude-haiku-4-5; override via IMPECCABLE_E2E_LLM_MODEL.

Prompt caching is on — the system prompt (instructions + the live-mode
spec from reference/live.md) is the cacheable prefix. First call writes
~10K tokens to cache; subsequent fixtures pay only the cache-read rate.
JSON output is validated for shape (scopedCss, variants[N].innerHtml),
with light error messages on parse failure.

tests/live-e2e.test.mjs: read IMPECCABLE_E2E_AGENT (fake|llm). When 'llm',
construct the LLM agent and skip the case cleanly if ANTHROPIC_API_KEY is
unset. Param-manifest assertions are gated to fake mode (LLM may emit
zero-param "fixed point" variants per the live.md spec). The accepted-h1
class assertion now allows hero-title as one of multiple classes so an
LLM agent that adds classes alongside the original still passes.

Test timeouts widen for LLM mode: 25s first-pass on conditional-render
fixtures (vs 5s for fake), 60s on direct waits (vs 30s). Without these,
the LLM's 3-8s generate latency races the orchestration's state-loss
recovery window.

tests/live-e2e/ui.mjs: clickGo retries up to 3× on stability failures.
Required because conditional-render fixtures (modal/tabs) animate the bar
mid-transition when preActions trigger framework HMR; a single click can
land during a re-render and Playwright's stability gate times out.

Pass rate on a typical sweep: 18/19 in LLM mode, 19/19 in fake mode.
The modal fixture's intrinsic state-loss flake (Fast Refresh resetting
useState(open) when source changes) is amplified by LLM latency and may
need a re-run; documented in CLAUDE.md.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

AGENTS.md

@@ -30,6 +30,8 @@ Tests use Bun’s test runner plus Node’s built-in `--test`. Name tests `*.tes
 
 For changes to `source/skills/impeccable/scripts/live-*.{mjs,js}`, also run `bun run test:live-e2e` (kept out of the default suite because it does real `npm install` per fixture and boots framework dev servers). Scope to one fixture with `IMPECCABLE_E2E_ONLY=<fixture-name>` while iterating; pass `IMPECCABLE_E2E_DEBUG=1` for page-DOM and dev-server-log dumps on failure. Schema and authoring guide for new fixtures live in `tests/framework-fixtures/README.md`.
 
+Set `IMPECCABLE_E2E_AGENT=llm` to swap the deterministic fake agent for a Claude-backed one (`tests/live-e2e/agents/llm-agent.mjs`, default Haiku 4.5, override via `IMPECCABLE_E2E_LLM_MODEL`). Requires `ANTHROPIC_API_KEY`; tests skip cleanly when it's unset. This path hits the API — use it for verification, not CI.
+
 ## Commit & Pull Request Guidelines
 
 Recent history favors short, imperative subjects such as `Fix: ...`, `Add ...`, `Improve ...`, or `Bump ...`. Keep commits focused and explain the user-facing impact when it is not obvious. PRs should summarize what changed, list validation performed, and call out regenerated artifacts like `dist/` or `build/`. Include screenshots for visible `public/` changes and mention affected providers when transform behavior changes.

CLAUDE.md

@@ -126,7 +126,9 @@ IMPECCABLE_E2E_DEBUG=1 bun run test:live-e2e                # dump page DOM + de
 
 **Kept out of the default `bun run test`** because (a) it does real `npm install` per fixture, (b) it boots framework dev servers, (c) wall time is ~2 minutes, and (d) it requires Playwright's browser cache. Run it locally before shipping changes to anything in `source/skills/impeccable/scripts/live-*.{mjs,js}`.
 
-The agent is pluggable via a one-method interface in `tests/live-e2e/agent.mjs`: `generateVariants(event, context) → { scopedCss, variants[] }`. The default fake agent emits canned variants that exercise all three param kinds (`range`, `steps`, `toggle`). A future LLM-backed agent slots in by implementing the same shape; the orchestrator (wrap, write, accept, carbonize) is agent-agnostic.
+The agent is pluggable via a one-method interface in `tests/live-e2e/agent.mjs`: `generateVariants(event, context) → { scopedCss, variants[] }`. The default fake agent emits canned variants that exercise all three param kinds (`range`, `steps`, `toggle`). The orchestrator (wrap, write, accept, carbonize) is agent-agnostic.
+
+**LLM agent (opt-in)**: set `IMPECCABLE_E2E_AGENT=llm` to swap the fake agent for `tests/live-e2e/agents/llm-agent.mjs`, which calls Claude (default Haiku 4.5) via `@anthropic-ai/sdk`. Requires `ANTHROPIC_API_KEY` in env; the test runner skips with a clear message when it's unset. Override the model with `IMPECCABLE_E2E_LLM_MODEL=claude-sonnet-4-6` if Haiku produces unreliable JSON. Caching is on — live.md is the cacheable prefix, and after the first call subsequent fixtures pay only the cache-read rate. Pass rate on a typical sweep is 18/19; the modal fixture's intrinsic state-loss flake is amplified by LLM latency and may need a re-run. **This path hits the API and costs money** — keep it out of CI unless you really want it there.
 
 Adding a new fixture is a matter of cloning a directory under `tests/framework-fixtures/`, swapping the source files, and writing a `fixture.json`. See `tests/framework-fixtures/README.md` for the full schema.
 

package.json

@@ -67,6 +67,7 @@
     "@ai-sdk/anthropic": "^3.0.69",
     "@ai-sdk/openai": "^3.0.53",
     "@anthropic-ai/claude-agent-sdk": "^0.2.110",
+    "@anthropic-ai/sdk": "^0.91.1",
     "@google/genai": "^1.50.1",
     "ai": "^6.0.162",
     "archiver": "^7.0.1",

tests/live-e2e.test.mjs

@@ -25,6 +25,7 @@ import { dirname, join } from 'node:path';
 import { fileURLToPath } from 'node:url';
 
 import { createFakeAgent } from './live-e2e/agent.mjs';
+import { createLlmAgent } from './live-e2e/agents/llm-agent.mjs';
 import { bootFixtureSession, FIXTURES_DIR } from './live-e2e/session.mjs';
 import {
   clickAccept,
@@ -102,12 +103,32 @@ for (const { name, fixture } of fixtures) {
       // the limitation is visible in the run output.
       const knownLimitation = fixture.runtime.knownLimitation;
 
+      // Pick the agent. `IMPECCABLE_E2E_AGENT=llm` opts into the real Claude
+      // API; everything else uses the deterministic fake. Skip rather than
+      // fail when LLM is requested but no API key is set so default suite
+      // runs in unauthenticated environments still pass.
+      const agentMode = process.env.IMPECCABLE_E2E_AGENT || 'fake';
+      let agent;
+      if (agentMode === 'llm') {
+        agent = await createLlmAgent({
+          model: process.env.IMPECCABLE_E2E_LLM_MODEL,
+          log: (m) => t.diagnostic('[llm] ' + m),
+        });
+        if (!agent) {
+          t.skip('IMPECCABLE_E2E_AGENT=llm requires ANTHROPIC_API_KEY');
+          return;
+        }
+        t.diagnostic(`Using LLM agent (model=${process.env.IMPECCABLE_E2E_LLM_MODEL || 'claude-haiku-4-5'})`);
+      } else {
+        agent = createFakeAgent();
+      }
+
       t.diagnostic(`Booting fixture ${name}`);
       const session = await bootFixtureSession({
         name,
         fixture,
         browser,
-        agent: createFakeAgent(),
+        agent,
         log: (m) => t.diagnostic(m),
       });
 
@@ -147,22 +168,33 @@ for (const { name, fixture } of fixtures) {
         //    For fixtures whose picked element lives inside a conditional
         //    render (modal, tab, route), HMR can remount the parent and lose
         //    the open/active state — the wrapper exists in source but isn't
-        //    in the DOM, so MutationObserver never fires. Live mode now
+        //    in the DOM, so MutationObserver never sees it. Live mode now
         //    surfaces a toast asking the user to retrace the path; we mirror
         //    that here by re-running preActions on the first short timeout.
+        //
+        //    The first-pass timeout has to be long enough to cover the agent's
+        //    generate latency before declaring "state was lost, retrace." A
+        //    fake agent finishes in <100ms; an LLM agent typically lands in
+        //    3-8s. Scale the gate accordingly.
         t.diagnostic(`Waiting for CYCLING state with ${expectedCount} variants`);
+        const firstPassTimeoutMs = agentMode === 'llm' ? 25_000 : 5_000;
         let cyclingReached = false;
         if (fixture.runtime.preActions) {
           try {
-            await waitForCycling(page, expectedCount, { timeout: 5_000 });
+            await waitForCycling(page, expectedCount, { timeout: firstPassTimeoutMs });
             cyclingReached = true;
           } catch {
-            t.diagnostic('Cycling not reached in 5s — retracing preActions');
+            t.diagnostic(`Cycling not reached in ${firstPassTimeoutMs}ms — retracing preActions`);
             await runPreActions(page, fixture.runtime.preActions);
           }
         }
         try {
-          if (!cyclingReached) await waitForCycling(page, expectedCount);
+          if (!cyclingReached) {
+            // Default 30s; LLM mode bumps to 60s to absorb API latency on
+            // top of HMR settle time.
+            const finalTimeoutMs = agentMode === 'llm' ? 60_000 : 30_000;
+            await waitForCycling(page, expectedCount, { timeout: finalTimeoutMs });
+          }
         } catch (err) {
           if (process.env.IMPECCABLE_E2E_DEBUG) {
             const variantCount = await page.evaluate(() =>
@@ -196,10 +228,15 @@ for (const { name, fixture } of fixtures) {
         assert.match(after, /@scope \(\[data-impeccable-variant="1"\]\)/, 'scoped CSS for variant 1');
         assert.match(after, /@scope \(\[data-impeccable-variant="2"\]\)/, 'scoped CSS for variant 2');
         assert.match(after, /@scope \(\[data-impeccable-variant="3"\]\)/, 'scoped CSS for variant 3');
-        assert.match(after, /data-impeccable-params=/, 'data-impeccable-params manifest emitted');
+        // Param manifest assertions are scoped to fake-agent mode. The fake
-        // Sanity-check the param manifest covers all three kinds across the set.
+        // agent deterministically emits one param per variant covering all
-        for (const kind of ['range', 'steps', 'toggle']) {
+        // three kinds; the LLM agent is non-deterministic and may legitimately
-          assert.match(after, new RegExp(`"kind"\\s*:\\s*"${kind}"`), `param kind ${kind} present`);
+        // emit no params per the live.md spec ("variants are fixed points").
+        if (agentMode === 'fake') {
+          assert.match(after, /data-impeccable-params=/, 'data-impeccable-params manifest emitted');
+          for (const kind of ['range', 'steps', 'toggle']) {
+            assert.match(after, new RegExp(`"kind"\\s*:\\s*"${kind}"`), `param kind ${kind} present`);
+          }
         }
 
         // 6. Cycle to variant 2 (the bold one in the fake agent)
@@ -222,7 +259,15 @@ for (const { name, fixture } of fixtures) {
         assert.doesNotMatch(final, /impeccable-carbonize-start/,     'carbonize-start marker removed');
         assert.doesNotMatch(final, /impeccable-carbonize-end/,       'carbonize-end marker removed');
         assert.doesNotMatch(final, /data-impeccable-variant="/,      'no leftover variant scaffolding');
-        assert.match(final, /<h1[^>]*(class|className)="hero-title"/, 'accepted h1 survives');
+        // Accept the original class as a substring of the className value so
+        // an LLM agent that adds classes around the original (e.g.
+        // class="hero-title bold red") still passes — only the literal
+        // class="hero-title" form would otherwise match.
+        assert.match(
+          final,
+          /<h1[^>]*(class|className)="[^"]*\bhero-title\b[^"]*"/,
+          'accepted h1 survives with hero-title class',
+        );
 
         // 9. DOM-side: at least one matching element, none inside any wrapper.
         await page.waitForFunction(

tests/live-e2e/agents/llm-agent.mjs

@@ -0,0 +1,186 @@
+/**
+ * LLM-backed VariantAgent for the live-mode E2E suite.
+ *
+ * Implements the same one-method interface as createFakeAgent() in
+ * tests/live-e2e/agent.mjs: generateVariants(event, context) returns
+ * { scopedCss, variants[] }. The orchestrator handles wrap, write, accept,
+ * and carbonize cleanup deterministically, so this module's only job is
+ * producing variant content for the wrapper.
+ *
+ * Default model: Claude Haiku 4.5 — fast, cheap, smart enough for variant
+ * generation in test fixtures. Override via { model } when constructing,
+ * or via the IMPECCABLE_E2E_LLM_MODEL env var at the call site (test runner).
+ *
+ * Prompt caching: live.md (the live-mode skill spec) is the bulk of the
+ * system prompt and is stable across calls. We mark a cache_control breakpoint
+ * on the last system block so both the JSON-contract instructions and the
+ * spec are cached as one prefix. Subsequent calls in the same run pay only
+ * the cache-read rate (~0.1× input).
+ *
+ * Returns null from createLlmAgent() when ANTHROPIC_API_KEY is unset; the
+ * test runner reads that and skips the case rather than failing.
+ */
+
+import fs from 'node:fs/promises';
+import path from 'node:path';
+import { fileURLToPath } from 'node:url';
+import Anthropic from '@anthropic-ai/sdk';
+
+const __dirname = path.dirname(fileURLToPath(import.meta.url));
+const REPO_ROOT = path.join(__dirname, '..', '..', '..');
+const LIVE_MD_PATH = path.join(
+  REPO_ROOT,
+  'source',
+  'skills',
+  'impeccable',
+  'reference',
+  'live.md',
+);
+
+const DEFAULT_MODEL = 'claude-haiku-4-5';
+
+const SYSTEM_INSTRUCTIONS = [
+  'You are an automated subagent inside Impeccable\'s live-mode test harness.',
+  'Given an element the user picked, an action, and a count, you produce variant DOM content in a strict JSON shape.',
+  '',
+  'OUTPUT CONTRACT — return ONLY a JSON object with this exact shape. No prose, no code fences, no commentary:',
+  '',
+  '{',
+  '  "scopedCss": "string — contents of a <style data-impeccable-css> block, with @scope ([data-impeccable-variant=\\"N\\"]) rules per variant",',
+  '  "variants": [',
+  '    {',
+  '      "innerHtml": "string — single top-level HTML element matching the picked element\'s tag, e.g. <h1 class=\\"hero-title\\">Title</h1>",',
+  '      "params": [/* optional 0-4 ParamSpec entries */]',
+  '    }',
+  '  ]',
+  '}',
+  '',
+  'ParamSpec is one of:',
+  '  { "id": "string", "kind": "range",  "min": number, "max": number, "step": number, "default": number, "label": "string" }',
+  '  { "id": "string", "kind": "steps",  "default": "string", "label": "string", "options": [{ "value": "string", "label": "string" }, ...] }',
+  '  { "id": "string", "kind": "toggle", "default": boolean, "label": "string" }',
+  '',
+  'REQUIREMENTS',
+  '- Each variant.innerHtml must be a single top-level HTML element. Use the EXACT same tag as the picked element.',
+  '- PRESERVE the original element\'s className verbatim. If the picked element\'s outerHTML contains class="hero-title", every variant\'s innerHtml MUST contain the same class="hero-title" string (you may add additional class names alongside, never remove or rename the original). This is a hard requirement — automated harnesses verify the original class survives across the variant set.',
+  '- Generate exactly event.count variants — no more, no fewer.',
+  '- Mix the param kinds across the variant set: include at least one range, one steps, and one toggle when count >= 3.',
+  '- The scopedCss should declare @scope ([data-impeccable-variant="N"]) rules wired against the params you emit (CSS vars for range/toggle, attribute selectors for steps/toggle).',
+  '- Use HTML attribute syntax in innerHtml (class=, not className=). The orchestrator translates per file syntax.',
+  '- Do NOT emit the wrapping <div data-impeccable-variant="N">. The orchestrator wraps your content.',
+  '- Do NOT emit the outer <style data-impeccable-css> tag. Only its contents go in scopedCss.',
+  '- Do NOT include any <!-- comments --> in scopedCss; CSS comments use /* */.',
+  '',
+  'CONTEXT — full live-mode skill spec follows. Use it as the source of truth for any nuance in the variant format.',
+].join('\n');
+
+/**
+ * @typedef {object} LlmAgentOptions
+ * @property {string=} apiKey  Override ANTHROPIC_API_KEY env var.
+ * @property {string=} model   Default 'claude-haiku-4-5'. Override to 'claude-sonnet-4-6' if Haiku produces unreliable JSON.
+ * @property {(msg: string) => void=} log  Optional logger for debug output.
+ */
+
+/**
+ * @param {LlmAgentOptions} [opts]
+ * @returns {Promise<{generateVariants: (event: object, context: object) => Promise<{scopedCss: string, variants: object[]}>} | null>}
+ */
+export async function createLlmAgent(opts = {}) {
+  const apiKey = opts.apiKey || process.env.ANTHROPIC_API_KEY;
+  if (!apiKey) return null;
+
+  const model = opts.model || DEFAULT_MODEL;
+  const log = opts.log || (() => {});
+
+  const liveMd = await fs.readFile(LIVE_MD_PATH, 'utf-8');
+  const client = new Anthropic({ apiKey });
+
+  return {
+    async generateVariants(event /*, context */) {
+      const userMessage = [
+        'Produce variants for the following pick. Reply with the JSON object only — no prose.',
+        '',
+        '```json',
+        JSON.stringify(
+          {
+            id: event.id,
+            action: event.action,
+            count: event.count,
+            element: {
+              outerHTML: event.element?.outerHTML,
+              tagName: event.element?.tagName,
+              className: event.element?.className,
+              textContent: event.element?.textContent?.slice(0, 200),
+            },
+          },
+          null,
+          2,
+        ),
+        '```',
+      ].join('\n');
+
+      const response = await client.messages.create({
+        model,
+        max_tokens: 16000,
+        system: [
+          { type: 'text', text: SYSTEM_INSTRUCTIONS },
+          // Cacheable: the entire stable prefix (instructions + spec) is
+          // cached up to this breakpoint. The user message holds all the
+          // per-call volatile content.
+          { type: 'text', text: liveMd, cache_control: { type: 'ephemeral' } },
+        ],
+        messages: [{ role: 'user', content: userMessage }],
+      });
+
+      const cacheRead = response.usage?.cache_read_input_tokens ?? 0;
+      const cacheWrite = response.usage?.cache_creation_input_tokens ?? 0;
+      const inputTokens = response.usage?.input_tokens ?? 0;
+      const outputTokens = response.usage?.output_tokens ?? 0;
+      log(
+        `model=${model} input=${inputTokens} output=${outputTokens} cache_read=${cacheRead} cache_write=${cacheWrite}`,
+      );
+
+      const text = response.content
+        .filter((b) => b.type === 'text')
+        .map((b) => b.text)
+        .join('');
+
+      const cleaned = stripCodeFence(text.trim());
+      let parsed;
+      try {
+        parsed = JSON.parse(cleaned);
+      } catch (err) {
+        throw new Error(
+          `LLM agent: response was not valid JSON (${err.message}). First 500 chars:\n${cleaned.slice(0, 500)}`,
+        );
+      }
+
+      if (typeof parsed.scopedCss !== 'string') {
+        throw new Error(`LLM agent: missing or non-string scopedCss in response`);
+      }
+      if (!Array.isArray(parsed.variants) || parsed.variants.length === 0) {
+        throw new Error(`LLM agent: variants must be a non-empty array`);
+      }
+      for (const [i, v] of parsed.variants.entries()) {
+        if (typeof v.innerHtml !== 'string' || !v.innerHtml.trim()) {
+          throw new Error(`LLM agent: variants[${i}].innerHtml missing or empty`);
+        }
+        if (v.params !== undefined && !Array.isArray(v.params)) {
+          throw new Error(`LLM agent: variants[${i}].params must be an array if present`);
+        }
+      }
+
+      return parsed;
+    },
+  };
+}
+
+/**
+ * Some models wrap JSON in ```json … ``` fences despite the instruction not to.
+ * Strip a single optional fence, leave anything else alone.
+ */
+function stripCodeFence(s) {
+  return s
+    .replace(/^```(?:json)?\s*\n/, '')
+    .replace(/\n```\s*$/, '');
+}

tests/live-e2e/ui.mjs

@@ -91,9 +91,26 @@ export async function setCount(page, count) {
 
 /**
  * Click Go. Browser POSTs the generate event; the agent picks it up.
+ *
+ * On fixtures whose preActions triggered a layout shift (modal/tab opening)
+ * the bar's open animation can still be running when we click, and
+ * Playwright's stability gate occasionally times out on the first attempt.
+ * Retry up to three times with a settle in between so a single race doesn't
+ * fail the test.
  */
 export async function clickGo(page) {
-  await page.locator(`${BAR_ID} button`, { hasText: /Go\b/ }).click();
+  const go = page.locator(`${BAR_ID} button`, { hasText: /Go\b/ });
+  let lastErr;
+  for (let attempt = 0; attempt < 3; attempt++) {
+    try {
+      await go.click({ timeout: 5_000 });
+      return;
+    } catch (err) {
+      lastErr = err;
+      await page.waitForTimeout(500);
+    }
+  }
+  throw lastErr;
 }
 
 /**