From 26f54d15c26e17aa7b1330231da3d4752f1ce396 Mon Sep 17 00:00:00 2001 From: Paul Bakaus Date: Mon, 27 Jul 2026 19:20:02 -0700 Subject: [PATCH] feat: just-in-time event instructions + frontier default for the LLM e2e agent Field feedback from two more Codex sessions drove both changes. JIT instructions (live/instructions.mjs): every event live-poll prints now carries _instructions, the authoritative next step for that exact situation with real ids, paths, and line numbers substituted, and only the active path's rules (a svelte-component session never sees JSX guidance). The boot payload carries loop instructions the same way. Instructions are versioned with the scripts, so they cannot drift from behavior, and live.md's plumbing can keep shrinking toward contract plus craft guidance. The Codex poll-discipline failure observed in the field ("the long poll was started, but I yielded the task instead of actively servicing its result") gets a named anti-pattern in both the harness policy and the boot instructions. LLM e2e agent: default provider/model moves from Claude Haiku 4.5 to OpenAI gpt-5.6-terra at medium reasoning effort via an Anthropic-shaped shim over the ai SDK (the three call sites stay provider-agnostic; Anthropic and DeepSeek remain selectable). The harness should exercise the model tier that actually drives live sessions. Both the react and sveltekit fixtures pass end to end with terra driving the trimmed live.md and the new _instructions. This work was produced with AI assistance (Claude Code). Co-Authored-By: Claude Code --- CLAUDE.md | 2 +- skill/reference/live.md | 4 +- skill/scripts/live-poll.mjs | 8 ++ skill/scripts/live.mjs | 2 + skill/scripts/live/instructions.mjs | 142 ++++++++++++++++++++++++++++ tests/live-e2e-llm-agent.test.mjs | 15 ++- tests/live-e2e/agents/llm-agent.mjs | 65 ++++++++++++- tests/live-poll.test.mjs | 63 ++++++++++++ tests/live-target-context.test.mjs | 2 +- 9 files changed, 294 insertions(+), 9 deletions(-) create mode 100644 skill/scripts/live/instructions.mjs diff --git a/CLAUDE.md b/CLAUDE.md index db9bb2345..51006e26d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -167,7 +167,7 @@ Three live-mode invariants worth knowing before editing (established by the 2026 The agent is pluggable via a one-method interface in `tests/live-e2e/agent.mjs`: `generateVariants(event, context) → { scopedCss, variants[] }`. The default fake agent emits canned variants that exercise all three param kinds (`range`, `steps`, `toggle`). The orchestrator (wrap, write, accept, carbonize) is agent-agnostic. -**LLM agent (opt-in)**: set `IMPECCABLE_E2E_AGENT=llm` to swap the fake agent for `tests/live-e2e/agents/llm-agent.mjs`, which calls Claude (default Haiku 4.5) via `@anthropic-ai/sdk`. Requires `ANTHROPIC_API_KEY` in env; the test runner skips with a clear message when it's unset. Override the model with `IMPECCABLE_E2E_LLM_MODEL=claude-sonnet-4-6` if Haiku produces unreliable JSON. Caching is on — live.md is the cacheable prefix, and after the first call subsequent fixtures pay only the cache-read rate. Pass rate on a typical sweep is 18/19; the modal fixture's intrinsic state-loss flake is amplified by LLM latency and may need a re-run. **This path hits the API and costs money** — keep it out of CI unless you really want it there. +**LLM agent (opt-in)**: set `IMPECCABLE_E2E_AGENT=llm` to swap the fake agent for `tests/live-e2e/agents/llm-agent.mjs`. Default provider/model: OpenAI `gpt-5.6-terra` at medium reasoning effort (a frontier tier, matching what drives real live sessions); Anthropic and DeepSeek remain selectable via `IMPECCABLE_E2E_LLM_PROVIDER`. Requires the selected provider's key in env (`OPENAI_API_KEY` by default); the test runner skips with a clear message when it's unset. Override the model with `IMPECCABLE_E2E_LLM_MODEL` and the effort with `IMPECCABLE_E2E_LLM_EFFORT`. Caching is on — live.md is the cacheable prefix, and after the first call subsequent fixtures pay only the cache-read rate. Pass rate on a typical sweep is 18/19; the modal fixture's intrinsic state-loss flake is amplified by LLM latency and may need a re-run. **This path hits the API and costs money** — keep it out of CI unless you really want it there. Adding a new fixture is a matter of cloning a directory under `tests/framework-fixtures/`, swapping the source files, and writing a `fixture.json`. See `tests/framework-fixtures/README.md` for the full schema. diff --git a/skill/reference/live.md b/skill/reference/live.md index dbab6a469..22f7999d9 100644 --- a/skill/reference/live.md +++ b/skill/reference/live.md @@ -10,7 +10,7 @@ Codex: run live helper commands, the app dev server, and any dependency-installi ## The contract (read once) -Execute in order. No step skipped, no step reordered. +Execute in order. No step skipped, no step reordered. Every tool output in live mode may carry an `_instructions` field: it is the authoritative next step for that exact situation, with real ids and paths substituted; when it conflicts with your recollection of this document, `_instructions` wins. 1. `live.mjs`: boot. If the request names or implies a file, route, or app inside a monorepo, infer the concrete path and run `node {{scripts_path}}/live.mjs --target ` instead; then run the rest of this live session from the returned `projectRoot`. The boot resolves the app root from dev-server config files and persists it in `.impeccable/live/roots.json`; every helper re-anchors to that manifest at startup (a wrong cwd cannot fork session state), PRODUCT.md / DESIGN.md are discovered upward to the git root, and relative helper args like `--file` resolve against the app root. 2. Open the app URL that serves `pageFile` (infer from `package.json`, docs, terminal output, or an open tab). Never use `serverPort`; it's the helper, not the app. **Cursor:** `browser_navigate` to that URL before polling; do not skip. **Other harnesses:** use the available browser tool; if the URL is uncertain, ask the user once. @@ -24,7 +24,7 @@ Execute in order. No step skipped, no step reordered. Harness policy: - **Claude Code**: run the poll as a **background task** (no short timeout); the harness notifies you on completion. Do not block the shell. - **Cursor**: **one-shot** poll in a **background terminal** with notify on `"type":"(steer|generate|accept|discard|exit)"`; handle, `--reply`, restart the poll. Do **not** use `--stream` on Cursor (measured ~5s pickup vs sub-second one-shot). -- **Codex**: default one-shot poll in a **yielded foreground exec session**. No `&`, no `--stream`, never leave Live without an active foreground poll. +- **Codex**: default one-shot poll in a **yielded foreground exec session**. No `&`, no `--stream`, never leave Live without an active foreground poll. Starting the poll is not enough: SERVICE it (keep reading the exec session until it returns an event). Never announce "waiting for the user" and idle; a yielded poll nobody reads is a dead session, and the user's Go sits unanswered. - **Other harnesses**: one-shot foreground unless you know stdout reliably returns when a shell exits. Delivery policy: atomic single-edit delivery everywhere; do not switch a harness to progressive publishing unless its poll loop is known not to block on the extra calls. diff --git a/skill/scripts/live-poll.mjs b/skill/scripts/live-poll.mjs index eb6f0be61..32e872a66 100644 --- a/skill/scripts/live-poll.mjs +++ b/skill/scripts/live-poll.mjs @@ -15,6 +15,7 @@ import { fileURLToPath } from 'node:url'; import { completionAckForAcceptResult, completionTypeForAcceptResult } from './live/completion.mjs'; import { readLiveServerInfo } from './lib/impeccable-paths.mjs'; import { enterLiveRoot } from './live/roots.mjs'; +import { instructionsForEvent } from './live/instructions.mjs'; // Absolute path to a sibling script in this skill's scripts dir, so runtime // error hints print a directly-runnable command instead of a placeholder. @@ -262,6 +263,13 @@ export function writeCarbonizeBanner(event) { } export function printPollEvent(event) { + // Situational plumbing rides with the event itself: `_instructions` is the + // authoritative next step, with real ids and paths substituted, so the + // reference doc can stay lean and can never drift from script behavior. + if (event && typeof event === 'object' && !event._instructions) { + const instructions = instructionsForEvent(event, { scriptsPath: SELF_DIR }); + if (instructions) event._instructions = instructions; + } console.log(JSON.stringify(event)); } diff --git a/skill/scripts/live.mjs b/skill/scripts/live.mjs index e1a7abe7e..306de3cae 100644 --- a/skill/scripts/live.mjs +++ b/skill/scripts/live.mjs @@ -26,6 +26,7 @@ import { resolveFiles } from './live-inject.mjs'; import { readLiveServerInfo } from './lib/impeccable-paths.mjs'; import { resolveSurfaceBrief } from './lib/surface-briefs.mjs'; import { resolveLiveTarget } from './live-target.mjs'; +import { bootInstructions } from './live/instructions.mjs'; import { resolveRoots, writeRootsManifest } from './live/roots.mjs'; const __dirname = path.dirname(fileURLToPath(import.meta.url)); @@ -196,6 +197,7 @@ The agent should then: hasSurfaceBrief: !!surfaceBrief, surfaceBrief, surfaceBriefPath, + _instructions: bootInstructions({ scriptsPath: __dirname }), }, null, 2)); } diff --git a/skill/scripts/live/instructions.mjs b/skill/scripts/live/instructions.mjs new file mode 100644 index 000000000..7c6599aa8 --- /dev/null +++ b/skill/scripts/live/instructions.mjs @@ -0,0 +1,142 @@ +/** + * Just-in-time agent instructions for live mode. + * + * The live scripts, not the reference doc, own situational plumbing: every + * event printed by live-poll carries an `_instructions` string describing + * exactly what to do NEXT, with real ids, paths, and line numbers already + * substituted and only the active path's rules included (a svelte-component + * session never sees JSX guidance, and vice versa). live.md stays lean: the + * session contract, harness policy, and design-quality guidance that is not + * situational (identity lock, variation axes, parameter budgets). + * + * Keep these strings imperative, concrete, and short. They are read by an + * agent mid-session; every sentence must earn its tokens. Instructions are + * versioned with the scripts, so they cannot drift from behavior the way a + * hand-maintained doc can. + */ + +const PLAN_POINTER = 'Plan per live.md section 4: extract the identity lock, pick default vs departure mode, commit each variant to a DIFFERENT primary axis, squint-test the trio. Size parameter knobs per section 7 budgets.'; + +function pollCmd(scriptsPath) { + return `node ${scriptsPath}/live-poll.mjs`; +} + +function replyCmd(scriptsPath, id, rest) { + return `${pollCmd(scriptsPath)} --reply ${id} ${rest}`; +} + +export function instructionsForEvent(event, { scriptsPath = '{{scripts_path}}' } = {}) { + if (!event || typeof event !== 'object') return undefined; + switch (event.type) { + case 'generate': + return generateInstructions(event, scriptsPath); + case 'steer': + return `Do what the message asks (page edits, navigation help, or a short answer). Then reply exactly once: ${replyCmd(scriptsPath, event.id, 'steer_done ["optional short toast"]')} (on failure: --reply ${event.id} error "Short reason"). No pickup ack; poll again immediately after.`; + case 'prefetch': + return `Speculative pre-read, no reply owed: resolve ${JSON.stringify(event.pageUrl || '/')} to its source file (root "/" is usually the boot's pageFile; multi-page sites map /foo to public/foo/index.html; SPAs map all routes to one entry), read it into context, then poll again. Skip if you cannot resolve it confidently.`; + case 'variant_mount_failed': + return `The browser could NOT render variant ${event.variant}${event.url ? ` (module: ${event.url})` : ''}${event.error ? `: ${String(event.error).slice(0, 200)}` : ''}. The user sees a persistent error card, not variants. Fix the variant source files, then reply ${replyCmd(scriptsPath, event.id, 'done --file ')}; the browser retries on its own. Poll again after the reply.`; + case 'accept': + return acceptInstructions(event, scriptsPath); + case 'discard': + return event?._completionAck?.ok === true + ? 'Original restored and durable completion acknowledged; nothing to do. Poll again.' + : `Completion was not acknowledged: run node ${scriptsPath}/live-complete.mjs --id ${event.id} --discarded, then poll again.`; + case 'manual_edit_apply': + return `The user already clicked Apply; never ask, discard, or redirect. Delegate the source edits to the impeccable_manual_edit_applier subagent when available (pass cwd, scripts path, event id, page URL, chunk/deadline, batch, evidencePath); it must not poll or reply. ${event.repair ? 'A `repair` payload is present: the previous Apply changed source but validation failed; fix the CURRENT source, never roll back yourself. ' : ''}Reply exactly once: ${replyCmd(scriptsPath, event.id, `done --data '{"status":"done","appliedEntryIds":[...],"failed":[],"files":[...],"notes":[]}'`)} (status "partial"/"error" with failed[] when not every entry applied). Then poll again.`; + case 'timeout': + return 'No event arrived; poll again immediately.'; + case 'exit': + return `Session over: kill any background poll, then node ${scriptsPath}/live-server.mjs stop (removes the injected script tag). Sweep leftover impeccable-variants-start / impeccable-carbonize-start markers from source.`; + default: + return undefined; + } +} + +function generateInstructions(event, scriptsPath) { + const id = event.id; + const scaffold = event.scaffold; + const steps = []; + + if (event.screenshotPath) { + steps.push(`Read the annotated screenshot first: ${event.screenshotPath}. Comment {x,y} positions bind text to the child under that point; strokes read by shape (loop = emphasis on this thing, arrow = direction, cross = delete).`); + } else { + steps.push('No screenshot was sent (the user did not annotate); do not ask for one and do not screenshot the page. Work from element.outerHTML, the computed styles, and the prompt.'); + } + + if (event.mode === 'insert') { + steps.push(insertScaffoldInstructions(event, scriptsPath)); + } else if (scaffold?.previewMode === 'svelte-component') { + steps.push(svelteComponentInstructions(event, scaffold, scriptsPath)); + } else if (scaffold && scaffold.sourceWritten === false) { + steps.push(deferredWrapperInstructions(event, scaffold, scriptsPath)); + } else if (scaffold) { + steps.push(`The wrapper is already written into ${scaffold.file}. Splice preview CSS plus all ${event.count} variants at line ${scaffold.insertLine} in ONE edit, following the returned cssAuthoring contract (styleTag, selector strategy, forbidden patterns). Each variant div holds exactly ONE top-level element (same tag as the original); first visible, others display: none.`); + } else { + steps.push(`Preflight could not scaffold${event.scaffoldError ? ` (${event.scaffoldError})` : ''}. Run node ${scriptsPath}/live-wrap.mjs --id ${id} --count ${event.count} --element-id "${event.element?.id || ''}" --classes "${(event.element?.classes || []).join(',')}" --tag "${event.element?.tagName || ''}" --text "". Keep the flags separate; --text disambiguates repeated siblings. On a fallback error, follow live.md's Handle fallback.`); + } + + steps.push(event.action && event.action !== 'impeccable' + ? `Action is "${event.action}": read reference/${event.action}.md before planning; its MUST params are non-negotiable. ${PLAN_POINTER}` + : `Freeform action: work from SKILL.md rules plus craft-floor.md; no sub-command file. ${PLAN_POINTER}`); + + steps.push(`When all ${event.count} variants are delivered: ${replyCmd(scriptsPath, id, 'done --file ')}. Then poll again. If generation fails after the browser flipped to GENERATING, reply --reply ${id} error "Short reason" so the bar resets (never live-accept --discard for this).`); + + return steps.map((s, i) => `${i + 1}. ${s}`).join('\n'); +} + +function svelteComponentInstructions(event, scaffold, scriptsPath) { + const dir = scaffold.componentDir; + const count = event.count; + return `Svelte component preview. EDIT the existing stubs ${dir}/v1.svelte ... v${count}.svelte in place; never delete or recreate them; do not read them back (the prop-substituted markup is in scaffold.componentStubMarkup). Keep the stub's control flow ({#each}, {#if}) and propContract prop names exactly; never flatten a loop into literal items. The stub