feat(live): annotation capture, comment pins, drawing, and halftone loading shader

Adds a full annotation pipeline to /impeccable live. On Go, the browser
captures the selected element as a PNG (with annotations composed in),
uploads it to the live helper, and sends the generate event with the
screenshot path so the agent reads user intent visually instead of from
HTML alone.

Annotation tools (while an element is picked):
- Click inside the outline to drop a magenta comment pin with a text input
- Drag to paint a magenta SVG stroke (5 px click-vs-drag threshold)
- Click a pin to edit; double-click to delete; drag a pin to reposition
- Click a stroke to delete it (wider invisible hit path)
- Clear chip top-right wipes everything; hidden when no annotations

Capture pipeline:
- modern-screenshot vendored as an IIFE (scripts/modern-screenshot.umd.js)
  and lazy-loaded from the live helper
- Font fix: cross-origin @font-face rules are fetched and fonts are inlined
  as base64 data URIs before being handed to modern-screenshot via
  font.cssText, since SVGs rasterized via canvas can't fetch external
  resources (fix for "Impeccable" rendering bold-serif and items wrapping
  wrong in the capture)
- Annotations are temporarily attached to the live element (not only the
  clone) so computed styles resolve during the embed pass
- Session screenshots live in .impeccable-live/annotations/session-*/ in
  the project root (gitignored) so the agent's Read tool doesn't trip a
  per-path permission prompt

Loading shader (activates during GENERATING):
- WebGL overlay rendering the captured PNG as a halftone — cells with
  luma-driven dot radius, rendered on paper-cream underneath a magenta
  roller that sweeps top-to-bottom with a 3.4s cycle and clean overshoot
- Fixed asymmetric bandAt() using one-sided smoothsteps (previous reversed
  smoothstep was undefined on d>0, giving "trail=1 everywhere below")
- Graceful <img> fallback when WebGL is unavailable; prefers-reduced-motion
  freezes the band at t=0

Server:
- POST /annotation endpoint (raw image/png body, token + eventId query),
  session-scoped tmpdir cleaned up on shutdown
- GET /modern-screenshot.js serves the vendored UMD with aggressive caching
- Optional screenshotPath / comments / strokes fields on generate events
- Fixed pre-existing /source crash on ENOENT (writeHead called twice)

Agent side:
- reference/live.md step 0 tells the agent to Read the screenshot first,
  with four rules for interpreting annotations: comments are position-
  anchored and scoped to the sub-element under their {x,y}; strokes are
  gestures (loop=focus, arrow=direction, cross=delete); comments and
  strokes are independent unless adjacent; don't silently guess on
  ambiguous strokes

Also:
- Generating bar no longer claims "Generating 1 of 3..." (variants arrive
  atomically) — now says "Generating N variants..."
- tests/live-server.test.mjs fixed to read the PID file from project root,
  matching the server; adds coverage for the new endpoints and validator
  fields
- .impeccable-live/ added to .gitignore

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Paul Bakaus
2026-04-21 09:38:48 -07:00
co-authored by Claude Opus 4.7
parent 51d28cf1eb
commit 81f880d030
52 changed files with 12583 additions and 361 deletions
+14 -1
View File
@@ -107,10 +107,23 @@ END LOOP
## Handle Generate
The event contains: `{id, action, freeformPrompt, count, pageUrl, element}`.
The event contains: `{id, action, freeformPrompt, count, pageUrl, element, screenshotPath?, comments?, strokes?}`.
**Speed matters.** The user is watching a spinner. Minimize tool calls by using the `wrap` helper and writing all variants in a single edit.
### Step 0: If `screenshotPath` is present, Read it
When the browser successfully captured the selected element, `event.screenshotPath` is an absolute path to a PNG showing the element as the user actually sees it — including any comment pins or drawn strokes the user placed before hitting Go. **Read it before planning variants.** The annotations encode user intent that is not recoverable from `element.outerHTML` alone (a circle around a piece of whitespace, an arrow pointing to an alignment issue, a "make this bolder" note on a specific sub-element).
If `event.comments` or `event.strokes` are set, they carry structured metadata (comment text + positions, stroke polylines) alongside the visual. Treat the screenshot as primary; use the structured data for specifics worth quoting verbatim (e.g. the exact text of a comment).
**Reading annotations precisely:**
- **A comment's position is load-bearing.** Its `{x, y}` (element-local CSS px, same coord space as `element.boundingRect`) tells you which sub-element it refers to. Find the child under that point and apply the comment text LOCALLY to that sub-element. A comment near the title is about the title, not a description of "the screenshot."
- **Treat comments and strokes as independent annotations** unless they are clearly paired by position (overlap or tight proximity). Do NOT let the visual weight of a prominent stroke override the precise location of a textually-specific comment elsewhere in the element.
- **Strokes are gestures — read them by shape, not as a mask.** A closed loop = "this thing" (emphasis / focus); an arrow = direction (move / point to); a cross or slash = delete; a free scribble = emphasis or delete depending on context. A loop around region X does NOT mean "only change pixels inside X"; it means "pay attention to X."
- **When a stroke's intent is ambiguous** (circle or arrow? emphasis or move?), state your reading in one sentence as part of your rationale rather than silently guessing. If the uncertainty materially changes the brief, ask the user for one quick clarification before generating.
### Step 1: Wrap the element (one CLI call)
Use the `wrap` helper to find the element and create the variant container:
File diff suppressed because it is too large Load Diff
+108 -11
View File
@@ -18,7 +18,6 @@ import { randomUUID } from 'node:crypto';
import { spawn, execFileSync } from 'node:child_process';
import fs from 'node:fs';
import path from 'node:path';
import os from 'node:os';
import net from 'node:net';
import { fileURLToPath } from 'node:url';
import { parseDesignMd } from './design-parser.mjs';
@@ -56,8 +55,13 @@ const state = {
pendingEvents: [], // browser events waiting for agent poll
pendingPolls: [], // agent poll callbacks waiting for browser events
exitTimer: null,
sessionDir: null, // per-session tmp dir for annotation screenshots
};
// Cap per-annotation upload size. A full 1920×1080 PNG is typically <1 MB;
// cap at 10 MB to guard against runaway writes from a misbehaving client.
const MAX_ANNOTATION_BYTES = 10 * 1024 * 1024;
function enqueueEvent(event) {
if (state.pendingPolls.length > 0) {
state.pendingPolls.shift()(event);
@@ -134,6 +138,10 @@ function validateEvent(msg) {
if (!msg.action || !VISUAL_ACTIONS.includes(msg.action)) return 'generate: invalid action';
if (!Number.isInteger(msg.count) || msg.count < 1 || msg.count > 8) return 'generate: count must be 1-8';
if (!msg.element || !msg.element.outerHTML) return 'generate: missing element context';
// Optional annotation fields (all-or-nothing: if any present, all must be well-formed).
if (msg.screenshotPath !== undefined && typeof msg.screenshotPath !== 'string') return 'generate: screenshotPath must be string';
if (msg.comments !== undefined && !Array.isArray(msg.comments)) return 'generate: comments must be array';
if (msg.strokes !== undefined && !Array.isArray(msg.strokes)) return 'generate: strokes must be array';
return null;
case 'accept':
if (!msg.id) return 'accept: missing id';
@@ -175,6 +183,83 @@ function createRequestHandler({ detectScript, liveScriptWithToken }) {
return;
}
// --- Vendored modern-screenshot (UMD build) ---
// Lazy-loaded by live.js when the user clicks Go; exposes
// window.modernScreenshot.domToBlob(...) for capture.
if (p === '/modern-screenshot.js') {
const vendorPath = path.join(__dirname, 'modern-screenshot.umd.js');
try {
res.writeHead(200, {
'Content-Type': 'application/javascript',
'Cache-Control': 'public, max-age=31536000, immutable',
});
res.end(fs.readFileSync(vendorPath));
} catch {
res.writeHead(404); res.end('Vendor script not found');
}
return;
}
// --- Annotation upload (browser → server, raw PNG body) ---
// Client generates the eventId, POSTs the PNG, then POSTs the generate
// event with screenshotPath already set. Keeps bytes out of the SSE/poll
// bridge and preserves the "one shot from the user's POV" UX.
if (p === '/annotation' && req.method === 'POST') {
const token = url.searchParams.get('token');
if (token !== state.token) { res.writeHead(401); res.end('Unauthorized'); return; }
const eventId = url.searchParams.get('eventId');
if (!eventId || !/^[A-Za-z0-9_-]{1,64}$/.test(eventId)) {
res.writeHead(400, { 'Content-Type': 'application/json' });
res.end(JSON.stringify({ error: 'Invalid eventId' }));
return;
}
if ((req.headers['content-type'] || '').toLowerCase() !== 'image/png') {
res.writeHead(415, { 'Content-Type': 'application/json' });
res.end(JSON.stringify({ error: 'Content-Type must be image/png' }));
return;
}
if (!state.sessionDir) {
res.writeHead(500, { 'Content-Type': 'application/json' });
res.end(JSON.stringify({ error: 'Session dir unavailable' }));
return;
}
const chunks = [];
let total = 0;
let aborted = false;
req.on('data', (c) => {
if (aborted) return;
total += c.length;
if (total > MAX_ANNOTATION_BYTES) {
aborted = true;
res.writeHead(413, { 'Content-Type': 'application/json' });
res.end(JSON.stringify({ error: 'Payload too large' }));
req.destroy();
return;
}
chunks.push(c);
});
req.on('end', () => {
if (aborted) return;
const absPath = path.join(state.sessionDir, eventId + '.png');
try {
fs.writeFileSync(absPath, Buffer.concat(chunks));
} catch (err) {
res.writeHead(500, { 'Content-Type': 'application/json' });
res.end(JSON.stringify({ error: 'Write failed: ' + err.message }));
return;
}
res.writeHead(200, { 'Content-Type': 'application/json' });
res.end(JSON.stringify({ ok: true, path: absPath }));
});
req.on('error', () => {
if (!aborted) {
res.writeHead(500, { 'Content-Type': 'application/json' });
res.end(JSON.stringify({ error: 'Upload failed' }));
}
});
return;
}
// --- Health ---
if (p === '/health') {
res.writeHead(200, { 'Content-Type': 'application/json' });
@@ -250,10 +335,11 @@ function createRequestHandler({ detectScript, liveScriptWithToken }) {
if (!filePath || filePath.includes('..')) { res.writeHead(400); res.end('Bad path'); return; }
const absPath = path.resolve(process.cwd(), filePath);
if (!absPath.startsWith(process.cwd())) { res.writeHead(403); res.end('Forbidden'); return; }
try {
res.writeHead(200, { 'Content-Type': 'text/html; charset=utf-8' });
res.end(fs.readFileSync(absPath, 'utf-8'));
} catch { res.writeHead(404); res.end('File not found'); }
let content;
try { content = fs.readFileSync(absPath, 'utf-8'); }
catch { res.writeHead(404); res.end('File not found'); return; }
res.writeHead(200, { 'Content-Type': 'text/html; charset=utf-8' });
res.end(content);
return;
}
@@ -411,6 +497,9 @@ let httpServer = null;
function shutdown() {
try { fs.unlinkSync(LIVE_PID_FILE); } catch {}
if (state.sessionDir) {
try { fs.rmSync(state.sessionDir, { recursive: true, force: true }); } catch {}
}
for (const res of state.sseClients) { try { res.end(); } catch {} }
state.sseClients.clear();
for (const resolve of state.pendingPolls) resolve({ type: 'exit' });
@@ -442,12 +531,14 @@ Options:
--help Show this help
Endpoints:
/live.js Browser script (element picker + variant cycling)
/detect.js Detection overlay (backwards compatible)
/events SSE stream (server→browser) + POST (browser→server)
/poll Long-poll for agent CLI
/source Raw source file reader (no-HMR fallback)
/health Health check`);
/live.js Browser script (element picker + variant cycling)
/detect.js Detection overlay (backwards compatible)
/modern-screenshot.js Vendored modern-screenshot UMD build (lazy-loaded by live.js)
/annotation POST raw image/png to stage a variant screenshot
/events SSE stream (server→browser) + POST (browser→server)
/poll Long-poll for agent CLI
/source Raw source file reader (no-HMR fallback)
/health Health check`);
process.exit(0);
}
@@ -531,6 +622,12 @@ try {
state.token = randomUUID();
const portArg = args.find(a => a.startsWith('--port='));
state.port = portArg ? parseInt(portArg.split('=')[1], 10) : await findOpenPort();
// Annotation screenshots live in the project root so the agent's Read tool
// doesn't trip a per-file permission prompt. Sessioned by token so concurrent
// projects (or quick restarts) don't collide.
const annotRoot = path.join(process.cwd(), '.impeccable-live', 'annotations');
fs.mkdirSync(annotRoot, { recursive: true });
state.sessionDir = fs.mkdtempSync(path.join(annotRoot, 'session-'));
const { detectScript, liveScript } = loadBrowserScripts();
const liveScriptWithToken =
File diff suppressed because one or more lines are too long