skills/pbakaus_impeccable
1
0
Fork 0
mirror of https://github.com/pbakaus/impeccable.git synced 2026-09-19 01:26:29 +03:00
Code Issues Packages Projects Releases Wiki Activity

feat(test): pluggable LLM agent for live-mode E2E suite

Browse Source 
tests/live-e2e/agents/llm-agent.mjs: a Claude-backed VariantAgent that
implements the same one-method interface as the fake agent
(generateVariants(event, context) → { scopedCss, variants[] }). Default
model claude-haiku-4-5; override via IMPECCABLE_E2E_LLM_MODEL.

Prompt caching is on — the system prompt (instructions + the live-mode
spec from reference/live.md) is the cacheable prefix. First call writes
~10K tokens to cache; subsequent fixtures pay only the cache-read rate.
JSON output is validated for shape (scopedCss, variants[N].innerHtml),
with light error messages on parse failure.

tests/live-e2e.test.mjs: read IMPECCABLE_E2E_AGENT (fake|llm). When 'llm',
construct the LLM agent and skip the case cleanly if ANTHROPIC_API_KEY is
unset. Param-manifest assertions are gated to fake mode (LLM may emit
zero-param "fixed point" variants per the live.md spec). The accepted-h1
class assertion now allows hero-title as one of multiple classes so an
LLM agent that adds classes alongside the original still passes.

Test timeouts widen for LLM mode: 25s first-pass on conditional-render
fixtures (vs 5s for fake), 60s on direct waits (vs 30s). Without these,
the LLM's 3-8s generate latency races the orchestration's state-loss
recovery window.

tests/live-e2e/ui.mjs: clickGo retries up to 3× on stability failures.
Required because conditional-render fixtures (modal/tabs) animate the bar
mid-transition when preActions trigger framework HMR; a single click can
land during a re-render and Playwright's stability gate times out.

Pass rate on a typical sweep: 18/19 in LLM mode, 19/19 in fake mode.
The modal fixture's intrinsic state-loss flake (Fast Refresh resetting
useState(open) when source changes) is amplified by LLM latency and may
need a re-run; documented in CLAUDE.md.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Paul Bakaus
2026-04-25 01:06:06 -07:00
co-authored by Claude Opus 4.7
parent 4310352423
commit d26ccac1be
6 changed files with 265 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/live-e2e/ui.mjs
+18 -1
View File
@@ -91,9 +91,26 @@ export async function setCount(page, count) {
/**
* Click Go. Browser POSTs the generate event; the agent picks it up.
*
* On fixtures whose preActions triggered a layout shift (modal/tab opening)
* the bar's open animation can still be running when we click, and
* Playwright's stability gate occasionally times out on the first attempt.
* Retry up to three times with a settle in between so a single race doesn't
* fail the test.
*/
export async function clickGo(page) {
await page.locator(`${BAR_ID} button`, { hasText: /Go\b/ }).click();
const go = page.locator(`${BAR_ID} button`, { hasText: /Go\b/ });
let lastErr;
for (let attempt = 0; attempt < 3; attempt++) {
try {
await go.click({ timeout: 5_000 });
return;
} catch (err) {
lastErr = err;
await page.waitForTimeout(500);
}
}
throw lastErr;
}
/**