mirror of
https://github.com/pbakaus/impeccable.git
synced 2026-09-11 21:57:14 +03:00
Paul BakausandClaude
4e381305e1
Addresses the review findings on #371, plus several the bots did not catch. All fixes have regression coverage that fails on the prior code. Source corruption: - Vue accept dropped valueless root attrs (disabled, v-cloak) and, worse, rewrote @click="x" as a literal click="x" DOM attribute, because the attr parser was name-anchored and skipped the sigil. Tokenize the whole Vue attr grammar and normalize shorthands so accept round-trips directives. - --variant was interpolated unescaped into a RegExp, so --variant '.*' matched the original block first and reported a successful accept while silently restoring the original. Validate against the digits pattern the browser and the /events schema already enforce. - --id reached path.join unvalidated, so --id ../../../../etc/evil wrote and read receipts outside the project. Hoist the existing safeSessionId check into impeccable-paths and apply it at every id-to-path sink. Accept/lock correctness: - Plain HTML/JSX accept and discard did not catch SOURCE_LOCKED, so contention exited non-zero with empty stdout and the agent got no JSON to retry on. - Lock staleness was mtime-only and never read the pid it records: a holder whose critical section outran 60s had its live lock swept, admitting a second writer to the same file, while a crashed holder blocked accepts for a full 60s. Decide staleness by owner liveness, and release only our own lock. Detector: - isNeutralColor only parses computed color forms, so routing authored CSS through it reported inset 4px 0 0 #000 / black / #e5e7eb as chromatic side-tab stripes. Add an authored-color neutrality test covering hex and named neutrals; the fixture had no literal-color cases at all. - Rule line numbers were off by one for every rule after the first, and commented-out CSS was scanned as live rules. Server: - An error reply carries no sourceEventType, and inferSourceEventType returned undefined, which acknowledgePendingEvent treats as a wildcard: a stale generate worker's failure consumed the user's queued Accept, which then reached no agent and left the browser in SAVING forever. - The generate preflight spawned live-wrap.mjs synchronously inside the request handler, freezing the single-threaded server for the whole scaffold (~7.6s measured on this repo, 15s ceiling) and stalling Accept/Discard/SSE. Make it async, claiming the lease before the first await so no event double-delivers. - Every browser checkpoint was echoed back as variant_progress, so a Tune slider drag remounted the preview under the user's cursor and latched the *_reviewable phases from the wrong trigger. Gate on the reason. Cleanup: - Collapse four divergent benchmark argv parsers into scripts/lib/cli-args.mjs. Three silently misread flags: --iterations 20 benchmarked 5, --agent llm ran the fake agent, --median-target=0.4 used the default threshold. - Drop a snapshot cache this branch made write-only (it grew per session for the server's lifetime and was never read), a dead exported reconcile helper, and the unused deferReply branch. Prepared with AI assistance under maintainer direction. Co-Authored-By: Claude <noreply@anthropic.com>
40 lines
1.5 KiB
JavaScript
40 lines
1.5 KiB
JavaScript
#!/usr/bin/env node
|
|
|
|
import { readFile } from 'node:fs/promises';
|
|
import { resolve } from 'node:path';
|
|
|
|
import { parseArgs } from './lib/cli-args.mjs';
|
|
import { compareModelBackedReports } from './lib/live-benchmark.mjs';
|
|
|
|
const args = parseArgs(process.argv.slice(2));
|
|
if (!args.atomic || !args.progressive) {
|
|
throw new Error('usage: node scripts/compare-live-benchmarks.mjs --atomic=<report.json> --progressive=<report.json>');
|
|
}
|
|
|
|
const [atomic, progressive] = await Promise.all([
|
|
readReport(args.atomic, 'atomic'),
|
|
readReport(args.progressive, 'progressive'),
|
|
]);
|
|
const comparison = compareModelBackedReports(atomic, progressive, {
|
|
medianTarget: ratioArg(args.medianTarget, 0.35),
|
|
p95Target: ratioArg(args.p95Target, 0.25),
|
|
});
|
|
|
|
process.stdout.write(JSON.stringify(comparison, null, 2) + '\n');
|
|
if (!comparison.passed) process.exitCode = 1;
|
|
|
|
async function readReport(file, delivery) {
|
|
const value = JSON.parse(await readFile(resolve(String(file)), 'utf-8'));
|
|
const reports = Array.isArray(value?.reports) ? value.reports : [value];
|
|
const report = reports.find((item) => item?.benchmark?.delivery === delivery);
|
|
if (!report) throw new Error(`${file} does not contain a ${delivery} benchmark report`);
|
|
return report;
|
|
}
|
|
|
|
function ratioArg(value, fallback) {
|
|
if (value == null || value === true) return fallback;
|
|
const parsed = Number(value);
|
|
if (!Number.isFinite(parsed) || parsed < 0 || parsed >= 1) throw new Error(`invalid threshold ratio: ${value}`);
|
|
return parsed;
|
|
}
|