Paul BakausandClaude Code
a91c226b2b
Brings the leak guard from #718 onto the branch and makes its guarantee hold
for the Rust engine instead of the Node scripts it was written against.
Conflicts and how each was resolved:
- tests/live-poll-stream.test.mjs, tests/live-server.test.mjs,
tests/live-target-context.test.mjs (modify/delete): kept deleted. They drove
skill/scripts/live-server.mjs, which does not exist here; the verb behavior
they covered is the oracle's job now. Their entries came out of
test-suites.mjs along with the rest of main's live list, which is Node-script
coverage this branch already retired.
- scripts/test-suites.mjs: took main's two new entries that still apply,
process-group.test.mjs into core and live-server-leak.test.mjs into live, plus
the infra trigger patterns for the three new scripts/lib modules. Dropped
main's pin.test.mjs (no such file here).
- package.json: kept test:cleanup, dropped test:cli-e2e (no cli-e2e suite here).
- scripts/run-tests.mjs: rewritten to hold both sides rather than picking one.
From #718: the createGroupShutdown state machine, the per-suite run-id marker
env, the post-suite leak check, and --cleanup. From 47f18713: the per-command
wall-clock cap with its per-suite wallClockMs override and
IMPECCABLE_TEST_WALL_CLOCK_MS, plus the killed-by-signal report. The two agree
on the detached process group, so they compose: the cap SIGKILLs that group
when a command wedges, the shutdown handler ends it on a signal, and both now
sweep for leaked servers before exiting. #718's handler replaces the old raw
signal forwarding, which sent one signal and never escalated.
- tests/live-e2e/session.mjs: kept both sides. The binary-driven boot
(runEngineSync, requireEngineBin, engineEnv) stands, with armLiveServerReaper
at module scope and trackServerChild around the fixture dev server.
Ported to the rest of the branch:
- tests/oracle/lib.mjs arms the reaper and tracks the daemon child. Its daemon
steps spawn live-server detached, so a SIGKILLed oracle run used to strand
one; buildInvocation already inherits process.env, so the marker reaches it.
- tests/live-server-leak.test.mjs now boots the engine binary through
tests/lib/engine-bin.mjs and skips cleanly without one.
No crate change was needed. The daemon spawn does env_clear().envs(env) against
Io::stdio()'s env, which is std::env::vars(), so the detached Rust process
carries the parent environment and the markers reach it. Verified against a
real --background daemon: found by run id and by repo marker, not found by an
adjacent checkout's marker. CLAUDE.md now says so, since narrowing that env
would make the guard silently blind.
Verified with a fresh cargo build --release -p impeccable:
- IMPECCABLE_BIN=... bun run test green end to end: core 90, oracle 1 (zero
unreviewed differences), detector 1, live 159 (157 pass, 2 skipped),
framework 186, plugin-e2e 4. Zero servers left.
- SIGKILL repro against impeccable live-server --background: 1 daemon up, 0
after with the reaper, 1 surviving with parent pid 1 under
IMPECCABLE_NO_TEST_REAPER=1. bun run test:cleanup then kills exactly that one.
- The leak test fails under IMPECCABLE_NO_TEST_REAPER=1 and passes with it.
- IMPECCABLE_E2E_ONLY=vite8-react-plain bun run test:live-e2e 4/4.
- bun run build green.
AI assistance: prepared by Claude Code under pbakaus's direction.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vau2X53xGTjjTCXWMVBoNY
Oracle: behavior goldens for every impeccable verb
lib.mjs runs each case (verb + args + staged workspace + stdin) against an
implementation and captures stdout, stderr, exit code, and named files, with
machine-specific paths and timestamps normalized.
- The goldens are frozen JS behavior: they were recorded from the Node
scripts (
skill/scripts,cli/bin) before those left the tree with the launcher swap, plus the reviewed deltas inDELTAS.md. They are the behavior contract the engine binary is held to. record.mjs --bin(with$IMPECCABLE_BINor--bin=/path) writes goldens from the binary, for new cases or a delta a review accepted. Plainrecord.mjsstill targets the JS scripts and only works on a checkout that has them (history before the swap).run.mjsreplays the corpus against$IMPECCABLE_BIN(or--jsfor a self-check on a pre-swap checkout) and diffs. Byte-equal is the bar;DELTAS.mdlists reviewed exceptions.tests/oracle.test.mjsrunsrun.mjsunderbun run testand skips when no binary is found (IMPECCABLE_BINorskill/scripts/bin/<os>-<arch>/, filled bybun run fetch:engine).cases/*.mjsdefine the corpus (default export: array or async function returning an array).workspaces/holds project fixtures that are copied to a temp dir per run, so cases can write freely.
Adding a case: append to the matching cases/*.mjs, run
node tests/oracle/record.mjs --bin <prefix>, review the golden by hand
(the binary is now the recorder, so a bug in it would be frozen too), commit
the golden.
Verb names are the binary's subcommands. cli-help and cli-version map to
impeccable --help / --version. lib.mjs still carries the JS_VERBS
table that maps each verb to the script it was recorded from.
vectors/ holds the function-level vectors recorded from the JS engine's pure
functions; see vectors/README.md.
Corpus files
cases/detect.mjs:detect,cli-help,cli-version,ignores.cases/hooks.mjs:hook,hook-before-edit,hook-admin.cases/context.mjs:context,doctor,pin,surface-brief,critique-storage,palette,embed-prompt,context-signals(id prefixsignals-),detect-csp(csp-),concept-seed(seed-),generate-image(genimg-),serve-question(question-). Only offline paths: the local catalog fixture or an unreachable roll API, fake image generation, and serve-question modes that never open a browser or listen. Workspaces areworkspaces/ctx-*; the header comment in the case file describes each one. Machine-specific env (OPENAI_API_KEY, catalog and context overrides,CI) is pinned per case so the recording host does not leak into goldens.
Normalizations
Beyond paths and ISO timestamps, normalize() masks these run- or
machine-dependent fragments. Each is targeted at one script's output:
IMAGE_TOOLS: <IMAGE_TOOLS_PROBE>:contextprobeswhich cwebp sips magick ffmpeg; the set found describes the machine, not the script."devServer": <DEV_SERVER_PROBE>:context-signalsprobes localhost ports 4321/3000/5173/5174/8080/8000/4200; whatever is listening on the recording host is not part of the contract.<STAMP>:critique-storagestamps snapshots with the wall clock in dash form (2026-05-12T18-30-00Z), in the file name and thetimestamp:frontmatter it writes. Cases that write a snapshot do not snapshot the file; they runlatest/trendafterwards instead."<finding-id>": <EPOCH>: the staleness notice cache (~/.impeccable/staleness-check.json) keys epoch stamps by finding id.<IMPECCABLE> <verb>/<HOOK_ADMIN_CMD>: self-referential command lines.
Not covered on purpose: palette with no --id / --from / env seed (random),
concept-seed against the live roll API, generate-image real mode,
serve-question --start / blocking mode (opens a browser and binds a port),
and unhandled-exception paths whose stack traces carry Node line numbers.
Live-mode cases (cases/live-*.mjs, workspaces live-*)
Helpers live in live-helpers.mjs (staged journals, buffers, wrapped source
files with the fake-agent variant block, a .git FILE pointing at a non-repo
gitdir so roots resolution sees a git boundary while git check-ignore exits
128 everywhere and the ignore block lands in the snapshotable
.gitfake/info/exclude). Svelte component preview cases symlink this repo's
node_modules/svelte into the staged app, exactly like the unit tests.
Harness additions made for live:
steps[]entries may carry their ownsetup(ws)(run right before that step) anddaemon: truewithreadyFile/readyTimeoutMs: the verb is spawned detached, the harness waits for the ready file, later steps run against it, and teardown SIGTERMs (then SIGKILLs) it. Its stdout/stderr land in the golden asdaemon: [{stdout, stderr}].normalize: [[regexSource, flags, replacement], ...]on a case applies extra masks to that case only. Live uses it for the dynamic helper port (localhost:<PORT>,"port": <PORT>), lease and phase stamps (<EPOCH>), and float durations (<N>).- Global masks added:
"pid": <PID>/(pid <PID>)and UUID tokens<UUID>. snapshotFileswalksnode_modules/.impeccable-live(the Svelte preview tree) and nothing else undernode_modules.
Deliberately not covered here (rely on tests/live-e2e): the browser
handshake and /live.js bundle, SSE, generate/accept round-trips through a
real browser, variant_mount_failed republish, manual-edit chat routing and
the codex/claude subprocess providers, Svelte revision-dir publishing, and
live.mjs's dev-server-dependent flows. Lock-file names hash the absolute
source path, so lock cases do not snapshot .impeccable/live/locks/.
live-poll-*-connection-refused assumes nothing listens on 127.0.0.1:65531.