mirror of
https://github.com/pbakaus/impeccable.git
synced 2026-09-18 00:56:30 +03:00
90 lines
4.8 KiB
Markdown
90 lines
4.8 KiB
Markdown
# Oracle: behavior goldens for every `impeccable` verb
|
|
|
|
`lib.mjs` runs each case (verb + args + staged workspace + stdin) against an
|
|
implementation and captures stdout, stderr, exit code, and named files, with
|
|
machine-specific paths and timestamps normalized.
|
|
|
|
- `record.mjs` writes goldens from the JS scripts (`skill/scripts`, `cli/bin`).
|
|
- `run.mjs` replays the corpus against `$IMPECCABLE_BIN` (or `--js` for a
|
|
self-check) and diffs. Byte-equal is the bar; `DELTAS.md` lists reviewed
|
|
exceptions.
|
|
- `cases/*.mjs` define the corpus (default export: array or async function
|
|
returning an array). `workspaces/` holds project fixtures that are copied to
|
|
a temp dir per run, so cases can write freely.
|
|
|
|
Adding a case: append to the matching `cases/*.mjs`, run
|
|
`node tests/oracle/record.mjs <prefix>`, commit the golden.
|
|
|
|
Verb names are the binary's subcommands. `cli-help` and `cli-version` map to
|
|
`impeccable --help` / `--version`.
|
|
|
|
## Corpus files
|
|
|
|
- `cases/detect.mjs`: `detect`, `cli-help`, `cli-version`, `ignores`.
|
|
- `cases/hooks.mjs`: `hook`, `hook-before-edit`, `hook-admin`.
|
|
- `cases/context.mjs`: `context`, `doctor`, `pin`, `surface-brief`,
|
|
`critique-storage`, `palette`, `embed-prompt`, `context-signals`
|
|
(id prefix `signals-`), `detect-csp` (`csp-`), `concept-seed` (`seed-`),
|
|
`generate-image` (`genimg-`), `serve-question` (`question-`). Only offline
|
|
paths: the local catalog fixture or an unreachable roll API, fake image
|
|
generation, and serve-question modes that never open a browser or listen.
|
|
Workspaces are `workspaces/ctx-*`; the header comment in the case file
|
|
describes each one. Machine-specific env (`OPENAI_API_KEY`, catalog and
|
|
context overrides, `CI`) is pinned per case so the recording host does not
|
|
leak into goldens.
|
|
|
|
## Normalizations
|
|
|
|
Beyond paths and ISO timestamps, `normalize()` masks these run- or
|
|
machine-dependent fragments. Each is targeted at one script's output:
|
|
|
|
- `IMAGE_TOOLS: <IMAGE_TOOLS_PROBE>`: `context` probes `which cwebp sips
|
|
magick ffmpeg`; the set found describes the machine, not the script.
|
|
- `"devServer": <DEV_SERVER_PROBE>`: `context-signals` probes localhost ports
|
|
4321/3000/5173/5174/8080/8000/4200; whatever is listening on the recording
|
|
host is not part of the contract.
|
|
- `<STAMP>`: `critique-storage` stamps snapshots with the wall clock in dash
|
|
form (`2026-05-12T18-30-00Z`), in the file name and the `timestamp:`
|
|
frontmatter it writes. Cases that write a snapshot do not snapshot the file;
|
|
they run `latest` / `trend` afterwards instead.
|
|
- `"<finding-id>": <EPOCH>`: the staleness notice cache
|
|
(`~/.impeccable/staleness-check.json`) keys epoch stamps by finding id.
|
|
- `<IMPECCABLE> <verb>` / `<HOOK_ADMIN_CMD>`: self-referential command lines.
|
|
|
|
Not covered on purpose: `palette` with no `--id` / `--from` / env seed (random),
|
|
`concept-seed` against the live roll API, `generate-image` real mode,
|
|
`serve-question --start` / blocking mode (opens a browser and binds a port),
|
|
and unhandled-exception paths whose stack traces carry Node line numbers.
|
|
|
|
## Live-mode cases (`cases/live-*.mjs`, workspaces `live-*`)
|
|
|
|
Helpers live in `live-helpers.mjs` (staged journals, buffers, wrapped source
|
|
files with the fake-agent variant block, a `.git` FILE pointing at a non-repo
|
|
gitdir so roots resolution sees a git boundary while `git check-ignore` exits
|
|
128 everywhere and the ignore block lands in the snapshotable
|
|
`.gitfake/info/exclude`). Svelte component preview cases symlink this repo's
|
|
`node_modules/svelte` into the staged app, exactly like the unit tests.
|
|
|
|
Harness additions made for live:
|
|
|
|
- `steps[]` entries may carry their own `setup(ws)` (run right before that
|
|
step) and `daemon: true` with `readyFile` / `readyTimeoutMs`: the verb is
|
|
spawned detached, the harness waits for the ready file, later steps run
|
|
against it, and teardown SIGTERMs (then SIGKILLs) it. Its stdout/stderr land
|
|
in the golden as `daemon: [{stdout, stderr}]`.
|
|
- `normalize: [[regexSource, flags, replacement], ...]` on a case applies extra
|
|
masks to that case only. Live uses it for the dynamic helper port
|
|
(`localhost:<PORT>`, `"port": <PORT>`), lease and phase stamps (`<EPOCH>`),
|
|
and float durations (`<N>`).
|
|
- Global masks added: `"pid": <PID>` / `(pid <PID>)` and UUID tokens `<UUID>`.
|
|
- `snapshotFiles` walks `node_modules/.impeccable-live` (the Svelte preview
|
|
tree) and nothing else under `node_modules`.
|
|
|
|
Deliberately not covered here (rely on `tests/live-e2e`): the browser
|
|
handshake and `/live.js` bundle, SSE, generate/accept round-trips through a
|
|
real browser, `variant_mount_failed` republish, manual-edit chat routing and
|
|
the codex/claude subprocess providers, Svelte revision-dir publishing, and
|
|
`live.mjs`'s dev-server-dependent flows. Lock-file names hash the absolute
|
|
source path, so lock cases do not snapshot `.impeccable/live/locks/`.
|
|
`live-poll-*-connection-refused` assumes nothing listens on 127.0.0.1:65531.
|