mirror of
https://github.com/pbakaus/impeccable.git
synced 2026-09-15 15:46:30 +03:00
Merge origin/main into rust-swap (#718 live-server leak guard)
Brings the leak guard from #718 onto the branch and makes its guarantee hold
for the Rust engine instead of the Node scripts it was written against.
Conflicts and how each was resolved:
- tests/live-poll-stream.test.mjs, tests/live-server.test.mjs,
tests/live-target-context.test.mjs (modify/delete): kept deleted. They drove
skill/scripts/live-server.mjs, which does not exist here; the verb behavior
they covered is the oracle's job now. Their entries came out of
test-suites.mjs along with the rest of main's live list, which is Node-script
coverage this branch already retired.
- scripts/test-suites.mjs: took main's two new entries that still apply,
process-group.test.mjs into core and live-server-leak.test.mjs into live, plus
the infra trigger patterns for the three new scripts/lib modules. Dropped
main's pin.test.mjs (no such file here).
- package.json: kept test:cleanup, dropped test:cli-e2e (no cli-e2e suite here).
- scripts/run-tests.mjs: rewritten to hold both sides rather than picking one.
From #718: the createGroupShutdown state machine, the per-suite run-id marker
env, the post-suite leak check, and --cleanup. From 47f18713: the per-command
wall-clock cap with its per-suite wallClockMs override and
IMPECCABLE_TEST_WALL_CLOCK_MS, plus the killed-by-signal report. The two agree
on the detached process group, so they compose: the cap SIGKILLs that group
when a command wedges, the shutdown handler ends it on a signal, and both now
sweep for leaked servers before exiting. #718's handler replaces the old raw
signal forwarding, which sent one signal and never escalated.
- tests/live-e2e/session.mjs: kept both sides. The binary-driven boot
(runEngineSync, requireEngineBin, engineEnv) stands, with armLiveServerReaper
at module scope and trackServerChild around the fixture dev server.
Ported to the rest of the branch:
- tests/oracle/lib.mjs arms the reaper and tracks the daemon child. Its daemon
steps spawn live-server detached, so a SIGKILLed oracle run used to strand
one; buildInvocation already inherits process.env, so the marker reaches it.
- tests/live-server-leak.test.mjs now boots the engine binary through
tests/lib/engine-bin.mjs and skips cleanly without one.
No crate change was needed. The daemon spawn does env_clear().envs(env) against
Io::stdio()'s env, which is std::env::vars(), so the detached Rust process
carries the parent environment and the markers reach it. Verified against a
real --background daemon: found by run id and by repo marker, not found by an
adjacent checkout's marker. CLAUDE.md now says so, since narrowing that env
would make the guard silently blind.
Verified with a fresh cargo build --release -p impeccable:
- IMPECCABLE_BIN=... bun run test green end to end: core 90, oracle 1 (zero
unreviewed differences), detector 1, live 159 (157 pass, 2 skipped),
framework 186, plugin-e2e 4. Zero servers left.
- SIGKILL repro against impeccable live-server --background: 1 daemon up, 0
after with the reaper, 1 surviving with parent pid 1 under
IMPECCABLE_NO_TEST_REAPER=1. bun run test:cleanup then kills exactly that one.
- The leak test fails under IMPECCABLE_NO_TEST_REAPER=1 and passes with it.
- IMPECCABLE_E2E_ONLY=vite8-react-plain bun run test:live-e2e 4/4.
- bun run build green.
AI assistance: prepared by Claude Code under pbakaus's direction.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vau2X53xGTjjTCXWMVBoNY
This commit is contained in:
@@ -0,0 +1,128 @@
|
||||
/**
|
||||
* Test-side guard against leaked live servers.
|
||||
*
|
||||
* Any test file that starts a live server (directly, through
|
||||
* `live-server --background`, or through a full `live` boot) imports this and
|
||||
* calls `armLiveServerReaper()` once at module scope. That does three things:
|
||||
*
|
||||
* 1. stamps this process's environment with a unique marker, so every server
|
||||
* it starts from then on is identifiable as belonging to this process;
|
||||
* 2. installs exit and signal handlers that kill those servers on the ways
|
||||
* out that JavaScript can still observe (assertion failure, thrown hook,
|
||||
* Ctrl-C, `SIGTERM`);
|
||||
* 3. spawns a detached reaper holding a pipe to this process, which covers
|
||||
* the way out that JavaScript cannot observe: `SIGKILL`, a runner killed
|
||||
* mid-test, a `node:test` timeout that never reaches the `after()` hook.
|
||||
*
|
||||
* Servers spawned as direct children are also tracked by handle so the common
|
||||
* case is a cheap `child.kill()` rather than a process-table sweep.
|
||||
*/
|
||||
|
||||
import { spawn } from 'node:child_process';
|
||||
import os from 'node:os';
|
||||
import path from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import {
|
||||
PROC_ID_ENV,
|
||||
REPO_ENV,
|
||||
REPO_PATH_ENV,
|
||||
RUN_ID_ENV,
|
||||
alive,
|
||||
findLiveServers,
|
||||
killLiveServers,
|
||||
makeProcId,
|
||||
makeRunId,
|
||||
repoMarker,
|
||||
} from '../../scripts/lib/live-server-processes.mjs';
|
||||
|
||||
const REPO_ROOT = path.resolve(fileURLToPath(new URL('../..', import.meta.url)));
|
||||
const REAPER = path.join(REPO_ROOT, 'scripts', 'lib', 'test-orphan-reaper.mjs');
|
||||
|
||||
const trackedChildren = new Set();
|
||||
let procId = null;
|
||||
let reaper = null;
|
||||
|
||||
/**
|
||||
* Idempotent. Safe to call from several modules in the same process.
|
||||
* @returns {string} the process marker every server started after this inherits.
|
||||
*/
|
||||
export function armLiveServerReaper() {
|
||||
if (procId) return procId;
|
||||
|
||||
procId = makeProcId();
|
||||
process.env[PROC_ID_ENV] = procId;
|
||||
// Standalone `node --test tests/live-server.test.mjs` gets the same coverage
|
||||
// as a run through scripts/run-tests.mjs.
|
||||
if (!process.env[RUN_ID_ENV]) process.env[RUN_ID_ENV] = makeRunId(REPO_ROOT);
|
||||
if (!process.env[REPO_ENV]) process.env[REPO_ENV] = repoMarker(REPO_ROOT);
|
||||
if (!process.env[REPO_PATH_ENV]) process.env[REPO_PATH_ENV] = REPO_ROOT;
|
||||
|
||||
if (process.platform !== 'win32' && process.env.IMPECCABLE_NO_TEST_REAPER !== '1') {
|
||||
try {
|
||||
reaper = spawn(process.execPath, [REAPER, procId], {
|
||||
detached: true,
|
||||
stdio: ['pipe', 'ignore', 'ignore'],
|
||||
});
|
||||
reaper.unref();
|
||||
reaper.on('error', () => { reaper = null; });
|
||||
} catch {
|
||||
reaper = null;
|
||||
}
|
||||
}
|
||||
|
||||
process.on('exit', () => cleanupSync());
|
||||
for (const sig of ['SIGINT', 'SIGTERM', 'SIGHUP']) {
|
||||
process.on(sig, () => {
|
||||
cleanupSync();
|
||||
// The shell convention for "killed by signal N": 130 SIGINT, 143 SIGTERM,
|
||||
// 129 SIGHUP.
|
||||
process.exit(128 + (os.constants.signals[sig] ?? 0));
|
||||
});
|
||||
}
|
||||
// An uncaught exception still fires 'exit', so no separate handler is owed.
|
||||
return procId;
|
||||
}
|
||||
|
||||
/**
|
||||
* Register a live server started as a direct child, so it is killed by handle
|
||||
* on the way out instead of waiting for a process-table sweep.
|
||||
*/
|
||||
export function trackServerChild(child) {
|
||||
if (!child || typeof child.kill !== 'function') return child;
|
||||
trackedChildren.add(child);
|
||||
child.once('exit', () => trackedChildren.delete(child));
|
||||
return child;
|
||||
}
|
||||
|
||||
export function untrackServerChild(child) {
|
||||
trackedChildren.delete(child);
|
||||
}
|
||||
|
||||
/** Live servers this process started that are still running. */
|
||||
export function findLeakedLiveServers() {
|
||||
if (!procId) return [];
|
||||
return findLiveServers({ procId });
|
||||
}
|
||||
|
||||
/**
|
||||
* Synchronous best-effort cleanup. Called from `process.on('exit')`, where the
|
||||
* event loop is closed, so everything here has to be synchronous.
|
||||
*/
|
||||
export function cleanupSync() {
|
||||
for (const child of trackedChildren) {
|
||||
try { if (child.exitCode == null && child.signalCode == null) child.kill('SIGKILL'); } catch { /* gone */ }
|
||||
}
|
||||
trackedChildren.clear();
|
||||
|
||||
if (procId) {
|
||||
try {
|
||||
const leaked = findLiveServers({ procId }).filter(({ pid }) => alive(pid));
|
||||
if (leaked.length) killLiveServers(leaked, { graceMs: 250 });
|
||||
} catch { /* the sweep is a safety net, not a requirement */ }
|
||||
}
|
||||
|
||||
if (reaper) {
|
||||
try { reaper.stdin?.end(); } catch { /* already closed */ }
|
||||
reaper = null;
|
||||
}
|
||||
}
|
||||
@@ -28,6 +28,7 @@ import { fileURLToPath } from 'node:url';
|
||||
|
||||
import { runAgentLoop } from './agent.mjs';
|
||||
import { ENGINE_MISSING_MESSAGE, engineEnv, findEngineBinary } from '../lib/engine-bin.mjs';
|
||||
import { armLiveServerReaper, trackServerChild } from '../lib/live-servers.mjs';
|
||||
|
||||
const __dirname = dirname(fileURLToPath(import.meta.url));
|
||||
const REPO_ROOT = join(__dirname, '..', '..');
|
||||
@@ -35,6 +36,13 @@ const SCRIPTS_DIR = join(REPO_ROOT, 'skill', 'scripts');
|
||||
const FIXTURES_DIR = join(REPO_ROOT, 'tests', 'framework-fixtures');
|
||||
const ENGINE_BIN = findEngineBinary();
|
||||
|
||||
// Live servers here are detached daemons (`impeccable live-server --background`,
|
||||
// or a full `live` boot), orphaned to pid 1 by design; teardown() is the only
|
||||
// thing that stops them. The reaper covers the runs where teardown never
|
||||
// happens, and it stamps this process's environment before any verb runs, so
|
||||
// the markers reach the daemon through engineEnv() below.
|
||||
armLiveServerReaper();
|
||||
|
||||
export { SCRIPTS_DIR, FIXTURES_DIR, REPO_ROOT, ENGINE_BIN, ENGINE_MISSING_MESSAGE };
|
||||
|
||||
/** The engine binary, or a thrown error naming how to get one. */
|
||||
@@ -204,14 +212,14 @@ export function runInject(tmp, port, token) {
|
||||
|
||||
export function startDevServer(tmp, runtime) {
|
||||
const [cmd, ...args] = runtime.devCommand;
|
||||
const child = spawn(cmd, args, {
|
||||
const child = trackServerChild(spawn(cmd, args, {
|
||||
cwd: tmp,
|
||||
// runtime.env lets a fixture pin framework behavior. Astro 7 needs
|
||||
// ASTRO_DEV_BACKGROUND set: it auto-detects AI-agent environments and
|
||||
// daemonizes `astro dev`, which the harness reads as a crashed server.
|
||||
env: { ...process.env, FORCE_COLOR: '0', NO_COLOR: '1', ...(runtime.env || {}) },
|
||||
stdio: ['ignore', 'pipe', 'pipe'],
|
||||
});
|
||||
}));
|
||||
|
||||
const readyRe = new RegExp(runtime.readyPattern);
|
||||
const bufLog = [];
|
||||
|
||||
@@ -0,0 +1,247 @@
|
||||
/**
|
||||
* The leak guard itself.
|
||||
*
|
||||
* A live server must not outlive the test process that started it, even when
|
||||
* that process is SIGKILLed and no `after()` hook, no `finally`, and no
|
||||
* `process.on('exit')` handler ever runs. That is the shape that left 197
|
||||
* orphaned servers squatting the live suite's ports (issue: live-server leak).
|
||||
*
|
||||
* Run with: node --test tests/live-server-leak.test.mjs
|
||||
*/
|
||||
|
||||
import { describe, it } from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { spawn } from 'node:child_process';
|
||||
import { mkdtempSync, rmSync, symlinkSync, writeFileSync } from 'node:fs';
|
||||
import { join } from 'node:path';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import {
|
||||
PROC_ID_ENV,
|
||||
REPO_ENV,
|
||||
RUN_ID_ENV,
|
||||
alive,
|
||||
envLineHasEntry,
|
||||
findLiveServers,
|
||||
killLiveServers,
|
||||
makeProcId,
|
||||
makeRunId,
|
||||
repoMarker,
|
||||
} from '../scripts/lib/live-server-processes.mjs';
|
||||
import { ENGINE_MISSING_MESSAGE, engineTarget, findEngineBinary } from './lib/engine-bin.mjs';
|
||||
|
||||
// The reaper is a POSIX mechanism (a detached process holding a pipe, killed by
|
||||
// signal). armLiveServerReaper() does not arm it on Windows, so the guarantee it
|
||||
// pins is not one Windows makes yet.
|
||||
const WINDOWS = process.platform === 'win32';
|
||||
|
||||
const REPO_ROOT = fileURLToPath(new URL('..', import.meta.url));
|
||||
const PORT = 8591;
|
||||
// The server under test is the engine's `live-server` verb, not a script: this
|
||||
// guarantee has to hold for whatever binary the harness is pointed at.
|
||||
const ENGINE_BIN = findEngineBinary();
|
||||
const NO_ENGINE = ENGINE_BIN ? false : `${ENGINE_MISSING_MESSAGE} (target ${engineTarget()})`;
|
||||
|
||||
function sleep(ms) {
|
||||
return new Promise((resolve) => setTimeout(resolve, ms));
|
||||
}
|
||||
|
||||
async function waitUntil(predicate, { timeoutMs = 10_000, stepMs = 100 } = {}) {
|
||||
const deadline = Date.now() + timeoutMs;
|
||||
while (Date.now() < deadline) {
|
||||
if (await predicate()) return true;
|
||||
await sleep(stepMs);
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
/**
|
||||
* A stand-in for a test file: arms the reaper, starts a detached live server
|
||||
* exactly as `live-server --background` does, prints its pid, then blocks
|
||||
* forever so the caller can decide how it dies.
|
||||
*/
|
||||
const VICTIM = `
|
||||
import { spawn } from 'node:child_process';
|
||||
import { armLiveServerReaper } from ${JSON.stringify(join(REPO_ROOT, 'tests/lib/live-servers.mjs'))};
|
||||
|
||||
armLiveServerReaper();
|
||||
|
||||
// Detached, exactly the shape \`impeccable live-server --background\` uses: the
|
||||
// server is orphaned to pid 1 from birth and nothing but an explicit stop, or
|
||||
// the reaper, ever ends it.
|
||||
const child = spawn(${JSON.stringify('__BIN__')}, ['live-server', '--port=${PORT}'], {
|
||||
detached: true,
|
||||
stdio: 'ignore',
|
||||
cwd: process.cwd(),
|
||||
env: { ...process.env, IMPECCABLE_SKILL_DIR: ${JSON.stringify(join(REPO_ROOT, 'skill'))}, IMPECCABLE_SELF: ${JSON.stringify('__BIN__')} },
|
||||
});
|
||||
child.unref();
|
||||
|
||||
// Wait until it is actually listening before reporting, so the assertion below
|
||||
// is about a running server rather than a pid that never came up.
|
||||
const deadline = Date.now() + 15000;
|
||||
while (Date.now() < deadline) {
|
||||
try {
|
||||
const res = await fetch('http://127.0.0.1:${PORT}/health');
|
||||
if (res.ok || res.status === 401 || res.status === 404) break;
|
||||
} catch {}
|
||||
await new Promise((r) => setTimeout(r, 50));
|
||||
}
|
||||
console.log(JSON.stringify({ serverPid: child.pid }));
|
||||
setInterval(() => {}, 1000);
|
||||
`;
|
||||
|
||||
function startVictim(cwd) {
|
||||
const script = join(cwd, 'victim.mjs');
|
||||
writeFileSync(script, VICTIM.replaceAll('__BIN__', ENGINE_BIN));
|
||||
const proc = spawn(process.execPath, [script], {
|
||||
cwd,
|
||||
stdio: ['ignore', 'pipe', 'pipe'],
|
||||
env: { ...process.env, [RUN_ID_ENV]: makeRunId(REPO_ROOT), [PROC_ID_ENV]: '' },
|
||||
});
|
||||
const ready = new Promise((resolve, reject) => {
|
||||
let out = '';
|
||||
let err = '';
|
||||
const timer = setTimeout(
|
||||
() => reject(new Error(`victim never reported a server\nstdout: ${out}\nstderr: ${err}`)),
|
||||
25_000,
|
||||
);
|
||||
timer.unref();
|
||||
proc.stdout.on('data', (d) => {
|
||||
out += d.toString();
|
||||
const line = out.split('\n').find((l) => l.trim().startsWith('{'));
|
||||
if (line) {
|
||||
clearTimeout(timer);
|
||||
resolve(JSON.parse(line));
|
||||
}
|
||||
});
|
||||
proc.stderr.on('data', (d) => { err += d.toString(); });
|
||||
proc.on('exit', (code) => {
|
||||
clearTimeout(timer);
|
||||
reject(new Error(`victim exited early (code=${code})\n${err}`));
|
||||
});
|
||||
});
|
||||
return { proc, ready };
|
||||
}
|
||||
|
||||
describe('live server leak guard', () => {
|
||||
it('kills the engine live server when the test process is SIGKILLed', {
|
||||
skip: WINDOWS
|
||||
? 'the reaper is POSIX-only; armLiveServerReaper() does not arm it on win32'
|
||||
: NO_ENGINE,
|
||||
}, async () => {
|
||||
const cwd = mkdtempSync(join(tmpdir(), 'impeccable-leak-'));
|
||||
let victim;
|
||||
let serverPid;
|
||||
try {
|
||||
victim = startVictim(cwd);
|
||||
({ serverPid } = await victim.ready);
|
||||
assert.ok(alive(serverPid), 'server should be running before the kill');
|
||||
|
||||
// The case no in-process cleanup can cover.
|
||||
victim.proc.kill('SIGKILL');
|
||||
|
||||
const reaped = await waitUntil(() => !alive(serverPid), { timeoutMs: 15_000 });
|
||||
assert.equal(reaped, true, `live server pid ${serverPid} outlived the SIGKILLed test process`);
|
||||
} finally {
|
||||
if (victim?.proc && victim.proc.exitCode == null) victim.proc.kill('SIGKILL');
|
||||
if (serverPid && alive(serverPid)) killLiveServers([{ pid: serverPid, command: '' }]);
|
||||
rmSync(cwd, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it('scopes a sweep to the marker, never to the name alone', () => {
|
||||
// No marker, no match: a sweep can never take out a live server that some
|
||||
// other checkout, or the user's own session, is running.
|
||||
assert.deepEqual(findLiveServers({}), []);
|
||||
assert.deepEqual(findLiveServers({ runId: 'no-such-run-id-' + Date.now() }), []);
|
||||
});
|
||||
});
|
||||
|
||||
describe('envLineHasEntry', () => {
|
||||
// Marker values are opaque tokens, never paths: repoMarker() hashes the
|
||||
// checkout so two adjacent checkouts get unrelated values, and the run and
|
||||
// process ids are random hex. Nothing a marker can hold contains whitespace,
|
||||
// which is the invariant this matcher rests on.
|
||||
const hash = repoMarker('/work/impeccable');
|
||||
const marker = `${REPO_ENV}=${hash}`;
|
||||
|
||||
it('matches the entry at the end of the line and between other entries', () => {
|
||||
assert.equal(envLineHasEntry(`node live-server.mjs PATH=/usr/bin ${marker}`, marker), true);
|
||||
assert.equal(envLineHasEntry(`node live-server.mjs ${marker} PATH=/usr/bin`, marker), true);
|
||||
assert.equal(envLineHasEntry(`node x ${marker} ${RUN_ID_ENV}=abc PATH=/usr/bin`, marker), true);
|
||||
});
|
||||
|
||||
it('does not match a value the marker is a strict prefix of', () => {
|
||||
// The shape the hash rules out at the source, asserted anyway: a longer
|
||||
// value starting with this one must not count as this entry.
|
||||
assert.equal(envLineHasEntry(`node x ${marker}ff PATH=/usr/bin`, marker), false);
|
||||
assert.equal(envLineHasEntry(`node x ${marker}-2`, marker), false);
|
||||
assert.equal(envLineHasEntry(`node x ${REPO_ENV}=${hash}0123456789abcdef`, marker), false);
|
||||
});
|
||||
|
||||
it('gives adjacent checkouts unrelated markers', () => {
|
||||
// The bug this all exists for: /work/impeccable is a substring of
|
||||
// /work/impeccable-copy. Hashing means the two markers no longer share a
|
||||
// prefix at all, so a substring can never arise in the first place.
|
||||
const neighbour = repoMarker('/work/impeccable-copy');
|
||||
assert.notEqual(neighbour, hash);
|
||||
assert.equal(neighbour.startsWith(hash), false);
|
||||
assert.equal(envLineHasEntry(`node x ${REPO_ENV}=${neighbour} PATH=/usr/bin`, marker), false);
|
||||
});
|
||||
|
||||
it('resolves one checkout to one marker through trailing slashes and dot segments', () => {
|
||||
const real = mkdtempSync(join(tmpdir(), 'impeccable-marker-'));
|
||||
try {
|
||||
const expected = repoMarker(real);
|
||||
for (const spelling of [`${real}/`, `${real}/.`, join(real, 'sub', '..')]) {
|
||||
assert.equal(repoMarker(spelling), expected, `${spelling} should hash like ${real}`);
|
||||
}
|
||||
} finally {
|
||||
rmSync(real, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it('resolves a symlinked checkout to the same marker as its target', () => {
|
||||
const real = mkdtempSync(join(tmpdir(), 'impeccable-marker-'));
|
||||
const link = join(mkdtempSync(join(tmpdir(), 'impeccable-link-')), 'checkout');
|
||||
try {
|
||||
// Windows refuses a directory symlink without Developer Mode; a junction
|
||||
// is the equivalent it does allow. Same call shape as concept-seed's.
|
||||
symlinkSync(real, link, process.platform === 'win32' ? 'junction' : 'dir');
|
||||
assert.equal(repoMarker(link), repoMarker(real));
|
||||
assert.equal(repoMarker(`${link}/`), repoMarker(real));
|
||||
} finally {
|
||||
rmSync(link, { force: true, recursive: true });
|
||||
rmSync(real, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it('does not match when the entry name only ends with the marker name', () => {
|
||||
// The marker is a substring here, but it does not start an entry.
|
||||
assert.equal(envLineHasEntry(`node x MY_${marker} PATH=/usr/bin`, marker), false);
|
||||
assert.equal(envLineHasEntry(`node x X${marker}`, marker), false);
|
||||
});
|
||||
|
||||
it('returns false when the marker is absent', () => {
|
||||
assert.equal(envLineHasEntry('node live-server.mjs PATH=/usr/bin', marker), false);
|
||||
assert.equal(envLineHasEntry('', marker), false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('marker values', () => {
|
||||
it('generates run and process ids from a whitespace-free alphabet', () => {
|
||||
for (const value of [makeRunId('/work/impeccable'), makeProcId(), repoMarker('/work/impeccable')]) {
|
||||
assert.match(value, /^[A-Za-z0-9_-]+$/, `marker value "${value}" must not need quoting`);
|
||||
}
|
||||
});
|
||||
|
||||
it('refuses to match on a value that could break out of its entry', () => {
|
||||
// The guard that keeps the matcher's invariant honest if someone later
|
||||
// passes a path where a token is expected.
|
||||
assert.throws(
|
||||
() => findLiveServers({ runId: '/work/my repo PATH=x' }),
|
||||
/marker value must be/,
|
||||
);
|
||||
});
|
||||
});
|
||||
+11
-2
@@ -28,6 +28,15 @@ import os from 'node:os';
|
||||
import path from 'node:path';
|
||||
import { spawn, spawnSync } from 'node:child_process';
|
||||
import { fileURLToPath, pathToFileURL } from 'node:url';
|
||||
import { armLiveServerReaper, trackServerChild } from '../lib/live-servers.mjs';
|
||||
|
||||
// Daemon steps spawn the engine's `live-server` detached, so it is orphaned to
|
||||
// pid 1 the moment this process dies and stopDaemon() is the only thing that
|
||||
// ever ends it. Arm the reaper before any case runs: it stamps this process's
|
||||
// environment, and buildInvocation() below inherits process.env, so the marker
|
||||
// reaches the daemon and a SIGKILLed oracle run leaves no server behind.
|
||||
// The marker vars are read by nothing in the engine, so goldens are unchanged.
|
||||
armLiveServerReaper();
|
||||
|
||||
export const ORACLE_DIR = path.dirname(fileURLToPath(import.meta.url));
|
||||
export const REPO_ROOT = path.resolve(ORACLE_DIR, '..', '..');
|
||||
@@ -341,9 +350,9 @@ function runDaemonStep(c, opts) {
|
||||
const errPath = path.join(outDir, `${n}.stderr`);
|
||||
const outFd = fs.openSync(outPath, 'w');
|
||||
const errFd = fs.openSync(errPath, 'w');
|
||||
const child = spawn(argv[0], argv.slice(1), {
|
||||
const child = trackServerChild(spawn(argv[0], argv.slice(1), {
|
||||
cwd, env, stdio: ['ignore', outFd, errFd], detached: true, windowsHide: true,
|
||||
});
|
||||
}));
|
||||
fs.closeSync(outFd);
|
||||
fs.closeSync(errFd);
|
||||
const readyFile = path.join(opts.ws, c.readyFile);
|
||||
|
||||
@@ -0,0 +1,200 @@
|
||||
/**
|
||||
* The runner's group shutdown.
|
||||
*
|
||||
* Two properties, and the first is the one that regressed: ending a group must
|
||||
* return as soon as the child is actually gone. A wait implemented as a poll on
|
||||
* `kill(pid, 0)` cannot do that, because a dead child is a zombie until this
|
||||
* process reaps it and a blocked event loop never reaps anything. Such a wait
|
||||
* always burns its whole grace period and always ends in a needless SIGKILL.
|
||||
*
|
||||
* Run with: node --test tests/process-group.test.mjs
|
||||
*/
|
||||
|
||||
import { describe, it } from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { spawn } from 'node:child_process';
|
||||
import {
|
||||
createGroupShutdown,
|
||||
killGroupSync,
|
||||
stopGroup,
|
||||
trackChildExit,
|
||||
} from '../scripts/lib/process-group.mjs';
|
||||
|
||||
const POSIX = process.platform !== 'win32';
|
||||
|
||||
/** A child in its own process group that exits on SIGTERM, the ordinary case. */
|
||||
function spawnObedient() {
|
||||
return spawnDetached('setInterval(() => {}, 1000);');
|
||||
}
|
||||
|
||||
/** A child that traps SIGTERM and keeps running, so only SIGKILL ends it. */
|
||||
function spawnStubborn() {
|
||||
return spawnDetached("process.on('SIGTERM', () => {}); setInterval(() => {}, 1000);");
|
||||
}
|
||||
|
||||
function spawnDetached(source) {
|
||||
const child = spawn(process.execPath, ['-e', source], {
|
||||
detached: true,
|
||||
stdio: 'ignore',
|
||||
});
|
||||
return trackChildExit(child);
|
||||
}
|
||||
|
||||
function isAlive(pid) {
|
||||
try {
|
||||
process.kill(pid, 0);
|
||||
return true;
|
||||
} catch (err) {
|
||||
return err.code === 'EPERM';
|
||||
}
|
||||
}
|
||||
|
||||
describe('stopGroup', () => {
|
||||
it('returns as soon as the child exits, not after the grace period', {
|
||||
skip: POSIX ? false : 'process groups and SIGTERM are POSIX-only',
|
||||
}, async () => {
|
||||
const running = spawnObedient();
|
||||
const startedAt = Date.now();
|
||||
const outcome = await stopGroup(running, { graceMs: 5_000 });
|
||||
const elapsed = Date.now() - startedAt;
|
||||
|
||||
assert.equal(outcome, 'exited');
|
||||
// The regression this pins: a poll on liveness could not see the reaped
|
||||
// child and would have taken the full 5s.
|
||||
assert.ok(elapsed < 1_000, `expected a prompt return, took ${elapsed}ms`);
|
||||
assert.equal(running.hasExited, true);
|
||||
});
|
||||
|
||||
it('escalates to SIGKILL when the child ignores SIGTERM', {
|
||||
skip: POSIX ? false : 'process groups and SIGTERM are POSIX-only',
|
||||
}, async () => {
|
||||
const running = spawnStubborn();
|
||||
const { pid } = running.child;
|
||||
// Give the trap a moment to be installed, so the SIGTERM lands on a child
|
||||
// that really is ignoring it.
|
||||
await new Promise((r) => setTimeout(r, 250));
|
||||
|
||||
const startedAt = Date.now();
|
||||
const outcome = await stopGroup(running, { graceMs: 400, killGraceMs: 2_000 });
|
||||
const elapsed = Date.now() - startedAt;
|
||||
|
||||
assert.equal(outcome, 'killed');
|
||||
assert.ok(elapsed >= 400, `should have waited out the grace period, took ${elapsed}ms`);
|
||||
assert.equal(running.hasExited, true);
|
||||
assert.equal(isAlive(pid), false);
|
||||
});
|
||||
|
||||
it('is a no-op for a child that has already exited', async () => {
|
||||
const running = spawnObedient();
|
||||
await stopGroup(running, { graceMs: 5_000 });
|
||||
assert.equal(await stopGroup(running, { graceMs: 5_000 }), 'already-gone');
|
||||
});
|
||||
});
|
||||
|
||||
describe('killGroupSync', () => {
|
||||
it('ends a stubborn child without awaiting anything', {
|
||||
skip: POSIX ? false : 'process groups and SIGTERM are POSIX-only',
|
||||
}, async () => {
|
||||
const running = spawnStubborn();
|
||||
const { pid } = running.child;
|
||||
await new Promise((r) => setTimeout(r, 250));
|
||||
|
||||
assert.equal(killGroupSync(running), true);
|
||||
// It cannot wait, so the death is observed here rather than there.
|
||||
await running.exited;
|
||||
assert.equal(isAlive(pid), false);
|
||||
});
|
||||
|
||||
it('reports nothing to do when the child has already exited', async () => {
|
||||
const running = spawnObedient();
|
||||
await stopGroup(running, { graceMs: 5_000 });
|
||||
assert.equal(killGroupSync(running), false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('createGroupShutdown', () => {
|
||||
const posixOnly = { skip: POSIX ? false : 'process groups and SIGTERM are POSIX-only' };
|
||||
|
||||
function spy() {
|
||||
const codes = [];
|
||||
return { codes, exit: (code) => codes.push(code) };
|
||||
}
|
||||
|
||||
it('a second signal kills the group instead of abandoning the escalation', posixOnly, async () => {
|
||||
// The regression: the first handler used to clear the only reference to
|
||||
// the group before awaiting, so the second signal killed nothing and its
|
||||
// exit walked away from an escalation that was still in flight. Because
|
||||
// the suite is spawned detached, it then outlived the runner, which is the
|
||||
// exact leak this whole change exists to close.
|
||||
const running = spawnStubborn();
|
||||
const { pid } = running.child;
|
||||
await new Promise((r) => setTimeout(r, 250));
|
||||
|
||||
const { codes, exit } = spy();
|
||||
const shutdown = createGroupShutdown({ exit, graceMs: 30_000 });
|
||||
shutdown.track(running);
|
||||
|
||||
const first = shutdown.onSignal(130);
|
||||
await new Promise((r) => setTimeout(r, 150));
|
||||
|
||||
const startedAt = Date.now();
|
||||
await shutdown.onSignal(130);
|
||||
await running.exited;
|
||||
const elapsed = Date.now() - startedAt;
|
||||
|
||||
assert.equal(isAlive(pid), false, 'the second signal must end the group');
|
||||
assert.ok(elapsed < 2_000, `should not wait out the 30s grace, took ${elapsed}ms`);
|
||||
assert.equal(codes[0], 130);
|
||||
await first;
|
||||
});
|
||||
|
||||
it('ends the group on the first signal when it exits promptly', posixOnly, async () => {
|
||||
const running = spawnObedient();
|
||||
const { pid } = running.child;
|
||||
const { codes, exit } = spy();
|
||||
const shutdown = createGroupShutdown({ exit, graceMs: 5_000 });
|
||||
shutdown.track(running);
|
||||
|
||||
const startedAt = Date.now();
|
||||
await shutdown.onSignal(143);
|
||||
assert.ok(Date.now() - startedAt < 1_000);
|
||||
assert.deepEqual(codes, [143]);
|
||||
assert.equal(isAlive(pid), false);
|
||||
});
|
||||
|
||||
it('onExit still reaches a group a shutdown is in the middle of stopping', posixOnly, async () => {
|
||||
const running = spawnStubborn();
|
||||
const { pid } = running.child;
|
||||
await new Promise((r) => setTimeout(r, 250));
|
||||
|
||||
const { exit } = spy();
|
||||
const shutdown = createGroupShutdown({ exit, graceMs: 30_000 });
|
||||
shutdown.track(running);
|
||||
const inFlight = shutdown.onSignal(130);
|
||||
await new Promise((r) => setTimeout(r, 150));
|
||||
|
||||
// `current` is null by now; the handle lives in `stopping`.
|
||||
assert.equal(shutdown.onExit(), true);
|
||||
await running.exited;
|
||||
assert.equal(isAlive(pid), false);
|
||||
await inFlight;
|
||||
});
|
||||
|
||||
it('exits cleanly when no group is running', async () => {
|
||||
const { codes, exit } = spy();
|
||||
const shutdown = createGroupShutdown({ exit });
|
||||
await shutdown.onSignal(129);
|
||||
assert.deepEqual(codes, [129]);
|
||||
assert.equal(shutdown.onExit(), false);
|
||||
});
|
||||
|
||||
it('reports that it is shutting down, so a normal exit is not misread', posixOnly, async () => {
|
||||
const running = spawnObedient();
|
||||
const shutdown = createGroupShutdown({ exit: () => {}, graceMs: 5_000 });
|
||||
shutdown.track(running);
|
||||
assert.equal(shutdown.shuttingDown, false);
|
||||
const done = shutdown.onSignal(130);
|
||||
assert.equal(shutdown.shuttingDown, true);
|
||||
await done;
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user