mirror of
https://github.com/pbakaus/impeccable.git
synced 2026-09-19 17:46:36 +03:00
Classify failed accepts as errors, and fix parallel lane race/all misuse
Two of the three new findings, plus the bug that chasing them exposed in my own earlier fix. The third is mitigated rather than broken; details below. Failed accepts reported success: live/completion.mjs only classifies a result as `error` when it carries `mode: 'error'`. Everything else unhandled falls through to `agent_done` with an ok ack, which is deliberate for the documented fallback paths (two tests pin it) but wrong for a real failure. So `accept_receipt_conflict` reported success, and reference/live.md's `handled: false` without `mode` bullet told the agent to "read file, find markers, edit" — hand-applying a second accept on top of the one the receipt already recorded. The same hole swallowed `source_locked`, which is mine: the earlier commit made lock contention return clean JSON so the agent could retry, but the classifier turned that failure into agent_done/ok, so the accept was dequeued and silently lost. Mark genuine failures with `mode: 'error'` through one `operationFailure` helper, and give live.md a `mode: "error"` bullet with per-error guidance: retry the same command on `source_locked`, never hand-edit, and on a receipt conflict report what the session actually resolved to. The deliberate fallback and markers-not-found handoffs stay untouched. parallel-compact lane orchestration: `Promise.race` settles on the first *settlement*, so one lane failing fast rejected the whole first-variant step while two lanes were still on their way to succeeding. `Promise.any` now takes the first success and only a total wipeout is fatal, reporting every lane's reason. The tail step's `Promise.all` surfaced a raw lane error non-deterministically; `Promise.allSettled` now reports how many lanes failed and why. Added a `requestImpl` seam so lane orchestration is testable without a provider key. Not a defect: the browser releasing Accept before the source write. That is the intended optimistic design, and it is safe because poll-lanes ranks accept at priority 0 against generate at 2, so a queued accept is always leased before a generate the user queues afterwards, even if the generate arrived first. Its source write lands inside the poll script before the next generate preflights. That invariant is load-bearing and had no tests at all; poll-lanes.mjs now has a suite covering it plus lease and type filtering. Prepared with AI assistance under maintainer direction. Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,99 @@
|
||||
/**
|
||||
* Tests for live/poll-lanes.mjs — which pending event a poll gets next.
|
||||
* Run with: node --test tests/live-poll-lanes.test.mjs
|
||||
*/
|
||||
|
||||
import { describe, it } from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
|
||||
import { eventPriority, selectAvailablePendingEvent } from '../skill/scripts/live/poll-lanes.mjs';
|
||||
|
||||
const entry = (type, seq, leaseUntil = 0, id = type + seq) => ({ event: { id, type }, leaseUntil, seq });
|
||||
|
||||
describe('poll lane priority', () => {
|
||||
it('puts terminal user actions ahead of generation', () => {
|
||||
for (const type of ['accept', 'discard', 'exit']) {
|
||||
assert.ok(
|
||||
eventPriority({ type }) < eventPriority({ type: 'generate' }),
|
||||
`${type} must outrank generate`,
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
it('ranks unknown event types last rather than first', () => {
|
||||
assert.ok(eventPriority({ type: 'something-new' }) > eventPriority({ type: 'generate' }));
|
||||
assert.ok(eventPriority({}) > eventPriority({ type: 'generate' }));
|
||||
});
|
||||
|
||||
// This is what makes the browser's optimistic Accept safe. The browser returns
|
||||
// to PICKING as soon as /events durably journals the accept, before the source
|
||||
// write happens, so the user can pick and hit Go while the accept is still
|
||||
// queued. If that generate were leased first, its preflight would wrap source
|
||||
// that still contains the previous session's variant markers.
|
||||
it('delivers a queued accept before a generate the user queued afterwards', () => {
|
||||
const selected = selectAvailablePendingEvent([
|
||||
entry('accept', 1),
|
||||
entry('generate', 2),
|
||||
]);
|
||||
assert.equal(selected.event.type, 'accept');
|
||||
});
|
||||
|
||||
it('delivers the accept first even when the generate was queued earlier', () => {
|
||||
const selected = selectAvailablePendingEvent([
|
||||
entry('generate', 1),
|
||||
entry('accept', 2),
|
||||
]);
|
||||
assert.equal(
|
||||
selected.event.type,
|
||||
'accept',
|
||||
'priority must beat arrival order, or a slow poller preflights against stale source',
|
||||
);
|
||||
});
|
||||
|
||||
it('breaks ties within one lane by arrival order', () => {
|
||||
const selected = selectAvailablePendingEvent([
|
||||
entry('generate', 7),
|
||||
entry('generate', 3),
|
||||
]);
|
||||
assert.equal(selected.seq, 3);
|
||||
});
|
||||
});
|
||||
|
||||
describe('poll lane availability', () => {
|
||||
it('skips an entry whose lease is still held', () => {
|
||||
const now = 1_000_000;
|
||||
const selected = selectAvailablePendingEvent([
|
||||
entry('accept', 1, now + 30_000),
|
||||
entry('generate', 2),
|
||||
], { now });
|
||||
assert.equal(selected.event.type, 'generate', 'a leased accept must not be handed out twice');
|
||||
});
|
||||
|
||||
it('re-offers an entry once its lease has expired', () => {
|
||||
const now = 1_000_000;
|
||||
const selected = selectAvailablePendingEvent([entry('accept', 1, now - 1)], { now });
|
||||
assert.equal(selected.event.type, 'accept');
|
||||
});
|
||||
|
||||
it('returns null when everything is leased', () => {
|
||||
const now = 1_000_000;
|
||||
assert.equal(selectAvailablePendingEvent([entry('accept', 1, now + 5_000)], { now }), null);
|
||||
});
|
||||
|
||||
it('returns null for an empty queue', () => {
|
||||
assert.equal(selectAvailablePendingEvent([]), null);
|
||||
});
|
||||
|
||||
it('restricts delivery to the requested types', () => {
|
||||
const entries = [entry('accept', 1), entry('generate', 2)];
|
||||
assert.equal(selectAvailablePendingEvent(entries, { types: ['generate'] }).event.type, 'generate');
|
||||
assert.equal(selectAvailablePendingEvent(entries, { types: new Set(['generate']) }).event.type, 'generate');
|
||||
assert.equal(selectAvailablePendingEvent(entries, { types: ['steer'] }), null);
|
||||
});
|
||||
|
||||
it('ignores an empty or absent type filter instead of starving the queue', () => {
|
||||
const entries = [entry('generate', 1)];
|
||||
assert.equal(selectAvailablePendingEvent(entries, { types: null }).event.type, 'generate');
|
||||
assert.equal(selectAvailablePendingEvent(entries, {}).event.type, 'generate');
|
||||
});
|
||||
});
|
||||
@@ -3,6 +3,7 @@ import { describe, it } from 'node:test';
|
||||
|
||||
import {
|
||||
STRATEGIES,
|
||||
createProviderLiveAgent,
|
||||
assembleProgressiveOutput,
|
||||
applyRuntimeSourceScore,
|
||||
estimateCostUsd,
|
||||
@@ -113,3 +114,67 @@ describe('cross-provider Live benchmark', () => {
|
||||
assert.equal(summary.estimatedCostUsd, 0.3);
|
||||
});
|
||||
});
|
||||
|
||||
describe('parallel-compact lane orchestration', () => {
|
||||
const laneOutput = (lane) => ({
|
||||
scopedCss: `@scope ([data-impeccable-variant="1"]) { .${lane} { color: var(--color-ink); } }`,
|
||||
variants: [{ innerHtml: VARIANT, params: [] }],
|
||||
});
|
||||
|
||||
// requestImpl stands in for the model call so lane timing/failure is exact.
|
||||
const agentWith = (behaviour) => createProviderLiveAgent({
|
||||
provider: 'anthropic',
|
||||
model: 'test-model',
|
||||
strategy: 'parallel-compact',
|
||||
liveSpec: '',
|
||||
requestImpl: ({ lane }) => behaviour(lane),
|
||||
});
|
||||
|
||||
const after = (ms, value) => new Promise((resolve) => setTimeout(() => resolve(value), ms));
|
||||
const failAfter = (ms, message) => new Promise((_, reject) => setTimeout(() => reject(new Error(message)), ms));
|
||||
|
||||
it('returns a slower lane rather than rejecting on the lane that fails first', async () => {
|
||||
// Promise.race settles on the first *settlement*, so the fast failure below
|
||||
// used to reject the whole first-variant step while two lanes were still on
|
||||
// their way to succeeding.
|
||||
const agent = agentWith((lane) => (
|
||||
lane === 'hierarchy' ? failAfter(2, 'hierarchy lane failed') : after(30, laneOutput(lane))
|
||||
));
|
||||
const first = await agent.generateFirstVariant({ id: 'par1', count: 3 });
|
||||
assert.ok(first.variants?.[0], 'a successful lane must still produce variant 1');
|
||||
});
|
||||
|
||||
it('fails with every reason when all lanes fail', async () => {
|
||||
const agent = agentWith((lane) => failAfter(2, `${lane} lane failed`));
|
||||
await assert.rejects(
|
||||
() => agent.generateFirstVariant({ id: 'par2', count: 3 }),
|
||||
(err) => {
|
||||
assert.match(err.message, /every parallel lane failed for par2/);
|
||||
for (const lane of ['hierarchy', 'layout', 'density']) {
|
||||
assert.match(err.message, new RegExp(`${lane} lane failed`), `${lane} reason must be reported`);
|
||||
}
|
||||
return true;
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
it('reports a late lane failure as a lane failure, not a raw rejection', async () => {
|
||||
// The tail step used to Promise.all the same lane promises, so a lane that
|
||||
// rejected after another won the race surfaced its bare error from here.
|
||||
const agent = agentWith((lane) => (
|
||||
lane === 'density' ? failAfter(40, 'density lane failed') : after(2, laneOutput(lane))
|
||||
));
|
||||
await agent.generateFirstVariant({ id: 'par3', count: 3 });
|
||||
await assert.rejects(
|
||||
() => agent.generateRemainingVariants({ id: 'par3', count: 3 }),
|
||||
/1 of 3 parallel lanes failed for par3.*density lane failed/s,
|
||||
);
|
||||
});
|
||||
|
||||
it('assembles all three lanes when every lane succeeds', async () => {
|
||||
const agent = agentWith((lane) => after(lane === 'layout' ? 1 : 10, laneOutput(lane)));
|
||||
await agent.generateFirstVariant({ id: 'par4', count: 3 });
|
||||
const output = await agent.generateRemainingVariants({ id: 'par4', count: 3 });
|
||||
assert.equal(output.variants.length, 3);
|
||||
});
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user