mirror of
https://github.com/pbakaus/impeccable.git
synced 2026-09-11 21:57:14 +03:00
Raise the skill-behavior timeout that was grading haste over thoroughness
`initialized natural build` looked like a third defect on main: sonnet began implementation before the attended concept checkpoint, three runs in a row. It is flaky, not broken, and the measurement setup was the larger problem. A run that stops to put the concept to the user before building takes about 579s on sonnet. A run that skips the checkpoint and fails the assertion finishes in 130-200s. The suite capped each test at 300s, so the thorough path was killed as a timeout and the hasty path was graded as a result: the cap was selecting for the behavior the scenario exists to forbid. Raised to 900s, with the reasoning recorded next to the number so it is not trimmed back as a mystery constant. The baseline is corrected accordingly: the scenario is flaky (1 of 4), not failing, and readers are told to check a duration against the cap before calling a slow failure a behavioral one. Written with AI assistance (Claude Code). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
07663f5fbd
commit
65de2d294b
@@ -311,7 +311,15 @@ export const SUITES = {
|
||||
commands: [
|
||||
{
|
||||
runner: 'node',
|
||||
timeoutMs: 300000,
|
||||
// 300000 was too low to measure what these scenarios assert. The
|
||||
// workflow-contract turns run 20+ steps against a frontier model, and
|
||||
// the *correct* path is the slow one: a run that stops to put the
|
||||
// concept to the user before building was measured at 579s, while the
|
||||
// runs that skipped that checkpoint and failed the assertion finished
|
||||
// in 130-200s. At a 300s cap the thorough path is killed and the hasty
|
||||
// path is graded, so the cap was selecting for the behavior the suite
|
||||
// exists to forbid.
|
||||
timeoutMs: 900000,
|
||||
files: [
|
||||
'tests/skill-behavior/scenarios.test.mjs',
|
||||
'tests/skill-behavior/workflow-contract.test.mjs',
|
||||
|
||||
Reference in New Issue
Block a user