Raise the skill-behavior timeout that was grading haste over thoroughness

`initialized natural build` looked like a third defect on main: sonnet began
implementation before the attended concept checkpoint, three runs in a row.
It is flaky, not broken, and the measurement setup was the larger problem.

A run that stops to put the concept to the user before building takes about
579s on sonnet. A run that skips the checkpoint and fails the assertion
finishes in 130-200s. The suite capped each test at 300s, so the thorough path
was killed as a timeout and the hasty path was graded as a result: the cap was
selecting for the behavior the scenario exists to forbid. Raised to 900s, with
the reasoning recorded next to the number so it is not trimmed back as a
mystery constant.

The baseline is corrected accordingly: the scenario is flaky (1 of 4), not
failing, and readers are told to check a duration against the cap before
calling a slow failure a behavioral one.

Written with AI assistance (Claude Code).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Paul Bakaus
2026-08-13 22:13:10 -04:00
co-authored by Claude Opus 5
parent 07663f5fbd
commit 65de2d294b
2 changed files with 47 additions and 18 deletions
+9 -1
View File
@@ -311,7 +311,15 @@ export const SUITES = {
commands: [
{
runner: 'node',
timeoutMs: 300000,
// 300000 was too low to measure what these scenarios assert. The
// workflow-contract turns run 20+ steps against a frontier model, and
// the *correct* path is the slow one: a run that stops to put the
// concept to the user before building was measured at 579s, while the
// runs that skipped that checkpoint and failed the assertion finished
// in 130-200s. At a 300s cap the thorough path is killed and the hasty
// path is graded, so the cap was selecting for the behavior the suite
// exists to forbid.
timeoutMs: 900000,
files: [
'tests/skill-behavior/scenarios.test.mjs',
'tests/skill-behavior/workflow-contract.test.mjs',