Export portable Live evidence bundles

AI-assisted implementation under maintainer direction.
This commit is contained in:
Paul Bakaus
2026-07-13 13:56:25 -07:00
parent 7a99e1725d
commit e0c19eff28
7 changed files with 161 additions and 12 deletions
+35
View File
@@ -120,3 +120,38 @@ When `preActions` is omitted, steer smoke inherits `runtime.preActions` to revea
| `nuxt-csp/` | Nuxt `routeRules` with literal CSP header in `nuxt.config.ts`. CSP shape `append-string`. |
Add new fixtures by cloning a directory, swapping files, and updating `fixture.json`.
## External quality-eval fixtures
The public Live benchmark can execute a fixture owned by another repository
without copying its task corpus or rubric into Impeccable:
```sh
bun run bench:live -- \
--fixture-dir=/absolute/path/to/private-fixture \
--agent=codex \
--action=bolder \
--iterations=1 \
--evidence-bundle=/absolute/path/to/output-bundle
```
An external fixture has the same shape as a directory in this folder:
`fixture.json`, `gitignore.txt`, and `files/`. Use the optional
`evidenceCapture` block in `fixture.json` for rubric-free capture metadata:
```json
{
"evidenceCapture": {
"captureSelector": "section.case-study",
"viewport": { "width": 1440, "height": 1080 },
"action": "bolder"
}
}
```
The bundle contains `report.json`, the original capture, each progressively
delivered variant capture, geometry/overflow facts, hashes, and timing data.
It deliberately cannot run `--judge-rendered`; comparative rubrics, private
fixtures, human calibration, and quality decisions belong in the consuming
evaluation harness. The normal public E2E suite remains responsible for Live
protocol, framework, source-commit, cleanup, and recovery correctness.