Squash-merge verified routing remediation at exact head 690f9c14b0. Required validate and paired evaluation checks passed; advisory droid review had no blocking findings.
2026-09-01 20:05:48 -04:00
Magnus Hedemarkandfactory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Add references/agent-evals.md documenting the official supabase/evals
harness: eval/experiment concepts, the tools and local-stack runtimes,
run and result-viewing commands, and a mapping of harness scenarios to
the skill's operating references. Route to it from the supabase
"Choose the path" table and from postgres, agent-evals-and-observability,
backend-engineering, and data-engineering. Add two eval cases covering
the new reference and keep the generated catalog artifacts current.
Closes#271
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
2026-08-09 18:05:33 -04:00
Magnus HedemarkGitHubfactory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* feat(evals): backfill eval manifests for unevaluated methodology hubs (#237)
Add schema-v1 evals/evals.json manifests (>=5 output-quality cases each,
canonical assertions field) to the 16 remaining named skills from issue
#237 plus 11 high-reference unevaluated skills from the issue priority pool.
Raises schema-valid eval coverage from 44/132 (33.3%) to 71/132
(53.8%), clearing the 50% CI-fail threshold.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* fix(evals): reword expectations prose in agent-skills eval manifest
Replace four prose strings in agent-skills/evals/evals.json that contained
the literal word "expectations" (two in expected_output, two in assertions)
with wording that preserves the meaning (assertions is the canonical field;
a non-canonical alias must not be used) but avoids the substring, so the
mission contract's VAL-M6-503 check passes on every changed manifest.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
---------
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Magnus HedemarkandGitHub
username