magnus919_agent-skills/supabase/references/agent-evals.md

Agent evals harness

Read this when measuring how well an agent can use Supabase, comparing agent setups on scored Supabase scenarios, or evaluating an agent's competence on a workflow this skill already covers (RLS/security, migrations, Auth, Storage, Functions, self-hosting).

What it is

supabase/evals is the official harness that answers "how well can agents use Supabase across various tasks" by running agents against scored scenarios. It is an external benchmark owned by Supabase, not part of this repository: this change adds no submodule, and the harness's results do not replace this skill's evals/evals.json output-quality contract. The harness sources its own skills from the separate supabase/agent-skills repository, pinned as a git submodule inside supabase/evals; do not confuse that submodule with this catalog skill.

Concepts

• An eval is one scenario under evals/<id>/. It contains evals/<id>/PROMPT.md (frontmatter metadata plus the task the agent sees), evals/<id>/EVAL.ts (a default-exported scorer), and optional starting state for two environments: remote/ (the hosted project's state, seeded into a platform-lite mock) and local/ (files copied into the agent's workspace).
• An experiment is one agent/runtime/model setup under experiments/<name>.ts.
• An eval suite is a named set of evals to run together; an experiment suite is a named set of experiments with related configurations for head-to-head comparison.
• An agent is the model driver that receives the eval prompt and calls the configured tools; a runtime is the local Supabase-like environment and tool surface an experiment gives to the agent.
• platform-lite exposes a Supabase Management API-compatible HTTP surface backed by @supabase/lite, so real tools such as @supabase/mcp-server-supabase can run against a lightweight project.

Frontmatter in evals/<id>/PROMPT.md (stage, suite, product, topic, motivation) drives eval discovery and web-app filters. suite is required on every eval and is one of benchmark, regression, or other.

Two runtimes

The harness picks the runtime per eval:

• Tools evals run the agent against the experiment's MCP/tool surface with no filesystem (no local/ directory and no interface: cli), then score the resulting project state or report.
• Local-stack evals run the agent inside a Docker sandbox that has a bash tool, file tools, and the real Supabase CLI installed, so it can run supabase init/start/db/test against a real local stack. An eval uses this runtime when it ships a local/ workspace or declares interface: cli; interface (mcp | cli) is otherwise a benchmark dimension, not the runtime switch.

Local-stack evals require a running Docker daemon. Each attempt boots a fresh sandbox container on host networking and mounts the host Docker socket, so supabase start spawns the stack as sibling containers on the sandbox's 127.0.0.1. Supabase default ports (54321-54329) must be free — stop any local supabase start stacks before running. A services: frontmatter list limits which local-stack services the scenario starts (an empty list starts only the database; omitting the key starts the full stack), and cliVersion: x.y.z pins a specific Supabase CLI release. After the agent finishes, the harness copies the workspace out of the sandbox so scorers can run host tooling (for example repo-root vite/vitest) against the produced files; scorers may also run commands and SQL inside the sandbox against the live stack.

How to run

git clone --recurse-submodules git@github.com:supabase/evals.git
cd evals
pnpm install
cp .env.example .env   # add the provider key(s) agent-backed runs need
pnpm eval -- --eval resolve-dataapi-001-empty-results --experiment claude-code-sonnet-5

--eval, --experiment, --suite, and --experiment-suite each accept repeated flags and comma-separated values. Example selections:

pnpm eval -- \
  --experiment claude-code-sonnet-5 \
  --experiment claude-code-opus-5 \
  --eval build-rls-002-own-todos-client \
  --eval resolve-security-001-rls-cross-user-leak

pnpm eval -- --suite benchmark --experiment-suite benchmark,no-skills

Runs write local result files under results/. After running, export and view results in the harness web app:

pnpm export-results   # writes eval-results.json for the web app
pnpm web

The sandbox plumbing has its own smoke test (pnpm --filter @supabase-evals/sandbox test:docker), and pnpm check runs typechecks plus local smoke tests. Verify current scenario IDs, commands, and metadata values against the repo before citing them; the harness evolves.

Mapping to this skill's workflows

The harness's evals/ scenarios map directly onto the skill's operating references, which is what makes running them actionable from this skill's contract:

Skill workflow	Reference	Example eval scenarios
RLS, security, and authorization testing	database development and testing	build-rls-002-own-todos-client, build-rls-003-org-roles-permissions, build-security-001-public-table, resolve-security-001-rls-cross-user-leak, resolve-security-002-rls-cross-tenant-leak, build-tests-001-rls-tenant-isolation
Migrations, bootstrap, and schema authoring	database development and testing, local development and CLI	build-cli-001-bootstrap-app, build-cli-002-declarative-schema, build-database-001-migrate-postgres-to-supabase, resolve-database-001-migration-history-mismatch
Auth	application services	build-auth-001-email-password-flow, investigate-auth-001-deleted-user-access, build-functions-005-dual-auth-user-secret
Storage	application services	build-storage-001-private-bucket-access, resolve-storage-001-upsert-missing-update-policy
Edge Functions	application services	build-functions-002-edge-auth-db, build-functions-004-service-role-bypass, deploy-functions-001-edge-function-secrets, investigate-functions-001-546-resource-limit
Realtime	application services	build-realtime-001-live-chat-updates, investigate-realtime-001-subscribed-no-events
Self-hosting	self-hosting deployment, administration and recovery	deploy-self-hosting-001-docker-compose, deploy-database-001-prometheus-metrics
Troubleshooting and diagnosis	troubleshooting	resolve-dataapi-001-empty-results, investigate-db-001-table-row-counts, investigate-logs-001-top-error-function, resolve-reliability-001-unhealthy-project-recovery

Use the scenario set that matches the workflow under evaluation, and prefer scenarios that exercise the delivery boundary the skill requires (real authorization paths, migration replay, live stack services) over scenarios that can be gamed with unverified claims.

Relationship to this skill's evals

• The harness is an external benchmark: it measures agent task competence on scored scenarios and is owned by Supabase.
• supabase/evals/evals.json in this catalog is this repository's schema-versioned output-quality contract for the skill's own cases.
• They are complementary, not duplicates. Run the harness when the question is "how well can an agent use Supabase?"; keep the catalog manifest as the regression contract for this skill's documented behaviors.

When not to use

• Do not use this reference to design evaluation methodology, datasets, or graders — that is agent-evals-and-observability.
• Do not confuse supabase/evals with this skill's own evals/evals.json: the former runs agents against scored scenarios, the latter is a static output-quality contract validated by this repository.

Attribution

Concepts, runtime descriptions, and commands in this reference are derived from the supabase/evals README, Copyright Supabase, licensed under Apache-2.0. Definitions are paraphrased or quoted for documentation; the harness and its skills submodule remain external to this repository.