feat(skill): incorporate supabase/evals harness into supabase skill

Add references/agent-evals.md documenting the official supabase/evals harness: eval/experiment concepts, the tools and local-stack runtimes, run and result-viewing commands, and a mapping of harness scenarios to the skill's operating references. Route to it from the supabase "Choose the path" table and from postgres, agent-evals-and-observability, backend-engineering, and data-engineering. Add two eval cases covering the new reference and keep the generated catalog artifacts current. Closes #271 Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
2026-09-11 19:47:12 +03:00 · 2026-08-09 18:05:33 -04:00
parent 6cdc1b1c32
commit 4a5f18e435
10 changed files with 117 additions and 5 deletions
@@ -1065,7 +1065,7 @@
        "./supabase"
      ],
      "strict": false,
-      "description": "Use this skill when developing applications with Supabase, running the Supabase CLI, designing migrations and RLS policies, testing database behavior, generating client types, deploying the official self-hosted Docker stack, or administering its Postgres, Auth, Storage, Realtime, Functions, API gateway, backups, upgrades, and security. Use it for managed and self-hosted projects. Do not use it for generic PostgreSQL work with no Supabase services or conventions."
+      "description": "Use this skill when developing applications with Supabase, running the Supabase CLI, designing migrations and RLS policies, testing database behavior, generating client types, deploying the official self-hosted Docker stack, or administering its Postgres, Auth, Storage, Realtime, Functions, API gateway, backups, upgrades, and security. Use it for managed and self-hosted projects. Do not use for generic PostgreSQL work with no Supabase services or conventions."
    },
    {
      "name": "systematic-debugging",
@@ -43,6 +43,7 @@ Stop when the supported decision, evidence gaps, residual risks, and responsible
 | Interoperate with OpenTelemetry | [references/opentelemetry-genai.md](references/opentelemetry-genai.md) |
 | Need framework-specific implementation | [references/framework-routing.md](references/framework-routing.md) |
 | Exercise the methodology with safe probes | [references/synthetic-scenarios.md](references/synthetic-scenarios.md) |
+| Benchmark an agent's Supabase task competence with the official harness | [supabase agent evals reference](../supabase/references/agent-evals.md) |
 | Verify a source claim or status | [references/source-index.md](references/source-index.md) |

 ## Templates
@@ -58,4 +59,4 @@ Use [templates/eval-plan.md](templates/eval-plan.md), [templates/dataset-manifes

 ## When Not To Use

-Use [systematic-debugging](../systematic-debugging/SKILL.md) to investigate one active defect before proposing fixes. Use [verification-methodology](../verification-methodology/SKILL.md) for general evidence-backed completion claims, [secure-software-engineering](../secure-software-engineering/SKILL.md) to design preventive controls, and framework skills for SDK-specific code.
+Use [systematic-debugging](../systematic-debugging/SKILL.md) to investigate one active defect before proposing fixes. Use [verification-methodology](../verification-methodology/SKILL.md) for general evidence-backed completion claims, [secure-software-engineering](../secure-software-engineering/SKILL.md) to design preventive controls, and framework skills for SDK-specific code. To benchmark an agent's competence on Supabase tasks with the official scored scenarios, use the [supabase agent evals reference](../supabase/references/agent-evals.md) — this skill covers methodology, not a named harness.
@@ -53,6 +53,7 @@ Backend engineering is the craft of building the server-side systems that power
 ## Related Skills

 - [postgres](../postgres/SKILL.md) — diagnosing the PostgreSQL side of a database problem: configuration review, index and query-plan issues, vacuum/bloat, backups/PITR, replication and failover. Application-level data access patterns stay here; engine-level operations route there.
+- [supabase](../supabase/SKILL.md) — building on Supabase: migrations, RLS, Auth, Storage, and Edge Functions. To measure an agent's Supabase task competence, use its [agent evals harness reference](../supabase/references/agent-evals.md).

 ## Core Principles

@@ -45,6 +45,7 @@ Data engineering is the operational backbone of data-driven systems. This method
 ## Related Skills

 - [postgres](../postgres/SKILL.md) — operating a PostgreSQL server itself: configuration review, index and query-plan diagnosis, vacuum/bloat management, WAL archiving and point-in-time recovery, replication and failover, upgrades. This skill owns the engine-specific runbooks; data-engineering owns the engine-neutral methodology.
+- [supabase](../supabase/SKILL.md) — Supabase platform operations: migrations, RLS, Auth, Storage, Functions, and self-hosting. To measure an agent's Supabase task competence, use its [agent evals harness reference](../supabase/references/agent-evals.md).

 ## Core Principles

@@ -119,7 +119,7 @@
 - [spec-driven-development](spec-driven-development/SKILL.md): Spec-Driven Development (SDD) methodology for AI software factories — where structured specifications are the input, AI agents generate the code, and quality gates enforce correctness at each pipeline phase. Use when designing a spec → review → decompose → implement → verify pipeline that any AI coding tool (Claude Code, Cursor, Hermes Agent, Devin, OpenHands) can follow.
 - [strategy-frameworks](strategy-frameworks/SKILL.md): Structure organizational strategy work: strategic direction, competitive and industry analysis, growth options, capital allocation, acquisitions, and portfolio choices. Use when framing consequential choices about where to compete, how to pursue an opportunity, or how to compare strategic options.
 - [stripe](stripe/SKILL.md): Read Stripe account state from a terminal or agent: balance, payment intents, and subscriptions — and perform guarded mutations like canceling a subscription — backed by a bundled stripe-cli script that is read-only first and gates every state-changing command behind a --dry-run/--yes confirmation. Use when an agent needs to answer questions about account balance, recent payments, active subscriptions, or apply a confirmed subscription cancellation. Do not use for building Stripe payments into an application (that is Stripe integration development), managing Stripe dashboard settings, or other payment processors (that is their own tooling).
- [supabase](supabase/SKILL.md): Use this skill when developing applications with Supabase, running the Supabase CLI, designing migrations and RLS policies, testing database behavior, generating client types, deploying the official self-hosted Docker stack, or administering its Postgres, Auth, Storage, Realtime, Functions, API gateway, backups, upgrades, and security. Use it for managed and self-hosted projects. Do not use it for generic PostgreSQL work with no Supabase services or conventions.
+- [supabase](supabase/SKILL.md): Use this skill when developing applications with Supabase, running the Supabase CLI, designing migrations and RLS policies, testing database behavior, generating client types, deploying the official self-hosted Docker stack, or administering its Postgres, Auth, Storage, Realtime, Functions, API gateway, backups, upgrades, and security. Use it for managed and self-hosted projects. Do not use for generic PostgreSQL work with no Supabase services or conventions.
 - [systematic-debugging](systematic-debugging/SKILL.md): 4-phase root cause debugging protocol: understand bugs before fixing. Use for ANY technical issue — test failures, production bugs, unexpected behavior, performance problems, build failures, or integration issues. ESPECIALLY when under time pressure, when "one quick fix" seems obvious, or when previous fix attempts have failed.
 - [tailscale](bundles/tailscale/SKILL.md): Self-hosted Tailscale/Headscale ecosystem: deploy and manage a Headscale control server, configure tailscale clients, manage ACL policies, node lifecycle, subnet routing, DERP relays, and backup/migration. Use when the user mentions Tailscale, Headscale, tailnet, mesh VPN, WireGuard mesh, or self-hosted VPN infrastructure.
 - [technical-documentation](technical-documentation/SKILL.md): Create and review technical documentation, including READMEs, agent-facing instructions, API references, and CLI help. Use when documentation must help someone complete real work. Do not use for marketing copy, brand messaging, or long-form editorial content.
@@ -164,5 +164,5 @@ Diagnose in evidence order: identity/version → configuration → connections
 - **Application-level data access patterns** (connection pooling in app code, ORM usage, query construction, transactions in services) — that is [backend-engineering](../backend-engineering/SKILL.md).
 - **Schema design and data modeling** (tables, keys, normalization, dimensional models) — that is [data-architect](../data-architect/SKILL.md) and [data-engineering](../data-engineering/SKILL.md).
 - **Database methodology across engines** (backup strategy, migration patterns, analytical SQL) — that is `data-engineering`.
- **Supabase platform administration** (managed projects, CLI stack, the self-hosted Supabase stack) — that is [supabase](../supabase/SKILL.md); plain PostgreSQL operations without Supabase conventions belong here.
+- **Supabase platform administration** (managed projects, CLI stack, the self-hosted Supabase stack) — that is [supabase](../supabase/SKILL.md); plain PostgreSQL operations without Supabase conventions belong here. To measure an agent's Supabase task competence, use the skill's [agent evals harness reference](../supabase/references/agent-evals.md).
 - **Other database engines** (Redis, MongoDB, Elasticsearch, vector stores) — those stay in `data-engineering` references; this skill owns PostgreSQL only.
@@ -13,6 +13,7 @@ This skill gives your agent one evidence-led workflow for managed projects, CLI-
 | Resource | Purpose |
 |---|---|
 | `SKILL.md` | Discovery-first workflow, safety boundaries, routing, and completion checks |
+| `references/agent-evals.md` | The official supabase/evals harness: eval/experiment concepts, the two runtimes, run commands, and scenario-to-workflow mapping |
 | `references/architecture-and-boundaries.md` | Service map, keys, trust boundaries, and environment differences |
 | `references/local-development-and-cli.md` | CLI installation, project layout, local workflow, linking, and deployment |
 | `references/database-development-and-testing.md` | Migrations, declarative schemas, seeds, RLS, pgTAP, and type generation |
@@ -1,7 +1,7 @@
 ---
 name: supabase
 description: >-
-  Use this skill when developing applications with Supabase, running the Supabase CLI, designing migrations and RLS policies, testing database behavior, generating client types, deploying the official self-hosted Docker stack, or administering its Postgres, Auth, Storage, Realtime, Functions, API gateway, backups, upgrades, and security. Use it for managed and self-hosted projects. Do not use it for generic PostgreSQL work with no Supabase services or conventions.
+  Use this skill when developing applications with Supabase, running the Supabase CLI, designing migrations and RLS policies, testing database behavior, generating client types, deploying the official self-hosted Docker stack, or administering its Postgres, Auth, Storage, Realtime, Functions, API gateway, backups, upgrades, and security. Use it for managed and self-hosted projects. Do not use for generic PostgreSQL work with no Supabase services or conventions.
 license: MIT
 compatibility: Requires network access for documentation lookup. Local development requires the Supabase CLI and a Docker-compatible runtime; self-hosting requires Linux, Git, Docker Engine, and Docker Compose.
 metadata:
@@ -33,6 +33,7 @@ Treat Supabase as a Postgres-centered system with multiple independently version
 | Deploy or harden the official Docker stack | [self-hosting deployment](references/self-hosting-deployment.md) |
 | Back up, restore, update, upgrade, monitor, or recover a self-hosted instance | [administration and recovery](references/administration-and-recovery.md) |
 | Diagnose unhealthy containers, bad URLs, auth failures, drift, or migration failures | [troubleshooting](references/troubleshooting.md) |
+| Evaluate how well an agent can use Supabase with scored scenarios | [agent evals harness](references/agent-evals.md) |
 | Check claim currency or authoritative source coverage | [source index](references/source-index.md) |

 ## First read-only discovery
@@ -91,6 +91,28 @@
        "Distinguishes genuinely new-install generation from explicit post-recovery rotation.",
        "Uses config, bounded logs, observed migration progress, health convergence, and smoke checks without invented durations or universal restart claims."
      ]
+    },
+    {
+      "id": "agent-eval-harness-run",
+      "prompt": "We want to know how well our coding agent can actually use Supabase end to end. Point me at the official way to benchmark agents against Supabase tasks and run one representative scenario.",
+      "expected_output": "The agent routes to the skill's agent evals harness reference, identifies supabase/evals as the official harness, distinguishes it from the skill's own evals/evals.json, picks an eval scenario and experiment matching the workflow under evaluation, and gives exact commands to clone, install, and run it.",
+      "assertions": [
+        "Routes to references/agent-evals.md rather than inventing a local benchmark.",
+        "Distinguishes the external supabase/evals harness from this skill's own evals/evals.json output-quality contract.",
+        "Uses the real harness commands (git clone --recurse-submodules, pnpm install, pnpm eval with --eval and --experiment) without inventing flags.",
+        "Verifies scenario IDs and metadata values against the current supabase/evals repository before citing them."
+      ]
+    },
+    {
+      "id": "agent-eval-competence-comparison",
+      "prompt": "Compare two agent configurations on Supabase RLS and security work. Tell me which one is better at writing secure policies, and how you measured it.",
+      "expected_output": "The agent maps RLS and security workflows to the harness's build-rls-* and resolve-security-* scenarios, runs a head-to-head experiment comparison, and reports results from the harness viewer with the evidence the skill's verification matrix requires.",
+      "assertions": [
+        "Selects harness scenarios that exercise RLS and authorization (for example build-rls-* and resolve-security-*) instead of generic prompts.",
+        "Uses --experiment-suite or repeated --experiment flags for head-to-head comparison and reports the comparison basis.",
+        "Does not present harness scores as proof beyond what the scenario scorer observes; keeps authorization claims tied to real policy behavior.",
+        "Confirms scenario IDs against the current supabase/evals repository before running."
+      ]
    }
  ]
 }
@@ -0,0 +1,85 @@
+# Agent evals harness
+
+Read this when measuring how well an agent can use Supabase, comparing agent setups on scored Supabase scenarios, or evaluating an agent's competence on a workflow this skill already covers (RLS/security, migrations, Auth, Storage, Functions, self-hosting).
+
+## What it is
+
+[supabase/evals](https://github.com/supabase/evals) is the official harness that answers "how well can agents use Supabase across various tasks" by running agents against scored scenarios. It is an external benchmark owned by Supabase, not part of this repository: this change adds no submodule, and the harness's results do not replace this skill's `evals/evals.json` output-quality contract. The harness sources its own skills from the separate [supabase/agent-skills](https://github.com/supabase/agent-skills) repository, pinned as a git submodule inside supabase/evals; do not confuse that submodule with this catalog skill.
+
+## Concepts
+
+- An **eval** is one scenario under `evals/<id>/`. It contains `evals/<id>/PROMPT.md` (frontmatter metadata plus the task the agent sees), `evals/<id>/EVAL.ts` (a default-exported scorer), and optional starting state for two environments: `remote/` (the hosted project's state, seeded into a platform-lite mock) and `local/` (files copied into the agent's workspace).
+- An **experiment** is one agent/runtime/model setup under `experiments/<name>.ts`.
+- An **eval suite** is a named set of evals to run together; an **experiment suite** is a named set of experiments with related configurations for head-to-head comparison.
+- An **agent** is the model driver that receives the eval prompt and calls the configured tools; a **runtime** is the local Supabase-like environment and tool surface an experiment gives to the agent.
+- `platform-lite` exposes a Supabase Management API-compatible HTTP surface backed by `@supabase/lite`, so real tools such as `@supabase/mcp-server-supabase` can run against a lightweight project.
+
+Frontmatter in `evals/<id>/PROMPT.md` (stage, suite, product, topic, motivation) drives eval discovery and web-app filters. `suite` is required on every eval and is one of `benchmark`, `regression`, or `other`.
+
+## Two runtimes
+
+The harness picks the runtime per eval:
+
+- **Tools evals** run the agent against the experiment's MCP/tool surface with no filesystem (no `local/` directory and no `interface: cli`), then score the resulting project state or report.
+- **Local-stack evals** run the agent inside a Docker sandbox that has a `bash` tool, file tools, and the real Supabase CLI installed, so it can run `supabase init/start/db/test` against a real local stack. An eval uses this runtime when it ships a `local/` workspace or declares `interface: cli`; `interface` (`mcp` | `cli`) is otherwise a benchmark dimension, not the runtime switch.
+
+Local-stack evals require a running Docker daemon. Each attempt boots a fresh sandbox container on host networking and mounts the host Docker socket, so `supabase start` spawns the stack as sibling containers on the sandbox's `127.0.0.1`. Supabase default ports (54321-54329) must be free — stop any local `supabase start` stacks before running. A `services:` frontmatter list limits which local-stack services the scenario starts (an empty list starts only the database; omitting the key starts the full stack), and `cliVersion: x.y.z` pins a specific Supabase CLI release. After the agent finishes, the harness copies the workspace out of the sandbox so scorers can run host tooling (for example repo-root `vite`/`vitest`) against the produced files; scorers may also run commands and SQL inside the sandbox against the live stack.
+
+## How to run
+
+```sh
+git clone --recurse-submodules git@github.com:supabase/evals.git
+cd evals
+pnpm install
+cp .env.example .env   # add the provider key(s) agent-backed runs need
+pnpm eval -- --eval resolve-dataapi-001-empty-results --experiment claude-code-sonnet-5
+```
+
+`--eval`, `--experiment`, `--suite`, and `--experiment-suite` each accept repeated flags and comma-separated values. Example selections:
+
+```sh
+pnpm eval -- \
+  --experiment claude-code-sonnet-5 \
+  --experiment claude-code-opus-5 \
+  --eval build-rls-002-own-todos-client \
+  --eval resolve-security-001-rls-cross-user-leak
+
+pnpm eval -- --suite benchmark --experiment-suite benchmark,no-skills
+```
+
+Runs write local result files under `results/`. After running, export and view results in the harness web app:
+
+```sh
+pnpm export-results   # writes eval-results.json for the web app
+pnpm web
+```
+
+The sandbox plumbing has its own smoke test (`pnpm --filter @supabase-evals/sandbox test:docker`), and `pnpm check` runs typechecks plus local smoke tests. Verify current scenario IDs, commands, and metadata values against the repo before citing them; the harness evolves.
+
+## Mapping to this skill's workflows
+
+The harness's `evals/` scenarios map directly onto the skill's operating references, which is what makes running them actionable from this skill's contract:
+
+| Skill workflow | Reference | Example eval scenarios |
+|---|---|---|
+| RLS, security, and authorization testing | [database development and testing](database-development-and-testing.md) | `build-rls-002-own-todos-client`, `build-rls-003-org-roles-permissions`, `build-security-001-public-table`, `resolve-security-001-rls-cross-user-leak`, `resolve-security-002-rls-cross-tenant-leak`, `build-tests-001-rls-tenant-isolation` |
+| Migrations, bootstrap, and schema authoring | [database development and testing](database-development-and-testing.md), [local development and CLI](local-development-and-cli.md) | `build-cli-001-bootstrap-app`, `build-cli-002-declarative-schema`, `build-database-001-migrate-postgres-to-supabase`, `resolve-database-001-migration-history-mismatch` |
+| Auth | [application services](application-services.md) | `build-auth-001-email-password-flow`, `investigate-auth-001-deleted-user-access`, `build-functions-005-dual-auth-user-secret` |
+| Storage | [application services](application-services.md) | `build-storage-001-private-bucket-access`, `resolve-storage-001-upsert-missing-update-policy` |
+| Edge Functions | [application services](application-services.md) | `build-functions-002-edge-auth-db`, `build-functions-004-service-role-bypass`, `deploy-functions-001-edge-function-secrets`, `investigate-functions-001-546-resource-limit` |
+| Realtime | [application services](application-services.md) | `build-realtime-001-live-chat-updates`, `investigate-realtime-001-subscribed-no-events` |
+| Self-hosting | [self-hosting deployment](self-hosting-deployment.md), [administration and recovery](administration-and-recovery.md) | `deploy-self-hosting-001-docker-compose`, `deploy-database-001-prometheus-metrics` |
+| Troubleshooting and diagnosis | [troubleshooting](troubleshooting.md) | `resolve-dataapi-001-empty-results`, `investigate-db-001-table-row-counts`, `investigate-logs-001-top-error-function`, `resolve-reliability-001-unhealthy-project-recovery` |
+
+Use the scenario set that matches the workflow under evaluation, and prefer scenarios that exercise the delivery boundary the skill requires (real authorization paths, migration replay, live stack services) over scenarios that can be gamed with unverified claims.
+
+## Relationship to this skill's evals
+
+- The harness is an external benchmark: it measures agent task competence on scored scenarios and is owned by Supabase.
+- `supabase/evals/evals.json` in this catalog is this repository's schema-versioned output-quality contract for the skill's own cases.
+- They are complementary, not duplicates. Run the harness when the question is "how well can an agent use Supabase?"; keep the catalog manifest as the regression contract for this skill's documented behaviors.
+
+## When not to use
+
+- Do not use this reference to design evaluation methodology, datasets, or graders — that is [agent-evals-and-observability](../../agent-evals-and-observability/SKILL.md).
+- Do not confuse supabase/evals with this skill's own `evals/evals.json`: the former runs agents against scored scenarios, the latter is a static output-quality contract validated by this repository.