magnus919_agent-skills/supabase/references/troubleshooting.md

# Troubleshooting

Read this when the local CLI stack or self-hosted deployment fails. Diagnose one layer at a time; do not restart or delete state before reading current evidence.

## Layered sequence

1. **Recent change:** inspect the diff in application migrations, `config.toml`, Compose, `.env.example` merge, gateway templates, image tags, proxy, DNS, and certificates.
2. **Resolved model:** `supabase --version`, `docker compose config --quiet`, active `COMPOSE_FILE`, services, ports, volumes, and interpolated non-secret settings.
3. **Container state:** `docker compose ps --format json`; inspect exit, health, restart count, and OOM state.
4. **Bounded logs:** `docker compose logs --tail=100 SERVICE`; use the official `tests/test-container-logs.sh` for the self-hosted stack.
5. **Internal health:** database, gateway, and failing service from inside the Compose network.
6. **Gateway route:** use the expected API key role and inspect status plus body.
7. **External boundary:** DNS, TLS, proxy headers, WebSocket upgrade, firewall, and callback URLs.
8. **Application semantics:** migration history, grants, RLS, user claims, Storage policy, Realtime publication, and Function code.

After two genuinely different approaches fail, report the evidence and ask before escalating to destructive recovery or security-boundary changes.

## CLI local stack

### `supabase start` fails

- Confirm Docker-compatible runtime is running and accessible.
- Check host RAM allocation and port conflicts.
- Verify the project was initialized and the command runs from the correct root.
- Read the failing container logs and CLI version before upgrading/restarting.
- Preserve uncommitted schema/data before `supabase stop --no-backup`.

### `db diff` reports no changes

If declarative schemas are configured, `db diff` reads `supabase/schemas/`, not ad hoc Studio/SQL changes. Apply the change to schema files. If the project uses imperative migrations, confirm the local database differs from migration history and pass scope explicitly.

### `db reset` fails

The first failing migration and SQL error are the root evidence. Fix the migration or ordering/dependency issue and replay from scratch. Do not patch only the resulting database; the migration chain is the deliverable.

### Local/remote migration mismatch

Run `supabase migration list`. Determine whether SQL is missing, remote schema drifted, or history alone is wrong. Use `db pull` for real schema drift. Use `migration repair` only after proving the actual SQL state; bookkeeping changes can hide defects.

## Self-hosted containers

### Service remains `created`, unhealthy, or restarting

On first boot, Postgres and service-owned migrations can outlast the initial healthcheck retry window. `run.sh start` may exit nonzero even though containers are still making forward progress and later become healthy. Read the migration logs and re-check health before restarting; distinguish slow initialization from a persistent error.

- Run `docker compose config --quiet` and `ps` first.
- Read its logs and dependency health; `depends_on` readiness works only where the official health conditions are present.
- Check database credentials, URL values, and mounted file permissions.
- Check host memory/OOM and disk/inode pressure.
- For Kong entrypoint errors, verify LF line endings; CRLF checkouts break shell entrypoints.
- For rootless Docker/Podman logging, verify `DOCKER_SOCKET_LOCATION` and current Compose interpolation support.

Do not delete volumes to clear a health failure.

### Auth/OAuth links point to the wrong place

Current self-hosted `API_EXTERNAL_URL` includes `/auth/v1`. Confirm `SUPABASE_PUBLIC_URL`, `API_EXTERNAL_URL`, `SITE_URL`, additional redirects, proxy scheme/host, and provider-console callback. Recreate services after environment changes.

### REST root returns `403` with public key

Current self-hosted gateway restricts `/rest/v1/` OpenAPI root to administrative keys. Test an intended table route with the publishable key and expected RLS role before diagnosing PostgREST as unavailable.

### RLS returns no rows or permits too much

- Confirm RLS is enabled on the exact table.
- Inspect grants and policies for the current role/operation.
- Test `auth.uid()`/claims and anonymous `null` behavior.
- Check `USING` versus `WITH CHECK`.
- Ensure the test is not using service role.
- Re-query after writes; an unauthorized update can be a zero-row no-op.

### Realtime connects but emits no changes

Check table publication, RLS visibility, user session/key, committed database change, Realtime logs, and proxy WebSocket forwarding. An HTTP 200 on a route is not subscription proof.

### Storage metadata works but objects are missing

Database restore recovers metadata/policies, not object bytes. Check the filesystem/S3 backend, `GLOBAL_S3_BUCKET`, credentials, endpoint/path style, volume mount, and separate object restore. Verify object checksum through the public/signed path.

### Functions ignore code or environment changes

Restart Functions for mounted code changes. Recreate for environment or secret changes. Verify the mounted path, gateway invocation, global JWT-verification setting, and Function logs.

## Upgrade and recovery failures

### Postgres 17 will not start

Never attach Postgres 17 to an old Postgres 15 data directory. For an actual upgrade, use the official script and preserved rollback data. A leftover `db-config` volume can break a genuinely fresh deployment, but deleting it also deletes custom config and the pgsodium key; only remove it after proving there is no data/Vault recovery value.

### Upgrade runs out of space

Stop and preserve state. The Postgres upgrade requires the source, copied destination, cached binaries, and staging space. Roll back, add capacity or move `TMPDIR`, revalidate `2x database size + 5 GB`, then retry.

### Restore fails across versions

Inventory Postgres version, extensions, service-internal tables/columns, and unsupported settings. Use `supabase db dump` rather than raw `pg_dump` for Supabase-aware filtering. A diagnostic non-transactional attempt may reveal all mismatches, but the final accepted restore must use error-stop and transactional behavior, followed by service-level checks.

## Evidence to report

Report target/environment, CLI or Docker release/commit, exact failing layer, command and exit status, redacted status/log excerpts, recent relevant change, what was verified, and the safest next step. Never include `.env`, keys, passwords, JWTs, email-provider secrets, or complete credential-bearing URLs.