mirror of
https://github.com/neondatabase/postgres-skills.git
synced 2026-09-11 19:46:49 +03:00
Merge pull request #1 from neondatabase/add-postgres-best-practices-references
Add comprehensive PostgreSQL best practices references
This commit is contained in:
sav-maya
@@ -7,8 +7,36 @@ description: Best practices and guidelines for working with Postgres. Covers sch
|
||||
|
||||
Guidelines and best practices for working with Postgres, covering schema design, indexing, query optimization, and common pitfalls.
|
||||
|
||||
## Supported Versions
|
||||
|
||||
This skill covers PostgreSQL 14 through 18. Version-specific features are tagged (e.g., `[PG15+]`, `[PG18+]`); environment-dependent examples identify required privileges, extensions, or multi-node setup.
|
||||
|
||||
PostgreSQL provides 5 years of support per major version. Always run the latest minor release.
|
||||
|
||||
| Version | Initial Release | End of Life |
|
||||
| ------- | ------------------ | ------------------ |
|
||||
| 18 | September 2025 | November 2030 |
|
||||
| 17 | September 2024 | November 2029 |
|
||||
| 16 | September 2023 | November 2028 |
|
||||
| 15 | October 2022 | November 2027 |
|
||||
| 14 | September 2021 | November 2026 |
|
||||
|
||||
Source: [postgresql.org/support/versioning](https://www.postgresql.org/support/versioning/)
|
||||
|
||||
## References
|
||||
|
||||
| Area | Resource | When to Use |
|
||||
| -------------- | --------------------------------- | ------------------------------------------------ |
|
||||
| Schema Design | `references/schema-design.md` | Designing tables, choosing data types, normalizing |
|
||||
| Area | Resource | When to Use |
|
||||
| ----------------------- | --------------------------------------- | ------------------------------------------------------------------ |
|
||||
| Schema Design | `references/schema-design.md` | Designing tables, choosing data types, normalizing, partitioning |
|
||||
| Indexing | `references/indexing.md` | Choosing index types, composite indexes, partial/covering indexes |
|
||||
| Query Optimization | `references/query-optimization.md` | Reading EXPLAIN ANALYZE, fixing bottlenecks, planner tuning |
|
||||
| Query Patterns | `references/query-patterns.md` | CTEs, window functions, lateral joins, UPSERT, JSONB, anti-patterns|
|
||||
| Performance Diagnostics | `references/performance-diagnostics.md` | pg_stat views, lock analysis, VACUUM, connection management |
|
||||
| Logical Replication | `references/logical-replication.md` | Pub/sub replication, live migrations, CDC |
|
||||
| Hot Standby | `references/hot-standby.md` | Streaming replication, read replicas, failover |
|
||||
| Transaction Isolation | `references/transaction-isolation.md` | Isolation levels, lost updates, serialization failures, retry logic |
|
||||
| Backup & Restore | `references/backup-restore.md` | pg_dump/pg_restore, pg_basebackup, PITR, recovery |
|
||||
| Security & Roles | `references/security-roles.md` | Privileges, RLS, pg_hba.conf, authentication, SSL |
|
||||
| Bulk Data Loading | `references/bulk-loading.md` | COPY patterns, ETL staging, optimizing large loads, batch ops |
|
||||
| Connection Pooling | `references/connection-pooling.md` | PgBouncer config, pool modes, prepared statements, sizing |
|
||||
| Major Version Upgrades | `references/major-version-upgrades.md` | pg_upgrade, logical replication migration, pre/post checklists |
|
||||
|
||||
@@ -0,0 +1,372 @@
|
||||
# Backup & Restore Reference
|
||||
|
||||
## Contents
|
||||
- Backup strategy overview
|
||||
- Logical backups (pg_dump / pg_restore)
|
||||
- Physical backups (pg_basebackup)
|
||||
- Point-in-Time Recovery (PITR)
|
||||
- Verification and testing
|
||||
- Automation patterns
|
||||
|
||||
## Backup Strategy Overview
|
||||
|
||||
| Method | What it captures | Granularity | Speed | Use case |
|
||||
|--------|-----------------|-------------|-------|----------|
|
||||
| `pg_dump` | Logical (SQL/custom) | Per-database, per-table | Slow on large DBs | Dev snapshots, migrations, selective restore |
|
||||
| `pg_dumpall` | All databases + globals | Entire cluster | Slow | Full cluster backup including roles/tablespaces |
|
||||
| `pg_basebackup` | Physical (file copy) | Entire cluster | Fast | Production backups, PITR base, replica setup |
|
||||
| Continuous archiving | WAL segments | Incremental | Continuous | PITR — restore to any point in time |
|
||||
|
||||
**Production recommendation**: `pg_basebackup` + continuous WAL archiving for PITR capability. Supplement with periodic `pg_dump` for portable, version-independent backups.
|
||||
|
||||
## Logical Backups (pg_dump / pg_restore)
|
||||
|
||||
### pg_dump Formats
|
||||
|
||||
| Format | Flag | Parallel restore? | Selective restore? | Notes |
|
||||
|--------|------|-------------------|-------------------|-------|
|
||||
| Custom | `-Fc` | Yes | Yes | **Recommended default** — compressed, flexible |
|
||||
| Directory | `-Fd` | Yes | Yes | One file per table, good for large DBs |
|
||||
| Plain SQL | `-Fp` | No | No (manual editing) | Human-readable, good for version control |
|
||||
| Tar | `-Ft` | No | Yes | Compatibility option |
|
||||
|
||||
### Common pg_dump Patterns
|
||||
|
||||
```bash
|
||||
# Full database backup (custom format — recommended)
|
||||
pg_dump -Fc -f backup.dump mydb
|
||||
|
||||
# With compression level (0-9, default varies by format)
|
||||
pg_dump -Fc -Z 6 -f backup.dump mydb
|
||||
|
||||
# Parallel dump (directory format only, 4 workers)
|
||||
pg_dump -Fd -j 4 -f backup_dir/ mydb
|
||||
|
||||
# Schema only (no data)
|
||||
pg_dump -Fc --schema-only -f schema.dump mydb
|
||||
|
||||
# Data only (no schema)
|
||||
pg_dump -Fc --data-only -f data.dump mydb
|
||||
|
||||
# Specific tables
|
||||
pg_dump -Fc -t orders -t customers -f subset.dump mydb
|
||||
|
||||
# Specific schema
|
||||
pg_dump -Fc -n public -f public_schema.dump mydb
|
||||
|
||||
# Exclude large tables
|
||||
pg_dump -Fc -T audit_log -T event_archive -f without_logs.dump mydb
|
||||
|
||||
# Include CREATE DATABASE in output
|
||||
pg_dump -Fc --create -f backup_with_create.dump mydb
|
||||
```
|
||||
|
||||
### pg_restore Patterns
|
||||
|
||||
```bash
|
||||
# Restore to an existing (empty) database
|
||||
pg_restore -d mydb backup.dump
|
||||
|
||||
# Parallel restore (4 workers — significantly faster for large DBs)
|
||||
pg_restore -d mydb -j 4 backup.dump
|
||||
|
||||
# Create the database during restore
|
||||
pg_restore --create -d postgres backup.dump
|
||||
|
||||
# List contents of a backup (inspect before restoring)
|
||||
pg_restore --list backup.dump
|
||||
|
||||
# Restore specific tables only
|
||||
pg_restore -d mydb -t orders -t customers backup.dump
|
||||
|
||||
# Schema only
|
||||
pg_restore -d mydb --schema-only backup.dump
|
||||
|
||||
# Data only (schema already exists)
|
||||
pg_restore -d mydb --data-only backup.dump
|
||||
|
||||
# Clean (drop) objects before recreating
|
||||
pg_restore -d mydb --clean --if-exists backup.dump
|
||||
|
||||
# Restore atomically and stop on the first error
|
||||
pg_restore -d mydb --single-transaction -j 1 backup.dump
|
||||
# Note: --single-transaction is incompatible with -j > 1
|
||||
```
|
||||
|
||||
Selective restore patterns do not automatically include dependencies. Schema-qualify table patterns when names may exist in multiple schemas, and verify required types, sequences, constraints, and referenced tables separately.
|
||||
|
||||
### Restoring Plain SQL Dumps
|
||||
|
||||
```bash
|
||||
# Plain SQL dumps are restored with psql, not pg_restore
|
||||
psql -d mydb -f backup.sql
|
||||
|
||||
# With error handling
|
||||
psql -d mydb -v ON_ERROR_STOP=1 -f backup.sql
|
||||
```
|
||||
|
||||
### pg_dumpall — Cluster-Wide Backup
|
||||
|
||||
`pg_dumpall` is the only way to back up global objects (roles, tablespaces):
|
||||
|
||||
```bash
|
||||
# Full cluster (all databases + globals) — plain SQL only
|
||||
pg_dumpall -f cluster_backup.sql
|
||||
|
||||
# Globals only (roles, tablespaces) — use alongside per-database pg_dump
|
||||
pg_dumpall --globals-only -f globals.sql
|
||||
|
||||
# Roles only
|
||||
pg_dumpall --roles-only -f roles.sql
|
||||
```
|
||||
|
||||
**Best practice**: Use `pg_dumpall --globals-only` for roles/tablespaces, then `pg_dump -Fc` per database for data. This gives you parallel restore capability while preserving globals.
|
||||
|
||||
### Optimizer Statistics in Backups (PG18+)
|
||||
|
||||
```bash
|
||||
# Dump optimizer statistics (speeds up post-restore query performance)
|
||||
pg_dump -Fc --statistics -f backup.dump mydb
|
||||
|
||||
# Statistics only (no schema or data)
|
||||
pg_dump -Fc --statistics-only -f stats.dump mydb
|
||||
|
||||
# Skip statistics
|
||||
pg_dump -Fc --no-statistics -f backup.dump mydb
|
||||
```
|
||||
|
||||
Without statistics, the planner uses default estimates after restore until `ANALYZE` runs on all tables. Dumping statistics avoids the post-restore performance dip.
|
||||
|
||||
### Performance Tips for Large Databases
|
||||
|
||||
1. **Use parallel dump/restore** (`-Fd -j N`) — scales well with CPU cores
|
||||
2. **Dump to fast storage** — local NVMe, not network mounts
|
||||
3. **Increase `maintenance_work_mem`** on the restore target for faster index rebuilds
|
||||
4. **Disable triggers during data-only restore**: `pg_restore --disable-triggers` (requires superuser)
|
||||
5. **Drop indexes before restore, recreate after** — faster than incremental index maintenance during bulk inserts
|
||||
6. **Restore schema first, then data, then indexes**:
|
||||
```bash
|
||||
pg_restore -d mydb --section=pre-data backup.dump
|
||||
pg_restore -d mydb --data-only -j 4 backup.dump
|
||||
pg_restore -d mydb --section=post-data -j 4 backup.dump
|
||||
```
|
||||
|
||||
## Physical Backups (pg_basebackup)
|
||||
|
||||
`pg_basebackup` creates a byte-for-byte copy of the entire cluster, suitable as a base for PITR or setting up replicas.
|
||||
|
||||
```bash
|
||||
# Basic backup (plain format)
|
||||
pg_basebackup -D /backup/base -Fp -Xs -P
|
||||
|
||||
# Compressed tar format
|
||||
pg_basebackup -D /backup/base -Ft -z -Xs -P
|
||||
|
||||
# With checkpoint mode (fast = don't wait for next scheduled checkpoint)
|
||||
pg_basebackup -D /backup/base -Fp -Xs -P --checkpoint=fast
|
||||
|
||||
# To a remote server
|
||||
pg_basebackup -h primary_host -U repl_user -D /backup/base -Fp -Xs -P
|
||||
```
|
||||
|
||||
| Flag | Meaning |
|
||||
|------|---------|
|
||||
| `-D` | Target directory |
|
||||
| `-Fp` | Plain format (ready-to-use data directory) |
|
||||
| `-Ft` | Tar format (one tar per tablespace) |
|
||||
| `-z` | Compress (with tar format) |
|
||||
| `-Xs` | Stream WAL during backup (ensures consistency) |
|
||||
| `-P` | Show progress |
|
||||
| `--checkpoint=fast` | Start backup immediately (don't wait for next checkpoint) |
|
||||
|
||||
### Prerequisites
|
||||
|
||||
The connection used by `pg_basebackup` must authenticate as a superuser or a role with `REPLICATION` privilege. `max_wal_senders` must also have enough capacity for the backup connection:
|
||||
|
||||
```sql
|
||||
-- On the primary
|
||||
CREATE ROLE backup_user WITH REPLICATION LOGIN PASSWORD '...';
|
||||
```
|
||||
|
||||
And `pg_hba.conf` must allow replication connections:
|
||||
|
||||
```
|
||||
host replication backup_user backup_server_ip/32 scram-sha-256
|
||||
```
|
||||
|
||||
[PG15+] `--target=server:/path` writes the backup on the database server. A non-superuser using this target needs both `REPLICATION` privilege for the backup connection and membership in `pg_write_server_files` for the server-side write. `--target` is unavailable in PG14 and cannot be combined with `-Xstream`; use `-Xfetch` or `-Xnone`.
|
||||
|
||||
## Point-in-Time Recovery (PITR)
|
||||
|
||||
PITR lets you restore a database to any specific point in time — invaluable for recovering from accidental data deletion or corruption.
|
||||
|
||||
### How PITR Works
|
||||
|
||||
1. Take a **base backup** (`pg_basebackup`)
|
||||
2. Continuously **archive WAL segments** as they're produced
|
||||
3. To recover: restore the base backup, then **replay WAL** up to the target time
|
||||
|
||||
### Step 1: Configure WAL Archiving
|
||||
|
||||
In `postgresql.conf`:
|
||||
|
||||
```
|
||||
archive_mode = on
|
||||
archive_command = 'cp %p /archive/wal/%f' # or use pgBackRest, barman, etc.
|
||||
# archive_library = '' # PG15+: use archive modules instead of shell commands
|
||||
```
|
||||
|
||||
Requires a restart after enabling `archive_mode`.
|
||||
|
||||
Verify archiving is working:
|
||||
|
||||
```sql
|
||||
SELECT * FROM pg_stat_archiver;
|
||||
-- Check: archived_count is increasing, last_failed_time is NULL
|
||||
```
|
||||
|
||||
### Step 2: Take Base Backups Regularly
|
||||
|
||||
```bash
|
||||
# Weekly base backup (adjust frequency based on WAL volume)
|
||||
pg_basebackup -D /backup/base_$(date +%Y%m%d) -Ft -z -Xs -P --checkpoint=fast
|
||||
```
|
||||
|
||||
### Step 3: Recovery
|
||||
|
||||
To recover to a specific point in time:
|
||||
|
||||
1. Stop PostgreSQL
|
||||
2. Replace the data directory with the base backup
|
||||
3. Create `recovery.signal` (PG12+) or `recovery.conf` (PG11-)
|
||||
4. Configure recovery target in `postgresql.conf`:
|
||||
|
||||
```
|
||||
# In postgresql.conf (PG12+):
|
||||
restore_command = 'cp /archive/wal/%f %p'
|
||||
recovery_target_time = '2024-06-15 14:30:00'
|
||||
recovery_target_action = 'promote' # 'pause' to inspect before promoting
|
||||
```
|
||||
|
||||
5. Start PostgreSQL — it replays WAL up to the target time, then promotes to read-write
|
||||
|
||||
### Recovery Target Options
|
||||
|
||||
```text
|
||||
-- Recover to a specific time
|
||||
recovery_target_time = '2024-06-15 14:30:00+00'
|
||||
|
||||
-- Recover to a specific transaction ID
|
||||
recovery_target_xid = '12345678'
|
||||
|
||||
-- Recover to a named restore point
|
||||
recovery_target_name = 'before_migration'
|
||||
|
||||
-- Stop as soon as a consistent state is reached
|
||||
-- (for an online backup, normally the point where the backup ended)
|
||||
recovery_target = 'immediate'
|
||||
|
||||
-- Recover to a specific WAL position
|
||||
recovery_target_lsn = '0/1A2B3C4D'
|
||||
```
|
||||
|
||||
To recover through the end of all available WAL, omit every `recovery_target*` setting. `recovery_target = 'immediate'` is an explicit early stopping target, so later available WAL can remain unapplied. PostgreSQL may still replay the WAL required to make an online backup consistent.
|
||||
|
||||
### Creating Named Restore Points
|
||||
|
||||
Before risky operations, create a named restore point:
|
||||
|
||||
```sql
|
||||
SELECT pg_create_restore_point('before_schema_migration');
|
||||
SELECT pg_create_restore_point('before_bulk_delete');
|
||||
```
|
||||
|
||||
This gives you an exact target to recover to if the operation goes wrong.
|
||||
|
||||
## Verification and Testing
|
||||
|
||||
### Verify Backup Integrity
|
||||
|
||||
```bash
|
||||
# List contents without restoring
|
||||
pg_restore --list backup.dump
|
||||
|
||||
# Verify a plain-format physical backup (PG13+)
|
||||
pg_verifybackup /backup/base_20240615
|
||||
|
||||
# PG18+: verify a tar-format backup directly
|
||||
pg_verifybackup --no-parse-wal /backup/base_tar_20240615
|
||||
```
|
||||
|
||||
### Test Restores Regularly
|
||||
|
||||
**A backup that hasn't been tested is not a backup.** Schedule regular restore tests:
|
||||
|
||||
```bash
|
||||
# Restore to a test database
|
||||
createdb mydb_restore_test
|
||||
pg_restore -d mydb_restore_test backup.dump
|
||||
|
||||
# Verify row counts
|
||||
psql -d mydb_restore_test -c "
|
||||
SELECT schemaname, relname, n_live_tup
|
||||
FROM pg_stat_user_tables
|
||||
ORDER BY n_live_tup DESC;
|
||||
"
|
||||
|
||||
# Clean up
|
||||
dropdb mydb_restore_test
|
||||
```
|
||||
|
||||
### Post-Restore Checklist
|
||||
|
||||
After restoring a backup:
|
||||
|
||||
1. **Run `ANALYZE`** on all tables — optimizer statistics may be stale or missing
|
||||
```sql
|
||||
ANALYZE; -- all tables
|
||||
```
|
||||
2. **Verify sequences** — if restoring data-only, sequences may not match the data
|
||||
```sql
|
||||
SELECT sequencename, last_value FROM pg_sequences WHERE schemaname = 'public';
|
||||
```
|
||||
3. **Check for invalid indexes** — concurrent index builds may have been in progress
|
||||
```sql
|
||||
SELECT indexrelid::regclass, indisvalid FROM pg_index WHERE NOT indisvalid;
|
||||
```
|
||||
4. **Verify replication slots are clean** — stale slots from the source won't work
|
||||
```sql
|
||||
SELECT slot_name, active FROM pg_replication_slots;
|
||||
```
|
||||
|
||||
## Automation Patterns
|
||||
|
||||
### Scripted Backup with Retention
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
BACKUP_DIR="/backup/pg"
|
||||
RETENTION_DAYS=30
|
||||
DB_NAME="mydb"
|
||||
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
|
||||
|
||||
# Dump
|
||||
pg_dump -Fc -Z 6 -f "${BACKUP_DIR}/${DB_NAME}_${TIMESTAMP}.dump" "${DB_NAME}"
|
||||
|
||||
# Verify dump was created and is non-empty
|
||||
if [ ! -s "${BACKUP_DIR}/${DB_NAME}_${TIMESTAMP}.dump" ]; then
|
||||
echo "ERROR: Backup file is empty or missing" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Clean old backups
|
||||
find "${BACKUP_DIR}" -name "${DB_NAME}_*.dump" -mtime +${RETENTION_DAYS} -delete
|
||||
```
|
||||
|
||||
### Dedicated Backup Tools
|
||||
|
||||
For production environments, consider purpose-built tools that handle scheduling, retention, compression, and PITR:
|
||||
|
||||
- **pgBackRest** — parallel backup/restore, incremental backups, S3/GCS/Azure support, built-in PITR
|
||||
- **Barman** — backup management with retention policies, remote backup, WAL archiving
|
||||
- **pg_probackup** — incremental backups with page-level tracking, merge, validation
|
||||
@@ -0,0 +1,315 @@
|
||||
# Bulk Data Loading Reference
|
||||
|
||||
## Contents
|
||||
- COPY vs INSERT performance
|
||||
- COPY FROM patterns
|
||||
- Optimizing large loads
|
||||
- ETL staging patterns
|
||||
- Bulk updates and deletes
|
||||
|
||||
## COPY vs INSERT Performance
|
||||
|
||||
| Method | Rows/sec (typical) | Use case |
|
||||
|--------|-------------------|----------|
|
||||
| Single-row INSERT | ~1,000-5,000 | Application writes |
|
||||
| Multi-row INSERT (VALUES) | ~10,000-50,000 | Batch inserts from code |
|
||||
| `COPY FROM` | ~100,000-500,000+ | Bulk loading from files or streams |
|
||||
| `COPY FROM` (binary) | ~200,000-1,000,000+ | Maximum throughput (binary format) |
|
||||
|
||||
`COPY` is **10-100x faster** than individual INSERTs because it bypasses per-row overhead (parsing, planning, WAL per statement).
|
||||
|
||||
## COPY FROM Patterns
|
||||
|
||||
### From a File
|
||||
|
||||
```sql
|
||||
-- CSV with header
|
||||
COPY orders FROM '/path/to/orders.csv' WITH (FORMAT csv, HEADER true);
|
||||
|
||||
-- Tab-delimited
|
||||
COPY orders FROM '/path/to/orders.tsv' WITH (FORMAT text);
|
||||
|
||||
-- Custom delimiter
|
||||
COPY orders FROM '/path/to/orders.dat' WITH (DELIMITER '|');
|
||||
|
||||
-- With NULL handling
|
||||
COPY orders FROM '/path/to/orders.csv'
|
||||
WITH (FORMAT csv, HEADER true, NULL '');
|
||||
```
|
||||
|
||||
### From STDIN (Piped Data)
|
||||
|
||||
```bash
|
||||
# Pipe from another command
|
||||
cat orders.csv | psql -d mydb -c "\COPY orders FROM STDIN WITH (FORMAT csv, HEADER true)"
|
||||
|
||||
# Pipe from gzip
|
||||
gunzip -c orders.csv.gz | psql -d mydb -c "\COPY orders FROM STDIN WITH (FORMAT csv, HEADER true)"
|
||||
```
|
||||
|
||||
### From Application Code
|
||||
|
||||
Most drivers support COPY protocol for streaming data:
|
||||
|
||||
```python
|
||||
# Python (psycopg 3)
|
||||
with conn.cursor() as cur:
|
||||
with cur.copy("COPY orders (id, customer_id, total) FROM STDIN") as copy:
|
||||
for row in data:
|
||||
copy.write_row(row)
|
||||
```
|
||||
|
||||
```javascript
|
||||
// Node.js (pg-copy-streams)
|
||||
const { from } = require('pg-copy-streams');
|
||||
const stream = client.query(from('COPY orders FROM STDIN WITH (FORMAT csv)'));
|
||||
fileStream.pipe(stream);
|
||||
```
|
||||
|
||||
### COPY TO (Export)
|
||||
|
||||
```sql
|
||||
-- Export to CSV
|
||||
COPY orders TO '/path/to/orders.csv' WITH (FORMAT csv, HEADER true);
|
||||
|
||||
-- Export query results
|
||||
COPY (SELECT id, total FROM orders WHERE created_at > '2024-01-01')
|
||||
TO '/path/to/recent.csv' WITH (FORMAT csv, HEADER true);
|
||||
```
|
||||
|
||||
### Error Handling (PG17+)
|
||||
|
||||
```sql
|
||||
-- Skip invalid rows instead of failing the entire COPY
|
||||
COPY orders FROM '/path/to/orders.csv'
|
||||
WITH (FORMAT csv, HEADER true, ON_ERROR ignore);
|
||||
|
||||
-- PG18+: limit how many errors to tolerate
|
||||
COPY orders FROM '/path/to/orders.csv'
|
||||
WITH (FORMAT csv, HEADER true, ON_ERROR ignore, REJECT_LIMIT 100);
|
||||
```
|
||||
|
||||
## Optimizing Large Loads
|
||||
|
||||
### 1. Drop Indexes, Load, Recreate
|
||||
|
||||
Building indexes incrementally during COPY is much slower than building them once after:
|
||||
|
||||
```sql
|
||||
-- Before load: drop non-essential indexes
|
||||
DROP INDEX idx_orders_customer_id;
|
||||
DROP INDEX idx_orders_created_at;
|
||||
|
||||
-- Load data
|
||||
COPY orders FROM '/path/to/orders.csv' WITH (FORMAT csv, HEADER true);
|
||||
|
||||
-- After load: recreate indexes (concurrently if table is in use)
|
||||
CREATE INDEX idx_orders_customer_id ON orders(customer_id);
|
||||
CREATE INDEX idx_orders_created_at ON orders(created_at);
|
||||
|
||||
-- Update statistics
|
||||
ANALYZE orders;
|
||||
```
|
||||
|
||||
### 2. Disable Triggers During Load
|
||||
|
||||
```sql
|
||||
-- Disable all triggers on the table
|
||||
ALTER TABLE orders DISABLE TRIGGER ALL;
|
||||
|
||||
-- Load data
|
||||
COPY orders FROM '/path/to/orders.csv' WITH (FORMAT csv, HEADER true);
|
||||
|
||||
-- Re-enable triggers
|
||||
ALTER TABLE orders ENABLE TRIGGER ALL;
|
||||
```
|
||||
|
||||
`DISABLE TRIGGER ALL` requires superuser privileges when the table has foreign-key or other internally generated constraint triggers. A table owner can use `DISABLE TRIGGER USER` to disable only user-defined triggers. Disabling constraint triggers can admit invalid data, so validate constraints before re-enabling writes.
|
||||
|
||||
### 3. Increase maintenance_work_mem
|
||||
|
||||
Larger `maintenance_work_mem` speeds up index creation after the load:
|
||||
|
||||
```sql
|
||||
SET maintenance_work_mem = '1GB'; -- for the duration of the load session
|
||||
```
|
||||
|
||||
### 4. Disable Autovacuum During Load
|
||||
|
||||
For very large bulk loads, temporarily disable autovacuum to avoid competing I/O:
|
||||
|
||||
```sql
|
||||
ALTER TABLE orders SET (autovacuum_enabled = false);
|
||||
|
||||
-- Load data...
|
||||
|
||||
ALTER TABLE orders SET (autovacuum_enabled = true);
|
||||
VACUUM ANALYZE orders;
|
||||
```
|
||||
|
||||
### 5. Use Unlogged Tables for Staging
|
||||
|
||||
Unlogged tables skip WAL writes — 2-3x faster for writes but **data is lost on crash**:
|
||||
|
||||
```sql
|
||||
CREATE UNLOGGED TABLE staging_orders (LIKE orders INCLUDING ALL);
|
||||
|
||||
COPY staging_orders FROM '/path/to/orders.csv' WITH (FORMAT csv, HEADER true);
|
||||
|
||||
-- Transform and move to the real table
|
||||
INSERT INTO orders SELECT * FROM staging_orders;
|
||||
|
||||
DROP TABLE staging_orders;
|
||||
```
|
||||
|
||||
### 6. Batch Size for Programmatic Inserts
|
||||
|
||||
When COPY isn't available, use multi-row INSERT with batches of 100-1,000 rows:
|
||||
|
||||
```sql
|
||||
-- Single round-trip for 1,000 rows
|
||||
INSERT INTO orders (customer_id, total, created_at)
|
||||
VALUES
|
||||
(1, 99.99, now()),
|
||||
(2, 149.99, now()),
|
||||
-- ... up to ~1,000 rows per statement
|
||||
(1000, 79.99, now());
|
||||
```
|
||||
|
||||
Beyond ~1,000 rows per statement, parse overhead increases. Use COPY for larger batches.
|
||||
|
||||
### 7. Parallel Loading into Partitioned Tables
|
||||
|
||||
Load data into individual partitions concurrently from separate sessions:
|
||||
|
||||
```bash
|
||||
# Session 1
|
||||
psql -c "COPY events_2024_q1 FROM '/data/q1.csv' WITH (FORMAT csv)"
|
||||
|
||||
# Session 2 (concurrent)
|
||||
psql -c "COPY events_2024_q2 FROM '/data/q2.csv' WITH (FORMAT csv)"
|
||||
```
|
||||
|
||||
## ETL Staging Patterns
|
||||
|
||||
### Staging Table with Upsert
|
||||
|
||||
```sql
|
||||
-- Create staging table (temporary or unlogged)
|
||||
CREATE TEMP TABLE staging_customers (LIKE customers INCLUDING DEFAULTS);
|
||||
|
||||
-- Load raw data
|
||||
COPY staging_customers FROM '/path/to/customers.csv' WITH (FORMAT csv, HEADER true);
|
||||
|
||||
-- Upsert into production table
|
||||
INSERT INTO customers (id, name, email, updated_at)
|
||||
SELECT id, name, email, now()
|
||||
FROM staging_customers
|
||||
ON CONFLICT (id)
|
||||
DO UPDATE SET
|
||||
name = EXCLUDED.name,
|
||||
email = EXCLUDED.email,
|
||||
updated_at = EXCLUDED.updated_at;
|
||||
```
|
||||
|
||||
### Staging Table with MERGE (PG15+)
|
||||
|
||||
```sql
|
||||
MERGE INTO customers AS target
|
||||
USING staging_customers AS source
|
||||
ON target.id = source.id
|
||||
WHEN MATCHED AND source.name IS DISTINCT FROM target.name THEN
|
||||
UPDATE SET name = source.name, email = source.email, updated_at = now()
|
||||
WHEN NOT MATCHED THEN
|
||||
INSERT (id, name, email, updated_at)
|
||||
VALUES (source.id, source.name, source.email, now());
|
||||
```
|
||||
|
||||
### Swap Table Pattern
|
||||
|
||||
For full-refresh loads where you replace all data:
|
||||
|
||||
```sql
|
||||
-- Load into a new table
|
||||
CREATE TABLE orders_new (LIKE orders INCLUDING ALL);
|
||||
COPY orders_new FROM '/path/to/orders.csv' WITH (FORMAT csv, HEADER true);
|
||||
ANALYZE orders_new;
|
||||
|
||||
-- Atomic swap (brief exclusive lock)
|
||||
BEGIN;
|
||||
ALTER TABLE orders RENAME TO orders_old;
|
||||
ALTER TABLE orders_new RENAME TO orders;
|
||||
DROP TABLE orders_old;
|
||||
COMMIT;
|
||||
```
|
||||
|
||||
`LIKE ... INCLUDING ALL` does not copy foreign keys, triggers, rules, grants, row-level security policies, or publication membership. Recreate and verify those objects before swapping, or use a data-only refresh when the original table's identity and dependencies must remain unchanged.
|
||||
|
||||
## Bulk Updates and Deletes
|
||||
|
||||
### Chunked Deletes
|
||||
|
||||
Large DELETEs lock rows and generate WAL. This loop limits each statement to 10,000 rows:
|
||||
|
||||
```sql
|
||||
-- Delete in batches of 10,000
|
||||
DO $$
|
||||
DECLARE
|
||||
rows_deleted int;
|
||||
BEGIN
|
||||
LOOP
|
||||
DELETE FROM audit_log
|
||||
WHERE id IN (
|
||||
SELECT id FROM audit_log
|
||||
WHERE created_at < now() - interval '1 year'
|
||||
LIMIT 10000
|
||||
);
|
||||
GET DIAGNOSTICS rows_deleted = ROW_COUNT;
|
||||
EXIT WHEN rows_deleted = 0;
|
||||
-- Optional: brief pause to reduce WAL pressure
|
||||
PERFORM pg_sleep(0.1);
|
||||
END LOOP;
|
||||
END $$;
|
||||
```
|
||||
|
||||
The entire `DO` block is still one transaction: locks, WAL, and dead rows accumulate until it finishes. For true transaction-level batching, execute one limited DELETE per client transaction and commit between batches.
|
||||
|
||||
### Chunked Updates
|
||||
|
||||
The same statement-size pattern works for large updates:
|
||||
|
||||
```sql
|
||||
DO $$
|
||||
DECLARE
|
||||
rows_updated int;
|
||||
BEGIN
|
||||
LOOP
|
||||
UPDATE orders
|
||||
SET status = 'archived'
|
||||
WHERE id IN (
|
||||
SELECT id FROM orders
|
||||
WHERE status = 'completed'
|
||||
AND created_at < now() - interval '2 years'
|
||||
LIMIT 10000
|
||||
FOR UPDATE SKIP LOCKED
|
||||
);
|
||||
GET DIAGNOSTICS rows_updated = ROW_COUNT;
|
||||
EXIT WHEN rows_updated = 0;
|
||||
END LOOP;
|
||||
END $$;
|
||||
```
|
||||
|
||||
As with the DELETE loop, the `DO` block commits only once. Drive batches from the client or a transaction-controlling procedure invoked outside an explicit transaction when each batch must commit independently.
|
||||
|
||||
### Partition Drop Instead of Delete
|
||||
|
||||
If data is partitioned by time, dropping a partition is instant vs. a slow DELETE:
|
||||
|
||||
```sql
|
||||
-- Instant: drop old partition
|
||||
ALTER TABLE events DETACH PARTITION events_2023_q1 CONCURRENTLY;
|
||||
DROP TABLE events_2023_q1;
|
||||
|
||||
-- vs. slow: delete rows
|
||||
-- DELETE FROM events WHERE occurred_at < '2024-01-01'; -- don't do this
|
||||
```
|
||||
@@ -0,0 +1,310 @@
|
||||
# Connection Pooling Reference
|
||||
|
||||
## Contents
|
||||
- Why connection pooling matters
|
||||
- PgBouncer configuration
|
||||
- Pool modes
|
||||
- Prepared statement handling
|
||||
- Monitoring and diagnostics
|
||||
- Application-side pooling
|
||||
|
||||
## Why Connection Pooling Matters
|
||||
|
||||
Each PostgreSQL connection spawns a dedicated backend process (~5-10 MB of memory). Without pooling:
|
||||
|
||||
- 500 application instances × 10 connections each = 5,000 backend processes
|
||||
- Memory: 5,000 × 10 MB = ~50 GB just for connection overhead
|
||||
- Context switching degrades performance above a few hundred active backends
|
||||
- `max_connections` must be set high, wasting shared memory
|
||||
|
||||
A connection pooler sits between the application and PostgreSQL, multiplexing many client connections onto fewer server connections.
|
||||
|
||||
### Connection Overhead
|
||||
|
||||
```sql
|
||||
-- Current connections vs. limit
|
||||
SELECT
|
||||
count(*) AS current,
|
||||
(SELECT setting::int FROM pg_settings WHERE name = 'max_connections') AS max,
|
||||
(SELECT setting::int FROM pg_settings WHERE name = 'superuser_reserved_connections') AS reserved
|
||||
FROM pg_stat_activity;
|
||||
|
||||
-- Potential per-operation and per-session memory settings
|
||||
SELECT
|
||||
name,
|
||||
current_setting(name) AS configured_value,
|
||||
pg_size_bytes(current_setting(name)) AS configured_bytes
|
||||
FROM pg_settings
|
||||
WHERE name IN ('work_mem', 'temp_buffers')
|
||||
ORDER BY name;
|
||||
```
|
||||
|
||||
These are not baseline allocations per backend. `work_mem` can be consumed by multiple query operations, while `temp_buffers` is allocated lazily when a session uses temporary tables.
|
||||
|
||||
## PgBouncer Configuration
|
||||
|
||||
PgBouncer is the most widely used connection pooler for PostgreSQL.
|
||||
|
||||
### Essential pgbouncer.ini Settings
|
||||
|
||||
```ini
|
||||
[databases]
|
||||
mydb = host=127.0.0.1 port=5432 dbname=mydb
|
||||
|
||||
[pgbouncer]
|
||||
listen_addr = 0.0.0.0
|
||||
listen_port = 6432
|
||||
auth_type = scram-sha-256
|
||||
auth_file = /etc/pgbouncer/userlist.txt
|
||||
|
||||
# Pool sizing
|
||||
pool_mode = transaction
|
||||
default_pool_size = 20
|
||||
min_pool_size = 5
|
||||
max_client_conn = 1000
|
||||
max_db_connections = 50
|
||||
|
||||
# Timeouts
|
||||
server_idle_timeout = 300
|
||||
client_idle_timeout = 0
|
||||
query_timeout = 0
|
||||
query_wait_timeout = 120
|
||||
server_login_retry = 15
|
||||
|
||||
# Logging
|
||||
log_connections = 0
|
||||
log_disconnections = 0
|
||||
stats_period = 60
|
||||
```
|
||||
|
||||
### Key Parameters
|
||||
|
||||
| Parameter | Recommended | Purpose |
|
||||
|-----------|------------|---------|
|
||||
| `default_pool_size` | 20-50 | Server connections per user/database pair |
|
||||
| `min_pool_size` | 5 | Minimum idle server connections to keep open |
|
||||
| `max_client_conn` | 1000-10000 | Max client connections PgBouncer accepts |
|
||||
| `max_db_connections` | 50-100 | Hard cap on server connections per database |
|
||||
| `reserve_pool_size` | 5 | Extra connections for burst traffic |
|
||||
| `reserve_pool_timeout` | 3 | Seconds before using reserve pool |
|
||||
| `query_wait_timeout` | 120 | Max time a client waits for a server connection |
|
||||
|
||||
### Sizing Rule of Thumb
|
||||
|
||||
**Server connections**: Set `default_pool_size` to roughly 2-4x the number of CPU cores on the database server. More connections don't help — they just increase lock contention and context switching.
|
||||
|
||||
**Client connections**: Set `max_client_conn` high enough for all application instances. PgBouncer handles thousands of idle client connections with minimal memory.
|
||||
|
||||
```
|
||||
App instances (500 clients) ──→ PgBouncer (max_client_conn=1000) ──→ PostgreSQL (pool_size=20)
|
||||
```
|
||||
|
||||
## Pool Modes
|
||||
|
||||
### Transaction Mode (Recommended for Most Applications)
|
||||
|
||||
```ini
|
||||
pool_mode = transaction
|
||||
```
|
||||
|
||||
Server connection is assigned when a transaction begins and returned when it commits/rolls back. Between transactions, the connection is available to other clients.
|
||||
|
||||
**Compatible with**: Standard SQL, parameterized queries, most ORMs.
|
||||
|
||||
**NOT compatible with** (session-level features):
|
||||
- `SET` / `RESET` (use `SET LOCAL` inside a transaction instead)
|
||||
- `LISTEN` / `NOTIFY`
|
||||
- SQL-level `PREPARE` / `DEALLOCATE` (use protocol-level prepared statements)
|
||||
- `DECLARE ... WITH HOLD` cursors
|
||||
- Temporary tables with `ON COMMIT PRESERVE ROWS`
|
||||
- Session-level advisory locks (`pg_advisory_lock` — use `pg_advisory_xact_lock` instead)
|
||||
- `LOAD` statement
|
||||
|
||||
### Session Mode
|
||||
|
||||
```ini
|
||||
pool_mode = session
|
||||
```
|
||||
|
||||
Server connection is held for the entire client session. Compatible with all PostgreSQL features but offers less multiplexing benefit.
|
||||
|
||||
**Use when**: Application relies on session-level features (temp tables, LISTEN/NOTIFY, SET parameters).
|
||||
|
||||
### Statement Mode
|
||||
|
||||
```ini
|
||||
pool_mode = statement
|
||||
```
|
||||
|
||||
Server connection released after every statement. Maximum multiplexing but **incompatible with multi-statement transactions**.
|
||||
|
||||
**Use when**: Application only runs autocommit single statements (rare).
|
||||
|
||||
### Choosing a Mode
|
||||
|
||||
| Application pattern | Recommended mode |
|
||||
|--------------------|-----------------|
|
||||
| Web apps, APIs, serverless | Transaction |
|
||||
| Applications using LISTEN/NOTIFY | Session |
|
||||
| Applications using temp tables across transactions | Session |
|
||||
| Applications with SET session variables | Session (or refactor to `SET LOCAL`) |
|
||||
| Simple autocommit queries | Statement |
|
||||
|
||||
## Prepared Statement Handling
|
||||
|
||||
### The Problem
|
||||
|
||||
SQL-level `PREPARE`/`EXECUTE` creates server-side prepared statements tied to a session. In transaction mode, the next transaction may use a different server connection where the prepared statement doesn't exist.
|
||||
|
||||
### Solutions
|
||||
|
||||
**Protocol-level prepared statements**: Most modern drivers use the PostgreSQL wire protocol's `Parse`/`Bind`/`Execute` messages, which PgBouncer (1.21+) can handle:
|
||||
|
||||
```ini
|
||||
# PgBouncer 1.21+
|
||||
max_prepared_statements = 100 # per server connection
|
||||
```
|
||||
|
||||
**Driver-level configuration** to avoid SQL-level PREPARE:
|
||||
|
||||
```python
|
||||
# Python psycopg: uses protocol-level by default — no change needed
|
||||
```
|
||||
|
||||
```javascript
|
||||
// Node.js pg: protocol-level by default
|
||||
const pool = new Pool({ ...config });
|
||||
// For explicit control:
|
||||
// statement_timeout via SET LOCAL, not SET
|
||||
```
|
||||
|
||||
```java
|
||||
// JDBC: protocol-level by default with prepareThreshold
|
||||
// ?prepareThreshold=5 (default: 5 uses before server-side prepare)
|
||||
// ?prepareThreshold=0 (disable server-side prepare entirely)
|
||||
```
|
||||
|
||||
```ruby
|
||||
# Ruby pg: protocol-level by default
|
||||
# ActiveRecord: no special config needed
|
||||
```
|
||||
|
||||
## Monitoring and Diagnostics
|
||||
|
||||
### PgBouncer Admin Console
|
||||
|
||||
Connect to PgBouncer's admin port:
|
||||
|
||||
```bash
|
||||
psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer
|
||||
```
|
||||
|
||||
### Key Commands
|
||||
|
||||
```text
|
||||
-- Pool status (most useful)
|
||||
SHOW POOLS;
|
||||
-- Columns: database, user, cl_active, cl_waiting, sv_active, sv_idle, sv_used, pool_mode
|
||||
|
||||
-- Active client and server connections
|
||||
SHOW CLIENTS;
|
||||
SHOW SERVERS;
|
||||
|
||||
-- Aggregate statistics
|
||||
SHOW STATS;
|
||||
-- Columns: total_xact_count, total_query_count, avg_xact_time, avg_query_time
|
||||
|
||||
-- Configuration
|
||||
SHOW CONFIG;
|
||||
|
||||
-- Memory usage
|
||||
SHOW MEM;
|
||||
```
|
||||
|
||||
These commands use PgBouncer's admin protocol and fail if sent directly to PostgreSQL.
|
||||
|
||||
### What to Watch
|
||||
|
||||
| Metric | Healthy | Problem |
|
||||
|--------|---------|---------|
|
||||
| `cl_waiting` | 0 | > 0 = clients waiting for server connections |
|
||||
| `sv_active` | < pool_size | = pool_size = pool exhausted |
|
||||
| `sv_idle` | > 0 | 0 = no spare connections |
|
||||
| `avg_xact_time` | < 100ms | High = long transactions hogging connections |
|
||||
| `avg_wait_time` | 0 | > 0 = pool too small or transactions too long |
|
||||
|
||||
### Common Issues
|
||||
|
||||
**Clients waiting (`cl_waiting > 0`)**:
|
||||
1. Check for long-running transactions: `SELECT * FROM pg_stat_activity WHERE state = 'idle in transaction';`
|
||||
2. Increase `default_pool_size` (but diminishing returns beyond ~4x CPU cores)
|
||||
3. Check if `query_wait_timeout` is being hit (errors in application logs)
|
||||
|
||||
**Server connections not being returned**:
|
||||
1. `idle in transaction` sessions hold connections — set `idle_in_transaction_session_timeout` in PostgreSQL
|
||||
2. Application not committing/rolling back — add explicit transaction management
|
||||
|
||||
## Application-Side Pooling
|
||||
|
||||
When PgBouncer isn't available, most drivers offer built-in connection pooling.
|
||||
|
||||
### Recommended Pool Sizes
|
||||
|
||||
**Per application instance**: 5-20 connections. Start low, increase only if you see connection wait times.
|
||||
|
||||
**Total across all instances**: Should not exceed ~4x database CPU cores for active connections.
|
||||
|
||||
### Common Driver Configuration
|
||||
|
||||
```python
|
||||
# Python (psycopg pool)
|
||||
from psycopg_pool import ConnectionPool
|
||||
|
||||
pool = ConnectionPool(
|
||||
conninfo="host=db port=5432 dbname=mydb",
|
||||
min_size=5,
|
||||
max_size=20,
|
||||
max_idle=300, # close idle connections after 5 min
|
||||
)
|
||||
|
||||
with pool.connection() as conn:
|
||||
conn.execute("SELECT ...")
|
||||
```
|
||||
|
||||
```javascript
|
||||
// Node.js (pg)
|
||||
const { Pool } = require('pg');
|
||||
const pool = new Pool({
|
||||
host: 'db',
|
||||
database: 'mydb',
|
||||
max: 20, // max connections in pool
|
||||
idleTimeoutMillis: 30000,
|
||||
connectionTimeoutMillis: 5000,
|
||||
});
|
||||
|
||||
const result = await pool.query('SELECT ...');
|
||||
```
|
||||
|
||||
```java
|
||||
// Java (HikariCP)
|
||||
HikariConfig config = new HikariConfig();
|
||||
config.setJdbcUrl("jdbc:postgresql://db:5432/mydb");
|
||||
config.setMaximumPoolSize(20);
|
||||
config.setMinimumIdle(5);
|
||||
config.setIdleTimeout(300000);
|
||||
config.setConnectionTimeout(5000);
|
||||
|
||||
HikariDataSource ds = new HikariDataSource(config);
|
||||
```
|
||||
|
||||
### Application Pool + PgBouncer
|
||||
|
||||
You can stack both. Each application instance pools locally (5-10 connections), and PgBouncer pools across all instances:
|
||||
|
||||
```
|
||||
App1 (pool: 10) ──┐
|
||||
App2 (pool: 10) ──┼──→ PgBouncer (pool: 30) ──→ PostgreSQL
|
||||
App3 (pool: 10) ──┘
|
||||
```
|
||||
|
||||
Set application pool sizes low when using PgBouncer — the point is to reduce total connections, not add them up.
|
||||
@@ -0,0 +1,221 @@
|
||||
# Hot Standby & Read Replicas Reference
|
||||
|
||||
## Contents
|
||||
- Streaming replication overview
|
||||
- Hot standby configuration
|
||||
- Monitoring replication lag
|
||||
- Replication conflicts
|
||||
- Synchronous vs asynchronous replication
|
||||
- Promoting a standby
|
||||
|
||||
## Streaming Replication Overview
|
||||
|
||||
Physical (streaming) replication creates an exact copy of the primary database on one or more standby servers. Unlike logical replication, it replicates the entire cluster (all databases, all objects) at the WAL level.
|
||||
|
||||
**Hot standby** allows read-only queries on the standby while it continuously replays WAL from the primary. This is the standard mechanism for PostgreSQL read replicas.
|
||||
|
||||
### Key Differences from Logical Replication
|
||||
|
||||
| Aspect | Physical (streaming) | Logical |
|
||||
|--------|---------------------|---------|
|
||||
| Scope | Entire cluster | Selected tables |
|
||||
| PG versions | Must match major version | Can differ |
|
||||
| Standby writable? | No (read-only) | Yes (for non-replicated tables) |
|
||||
| DDL replicated? | Yes (via WAL) | No |
|
||||
| Use case | HA, read replicas | CDC, selective sync, cross-version migration |
|
||||
|
||||
## Hot Standby Configuration
|
||||
|
||||
### On the Primary
|
||||
|
||||
```sql
|
||||
-- Check current settings
|
||||
SHOW wal_level; -- must be 'replica' or 'logical'
|
||||
SHOW max_wal_senders; -- must have available sender slots
|
||||
SHOW max_replication_slots; -- one per standby for slot-based replication
|
||||
```
|
||||
|
||||
Key settings (in `postgresql.conf`):
|
||||
- `wal_level = replica` (default)
|
||||
- `max_wal_senders = 10` (enough for all standbys)
|
||||
- `max_replication_slots = 10` (optional but recommended — prevents WAL removal before standby catches up)
|
||||
|
||||
### On the Standby
|
||||
|
||||
Key settings:
|
||||
- `hot_standby = on` (allows read queries during recovery — default)
|
||||
- `primary_conninfo = 'host=primary_host user=repl_user ...'` (connection to primary)
|
||||
- `primary_slot_name = 'standby_slot'` (optional — use a replication slot)
|
||||
|
||||
### Replication Slots (Recommended)
|
||||
|
||||
Slots prevent the primary from removing WAL segments before the standby has replayed them:
|
||||
|
||||
```sql
|
||||
-- On the primary: create a physical replication slot
|
||||
SELECT pg_create_physical_replication_slot('standby1_slot');
|
||||
|
||||
-- List slots
|
||||
SELECT slot_name, slot_type, active, restart_lsn
|
||||
FROM pg_replication_slots;
|
||||
|
||||
-- Drop a slot (if standby is permanently removed)
|
||||
SELECT pg_drop_replication_slot('standby1_slot');
|
||||
```
|
||||
|
||||
**Warning**: An inactive slot causes WAL to accumulate indefinitely on the primary, potentially filling the disk.
|
||||
|
||||
## Monitoring Replication Lag
|
||||
|
||||
### On the Primary: pg_stat_replication
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
pid,
|
||||
application_name,
|
||||
client_addr,
|
||||
state,
|
||||
sent_lsn,
|
||||
write_lsn,
|
||||
flush_lsn,
|
||||
replay_lsn,
|
||||
pg_size_pretty(pg_wal_lsn_diff(sent_lsn, replay_lsn)) AS replay_lag,
|
||||
pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), sent_lsn)) AS send_lag,
|
||||
write_lag,
|
||||
flush_lag,
|
||||
replay_lag AS replay_lag_time
|
||||
FROM pg_stat_replication;
|
||||
```
|
||||
|
||||
**LSN progression**: `pg_current_wal_lsn()` → `sent_lsn` → `write_lsn` → `flush_lsn` → `replay_lsn`. The gap between any two is a measure of lag at that stage.
|
||||
|
||||
### On the Standby: pg_stat_wal_receiver
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
status,
|
||||
written_lsn,
|
||||
flushed_lsn,
|
||||
latest_end_lsn,
|
||||
latest_end_time,
|
||||
slot_name,
|
||||
conninfo
|
||||
FROM pg_stat_wal_receiver;
|
||||
```
|
||||
|
||||
### On the Standby: Lag in Seconds
|
||||
|
||||
```sql
|
||||
-- How far behind is the standby?
|
||||
SELECT
|
||||
now() - pg_last_xact_replay_timestamp() AS replay_lag,
|
||||
pg_last_wal_receive_lsn() AS received_lsn,
|
||||
pg_last_wal_replay_lsn() AS replayed_lsn,
|
||||
pg_wal_lsn_diff(pg_last_wal_receive_lsn(), pg_last_wal_replay_lsn()) AS replay_lag_bytes;
|
||||
```
|
||||
|
||||
**Caveat**: `pg_last_xact_replay_timestamp()` only updates when transactions are replayed. On an idle primary, it may show a large lag even though the standby is fully caught up. Check `received_lsn = replayed_lsn` for actual status.
|
||||
|
||||
### Is the Standby in Recovery?
|
||||
|
||||
```sql
|
||||
-- Returns true on a standby, false on a primary
|
||||
SELECT pg_is_in_recovery();
|
||||
```
|
||||
|
||||
## Replication Conflicts
|
||||
|
||||
On a hot standby, long-running read queries can conflict with WAL replay. When replay needs to apply changes that conflict with an active query (e.g., dropping a table, vacuuming rows the query is reading), PostgreSQL must choose: wait for the query or cancel it.
|
||||
|
||||
### max_standby_streaming_delay
|
||||
|
||||
Controls how long the standby waits for conflicting queries before cancelling them:
|
||||
|
||||
```sql
|
||||
SHOW max_standby_streaming_delay; -- default: 30s
|
||||
```
|
||||
|
||||
- `30s` (default): Standby waits up to 30 seconds, then cancels conflicting queries
|
||||
- `-1`: Wait forever (replay pauses until the query finishes — can cause unbounded lag)
|
||||
- `0`: Cancel conflicting queries immediately (minimal lag, but queries may fail)
|
||||
|
||||
### max_standby_archive_delay
|
||||
|
||||
Same concept but for WAL segments being replayed from archive (rather than streaming):
|
||||
|
||||
```sql
|
||||
SHOW max_standby_archive_delay; -- default: 30s
|
||||
```
|
||||
|
||||
### The Common Error
|
||||
|
||||
When a query on the standby is cancelled due to a conflict:
|
||||
|
||||
```
|
||||
ERROR: canceling statement due to conflict with recovery
|
||||
DETAIL: User was holding shared buffer pin for too long.
|
||||
```
|
||||
|
||||
**Solutions (in order of preference):**
|
||||
1. Keep queries on the standby short
|
||||
2. Increase `max_standby_streaming_delay` (trades lag for query stability)
|
||||
3. Enable `hot_standby_feedback` (see below)
|
||||
4. Use a logical replica instead of hot standby for long-running analytics
|
||||
|
||||
### hot_standby_feedback
|
||||
|
||||
When enabled, the standby informs the primary about which rows it still needs, preventing the primary's VACUUM from removing them:
|
||||
|
||||
```sql
|
||||
SHOW hot_standby_feedback; -- default: off
|
||||
```
|
||||
|
||||
**Pros**: Eliminates most replication conflicts — long queries on the standby won't be cancelled.
|
||||
|
||||
**Cons**: Can cause table bloat on the primary because VACUUM can't remove dead rows that the standby still references.
|
||||
|
||||
**Recommendation**: Enable only if you have long-running analytical queries on the standby and can tolerate some extra bloat on the primary.
|
||||
|
||||
## Synchronous vs Asynchronous Replication
|
||||
|
||||
### Asynchronous (Default)
|
||||
|
||||
The primary doesn't wait for standby acknowledgment before committing. Fastest, but a primary failure can lose recently committed transactions not yet replicated.
|
||||
|
||||
### Synchronous
|
||||
|
||||
The primary waits for at least one standby to confirm before returning commit success:
|
||||
|
||||
```sql
|
||||
-- On the primary
|
||||
SHOW synchronous_standby_names; -- e.g., 'standby1'
|
||||
SHOW synchronous_commit; -- 'on', 'remote_write', 'remote_apply', etc.
|
||||
```
|
||||
|
||||
Synchronous commit levels:
|
||||
|
||||
| Level | Primary waits for | Durability | Latency impact |
|
||||
|-------|-------------------|------------|----------------|
|
||||
| `on` (default with sync standbys) | Standby WAL flush | Strong | Moderate |
|
||||
| `remote_write` | Standby WAL write (not fsync) | Good | Lower |
|
||||
| `remote_apply` | Standby WAL replay | Strongest (read-your-writes on standby) | Highest |
|
||||
| `local` | Local WAL flush only | Primary only | None |
|
||||
| `off` | Nothing | Weakest | None |
|
||||
|
||||
## Promoting a Standby
|
||||
|
||||
To promote a standby to become the new primary (failover):
|
||||
|
||||
```sql
|
||||
-- On the standby:
|
||||
SELECT pg_promote();
|
||||
|
||||
-- Or from the command line:
|
||||
-- pg_ctl promote -D /path/to/data
|
||||
```
|
||||
|
||||
After promotion:
|
||||
- The standby stops replay and opens for writes
|
||||
- Applications must be redirected to the new primary
|
||||
- Other standbys must be reconfigured to follow the new primary
|
||||
- The old primary must not be restarted without reconfiguring (risk of split-brain)
|
||||
@@ -0,0 +1,318 @@
|
||||
# Indexing Reference
|
||||
|
||||
## Contents
|
||||
- Index types and when to use each
|
||||
- Composite index column ordering
|
||||
- Partial indexes
|
||||
- Covering indexes (INCLUDE)
|
||||
- Expression indexes
|
||||
- Index maintenance and bloat
|
||||
- Finding unused and duplicate indexes
|
||||
|
||||
## Index Types
|
||||
|
||||
### B-tree (Default)
|
||||
|
||||
Supports: `=`, `<`, `>`, `<=`, `>=`, `BETWEEN`, `IN`, `IS NULL`, `IS NOT NULL`
|
||||
|
||||
```sql
|
||||
CREATE INDEX idx_orders_created ON orders(created_at);
|
||||
```
|
||||
|
||||
Best for: equality and range queries on scalar types. The default and most common choice.
|
||||
|
||||
### GIN (Generalized Inverted Index)
|
||||
|
||||
Supports: containment operators on composite values.
|
||||
|
||||
```sql
|
||||
-- JSONB containment
|
||||
CREATE INDEX idx_data_gin ON events USING gin(payload);
|
||||
-- Matches: WHERE payload @> '{"status": "active"}'
|
||||
|
||||
-- Array containment
|
||||
CREATE INDEX idx_tags_gin ON articles USING gin(tags);
|
||||
-- Matches: WHERE tags @> ARRAY['postgres']
|
||||
|
||||
-- Full-text search
|
||||
CREATE INDEX idx_fts ON articles USING gin(to_tsvector('english', body));
|
||||
-- Matches: WHERE to_tsvector('english', body) @@ to_tsquery('postgres & index')
|
||||
|
||||
-- Trigram (fuzzy text search)
|
||||
CREATE EXTENSION IF NOT EXISTS pg_trgm;
|
||||
CREATE INDEX idx_name_trgm ON users USING gin(name gin_trgm_ops);
|
||||
-- Matches: WHERE name ILIKE '%pattern%'
|
||||
```
|
||||
|
||||
**Operator class tip**: For JSONB, `jsonb_path_ops` is 2-3x smaller than the default `jsonb_ops` but only supports `@>` (containment). Use it when you only need containment queries:
|
||||
|
||||
```sql
|
||||
CREATE INDEX idx_events_gin_path ON events USING gin(payload jsonb_path_ops);
|
||||
-- Only supports: WHERE payload @> '{"type": "click"}'
|
||||
-- Does NOT support: WHERE payload ? 'type'
|
||||
```
|
||||
|
||||
**Stored generated column for FTS**:
|
||||
|
||||
```sql
|
||||
ALTER TABLE articles ADD COLUMN search_vector tsvector
|
||||
GENERATED ALWAYS AS (to_tsvector('english', title || ' ' || body)) STORED;
|
||||
CREATE INDEX idx_articles_search ON articles USING gin(search_vector);
|
||||
-- Query: WHERE search_vector @@ to_tsquery('english', 'postgres & optimization')
|
||||
```
|
||||
|
||||
GIN indexes are larger and slower to update than B-tree but excel at multi-valued containment queries.
|
||||
|
||||
**Parallel GIN builds (PG18+)**: GIN indexes can now be built in parallel, significantly speeding up index creation on large tables.
|
||||
|
||||
### GiST (Generalized Search Tree)
|
||||
|
||||
Supports: overlap, containment, nearest-neighbor on geometric, range, and full-text types.
|
||||
|
||||
```sql
|
||||
-- Range overlap
|
||||
CREATE INDEX idx_booking_range ON bookings USING gist(during);
|
||||
-- Matches: WHERE during && '[2024-01-01, 2024-02-01)'
|
||||
|
||||
-- Used in exclusion constraints
|
||||
CREATE EXTENSION IF NOT EXISTS btree_gist;
|
||||
ALTER TABLE bookings
|
||||
ADD CONSTRAINT bookings_no_overlap
|
||||
EXCLUDE USING gist (room_id WITH =, during WITH &&);
|
||||
```
|
||||
|
||||
### BRIN (Block Range Index)
|
||||
|
||||
Tiny index for naturally ordered data (e.g., append-only timestamp columns).
|
||||
|
||||
```sql
|
||||
CREATE INDEX idx_events_ts_brin ON events USING brin(occurred_at);
|
||||
```
|
||||
|
||||
BRIN stores min/max per block range. Effective when physical row order correlates with column value. Very small (~0.1% of B-tree size) but less precise — may read extra blocks.
|
||||
|
||||
### Hash
|
||||
|
||||
Only supports `=`. Rarely better than B-tree.
|
||||
|
||||
```sql
|
||||
CREATE INDEX idx_session_hash ON sessions USING hash(session_token);
|
||||
```
|
||||
|
||||
## Composite Index Column Ordering
|
||||
|
||||
Order matters. The index is a sorted tree, leftmost column first.
|
||||
|
||||
**Rule of thumb**: equality columns first, then range/sort columns.
|
||||
|
||||
```sql
|
||||
-- Query: WHERE tenant_id = 1 AND created_at > '2024-01-01' ORDER BY created_at
|
||||
CREATE INDEX idx_tenant_created ON orders(tenant_id, created_at);
|
||||
```
|
||||
|
||||
The index skips to `tenant_id = 1` (equality), then range-scans `created_at` in order.
|
||||
|
||||
**Leading column rule**: A composite index on `(a, b, c)` can serve queries on:
|
||||
- `a` alone
|
||||
- `a, b`
|
||||
- `a, b, c`
|
||||
|
||||
But NOT `b` alone or `c` alone (pre-PG18).
|
||||
|
||||
**B-tree skip scan (PG18+)**: PG18 can skip through distinct values of leading columns, so an index on `(a, b, c)` can now serve queries on `b` or `c` alone — by scanning each distinct `a` value. This works best when the leading column has low cardinality (few distinct values). It eliminates many cases where you previously needed a separate single-column index.
|
||||
|
||||
## Partial Indexes
|
||||
|
||||
Index only the rows that matter. Smaller index = faster lookups, less maintenance.
|
||||
|
||||
```sql
|
||||
-- Only index active orders (95% of queries filter on active)
|
||||
CREATE INDEX idx_orders_active ON orders(customer_id)
|
||||
WHERE status = 'active';
|
||||
|
||||
-- Only index non-null values
|
||||
CREATE INDEX idx_orders_shipped ON orders(shipped_at)
|
||||
WHERE shipped_at IS NOT NULL;
|
||||
```
|
||||
|
||||
The query's WHERE clause must match (or imply) the index predicate for the planner to use it.
|
||||
|
||||
## Covering Indexes (INCLUDE)
|
||||
|
||||
Add non-key columns to enable index-only scans without bloating the B-tree structure.
|
||||
|
||||
```sql
|
||||
-- Query: SELECT email, name FROM users WHERE email = ?
|
||||
CREATE UNIQUE INDEX idx_users_email ON users(email) INCLUDE (name);
|
||||
```
|
||||
|
||||
The `name` column is stored in the index leaf pages but not in the B-tree structure. This enables an index-only scan (no heap fetch) without affecting index ordering or uniqueness.
|
||||
|
||||
## Expression Indexes
|
||||
|
||||
Index the result of an expression or function.
|
||||
|
||||
```sql
|
||||
-- Case-insensitive email lookup
|
||||
CREATE UNIQUE INDEX idx_users_email_lower ON users(lower(email));
|
||||
-- Query must match: WHERE lower(email) = lower($1)
|
||||
|
||||
-- JSONB field extraction
|
||||
CREATE INDEX idx_events_type ON events((payload->>'type'));
|
||||
-- Query: WHERE payload->>'type' = 'click'
|
||||
|
||||
-- Date truncation
|
||||
-- timestamptz must be made timezone-independent for an immutable expression
|
||||
CREATE INDEX idx_orders_month
|
||||
ON orders(date_trunc('month', created_at AT TIME ZONE 'UTC'));
|
||||
```
|
||||
|
||||
The query must use the same expression for the planner to match the index.
|
||||
|
||||
## Indexes on Partitioned Tables
|
||||
|
||||
Indexes defined on a partitioned parent table are **automatically created on all existing and future child partitions**.
|
||||
|
||||
```sql
|
||||
-- Create index on the partitioned parent — propagates to all partitions
|
||||
CREATE INDEX idx_events_type ON events(occurred_at, (payload->>'type'));
|
||||
```
|
||||
|
||||
### Key Rules
|
||||
|
||||
- **Partition key in unique indexes**: Any UNIQUE or PRIMARY KEY index on a partitioned table must include all partition key columns.
|
||||
|
||||
```sql
|
||||
-- Partitioned by occurred_at — PK must include it
|
||||
CREATE TABLE events (
|
||||
id bigint GENERATED ALWAYS AS IDENTITY,
|
||||
occurred_at timestamptz NOT NULL,
|
||||
payload jsonb
|
||||
) PARTITION BY RANGE (occurred_at);
|
||||
|
||||
-- This works:
|
||||
ALTER TABLE events ADD PRIMARY KEY (id, occurred_at);
|
||||
|
||||
-- This fails: partition key 'occurred_at' not in the constraint
|
||||
-- ALTER TABLE events ADD PRIMARY KEY (id);
|
||||
```
|
||||
|
||||
- **Per-partition indexes**: You can also create indexes on individual partitions for partition-specific optimizations. These won't propagate to other partitions.
|
||||
|
||||
```sql
|
||||
-- Extra index only on the hot partition
|
||||
CREATE INDEX idx_events_q1_status ON events_2024_q1((payload->>'status'));
|
||||
```
|
||||
|
||||
- **CONCURRENTLY on partitioned tables**: PostgreSQL does not support `CREATE INDEX CONCURRENTLY` directly on a partitioned parent. Create an invalid parent index with `ON ONLY`, build matching indexes concurrently on each partition, then attach them:
|
||||
|
||||
```sql
|
||||
CREATE INDEX idx_events_customer
|
||||
ON ONLY events ((payload->>'customer_id'));
|
||||
|
||||
CREATE INDEX CONCURRENTLY idx_events_q1_customer
|
||||
ON events_2024_q1 ((payload->>'customer_id'));
|
||||
|
||||
ALTER INDEX idx_events_customer
|
||||
ATTACH PARTITION idx_events_q1_customer;
|
||||
```
|
||||
|
||||
Repeat the concurrent build and attach steps for every partition. The parent index becomes valid after all partition indexes are attached. If a child build fails, drop or rebuild that invalid child index before attaching it.
|
||||
|
||||
- **Index-only scans**: Work across partitions. The planner prunes irrelevant partitions first, then uses index-only scans on the remaining ones.
|
||||
|
||||
- **REINDEX on partitioned tables**: `REINDEX TABLE` on a partitioned table reindexes all partitions.
|
||||
|
||||
```sql
|
||||
REINDEX TABLE CONCURRENTLY events;
|
||||
```
|
||||
|
||||
## Concurrent Index Creation
|
||||
|
||||
For production use, always create indexes concurrently to avoid locking writes:
|
||||
|
||||
```sql
|
||||
CREATE INDEX CONCURRENTLY idx_orders_customer ON orders(customer_id);
|
||||
```
|
||||
|
||||
Caveats:
|
||||
- Takes longer (two table scans instead of one)
|
||||
- Cannot run inside a transaction block
|
||||
- If it fails, leaves an `INVALID` index — drop and retry
|
||||
- Check for invalid indexes: `SELECT * FROM pg_index WHERE NOT indisvalid;`
|
||||
|
||||
## Index Maintenance
|
||||
|
||||
### Reindexing Bloated Indexes
|
||||
|
||||
```sql
|
||||
-- Concurrent reindex
|
||||
REINDEX INDEX CONCURRENTLY idx_orders_customer;
|
||||
|
||||
-- Or reindex all indexes on a table
|
||||
REINDEX TABLE CONCURRENTLY orders;
|
||||
```
|
||||
|
||||
### Monitoring Index Size
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
indexrelid::regclass AS index_name,
|
||||
pg_size_pretty(pg_relation_size(indexrelid)) AS size,
|
||||
idx_scan AS scans,
|
||||
idx_tup_read AS tuples_read
|
||||
FROM pg_stat_user_indexes
|
||||
ORDER BY pg_relation_size(indexrelid) DESC;
|
||||
```
|
||||
|
||||
## Finding Unused Indexes
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
schemaname,
|
||||
relname AS table_name,
|
||||
indexrelname AS index_name,
|
||||
pg_size_pretty(pg_relation_size(indexrelid)) AS index_size,
|
||||
idx_scan AS times_used
|
||||
FROM pg_stat_user_indexes
|
||||
WHERE idx_scan = 0
|
||||
AND indexrelname NOT LIKE '%_pkey' -- exclude primary keys
|
||||
AND indexrelname NOT LIKE '%unique%' -- exclude unique constraints
|
||||
ORDER BY pg_relation_size(indexrelid) DESC;
|
||||
```
|
||||
|
||||
Reset statistics after a representative period: `SELECT pg_stat_reset();`
|
||||
|
||||
## Finding Duplicate Indexes
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
array_agg(indexrelid::regclass) AS indexes,
|
||||
indrelid::regclass AS table_name,
|
||||
indkey AS column_numbers
|
||||
FROM pg_index
|
||||
GROUP BY indrelid, indkey
|
||||
HAVING count(*) > 1;
|
||||
```
|
||||
|
||||
Also check for indexes that are a prefix of another:
|
||||
- `(a)` is redundant if `(a, b)` exists
|
||||
- `(a, b)` is NOT redundant if `(a, b, c)` exists and you need index-only scans on just `(a, b)`
|
||||
|
||||
## Index Selection Decision Tree
|
||||
|
||||
1. **What operator?**
|
||||
- `=`, `<`, `>`, `BETWEEN` → B-tree
|
||||
- `@>`, `?`, `&&` on jsonb/array → GIN
|
||||
- `@@` full-text → GIN
|
||||
- `ILIKE '%x%'` → GIN + pg_trgm
|
||||
- Range/geometric overlap → GiST
|
||||
- Naturally ordered append-only → BRIN
|
||||
|
||||
2. **How many rows match?** If > 10-20% of table, index may not help (seq scan is cheaper).
|
||||
|
||||
3. **Can you narrow the index?** Use a partial index if most queries filter on a subset.
|
||||
|
||||
4. **Do you need index-only scans?** Add `INCLUDE` columns.
|
||||
|
||||
5. **Is the column an expression?** Use an expression index.
|
||||
@@ -0,0 +1,610 @@
|
||||
# Logical Replication & Migrations Reference
|
||||
|
||||
## Contents
|
||||
- Overview and prerequisites
|
||||
- Publisher setup
|
||||
- Subscriber setup
|
||||
- Managing publications (add/remove tables)
|
||||
- Managing subscriptions
|
||||
- Monitoring replication progress
|
||||
- Schema changes during replication
|
||||
- Live migration patterns
|
||||
- Troubleshooting
|
||||
|
||||
## Overview
|
||||
|
||||
Logical replication streams row-level changes (INSERT, UPDATE, DELETE) from a **publisher** to one or more **subscribers** using the publish/subscribe model. Unlike physical replication, it:
|
||||
|
||||
- Replicates specific tables, not the entire cluster
|
||||
- Allows different indexes, security policies, or schemas on the subscriber
|
||||
- Works across different PG major versions (useful for upgrades)
|
||||
- Allows the subscriber to be writable (for other tables)
|
||||
|
||||
### Prerequisites
|
||||
|
||||
**On the publisher:**
|
||||
- `wal_level = logical` (requires restart if changing)
|
||||
- Sufficient `max_replication_slots` (one per subscription)
|
||||
- Sufficient `max_wal_senders` (one per subscription + headroom)
|
||||
- Tables must have a replica identity (primary key by default)
|
||||
- The replication role needs `REPLICATION` privilege, plus `USAGE` on the schema and `SELECT` on replicated tables
|
||||
|
||||
**On the subscriber:**
|
||||
- Target tables must already exist with compatible schema
|
||||
- Sufficient `max_logical_replication_workers`
|
||||
- Sufficient `max_worker_processes`
|
||||
- The role needs `pg_create_subscription` membership (PG16+)
|
||||
|
||||
Check current settings:
|
||||
|
||||
```sql
|
||||
SHOW wal_level; -- must be 'logical'
|
||||
SHOW max_replication_slots; -- must have available slots
|
||||
SHOW max_wal_senders; -- must have available senders
|
||||
|
||||
-- Check how many slots are already in use
|
||||
SELECT count(*) AS used_slots FROM pg_replication_slots;
|
||||
```
|
||||
|
||||
**How many slots does a subscription need?**
|
||||
|
||||
During **initial sync**, Postgres creates one temporary replication slot per table being copied in parallel, plus one permanent slot for the subscription itself. For example, a subscription syncing 10 tables with `max_sync_workers_per_subscription = 2` (default) uses up to **3 slots** at peak: 1 permanent + 2 temporary for parallel table copy.
|
||||
|
||||
Once initial sync completes and the subscription enters **streaming (CDC) mode**, only the **1 permanent slot** per subscription is used. The temporary per-table slots are dropped.
|
||||
|
||||
**Rule of thumb**: `max_replication_slots` >= (number of subscriptions) + (max_sync_workers_per_subscription) + headroom for physical replication slots. The default of 10 is sufficient for most setups.
|
||||
|
||||
## Publisher Setup
|
||||
|
||||
### Create a Publication
|
||||
|
||||
```sql
|
||||
-- Publish specific tables
|
||||
CREATE PUBLICATION my_pub FOR TABLE orders, customers, products;
|
||||
|
||||
-- Publish all tables in a schema (PG15+)
|
||||
CREATE PUBLICATION my_pub FOR TABLES IN SCHEMA public;
|
||||
|
||||
-- Publish all tables in the database
|
||||
-- WARNING: FOR ALL TABLES prevents later ADD/DROP TABLE modifications.
|
||||
-- Prefer listing tables explicitly if you may need to change the set later.
|
||||
CREATE PUBLICATION my_pub FOR ALL TABLES;
|
||||
|
||||
-- Publish only specific operations
|
||||
CREATE PUBLICATION inserts_only FOR TABLE events
|
||||
WITH (publish = 'insert');
|
||||
|
||||
-- Publish with row filter (PG15+)
|
||||
CREATE PUBLICATION active_orders FOR TABLE orders
|
||||
WHERE (status = 'active');
|
||||
|
||||
-- Publish specific columns only (PG15+)
|
||||
CREATE PUBLICATION partial_customers FOR TABLE customers (id, name, email);
|
||||
```
|
||||
|
||||
### Replica Identity
|
||||
|
||||
Logical replication needs a way to identify rows for UPDATE and DELETE. By default, it uses the primary key.
|
||||
|
||||
```sql
|
||||
-- Check current replica identity
|
||||
SELECT relname, relreplident
|
||||
FROM pg_class
|
||||
WHERE relname IN ('orders', 'customers');
|
||||
-- 'd' = default (primary key), 'f' = full, 'n' = nothing, 'i' = index
|
||||
|
||||
-- If a table has no primary key, use FULL (sends entire old row)
|
||||
ALTER TABLE legacy_table REPLICA IDENTITY FULL;
|
||||
|
||||
-- Or use a unique index
|
||||
CREATE UNIQUE INDEX idx_legacy_key ON legacy_table(external_id);
|
||||
ALTER TABLE legacy_table REPLICA IDENTITY USING INDEX idx_legacy_key;
|
||||
```
|
||||
|
||||
**Without a replica identity, UPDATE and DELETE will fail on the publisher** for that table.
|
||||
|
||||
### Grant Permissions to the Replication Role
|
||||
|
||||
The replication role needs schema access and SELECT on the published tables:
|
||||
|
||||
```sql
|
||||
GRANT USAGE ON SCHEMA public TO repl_user;
|
||||
GRANT SELECT ON ALL TABLES IN SCHEMA public TO repl_user;
|
||||
|
||||
-- Also grant for future tables
|
||||
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO repl_user;
|
||||
```
|
||||
|
||||
### Decoder Plugins
|
||||
|
||||
Postgres supports logical decoding output plugins, including:
|
||||
- **`pgoutput`** (default): built into Postgres, used by native logical replication
|
||||
- **`wal2json`**: an optional third-party plugin that converts WAL to JSON format for CDC integrations
|
||||
|
||||
The decoder is specified when creating a replication slot manually:
|
||||
|
||||
```sql
|
||||
SELECT pg_create_logical_replication_slot('my_slot', 'pgoutput');
|
||||
```
|
||||
|
||||
After installing `wal2json` on the database server:
|
||||
|
||||
```sql
|
||||
SELECT pg_create_logical_replication_slot('my_slot', 'wal2json');
|
||||
```
|
||||
|
||||
When using `CREATE SUBSCRIPTION`, the default `pgoutput` plugin is used automatically.
|
||||
|
||||
### List Publications
|
||||
|
||||
```sql
|
||||
SELECT * FROM pg_publication;
|
||||
|
||||
-- See which tables are in a publication
|
||||
SELECT pubname, schemaname, tablename
|
||||
FROM pg_publication_tables
|
||||
ORDER BY pubname, tablename;
|
||||
```
|
||||
|
||||
## Subscriber Setup
|
||||
|
||||
### Create a Subscription
|
||||
|
||||
```sql
|
||||
-- Basic subscription (triggers initial data copy)
|
||||
CREATE SUBSCRIPTION my_sub
|
||||
CONNECTION 'host=publisher_host dbname=source_db user=repl_user password=...'
|
||||
PUBLICATION my_pub;
|
||||
|
||||
-- Subscribe without initial data copy (tables already have data)
|
||||
CREATE SUBSCRIPTION my_sub
|
||||
CONNECTION 'host=publisher_host dbname=source_db user=repl_user password=...'
|
||||
PUBLICATION my_pub
|
||||
WITH (copy_data = false);
|
||||
|
||||
-- Subscribe to multiple publications
|
||||
CREATE SUBSCRIPTION my_sub
|
||||
CONNECTION 'host=publisher_host dbname=source_db user=repl_user password=...'
|
||||
PUBLICATION pub_orders, pub_customers;
|
||||
|
||||
-- Create disabled (activate later)
|
||||
CREATE SUBSCRIPTION my_sub
|
||||
CONNECTION 'host=publisher_host dbname=source_db user=repl_user password=...'
|
||||
PUBLICATION my_pub
|
||||
WITH (enabled = false);
|
||||
```
|
||||
|
||||
### List Subscriptions
|
||||
|
||||
```sql
|
||||
SELECT * FROM pg_subscription;
|
||||
|
||||
-- Subscription status and worker info
|
||||
SELECT subname, pid, relid::regclass, received_lsn, latest_end_lsn,
|
||||
latest_end_time
|
||||
FROM pg_stat_subscription;
|
||||
```
|
||||
|
||||
## Managing Publications
|
||||
|
||||
### Add Tables
|
||||
|
||||
```sql
|
||||
ALTER PUBLICATION my_pub ADD TABLE new_table;
|
||||
|
||||
-- Add with row filter (PG15+)
|
||||
ALTER PUBLICATION my_pub ADD TABLE audit_log WHERE (created_at > '2024-01-01');
|
||||
|
||||
-- Add multiple tables at once
|
||||
ALTER PUBLICATION my_pub ADD TABLE table_a, table_b, table_c;
|
||||
```
|
||||
|
||||
### Remove Tables
|
||||
|
||||
```sql
|
||||
ALTER PUBLICATION my_pub DROP TABLE old_table;
|
||||
|
||||
-- Remove multiple
|
||||
ALTER PUBLICATION my_pub DROP TABLE table_a, table_b;
|
||||
```
|
||||
|
||||
### Replace Entire Table List
|
||||
|
||||
```sql
|
||||
ALTER PUBLICATION my_pub SET TABLE orders, customers, products, shipments;
|
||||
```
|
||||
|
||||
### Change Published Operations
|
||||
|
||||
```sql
|
||||
-- Only publish inserts and updates (no deletes)
|
||||
ALTER PUBLICATION my_pub SET (publish = 'insert, update');
|
||||
|
||||
-- Restore all operations
|
||||
ALTER PUBLICATION my_pub SET (publish = 'insert, update, delete, truncate');
|
||||
```
|
||||
|
||||
### After Adding Tables — Refresh the Subscriber
|
||||
|
||||
After adding tables to a publication, the subscriber must refresh:
|
||||
|
||||
```sql
|
||||
-- On the subscriber: pick up new tables and copy initial data
|
||||
ALTER SUBSCRIPTION my_sub REFRESH PUBLICATION;
|
||||
|
||||
-- Refresh without copying data for new tables
|
||||
ALTER SUBSCRIPTION my_sub REFRESH PUBLICATION WITH (copy_data = false);
|
||||
```
|
||||
|
||||
### Drop a Publication
|
||||
|
||||
```sql
|
||||
DROP PUBLICATION my_pub;
|
||||
-- or
|
||||
DROP PUBLICATION IF EXISTS my_pub;
|
||||
```
|
||||
|
||||
## Managing Subscriptions
|
||||
|
||||
### Enable / Disable
|
||||
|
||||
```sql
|
||||
-- Pause replication
|
||||
ALTER SUBSCRIPTION my_sub DISABLE;
|
||||
|
||||
-- Resume replication
|
||||
ALTER SUBSCRIPTION my_sub ENABLE;
|
||||
```
|
||||
|
||||
### Change Connection
|
||||
|
||||
```sql
|
||||
ALTER SUBSCRIPTION my_sub CONNECTION 'host=new_host dbname=source_db user=repl_user password=...';
|
||||
```
|
||||
|
||||
### Change Publications
|
||||
|
||||
```sql
|
||||
-- Switch to different publications
|
||||
ALTER SUBSCRIPTION my_sub SET PUBLICATION new_pub;
|
||||
|
||||
-- Add a publication
|
||||
ALTER SUBSCRIPTION my_sub ADD PUBLICATION extra_pub;
|
||||
|
||||
-- Remove a publication
|
||||
ALTER SUBSCRIPTION my_sub DROP PUBLICATION old_pub;
|
||||
```
|
||||
|
||||
### Drop a Subscription
|
||||
|
||||
```sql
|
||||
-- This also drops the replication slot on the publisher
|
||||
DROP SUBSCRIPTION my_sub;
|
||||
```
|
||||
|
||||
If the publisher is unreachable, disable first then drop:
|
||||
|
||||
```sql
|
||||
ALTER SUBSCRIPTION my_sub DISABLE;
|
||||
ALTER SUBSCRIPTION my_sub SET (slot_name = NONE);
|
||||
DROP SUBSCRIPTION my_sub;
|
||||
-- Then manually drop the orphaned slot on the publisher when it's back:
|
||||
-- SELECT pg_drop_replication_slot('my_sub');
|
||||
```
|
||||
|
||||
## Monitoring Replication Progress
|
||||
|
||||
### On the Publisher: Replication Slots and Lag
|
||||
|
||||
```sql
|
||||
-- Replication slot status
|
||||
SELECT
|
||||
slot_name,
|
||||
plugin,
|
||||
slot_type,
|
||||
active,
|
||||
pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn)) AS replication_lag
|
||||
FROM pg_replication_slots
|
||||
WHERE slot_type = 'logical';
|
||||
|
||||
-- Active WAL senders
|
||||
SELECT
|
||||
pid,
|
||||
application_name,
|
||||
client_addr,
|
||||
state,
|
||||
sent_lsn,
|
||||
write_lsn,
|
||||
flush_lsn,
|
||||
replay_lsn,
|
||||
pg_size_pretty(pg_wal_lsn_diff(sent_lsn, replay_lsn)) AS lag
|
||||
FROM pg_stat_replication;
|
||||
```
|
||||
|
||||
### On the Subscriber: Subscription Status
|
||||
|
||||
```sql
|
||||
-- Overall subscription status
|
||||
SELECT
|
||||
subname,
|
||||
pid,
|
||||
received_lsn,
|
||||
latest_end_lsn,
|
||||
latest_end_time,
|
||||
last_msg_send_time,
|
||||
last_msg_receipt_time
|
||||
FROM pg_stat_subscription
|
||||
WHERE subname IS NOT NULL;
|
||||
|
||||
-- Per-table sync state (initial copy progress)
|
||||
SELECT
|
||||
s.subname AS subscription,
|
||||
sr.srrelid::regclass AS table_name,
|
||||
sr.srsubstate AS state,
|
||||
CASE sr.srsubstate
|
||||
WHEN 'i' THEN 'initialize'
|
||||
WHEN 'd' THEN 'data is being copied'
|
||||
WHEN 'f' THEN 'finished table copy'
|
||||
WHEN 's' THEN 'synchronized'
|
||||
WHEN 'r' THEN 'ready (normal replication)'
|
||||
ELSE 'unknown'
|
||||
END AS state_meaning,
|
||||
sr.srsublsn AS lsn
|
||||
FROM pg_catalog.pg_subscription_rel AS sr
|
||||
JOIN pg_catalog.pg_subscription AS s
|
||||
ON s.oid = sr.srsubid
|
||||
ORDER BY s.subname, sr.srrelid::regclass::text;
|
||||
```
|
||||
|
||||
**State codes for `pg_subscription_rel.srsubstate`:**
|
||||
|
||||
| Code | Meaning |
|
||||
|------|---------|
|
||||
| `i` | Initializing |
|
||||
| `d` | Copying data (initial sync) |
|
||||
| `f` | Finished table copy, waiting for sync |
|
||||
| `s` | Synced with publisher |
|
||||
| `r` | Ready (streaming) |
|
||||
|
||||
These codes describe each table's initialization state. `s` means the table synchronized during initialization; it does not prove that current replication lag is zero.
|
||||
|
||||
### Replication Lag Monitoring Query
|
||||
|
||||
Run on the subscriber to check how far behind it is:
|
||||
|
||||
```sql
|
||||
-- Lag in bytes and time
|
||||
SELECT
|
||||
s.subname,
|
||||
st.received_lsn,
|
||||
st.latest_end_lsn,
|
||||
st.latest_end_time,
|
||||
now() - st.latest_end_time AS time_lag
|
||||
FROM pg_subscription s
|
||||
JOIN pg_stat_subscription st ON st.subid = s.oid
|
||||
WHERE st.pid IS NOT NULL
|
||||
AND st.relid IS NULL
|
||||
-- PG17+ can also expose parallel apply workers; keep only the leader.
|
||||
AND coalesce(to_jsonb(st)->>'worker_type', 'apply') = 'apply';
|
||||
```
|
||||
|
||||
## Schema Changes During Replication
|
||||
|
||||
Logical replication does **NOT** replicate DDL. Schema changes must be applied manually on both sides.
|
||||
|
||||
### Safe Pattern for Adding a Column
|
||||
|
||||
```sql
|
||||
-- 1. Add column on SUBSCRIBER first (nullable, no default)
|
||||
ALTER TABLE orders ADD COLUMN priority int;
|
||||
|
||||
-- 2. Add column on PUBLISHER
|
||||
ALTER TABLE orders ADD COLUMN priority int;
|
||||
|
||||
-- 3. New rows will now include the column
|
||||
-- Existing replicated rows will have NULL for the new column
|
||||
```
|
||||
|
||||
**Add on subscriber first** to avoid errors when the publisher starts sending the new column before the subscriber schema is updated.
|
||||
|
||||
### Safe Pattern for Dropping a Column
|
||||
|
||||
```sql
|
||||
-- 1. Remove column from PUBLICATION (PG15+, if using column lists)
|
||||
ALTER PUBLICATION my_pub SET TABLE orders (id, customer_id, total, created_at);
|
||||
|
||||
-- 2. Drop column on PUBLISHER
|
||||
ALTER TABLE orders DROP COLUMN old_column;
|
||||
|
||||
-- 3. Drop column on SUBSCRIBER
|
||||
ALTER TABLE orders DROP COLUMN old_column;
|
||||
```
|
||||
|
||||
### Adding a New Table to Replication
|
||||
|
||||
```sql
|
||||
-- 1. Create table on SUBSCRIBER with matching schema
|
||||
CREATE TABLE shipments (...);
|
||||
|
||||
-- 2. Add table to publication on PUBLISHER
|
||||
ALTER PUBLICATION my_pub ADD TABLE shipments;
|
||||
|
||||
-- 3. Refresh on SUBSCRIBER (copies existing data)
|
||||
ALTER SUBSCRIPTION my_sub REFRESH PUBLICATION;
|
||||
```
|
||||
|
||||
## Live Migration Patterns
|
||||
|
||||
### Migration to a New Database (Minimal Downtime)
|
||||
|
||||
1. **Set up target**: Create schema and constraints on the new database. **Defer index creation** until after initial data copy completes — this significantly speeds up the initial sync.
|
||||
2. **Start replication**: Create publication on source, subscription on target
|
||||
3. **Wait for sync**: Monitor until all tables reach `r` (ready) state
|
||||
4. **Verify**: Compare row counts, spot-check data
|
||||
5. **Cutover**:
|
||||
- Stop writes to source (set `default_transaction_read_only = on` or revoke write access)
|
||||
- Wait for final lag to drain to zero
|
||||
- Verify sequences: advance sequences on target to match source
|
||||
- Switch application connection strings
|
||||
6. **Cleanup**: Drop subscription, drop publication, drop replication slot
|
||||
|
||||
### Sequence Synchronization
|
||||
|
||||
Logical replication does **NOT** replicate sequences. Before cutover, sync them:
|
||||
|
||||
```sql
|
||||
-- On SOURCE: get current sequence values
|
||||
SELECT schemaname, sequencename, last_value
|
||||
FROM pg_sequences
|
||||
WHERE schemaname = 'public';
|
||||
|
||||
-- On TARGET: substitute the value obtained from the source.
|
||||
-- Add a buffer only if writes can still reach the source during cutover.
|
||||
SELECT pg_catalog.setval(
|
||||
'public.orders_id_seq'::regclass,
|
||||
<source_last_value> + 1000,
|
||||
true
|
||||
);
|
||||
```
|
||||
|
||||
### Row Count Verification
|
||||
|
||||
```sql
|
||||
-- Run on both source and target, compare results
|
||||
SELECT
|
||||
schemaname,
|
||||
relname,
|
||||
n_live_tup AS approx_row_count
|
||||
FROM pg_stat_user_tables
|
||||
WHERE schemaname = 'public'
|
||||
ORDER BY relname;
|
||||
```
|
||||
|
||||
For exact counts (slower):
|
||||
|
||||
```sql
|
||||
CREATE TEMP TABLE exact_row_counts (
|
||||
schemaname name NOT NULL,
|
||||
tablename name NOT NULL,
|
||||
exact_count bigint NOT NULL
|
||||
);
|
||||
|
||||
DO $$
|
||||
DECLARE r record;
|
||||
BEGIN
|
||||
FOR r IN
|
||||
SELECT schemaname, tablename
|
||||
FROM pg_catalog.pg_tables
|
||||
WHERE schemaname = 'public'
|
||||
ORDER BY tablename
|
||||
LOOP
|
||||
EXECUTE format(
|
||||
'INSERT INTO pg_temp.exact_row_counts
|
||||
SELECT %L::name, %L::name, count(*)
|
||||
FROM %I.%I',
|
||||
r.schemaname, r.tablename,
|
||||
r.schemaname, r.tablename
|
||||
);
|
||||
END LOOP;
|
||||
END $$;
|
||||
|
||||
SELECT
|
||||
format('%I.%I', schemaname, tablename) AS table_name,
|
||||
exact_count
|
||||
FROM pg_temp.exact_row_counts
|
||||
ORDER BY schemaname, tablename;
|
||||
|
||||
DROP TABLE pg_temp.exact_row_counts;
|
||||
```
|
||||
|
||||
Run the exact-count query on both source and target only after stopping writes and allowing replication lag to drain. It performs a full scan of every selected table.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Replication Slot Growing / WAL Accumulation
|
||||
|
||||
If a subscriber falls behind or is disconnected, WAL accumulates on the publisher:
|
||||
|
||||
```sql
|
||||
-- Check slot lag on publisher
|
||||
SELECT slot_name, active,
|
||||
pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained_wal
|
||||
FROM pg_replication_slots;
|
||||
```
|
||||
|
||||
**Fix**: Reconnect the subscriber, or drop the slot if the subscription is no longer needed:
|
||||
|
||||
```sql
|
||||
-- Drop an inactive slot (publisher)
|
||||
SELECT pg_drop_replication_slot('orphaned_slot_name');
|
||||
```
|
||||
|
||||
### Initial Sync Stuck or Slow
|
||||
|
||||
```sql
|
||||
-- Check which tables are still copying
|
||||
SELECT srrelid::regclass AS table_name, srsubstate AS state
|
||||
FROM pg_subscription_rel
|
||||
WHERE srsubstate != 'r';
|
||||
-- 'd' = still copying data, 'i' = initializing
|
||||
```
|
||||
|
||||
Initial sync speed depends on table size and network. For very large tables, consider:
|
||||
- Dump/restore the data first, then create subscription with `copy_data = false`
|
||||
- Increase `max_logical_replication_workers` if many tables need initial sync in parallel
|
||||
|
||||
### Conflict Errors
|
||||
|
||||
If the subscriber has conflicting data (e.g., duplicate key):
|
||||
|
||||
```sql
|
||||
-- PG15+: cumulative subscription apply-error counters
|
||||
SELECT * FROM pg_stat_subscription_stats;
|
||||
```
|
||||
|
||||
This view reports counters rather than detailed error messages. Check PostgreSQL server logs for the specific conflict; on PG14, logs are the primary source because `pg_stat_subscription_stats` is unavailable.
|
||||
|
||||
**PG16+**: Skip a conflicting transaction:
|
||||
|
||||
```sql
|
||||
ALTER SUBSCRIPTION my_sub SKIP (lsn = '0/12345678');
|
||||
```
|
||||
|
||||
**Pre-PG16**: Delete the conflicting row on the subscriber, then replication will proceed.
|
||||
|
||||
### "Publisher Does Not Exist" After Refresh
|
||||
|
||||
If you renamed or recreated a publication:
|
||||
|
||||
```sql
|
||||
-- Subscriber needs to be pointed to the new publication name
|
||||
ALTER SUBSCRIPTION my_sub SET PUBLICATION new_pub_name;
|
||||
ALTER SUBSCRIPTION my_sub REFRESH PUBLICATION;
|
||||
```
|
||||
|
||||
### Inactive Slot Timeout (PG18+)
|
||||
|
||||
PG18 adds `idle_replication_slot_timeout` to automatically invalidate inactive replication slots, preventing unbounded WAL accumulation:
|
||||
|
||||
```sql
|
||||
SHOW idle_replication_slot_timeout; -- auto-invalidates stale slots
|
||||
```
|
||||
|
||||
### Generated Column Replication (PG18+)
|
||||
|
||||
Logical replication can now replicate generated column values via the `publish_generated_columns` publication option:
|
||||
|
||||
```sql
|
||||
CREATE PUBLICATION my_pub FOR TABLE orders
|
||||
WITH (publish_generated_columns = true);
|
||||
```
|
||||
|
||||
### Check if wal_level is Logical
|
||||
|
||||
```sql
|
||||
SHOW wal_level;
|
||||
-- If not 'logical', it requires a restart to change
|
||||
-- In postgresql.conf: wal_level = logical
|
||||
```
|
||||
|
||||
On managed platforms, `wal_level` may be controlled by the platform's settings UI or API rather than `postgresql.conf`.
|
||||
@@ -0,0 +1,334 @@
|
||||
# Major Version Upgrades Reference
|
||||
|
||||
## Contents
|
||||
- Upgrade methods overview
|
||||
- pg_upgrade (in-place)
|
||||
- Logical replication (minimal downtime)
|
||||
- Pre-upgrade checklist
|
||||
- Post-upgrade checklist
|
||||
- Testing strategy
|
||||
|
||||
## Upgrade Methods Overview
|
||||
|
||||
| Method | Downtime | Complexity | Rollback | Best for |
|
||||
|--------|----------|------------|----------|----------|
|
||||
| `pg_upgrade` | Minutes to hours | Low | Restore from backup | Most upgrades, moderate database sizes |
|
||||
| `pg_upgrade --swap` (PG18+) | Minutes | Low | Restore from backup | Fastest in-place upgrade |
|
||||
| Logical replication | Seconds | High | Switch back to old primary | Large databases requiring near-zero downtime |
|
||||
| pg_dump/pg_restore | Hours to days | Low | Old cluster still running | Small databases, or when other methods fail |
|
||||
|
||||
**Default recommendation**: `pg_upgrade` for most cases. Use logical replication only when you need near-zero downtime on a large database.
|
||||
|
||||
## pg_upgrade (In-Place)
|
||||
|
||||
`pg_upgrade` replaces the old cluster's data files with the new version in place, without dumping and reloading data. PG18 supports five transfer modes:
|
||||
|
||||
| Mode | Flag | Speed | Disk usage |
|
||||
|------|------|-------|------------|
|
||||
| Copy | (default) | Moderate | 2x disk during upgrade |
|
||||
| Copy file range | `--copy-file-range` (PG18+) | Fast where supported | 2x disk during upgrade |
|
||||
| Link | `--link` | Fast | Minimal extra disk (hard links) |
|
||||
| Swap | `--swap` (PG18+) | Fastest | No extra disk (swaps data dirs) |
|
||||
| Clone | `--clone` | Fast (if filesystem supports reflinks) | Minimal extra disk |
|
||||
|
||||
### Step-by-Step: pg_upgrade
|
||||
|
||||
#### 1. Install the New Version
|
||||
|
||||
Install the new PostgreSQL version alongside the old one. Do not remove the old version yet.
|
||||
|
||||
```bash
|
||||
# Example: upgrading from PG16 to PG17
|
||||
# Install PG17 (method depends on your OS/package manager)
|
||||
# Both versions coexist with different binary directories
|
||||
```
|
||||
|
||||
#### 2. Run Pre-Upgrade Check
|
||||
|
||||
```bash
|
||||
# Dry run — checks compatibility without making changes
|
||||
pg_upgrade \
|
||||
--old-datadir /var/lib/postgresql/16/main \
|
||||
--new-datadir /var/lib/postgresql/17/main \
|
||||
--old-bindir /usr/lib/postgresql/16/bin \
|
||||
--new-bindir /usr/lib/postgresql/17/bin \
|
||||
--check
|
||||
```
|
||||
|
||||
Fix any issues reported before proceeding. Common issues:
|
||||
- Extensions not available in the new version
|
||||
- Custom data types with incompatible binary formats
|
||||
- `contrib` modules that need updating
|
||||
|
||||
#### 3. Stop the Old Cluster
|
||||
|
||||
```bash
|
||||
pg_ctl stop -D /var/lib/postgresql/16/main
|
||||
```
|
||||
|
||||
#### 4. Run pg_upgrade
|
||||
|
||||
```bash
|
||||
# Copy mode (safe, uses more disk)
|
||||
pg_upgrade \
|
||||
--old-datadir /var/lib/postgresql/16/main \
|
||||
--new-datadir /var/lib/postgresql/17/main \
|
||||
--old-bindir /usr/lib/postgresql/16/bin \
|
||||
--new-bindir /usr/lib/postgresql/17/bin
|
||||
|
||||
# Link mode (faster, old cluster becomes unusable)
|
||||
pg_upgrade \
|
||||
--old-datadir /var/lib/postgresql/16/main \
|
||||
--new-datadir /var/lib/postgresql/17/main \
|
||||
--old-bindir /usr/lib/postgresql/16/bin \
|
||||
--new-bindir /usr/lib/postgresql/17/bin \
|
||||
--link
|
||||
|
||||
# PG18+: Swap mode (fastest, swaps data directories)
|
||||
pg_upgrade \
|
||||
--old-datadir /var/lib/postgresql/17/main \
|
||||
--new-datadir /var/lib/postgresql/18/main \
|
||||
--old-bindir /usr/lib/postgresql/17/bin \
|
||||
--new-bindir /usr/lib/postgresql/18/bin \
|
||||
--swap
|
||||
|
||||
# Parallel checking: speed up pre-checks
|
||||
pg_upgrade ... --jobs 4
|
||||
```
|
||||
|
||||
#### 5. Start the New Cluster
|
||||
|
||||
```bash
|
||||
pg_ctl start -D /var/lib/postgresql/17/main
|
||||
```
|
||||
|
||||
#### 6. Post-Upgrade Tasks
|
||||
|
||||
After a successful upgrade, refresh optimizer statistics and remove the old cluster only after verification:
|
||||
|
||||
```bash
|
||||
# Update optimizer statistics in stages
|
||||
vacuumdb --all --analyze-in-stages
|
||||
|
||||
# Generated by pg_upgrade — delete old cluster data (only after confirming the upgrade works)
|
||||
./delete_old_cluster.sh
|
||||
```
|
||||
|
||||
Current `pg_upgrade` releases generate the deletion script but may instruct you to run `vacuumdb` directly instead of generating `analyze_new_cluster.sh`.
|
||||
|
||||
### Optimizer Statistics Preservation (PG18+)
|
||||
|
||||
PG18's `pg_upgrade` preserves most optimizer statistics by default, reducing the post-upgrade performance dip:
|
||||
|
||||
```bash
|
||||
# Default in PG18: statistics are preserved
|
||||
pg_upgrade ...
|
||||
|
||||
# Opt out (if you want fresh statistics)
|
||||
pg_upgrade ... --no-statistics
|
||||
```
|
||||
|
||||
Some statistics, including cumulative statistics and statistics that cannot be transferred, still need to be rebuilt. On older versions, always run `ANALYZE` on all databases immediately after upgrading.
|
||||
|
||||
## Logical Replication (Minimal Downtime)
|
||||
|
||||
For near-zero downtime upgrades on large databases, use logical replication to stream changes from the old version to the new version, then cut over.
|
||||
|
||||
### Overview
|
||||
|
||||
1. Set up new cluster on the target PG version
|
||||
2. Create schema on the new cluster
|
||||
3. Set up logical replication (old → new)
|
||||
4. Wait for initial sync + streaming to catch up
|
||||
5. Cut over applications to the new cluster
|
||||
|
||||
See `logical-replication.md` for the full live migration pattern.
|
||||
|
||||
### Advantages Over pg_upgrade
|
||||
|
||||
- Downtime measured in seconds (just the application switchover)
|
||||
- Can upgrade across multiple major versions in one step
|
||||
- Allows testing the new cluster while the old one is still serving traffic
|
||||
- Rollback is simple: point applications back to the old cluster
|
||||
|
||||
### Disadvantages
|
||||
|
||||
- More complex setup
|
||||
- Sequences must be synchronized manually
|
||||
- DDL is not replicated — schema must be created manually on the target
|
||||
- Large objects (lo) are not replicated
|
||||
- Some extensions may behave differently across versions
|
||||
|
||||
### pg_createsubscriber (PG17+)
|
||||
|
||||
Converts a physical standby into a logical subscriber, simplifying the setup:
|
||||
|
||||
```bash
|
||||
# PG18+: --all flag converts all databases at once
|
||||
pg_createsubscriber \
|
||||
--pgdata /var/lib/postgresql/18/main \
|
||||
--publisher-server "host=old_primary dbname=mydb" \
|
||||
--all
|
||||
```
|
||||
|
||||
## Pre-Upgrade Checklist
|
||||
|
||||
### 1. Review Release Notes
|
||||
|
||||
Read the release notes for **every version** between your current and target version. Pay attention to:
|
||||
- Removed features or changed defaults
|
||||
- Extension compatibility changes
|
||||
- Authentication changes (e.g., PG18 deprecates md5)
|
||||
- Behavior changes that could affect queries
|
||||
|
||||
### 2. Check Extension Compatibility
|
||||
|
||||
```sql
|
||||
-- List installed extensions and versions
|
||||
SELECT extname, extversion FROM pg_extension ORDER BY extname;
|
||||
|
||||
-- Check if extensions are available in the new version
|
||||
-- (run against the new cluster after initdb)
|
||||
SELECT name, default_version FROM pg_available_extensions WHERE name IN (
|
||||
SELECT extname FROM pg_extension
|
||||
) ORDER BY name;
|
||||
```
|
||||
|
||||
### 3. Check for Deprecated Features
|
||||
|
||||
```sql
|
||||
-- Check for md5 passwords (deprecated in PG18)
|
||||
SELECT rolname
|
||||
FROM pg_authid
|
||||
WHERE rolcanlogin AND rolpassword LIKE 'md5%';
|
||||
|
||||
-- Check for removed/renamed GUC settings
|
||||
-- (pg_upgrade --check will catch these)
|
||||
```
|
||||
|
||||
### 4. Take a Full Backup
|
||||
|
||||
**Non-negotiable**. Before any upgrade:
|
||||
|
||||
```bash
|
||||
# Full cluster backup
|
||||
pg_dumpall -f pre_upgrade_backup.sql
|
||||
|
||||
# Or physical backup
|
||||
pg_basebackup -D /backup/pre_upgrade -Ft -z -Xs -P
|
||||
```
|
||||
|
||||
### 5. Test on a Clone
|
||||
|
||||
Never upgrade production first. Test on a copy:
|
||||
|
||||
```bash
|
||||
# Create a test copy
|
||||
pg_basebackup -D /tmp/upgrade_test -Fp -Xs -P
|
||||
|
||||
# Initialize an empty target cluster with the new version's binaries
|
||||
/usr/lib/postgresql/17/bin/initdb -D /tmp/new_cluster
|
||||
|
||||
# Run pg_upgrade --check against the copy
|
||||
/usr/lib/postgresql/17/bin/pg_upgrade --check \
|
||||
--old-datadir /tmp/upgrade_test \
|
||||
--new-datadir /tmp/new_cluster \
|
||||
--old-bindir /usr/lib/postgresql/16/bin \
|
||||
--new-bindir /usr/lib/postgresql/17/bin
|
||||
```
|
||||
|
||||
### 6. Plan for Replication
|
||||
|
||||
If using streaming replication:
|
||||
- Standbys must be rebuilt after pg_upgrade (they can't follow a pg_upgraded primary)
|
||||
- Alternative: upgrade standbys with `pg_upgrade` too (stop all, upgrade all, restart)
|
||||
- Or use `pg_createsubscriber` (PG17+) to convert to logical replication
|
||||
|
||||
## Post-Upgrade Checklist
|
||||
|
||||
### 1. Update Optimizer Statistics
|
||||
|
||||
```bash
|
||||
# Rebuild optimizer statistics in stages
|
||||
vacuumdb --all --analyze-in-stages --jobs 4
|
||||
```
|
||||
|
||||
On PG18+ with `pg_upgrade`, most optimizer statistics are preserved by default, but analyze tables whose statistics were not transferred and tables changed after the upgrade.
|
||||
|
||||
### 2. Check for Invalid Indexes
|
||||
|
||||
```sql
|
||||
SELECT indexrelid::regclass, indisvalid
|
||||
FROM pg_index
|
||||
WHERE NOT indisvalid;
|
||||
```
|
||||
|
||||
### 3. Update Extensions
|
||||
|
||||
```sql
|
||||
-- Update extensions to latest available version
|
||||
ALTER EXTENSION pg_stat_statements UPDATE;
|
||||
ALTER EXTENSION postgis UPDATE;
|
||||
-- Repeat for each extension
|
||||
```
|
||||
|
||||
### 4. Review Changed Defaults
|
||||
|
||||
Check if new default values for GUC parameters affect your workload:
|
||||
|
||||
```sql
|
||||
-- Compare settings that differ from defaults
|
||||
SELECT name, setting, boot_val, source
|
||||
FROM pg_settings
|
||||
WHERE setting != boot_val
|
||||
AND source != 'default'
|
||||
ORDER BY name;
|
||||
```
|
||||
|
||||
### 5. Monitor Performance
|
||||
|
||||
After upgrade, monitor for:
|
||||
- Query plan regressions (optimizer changes across versions)
|
||||
- Connection behavior changes
|
||||
- Extension behavior changes
|
||||
|
||||
```sql
|
||||
-- Compare top queries before/after via pg_stat_statements
|
||||
SELECT left(query, 80), calls,
|
||||
round(mean_exec_time::numeric, 1) AS avg_ms
|
||||
FROM pg_stat_statements
|
||||
WHERE dbid = (SELECT oid FROM pg_database WHERE datname = current_database())
|
||||
ORDER BY total_exec_time DESC
|
||||
LIMIT 20;
|
||||
```
|
||||
|
||||
### 6. Clean Up
|
||||
|
||||
```bash
|
||||
# Remove old cluster (only after confirming upgrade is stable)
|
||||
./delete_old_cluster.sh
|
||||
|
||||
# Or manually:
|
||||
rm -rf /var/lib/postgresql/16/main # be very careful with this
|
||||
|
||||
# Uninstall old PostgreSQL version
|
||||
# (method depends on your package manager)
|
||||
```
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Regression Testing
|
||||
|
||||
1. **Dump queries from pg_stat_statements** before upgrade
|
||||
2. **Replay on the test cluster** (or use `pg_stat_statements` to compare plans)
|
||||
3. **Compare EXPLAIN plans** for your top-20 queries by total time
|
||||
4. **Run application test suite** against the new version
|
||||
5. **Load test** at production-like traffic levels
|
||||
|
||||
### Rollback Plan
|
||||
|
||||
- **pg_upgrade (copy mode)**: Old cluster is intact — start it back up
|
||||
- **pg_upgrade (link mode)**: Old cluster is unusable — restore from backup
|
||||
- **pg_upgrade (swap mode)**: The file transfer is destructive; restore the old cluster from backup
|
||||
- **Logical replication**: Point applications back to the old cluster
|
||||
- **Always have a tested backup** regardless of method
|
||||
@@ -0,0 +1,553 @@
|
||||
# Performance Diagnostics Reference
|
||||
|
||||
## Contents
|
||||
- Essential pg_stat views
|
||||
- Table health diagnostics
|
||||
- Index health diagnostics
|
||||
- Active query analysis
|
||||
- Lock analysis
|
||||
- VACUUM and bloat
|
||||
- Connection management
|
||||
- pg_stat_statements setup and queries
|
||||
|
||||
## Essential pg_stat Views
|
||||
|
||||
| View | What it tells you |
|
||||
|------|------------------|
|
||||
| `pg_stat_user_tables` | Seq scans, index scans, row counts, dead tuples, last vacuum/analyze |
|
||||
| `pg_stat_user_indexes` | Index usage counts, tuple reads |
|
||||
| `pg_stat_activity` | Currently running queries, wait events, state |
|
||||
| `pg_stat_statements` | Top queries by time, calls, rows (extension) |
|
||||
| `pg_stat_bgwriter` | Checkpoint frequency, buffer allocation |
|
||||
| `pg_stat_io` (PG16+) | I/O statistics by backend type. PG18+ adds byte-level columns and per-backend stats |
|
||||
| `pg_locks` | Current locks held and awaited |
|
||||
|
||||
## Table Health Diagnostics
|
||||
|
||||
### Tables with Most Sequential Scans
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
schemaname,
|
||||
relname AS table_name,
|
||||
seq_scan,
|
||||
seq_tup_read,
|
||||
idx_scan,
|
||||
pg_size_pretty(pg_total_relation_size(relid)) AS total_size,
|
||||
CASE WHEN seq_scan + idx_scan > 0
|
||||
THEN round(100.0 * idx_scan / (seq_scan + idx_scan), 1)
|
||||
ELSE 0 END AS idx_scan_pct
|
||||
FROM pg_stat_user_tables
|
||||
WHERE seq_scan > 0
|
||||
ORDER BY seq_tup_read DESC
|
||||
LIMIT 20;
|
||||
```
|
||||
|
||||
High `seq_tup_read` with low `idx_scan_pct` = missing index opportunity.
|
||||
|
||||
### Tables with High Dead Tuple Ratio
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
schemaname,
|
||||
relname AS table_name,
|
||||
n_live_tup,
|
||||
n_dead_tup,
|
||||
CASE WHEN n_live_tup > 0
|
||||
THEN round(100.0 * n_dead_tup / n_live_tup, 1)
|
||||
ELSE 0 END AS dead_pct,
|
||||
last_autovacuum,
|
||||
last_autoanalyze
|
||||
FROM pg_stat_user_tables
|
||||
WHERE n_dead_tup > 1000
|
||||
ORDER BY n_dead_tup DESC
|
||||
LIMIT 20;
|
||||
```
|
||||
|
||||
`dead_pct` > 20% = VACUUM is falling behind. Check autovacuum settings.
|
||||
|
||||
### Relation Storage Breakdown
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
schemaname,
|
||||
tablename,
|
||||
pg_size_pretty(pg_total_relation_size(schemaname || '.' || tablename)) AS total_size,
|
||||
pg_size_pretty(
|
||||
pg_total_relation_size(schemaname || '.' || tablename) -
|
||||
pg_relation_size(schemaname || '.' || tablename)
|
||||
) AS index_and_toast_size
|
||||
FROM pg_tables
|
||||
WHERE schemaname NOT IN ('pg_catalog', 'information_schema')
|
||||
ORDER BY pg_total_relation_size(schemaname || '.' || tablename) DESC
|
||||
LIMIT 20;
|
||||
```
|
||||
|
||||
This separates heap size from indexes and TOAST; it is not a bloat estimate. For bloat measurements, use the `pgstattuple` extension (requires superuser or elevated privileges):
|
||||
|
||||
```sql
|
||||
CREATE EXTENSION IF NOT EXISTS pgstattuple;
|
||||
SELECT * FROM pgstattuple('orders');
|
||||
-- dead_tuple_percent > 20% = significant bloat
|
||||
```
|
||||
|
||||
## Index Health Diagnostics
|
||||
|
||||
### Unused Indexes (Wasting Space and Slowing Writes)
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
schemaname,
|
||||
relname AS table_name,
|
||||
indexrelname AS index_name,
|
||||
pg_size_pretty(pg_relation_size(indexrelid)) AS index_size,
|
||||
idx_scan AS times_used
|
||||
FROM pg_stat_user_indexes
|
||||
WHERE idx_scan = 0
|
||||
AND indexrelname NOT LIKE '%_pkey' -- exclude primary keys
|
||||
ORDER BY pg_relation_size(indexrelid) DESC;
|
||||
```
|
||||
|
||||
### Index Hit Rate (Should Be > 99%)
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
sum(idx_blks_hit) AS idx_hit,
|
||||
sum(idx_blks_read) AS idx_read,
|
||||
CASE WHEN sum(idx_blks_hit + idx_blks_read) > 0
|
||||
THEN round(100.0 * sum(idx_blks_hit) / sum(idx_blks_hit + idx_blks_read), 2)
|
||||
ELSE 100 END AS hit_rate_pct
|
||||
FROM pg_statio_user_indexes;
|
||||
```
|
||||
|
||||
### Table Cache Hit Rate (Should Be > 99%)
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
sum(heap_blks_hit) AS heap_hit,
|
||||
sum(heap_blks_read) AS heap_read,
|
||||
CASE WHEN sum(heap_blks_hit + heap_blks_read) > 0
|
||||
THEN round(100.0 * sum(heap_blks_hit) / sum(heap_blks_hit + heap_blks_read), 2)
|
||||
ELSE 100 END AS hit_rate_pct
|
||||
FROM pg_statio_user_tables;
|
||||
```
|
||||
|
||||
Hit rate < 99% = consider increasing `shared_buffers` or optimizing queries.
|
||||
|
||||
### shared_buffers Tuning
|
||||
|
||||
`shared_buffers` is the primary in-memory page cache. Every page read from disk passes through it, and frequently accessed pages stay cached here.
|
||||
|
||||
**Sizing rule of thumb**: Start at **25% of total RAM**. Going higher (up to ~40%) can help on read-heavy workloads with large working sets, but beyond that the OS page cache becomes less effective and returns diminish.
|
||||
|
||||
```sql
|
||||
-- Current setting
|
||||
SHOW shared_buffers;
|
||||
|
||||
-- Effective cache (shared_buffers + OS page cache estimate — planner hint only)
|
||||
SHOW effective_cache_size;
|
||||
```
|
||||
|
||||
**Diagnostic: Is shared_buffers large enough?**
|
||||
|
||||
Combine the cache hit rate above with a working set estimate:
|
||||
|
||||
```sql
|
||||
-- Total size of all user tables and indexes
|
||||
SELECT
|
||||
pg_size_pretty(sum(pg_total_relation_size(relid))) AS total_data_size
|
||||
FROM pg_stat_user_tables;
|
||||
|
||||
-- Compare to shared_buffers
|
||||
SELECT
|
||||
setting || ' ' || unit AS shared_buffers,
|
||||
pg_size_pretty(setting::bigint * 8192) AS shared_buffers_bytes
|
||||
FROM pg_settings
|
||||
WHERE name = 'shared_buffers';
|
||||
```
|
||||
|
||||
If your hot data (frequently accessed tables + their indexes) significantly exceeds `shared_buffers`, the hit rate drops and you'll see more `shared read` in EXPLAIN output.
|
||||
|
||||
**When to increase shared_buffers:**
|
||||
- Table/index cache hit rate consistently below 99%
|
||||
- `shared read` dominates `shared hit` in EXPLAIN plans for hot queries
|
||||
- Server has available RAM (check OS isn't swapping)
|
||||
|
||||
**When NOT to increase shared_buffers:**
|
||||
- Low hit rate caused by full table scans (fix with indexes or query changes, not more cache)
|
||||
- Server is already memory-constrained (each connection also uses `work_mem`, `maintenance_work_mem`, etc.)
|
||||
- Hit rate is already 99%+ (adding more cache won't help)
|
||||
|
||||
**After changing shared_buffers:**
|
||||
- Requires a server restart (`pg_ctl restart`)
|
||||
- Update `effective_cache_size` to roughly `shared_buffers + estimated OS page cache` (typically ~75% of total RAM)
|
||||
- Monitor hit rates for a representative period after the change
|
||||
|
||||
**Per-table cache usage** (requires `pg_buffercache` extension):
|
||||
|
||||
```sql
|
||||
CREATE EXTENSION IF NOT EXISTS pg_buffercache;
|
||||
|
||||
SELECT
|
||||
c.relname,
|
||||
pg_size_pretty(count(*) * current_setting('block_size')::bigint) AS buffered,
|
||||
round(100.0 * count(*) / (SELECT setting::int FROM pg_settings WHERE name = 'shared_buffers'), 1) AS pct_of_cache
|
||||
FROM pg_buffercache b
|
||||
JOIN pg_class c ON pg_relation_filenode(c.oid) = b.relfilenode
|
||||
WHERE b.reldatabase = (SELECT oid FROM pg_database WHERE datname = current_database())
|
||||
GROUP BY c.relname
|
||||
ORDER BY count(*) DESC
|
||||
LIMIT 20;
|
||||
```
|
||||
|
||||
This shows which tables/indexes are consuming the most shared_buffers — useful for identifying cache hogs or verifying that hot tables are actually cached.
|
||||
|
||||
### Finding Missing Foreign Key Indexes
|
||||
|
||||
FK columns without indexes cause slow JOINs and slow CASCADE deletes:
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
c.conrelid::regclass AS table_name,
|
||||
c.conname AS constraint_name,
|
||||
pg_get_constraintdef(c.oid) AS constraint_definition
|
||||
FROM pg_constraint c
|
||||
WHERE c.contype = 'f'
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM pg_index i
|
||||
WHERE i.indrelid = c.conrelid
|
||||
AND i.indisvalid
|
||||
AND i.indpred IS NULL
|
||||
AND i.indnkeyatts >= cardinality(c.conkey)
|
||||
AND (
|
||||
SELECT array_agg(key_attnum ORDER BY ordinality)
|
||||
FROM unnest(i.indkey::smallint[]) WITH ORDINALITY
|
||||
AS keys(key_attnum, ordinality)
|
||||
WHERE ordinality <= cardinality(c.conkey)
|
||||
) = c.conkey
|
||||
);
|
||||
```
|
||||
|
||||
### pg_stat_io (PG16+)
|
||||
|
||||
I/O statistics by backend type and context — helps identify I/O-heavy operations:
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
backend_type, object, context,
|
||||
reads, read_time,
|
||||
writes, write_time,
|
||||
hits
|
||||
FROM pg_stat_io
|
||||
WHERE reads > 0 OR writes > 0
|
||||
ORDER BY reads + writes DESC
|
||||
LIMIT 10;
|
||||
```
|
||||
|
||||
## Active Query Analysis
|
||||
|
||||
### Currently Running Queries
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
pid,
|
||||
now() - query_start AS duration,
|
||||
state,
|
||||
wait_event_type,
|
||||
wait_event,
|
||||
left(query, 100) AS query_preview
|
||||
FROM pg_stat_activity
|
||||
WHERE state != 'idle'
|
||||
AND pid != pg_backend_pid()
|
||||
ORDER BY duration DESC;
|
||||
```
|
||||
|
||||
### Long-Running Queries (> 5 minutes)
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
pid,
|
||||
now() - query_start AS duration,
|
||||
usename,
|
||||
application_name,
|
||||
client_addr,
|
||||
left(query, 200) AS query
|
||||
FROM pg_stat_activity
|
||||
WHERE state = 'active'
|
||||
AND now() - query_start > interval '5 minutes'
|
||||
AND pid != pg_backend_pid()
|
||||
ORDER BY duration DESC;
|
||||
```
|
||||
|
||||
### Cancel or Terminate a Query
|
||||
|
||||
```sql
|
||||
-- Replace 12345 with a PID selected from pg_stat_activity.
|
||||
-- Graceful cancel (sends a cancel signal)
|
||||
SELECT pg_cancel_backend(12345);
|
||||
|
||||
-- Force terminate (kills the connection)
|
||||
SELECT pg_terminate_backend(12345);
|
||||
```
|
||||
|
||||
Do not target your current session (`pg_backend_pid()`). Prefer cancellation first; terminate only when cancellation does not resolve the problem.
|
||||
|
||||
## Lock Analysis
|
||||
|
||||
### Blocked Queries and What's Blocking Them
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
blocked.pid AS blocked_pid,
|
||||
blocked.query AS blocked_query,
|
||||
blocking.pid AS blocking_pid,
|
||||
blocking.query AS blocking_query,
|
||||
now() - blocked.query_start AS blocked_duration
|
||||
FROM pg_stat_activity blocked
|
||||
JOIN pg_locks bl ON bl.pid = blocked.pid
|
||||
JOIN pg_locks kl ON kl.locktype = bl.locktype
|
||||
AND kl.database IS NOT DISTINCT FROM bl.database
|
||||
AND kl.relation IS NOT DISTINCT FROM bl.relation
|
||||
AND kl.page IS NOT DISTINCT FROM bl.page
|
||||
AND kl.tuple IS NOT DISTINCT FROM bl.tuple
|
||||
AND kl.virtualxid IS NOT DISTINCT FROM bl.virtualxid
|
||||
AND kl.transactionid IS NOT DISTINCT FROM bl.transactionid
|
||||
AND kl.classid IS NOT DISTINCT FROM bl.classid
|
||||
AND kl.objid IS NOT DISTINCT FROM bl.objid
|
||||
AND kl.objsubid IS NOT DISTINCT FROM bl.objsubid
|
||||
AND kl.pid != bl.pid
|
||||
JOIN pg_stat_activity blocking ON blocking.pid = kl.pid
|
||||
WHERE NOT bl.granted AND kl.granted;
|
||||
```
|
||||
|
||||
### Lock Types Quick Reference
|
||||
|
||||
| Lock | Acquired by | Conflicts with |
|
||||
|------|------------|----------------|
|
||||
| `AccessShareLock` | SELECT | AccessExclusiveLock |
|
||||
| `RowShareLock` | SELECT FOR UPDATE/SHARE | ExclusiveLock, AccessExclusiveLock |
|
||||
| `RowExclusiveLock` | INSERT, UPDATE, DELETE | ShareLock, ShareRowExclusiveLock, ExclusiveLock, AccessExclusiveLock |
|
||||
| `ShareLock` | CREATE INDEX (non-concurrent) | RowExclusiveLock and above |
|
||||
| `AccessExclusiveLock` | ALTER TABLE, DROP TABLE, VACUUM FULL | Everything |
|
||||
|
||||
### Advisory Locks
|
||||
|
||||
For application-level coordination without row locking:
|
||||
|
||||
```sql
|
||||
-- Choose one acquisition method, not both.
|
||||
-- Blocking acquisition:
|
||||
SELECT pg_advisory_lock(hashtext('my_job_name'));
|
||||
|
||||
-- Release once for each successful session-level acquisition:
|
||||
SELECT pg_advisory_unlock(hashtext('my_job_name'));
|
||||
|
||||
-- Or use a non-blocking acquisition:
|
||||
SELECT pg_try_advisory_lock(hashtext('my_job_name'));
|
||||
SELECT pg_advisory_unlock(hashtext('my_job_name'));
|
||||
```
|
||||
|
||||
## VACUUM and Bloat
|
||||
|
||||
### Check Autovacuum Status
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
schemaname,
|
||||
relname,
|
||||
n_dead_tup,
|
||||
last_autovacuum,
|
||||
last_autoanalyze,
|
||||
autovacuum_count,
|
||||
autoanalyze_count
|
||||
FROM pg_stat_user_tables
|
||||
WHERE n_dead_tup > 0
|
||||
ORDER BY n_dead_tup DESC
|
||||
LIMIT 20;
|
||||
```
|
||||
|
||||
### VACUUM Time Tracking (PG18+)
|
||||
|
||||
PG18 adds cumulative timing columns to `pg_stat_all_tables`:
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
schemaname,
|
||||
relname,
|
||||
total_vacuum_time,
|
||||
total_autovacuum_time,
|
||||
total_analyze_time,
|
||||
total_autoanalyze_time
|
||||
FROM pg_stat_user_tables
|
||||
WHERE total_autovacuum_time > 0
|
||||
ORDER BY total_autovacuum_time DESC
|
||||
LIMIT 10;
|
||||
```
|
||||
|
||||
### Autovacuum Tuning for High-Write Tables
|
||||
|
||||
```sql
|
||||
-- Per-table autovacuum settings for a high-churn table
|
||||
ALTER TABLE hot_table SET (
|
||||
autovacuum_vacuum_scale_factor = 0.01, -- trigger at 1% dead tuples (default 20%)
|
||||
autovacuum_vacuum_cost_delay = 2, -- faster vacuum (default 2ms)
|
||||
autovacuum_analyze_scale_factor = 0.005 -- trigger analyze more often
|
||||
);
|
||||
|
||||
-- PG18+: fixed dead-tuple threshold (in addition to percentage-based)
|
||||
-- autovacuum_vacuum_max_threshold = 100000000 -- trigger at fixed count regardless of table size
|
||||
```
|
||||
|
||||
**PG18 vacuum improvements:**
|
||||
- **Eager freezing**: Normal vacuums can freeze some pages opportunistically, reducing later freeze-only vacuum passes. Controlled by `vacuum_max_eager_freeze_failure_rate`.
|
||||
- **`autovacuum_worker_slots`**: New GUC specifying max background worker slots; `autovacuum_max_workers` is now adjustable at runtime.
|
||||
- **VACUUM/ANALYZE processes inheritance children by default**: Use `VACUUM (ONLY) tablename` for old behavior (parent table only).
|
||||
|
||||
### Manual VACUUM Operations
|
||||
|
||||
```sql
|
||||
-- Standard VACUUM (reclaims space for reuse, doesn't lock)
|
||||
VACUUM orders;
|
||||
|
||||
-- VACUUM with buffer usage limit (PG16+): limit shared buffer impact
|
||||
VACUUM (BUFFER_USAGE_LIMIT '256kB') orders;
|
||||
|
||||
-- VACUUM with analysis
|
||||
VACUUM ANALYZE orders;
|
||||
|
||||
-- VACUUM FULL (rewrites entire table — LOCKS TABLE, use as last resort)
|
||||
VACUUM FULL orders;
|
||||
|
||||
-- VACUUM VERBOSE (show progress)
|
||||
VACUUM VERBOSE orders;
|
||||
```
|
||||
|
||||
### Monitoring VACUUM Progress
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
relid::regclass AS table_name,
|
||||
phase,
|
||||
heap_blks_total,
|
||||
heap_blks_scanned,
|
||||
heap_blks_vacuumed,
|
||||
CASE WHEN heap_blks_total > 0
|
||||
THEN round(100.0 * heap_blks_vacuumed / heap_blks_total, 1)
|
||||
ELSE 0 END AS pct_complete
|
||||
FROM pg_stat_progress_vacuum;
|
||||
```
|
||||
|
||||
## Connection Management
|
||||
|
||||
### Connection Overview
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
state,
|
||||
count(*) AS connections,
|
||||
max(now() - state_change) AS longest_in_state
|
||||
FROM pg_stat_activity
|
||||
GROUP BY state
|
||||
ORDER BY count(*) DESC;
|
||||
```
|
||||
|
||||
### Idle Connections Holding Resources
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
pid,
|
||||
usename,
|
||||
application_name,
|
||||
client_addr,
|
||||
now() - state_change AS idle_duration,
|
||||
left(query, 100) AS last_query
|
||||
FROM pg_stat_activity
|
||||
WHERE state = 'idle'
|
||||
AND now() - state_change > interval '10 minutes'
|
||||
ORDER BY idle_duration DESC;
|
||||
```
|
||||
|
||||
### Connection Limits
|
||||
|
||||
```sql
|
||||
-- Current vs max connections
|
||||
SELECT
|
||||
count(*) AS current_connections,
|
||||
(SELECT setting::int FROM pg_settings WHERE name = 'max_connections') AS max_connections;
|
||||
```
|
||||
|
||||
If hitting connection limits, use PgBouncer or application-side connection pooling rather than increasing `max_connections`.
|
||||
|
||||
## pg_stat_statements
|
||||
|
||||
### Setup
|
||||
|
||||
```sql
|
||||
-- In postgresql.conf:
|
||||
-- shared_preload_libraries = 'pg_stat_statements'
|
||||
|
||||
CREATE EXTENSION IF NOT EXISTS pg_stat_statements;
|
||||
```
|
||||
|
||||
### Top Queries by Total Time
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
left(query, 100) AS query,
|
||||
calls,
|
||||
round(total_exec_time::numeric, 1) AS total_ms,
|
||||
round(mean_exec_time::numeric, 1) AS avg_ms,
|
||||
round((100.0 * total_exec_time / nullif(sum(total_exec_time) OVER (), 0))::numeric, 1) AS pct_total,
|
||||
rows
|
||||
FROM pg_stat_statements
|
||||
WHERE dbid = (SELECT oid FROM pg_database WHERE datname = current_database())
|
||||
ORDER BY total_exec_time DESC
|
||||
LIMIT 20;
|
||||
```
|
||||
|
||||
### Top Queries by Mean Time (Slowest Individual Executions)
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
left(query, 100) AS query,
|
||||
calls,
|
||||
round(mean_exec_time::numeric, 1) AS avg_ms,
|
||||
round(min_exec_time::numeric, 1) AS min_ms,
|
||||
round(max_exec_time::numeric, 1) AS max_ms,
|
||||
round(stddev_exec_time::numeric, 1) AS stddev_ms
|
||||
FROM pg_stat_statements
|
||||
WHERE calls >= 10 -- ignore rare queries
|
||||
AND dbid = (SELECT oid FROM pg_database WHERE datname = current_database())
|
||||
ORDER BY mean_exec_time DESC
|
||||
LIMIT 20;
|
||||
```
|
||||
|
||||
### Parallel Worker Usage (PG18+)
|
||||
|
||||
PG18 adds parallel worker tracking to `pg_stat_statements`:
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
left(query, 80) AS query,
|
||||
calls,
|
||||
round(mean_exec_time::numeric, 1) AS avg_ms,
|
||||
parallel_workers_to_launch,
|
||||
parallel_workers_launched
|
||||
FROM pg_stat_statements
|
||||
WHERE parallel_workers_to_launch > 0
|
||||
AND dbid = (SELECT oid FROM pg_database WHERE datname = current_database())
|
||||
ORDER BY parallel_workers_to_launch - parallel_workers_launched DESC
|
||||
LIMIT 10;
|
||||
```
|
||||
|
||||
A gap between `to_launch` and `launched` indicates insufficient parallel workers.
|
||||
|
||||
### Reset Statistics
|
||||
|
||||
```sql
|
||||
-- Reset pg_stat_statements (do this after config changes or deployments)
|
||||
SELECT pg_stat_statements_reset();
|
||||
|
||||
-- Reset table/index stats
|
||||
SELECT pg_stat_reset();
|
||||
```
|
||||
@@ -0,0 +1,294 @@
|
||||
# Query Optimization Reference
|
||||
|
||||
## Contents
|
||||
- Running EXPLAIN ANALYZE
|
||||
- Reading execution plans
|
||||
- Plan node reference
|
||||
- Join strategy selection
|
||||
- Common bottlenecks and fixes
|
||||
- Statistics and the planner
|
||||
- Configuration tuning knobs
|
||||
|
||||
## Running EXPLAIN ANALYZE
|
||||
|
||||
Always use this form for real diagnostics:
|
||||
|
||||
```sql
|
||||
EXPLAIN (ANALYZE, BUFFERS, FORMAT TEXT) SELECT ...;
|
||||
```
|
||||
|
||||
| Flag | Purpose |
|
||||
|------|---------|
|
||||
| `ANALYZE` | Actually executes the query (required for actual times/rows) |
|
||||
| `BUFFERS` | Shows shared/local buffer hits and reads (I/O insight). **PG18+**: included by default with `ANALYZE` |
|
||||
| `FORMAT TEXT` | Human-readable output (default) |
|
||||
|
||||
Additional EXPLAIN options:
|
||||
|
||||
| Flag | PG Version | Purpose |
|
||||
|------|-----------|---------|
|
||||
| `GENERIC_PLAN` | PG16+ | Show plan for parameterized queries (with `$1` params) without executing |
|
||||
| `MEMORY` | PG17+ | Report planner memory usage |
|
||||
| `SERIALIZE` | PG17+ | Show cost of converting result data for network transmission |
|
||||
|
||||
**PG18 EXPLAIN improvements:**
|
||||
- `BUFFERS` is now included automatically with `ANALYZE` — no need to specify it separately
|
||||
- Index scan nodes report the number of index lookups
|
||||
- Memory/disk usage shown for Materialize, Window Aggregate, and CTE Scan nodes
|
||||
- Fractional row counts in output (instead of rounding to integers)
|
||||
- Disabled nodes are explicitly indicated in output
|
||||
|
||||
For queries that modify data, wrap in a transaction and rollback:
|
||||
|
||||
```sql
|
||||
BEGIN;
|
||||
EXPLAIN (ANALYZE, BUFFERS) DELETE FROM orders WHERE ...;
|
||||
ROLLBACK;
|
||||
```
|
||||
|
||||
### Key Metrics in Output
|
||||
|
||||
```
|
||||
Seq Scan on orders (cost=0.00..1520.00 rows=50000 width=64)
|
||||
(actual time=0.012..15.432 rows=48573 loops=1)
|
||||
Buffers: shared hit=520 read=200
|
||||
```
|
||||
|
||||
- **cost**: Estimated startup..total cost (arbitrary planner units)
|
||||
- **rows**: Estimated row count
|
||||
- **actual time**: Real time in ms (startup..total)
|
||||
- **actual rows**: Real row count — compare to estimate
|
||||
- **loops**: How many times this node executed
|
||||
- **Buffers shared hit**: Pages found in cache
|
||||
- **Buffers shared read**: Pages read from disk
|
||||
|
||||
**Critical check**: If estimated `rows` and actual `rows` differ by 10x+, statistics are stale or the planner is misestimating.
|
||||
|
||||
## Reading Execution Plans
|
||||
|
||||
Read bottom-up, innermost to outermost. Each node:
|
||||
1. Scans or receives input rows
|
||||
2. Applies filtering or transformation
|
||||
3. Passes rows to the parent node
|
||||
|
||||
### Identifying the Bottleneck
|
||||
|
||||
1. Find the node with the highest `actual time` (total, not startup)
|
||||
2. Check if actual rows >> estimated rows (bad statistics)
|
||||
3. Check `Buffers: shared read` — high reads = cold cache or missing index
|
||||
4. Look for `Rows Removed by Filter` — high values mean the scan is too broad
|
||||
|
||||
## Plan Node Reference
|
||||
|
||||
### Scan Nodes
|
||||
|
||||
| Node | Meaning | When it's a problem |
|
||||
|------|---------|-------------------|
|
||||
| `Seq Scan` | Full table scan | On large tables when only a few rows match |
|
||||
| `Index Scan` | B-tree lookup + heap fetch | Normal and expected |
|
||||
| `Index Only Scan` | B-tree lookup, no heap fetch | Best case — all columns in index |
|
||||
| `Bitmap Index Scan` + `Bitmap Heap Scan` | Index lookup → bitmap → heap | Good for medium selectivity |
|
||||
| `CTE Scan` | Reads from materialized CTE | Check if CTE materialization is needed |
|
||||
|
||||
**When is Seq Scan OK?**
|
||||
- Table is small (< few thousand rows)
|
||||
- Query returns > 10-20% of rows
|
||||
- No usable index exists and adding one isn't warranted
|
||||
|
||||
### Join Nodes
|
||||
|
||||
| Node | How it works | Best for |
|
||||
|------|-------------|----------|
|
||||
| `Nested Loop` | For each outer row, scan inner | Small outer set + indexed inner |
|
||||
| `Hash Join` | Build hash table on inner, probe with outer | Medium-large equijoins, enough work_mem |
|
||||
| `Merge Join` | Both inputs sorted, merge | Pre-sorted inputs or sorted output needed |
|
||||
|
||||
**Nested Loop red flags**: high outer row count with no index on the inner side.
|
||||
|
||||
**Hash Join red flags**: `Batches: N` where N > 1 means hash table spilled to disk (increase `work_mem`).
|
||||
|
||||
### Sort and Aggregate Nodes
|
||||
|
||||
| Node | Notes |
|
||||
|------|-------|
|
||||
| `Sort` | Check `Sort Method: external merge` = disk spill (increase `work_mem`) |
|
||||
| `HashAggregate` | Groups via hash table — check for disk spill batches |
|
||||
| `GroupAggregate` | Groups pre-sorted input — needs sorted input |
|
||||
| `Incremental Sort` | Sorts remaining columns when leading columns already sorted |
|
||||
|
||||
### Other Nodes
|
||||
|
||||
| Node | Notes |
|
||||
|------|-------|
|
||||
| `Materialize` | Caches a subplan's output for re-scan |
|
||||
| `Memoize` (PG14+) | Caches parameterized nested loop inner-side results. Look for `Hits: N Misses: N` — high hit ratio means effective caching. Poor cache ratio may indicate high cardinality on the join key |
|
||||
| `Gather` / `Gather Merge` | Collects results from parallel workers |
|
||||
| `Append` | Concatenates results (UNION ALL, partitioned tables) |
|
||||
| `SubPlan` | Correlated subquery — potentially executed per row |
|
||||
|
||||
## Common Bottlenecks and Fixes
|
||||
|
||||
### Sequential Scan on Large Table
|
||||
|
||||
**Symptom**: `Seq Scan` with `Rows Removed by Filter: 999000` (scanned 1M, kept 1K)
|
||||
|
||||
**Fixes**:
|
||||
1. Add an index on the WHERE clause columns
|
||||
2. If query returns > 20% of rows, seq scan may actually be optimal
|
||||
3. Check `enable_seqscan = on` isn't masking the real issue
|
||||
|
||||
### Nested Loop with High Row Count
|
||||
|
||||
**Symptom**: `Nested Loop (actual loops=50000)` with inner `Seq Scan`
|
||||
|
||||
**Fixes**:
|
||||
1. Add index on the inner table's join column
|
||||
2. If both sides are large, the planner should choose hash/merge join — check statistics
|
||||
|
||||
### Sort Spilling to Disk
|
||||
|
||||
**Symptom**: `Sort Method: external merge Disk: 125MB`
|
||||
|
||||
**Fixes**:
|
||||
1. Increase `work_mem` (per-operation, not global): `SET work_mem = '256MB';`
|
||||
2. Add an index matching the ORDER BY to avoid sorting entirely
|
||||
3. If in a CTE or subquery, consider whether the sort is necessary
|
||||
|
||||
### Bad Row Estimates
|
||||
|
||||
**Symptom**: Estimated rows: 1, Actual rows: 500,000
|
||||
|
||||
**Fixes**:
|
||||
1. `ANALYZE table_name;` — refresh statistics
|
||||
2. Increase `default_statistics_target` for columns with skewed distributions:
|
||||
```sql
|
||||
ALTER TABLE orders ALTER COLUMN status SET STATISTICS 1000;
|
||||
ANALYZE orders;
|
||||
```
|
||||
3. Create extended statistics for correlated columns:
|
||||
```sql
|
||||
CREATE STATISTICS orders_status_date (dependencies)
|
||||
ON status, created_at FROM orders;
|
||||
ANALYZE orders;
|
||||
```
|
||||
|
||||
### Correlated SubPlan Executed Per Row
|
||||
|
||||
**Symptom**: `SubPlan` node with `loops=100000`
|
||||
|
||||
**Fix**: Rewrite as a JOIN or lateral join:
|
||||
|
||||
```sql
|
||||
-- Bad: correlated subquery
|
||||
SELECT *, (SELECT name FROM departments d WHERE d.id = e.dept_id) AS dept_name
|
||||
FROM employees e;
|
||||
|
||||
-- Good: join
|
||||
SELECT e.*, d.name AS dept_name
|
||||
FROM employees e
|
||||
JOIN departments d ON d.id = e.dept_id;
|
||||
```
|
||||
|
||||
### PG18 Optimizer Improvements
|
||||
|
||||
**B-tree skip scan (PG18+)**: Multi-column B-tree indexes can now be used even when there are no restrictions on the leading columns. Previously a composite index on `(tenant_id, created_at)` was useless for `WHERE created_at > '2024-01-01'` without a `tenant_id` filter. PG18 can skip through distinct `tenant_id` values — eliminating many cases where you needed a separate single-column index.
|
||||
|
||||
**Self-join elimination**: The optimizer automatically removes unnecessary self-joins. Controlled by `enable_self_join_elimination`.
|
||||
|
||||
**IN to ANY conversion**: `WHERE x IN (VALUES ...)` is converted to `x = ANY(...)` for better use of optimizer statistics.
|
||||
|
||||
**OR-clause to array transformation**: OR clauses like `WHERE x = 1 OR x = 2 OR x = 3` are transformed to arrays for faster index processing.
|
||||
|
||||
**DISTINCT reordering**: Keys in `SELECT DISTINCT` can be reordered internally to match an existing index and avoid sorting. Controlled by `enable_distinct_reordering`.
|
||||
|
||||
**Improved partition planning**: More efficient planning for queries accessing many partitions, with reduced memory usage. Partitionwise joins allowed in more cases.
|
||||
|
||||
## Statistics and the Planner
|
||||
|
||||
### Manual ANALYZE
|
||||
|
||||
```sql
|
||||
-- Analyze one table
|
||||
ANALYZE orders;
|
||||
|
||||
-- Analyze specific columns
|
||||
ANALYZE orders(status, created_at);
|
||||
```
|
||||
|
||||
Autovacuum runs ANALYZE automatically, but after bulk loads or major changes, run it manually.
|
||||
|
||||
### Extended Statistics
|
||||
|
||||
For correlated columns that the planner estimates independently:
|
||||
|
||||
```sql
|
||||
-- Functional dependency: knowing city tells you the state
|
||||
CREATE STATISTICS city_state_dep (dependencies) ON city, state FROM addresses;
|
||||
|
||||
-- N-distinct: correct group count estimates
|
||||
CREATE STATISTICS city_state_ndist (ndistinct) ON city, state FROM addresses;
|
||||
|
||||
-- MCV lists: track most common value combinations
|
||||
CREATE STATISTICS city_state_mcv (mcv) ON city, state FROM addresses;
|
||||
|
||||
ANALYZE addresses;
|
||||
```
|
||||
|
||||
### Checking Current Statistics
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
attname,
|
||||
n_distinct,
|
||||
most_common_vals,
|
||||
most_common_freqs,
|
||||
correlation -- physical vs. logical ordering (affects index scan cost)
|
||||
FROM pg_stats
|
||||
WHERE tablename = 'orders' AND attname = 'status';
|
||||
```
|
||||
|
||||
## Configuration Tuning Knobs
|
||||
|
||||
Except for `shared_buffers`, these parameters can be overridden in the current session with `SET` for workload-specific testing. `shared_buffers` is allocated at server start; persisting a change in `postgresql.conf` or with `ALTER SYSTEM` requires a restart before it takes effect.
|
||||
|
||||
| Parameter | Default | When to increase |
|
||||
|-----------|---------|-----------------|
|
||||
| `shared_buffers` | 128MB | Server-wide, restart-required setting; start around 25% of total RAM and validate for the workload. See performance-diagnostics for cache hit rate queries |
|
||||
| `work_mem` | 4MB | Sort/hash spilling to disk |
|
||||
| `maintenance_work_mem` | 64MB | Slow VACUUM, CREATE INDEX, ALTER TABLE |
|
||||
| `effective_cache_size` | 4GB | Set to ~75% of total RAM (hint to planner, no allocation) |
|
||||
| `random_page_cost` | 4.0 | SSD storage → set to 1.1-1.5 |
|
||||
| `effective_io_concurrency` | 1 (16 in PG18) | SSD storage → set to 200 |
|
||||
| `default_statistics_target` | 100 | Bad estimates on skewed columns → 500-1000 |
|
||||
| `max_parallel_workers_per_gather` | 2 | Large analytical queries benefit from more parallelism |
|
||||
|
||||
### SSD-Optimized Settings
|
||||
|
||||
```sql
|
||||
SET random_page_cost = 1.1;
|
||||
-- Some platforms without posix_fadvise only support 0.
|
||||
DO $$
|
||||
BEGIN
|
||||
PERFORM set_config('effective_io_concurrency', '200', false);
|
||||
EXCEPTION
|
||||
WHEN invalid_parameter_value THEN
|
||||
PERFORM set_config('effective_io_concurrency', '0', false);
|
||||
END
|
||||
$$;
|
||||
SET effective_cache_size = '24GB'; -- adjust to your server
|
||||
```
|
||||
|
||||
These make the planner more willing to use index scans when measurements show SSD random reads are fast. Some platforms without asynchronous prefetch support cap `effective_io_concurrency` at `0`.
|
||||
|
||||
### Asynchronous I/O (PG18+)
|
||||
|
||||
PG18 introduces a native async I/O subsystem that improves sequential scan, bitmap heap scan, and VACUUM performance.
|
||||
|
||||
| Parameter | Default (PG18) | Purpose |
|
||||
|-----------|----------------|---------|
|
||||
| `io_method` | platform-dependent | I/O method: `sync`, `io_uring` (Linux), `posix_aio` |
|
||||
| `io_combine_limit` | 128kB | Max size of combined I/O operations |
|
||||
| `effective_io_concurrency` | 16 (was 1) | Default raised significantly in PG18 |
|
||||
| `maintenance_io_concurrency` | 16 (was 10) | Default raised for maintenance operations |
|
||||
|
||||
The raised defaults mean PG18 out-of-the-box performance for sequential scans and VACUUM is significantly better than prior versions.
|
||||
@@ -0,0 +1,566 @@
|
||||
# SQL Query Patterns Reference
|
||||
|
||||
## Contents
|
||||
- Common Table Expressions (CTEs)
|
||||
- Window functions
|
||||
- Lateral joins
|
||||
- Recursive queries
|
||||
- UPSERT (INSERT ON CONFLICT)
|
||||
- Bulk operations
|
||||
- JSONB queries
|
||||
- Date/time patterns
|
||||
- Anti-patterns to avoid
|
||||
|
||||
## Common Table Expressions (CTEs)
|
||||
|
||||
### Readability CTEs
|
||||
|
||||
```sql
|
||||
WITH active_customers AS (
|
||||
SELECT id, name, email
|
||||
FROM customers
|
||||
WHERE status = 'active'
|
||||
),
|
||||
recent_orders AS (
|
||||
SELECT customer_id, count(*) AS order_count, max(created_at) AS last_order
|
||||
FROM orders
|
||||
WHERE created_at > now() - interval '90 days'
|
||||
GROUP BY customer_id
|
||||
)
|
||||
SELECT ac.name, ac.email, ro.order_count, ro.last_order
|
||||
FROM active_customers ac
|
||||
JOIN recent_orders ro ON ro.customer_id = ac.id
|
||||
ORDER BY ro.order_count DESC;
|
||||
```
|
||||
|
||||
### CTE Materialization
|
||||
|
||||
By default the optimizer may inline CTEs. Force materialization when:
|
||||
- The CTE is referenced multiple times
|
||||
- You want to create an optimization fence
|
||||
|
||||
```sql
|
||||
WITH expensive_calc AS MATERIALIZED (
|
||||
SELECT ... -- complex aggregation
|
||||
)
|
||||
SELECT * FROM expensive_calc WHERE ...
|
||||
UNION ALL
|
||||
SELECT * FROM expensive_calc WHERE ...;
|
||||
```
|
||||
|
||||
Force inlining (default for single-use):
|
||||
```sql
|
||||
WITH simple_filter AS NOT MATERIALIZED (
|
||||
SELECT * FROM large_table WHERE status = 'active'
|
||||
)
|
||||
SELECT * FROM simple_filter WHERE created_at > '2024-01-01';
|
||||
```
|
||||
|
||||
## Window Functions
|
||||
|
||||
### Ranking
|
||||
|
||||
```sql
|
||||
-- Row number (no ties)
|
||||
SELECT *, row_number() OVER (PARTITION BY department ORDER BY salary DESC) AS rn
|
||||
FROM employees;
|
||||
|
||||
-- Rank (ties get same rank, gaps after)
|
||||
SELECT *, rank() OVER (ORDER BY score DESC) AS rank
|
||||
FROM leaderboard;
|
||||
|
||||
-- Dense rank (no gaps)
|
||||
SELECT *, dense_rank() OVER (ORDER BY score DESC) AS dense_rank
|
||||
FROM leaderboard;
|
||||
```
|
||||
|
||||
### Running Totals and Moving Averages
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
date,
|
||||
revenue,
|
||||
sum(revenue) OVER (ORDER BY date) AS running_total,
|
||||
avg(revenue) OVER (
|
||||
ORDER BY date
|
||||
ROWS BETWEEN 6 PRECEDING AND CURRENT ROW
|
||||
) AS moving_avg_7d
|
||||
FROM daily_revenue;
|
||||
```
|
||||
|
||||
### Lead/Lag (Access Adjacent Rows)
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
event_time,
|
||||
event_type,
|
||||
lag(event_time) OVER (PARTITION BY user_id ORDER BY event_time) AS prev_event,
|
||||
event_time - lag(event_time) OVER (PARTITION BY user_id ORDER BY event_time) AS time_since_last
|
||||
FROM user_events;
|
||||
```
|
||||
|
||||
### First/Last Value
|
||||
|
||||
```sql
|
||||
SELECT DISTINCT
|
||||
department,
|
||||
first_value(name) OVER (
|
||||
PARTITION BY department ORDER BY salary DESC
|
||||
ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING
|
||||
) AS highest_paid
|
||||
FROM employees;
|
||||
```
|
||||
|
||||
### Gaps and Islands
|
||||
|
||||
Find consecutive sequences:
|
||||
|
||||
```sql
|
||||
WITH numbered AS (
|
||||
SELECT *,
|
||||
date - (row_number() OVER (PARTITION BY user_id ORDER BY date))::int * interval '1 day' AS grp
|
||||
FROM daily_logins
|
||||
)
|
||||
SELECT user_id, min(date) AS streak_start, max(date) AS streak_end,
|
||||
count(*) AS streak_length
|
||||
FROM numbered
|
||||
GROUP BY user_id, grp
|
||||
HAVING count(*) >= 7; -- streaks of 7+ days
|
||||
```
|
||||
|
||||
## Lateral Joins
|
||||
|
||||
LATERAL lets a subquery reference columns from preceding tables. Essential for "top-N per group" queries.
|
||||
|
||||
### Top-N Per Group
|
||||
|
||||
```sql
|
||||
-- Get latest 3 orders per customer
|
||||
SELECT c.name, o.*
|
||||
FROM customers c
|
||||
CROSS JOIN LATERAL (
|
||||
SELECT id, total, created_at
|
||||
FROM orders
|
||||
WHERE customer_id = c.id
|
||||
ORDER BY created_at DESC
|
||||
LIMIT 3
|
||||
) o;
|
||||
```
|
||||
|
||||
This is much faster than window function approaches when you have an index on `orders(customer_id, created_at DESC)`.
|
||||
|
||||
### Calling Set-Returning Functions
|
||||
|
||||
```sql
|
||||
SELECT j.id, elem.key, elem.value
|
||||
FROM journal_entries j
|
||||
CROSS JOIN LATERAL jsonb_each_text(j.metadata) AS elem;
|
||||
```
|
||||
|
||||
## Recursive Queries
|
||||
|
||||
### Tree Traversal (Adjacency List)
|
||||
|
||||
```sql
|
||||
WITH RECURSIVE tree AS (
|
||||
-- Base case: root nodes
|
||||
SELECT id, name, parent_id, 0 AS depth, ARRAY[id] AS path
|
||||
FROM categories
|
||||
WHERE parent_id IS NULL
|
||||
|
||||
UNION ALL
|
||||
|
||||
-- Recursive step
|
||||
SELECT c.id, c.name, c.parent_id, t.depth + 1, t.path || c.id
|
||||
FROM categories c
|
||||
JOIN tree t ON t.id = c.parent_id
|
||||
WHERE NOT c.id = ANY(t.path) -- cycle prevention
|
||||
)
|
||||
SELECT * FROM tree ORDER BY path;
|
||||
```
|
||||
|
||||
### Generating Series
|
||||
|
||||
```sql
|
||||
-- Date series for gap-filling
|
||||
SELECT d::date, coalesce(o.total, 0) AS total
|
||||
FROM generate_series('2024-01-01'::date, '2024-12-31'::date, '1 day') AS d
|
||||
LEFT JOIN (
|
||||
SELECT created_at::date AS day, sum(total) AS total
|
||||
FROM orders
|
||||
GROUP BY 1
|
||||
) o ON o.day = d;
|
||||
```
|
||||
|
||||
## UPSERT (INSERT ON CONFLICT)
|
||||
|
||||
### Basic Upsert
|
||||
|
||||
```sql
|
||||
INSERT INTO user_settings (user_id, key, value)
|
||||
VALUES (1, 'theme', 'dark')
|
||||
ON CONFLICT (user_id, key)
|
||||
DO UPDATE SET
|
||||
value = EXCLUDED.value,
|
||||
updated_at = now();
|
||||
```
|
||||
|
||||
### Insert-or-Ignore
|
||||
|
||||
```sql
|
||||
INSERT INTO tags (name)
|
||||
VALUES ('postgres'), ('sql'), ('database')
|
||||
ON CONFLICT (name) DO NOTHING;
|
||||
```
|
||||
|
||||
### Upsert with Conditional Update
|
||||
|
||||
```sql
|
||||
INSERT INTO inventory (sku, quantity, last_restock)
|
||||
VALUES ('ABC-123', 50, now())
|
||||
ON CONFLICT (sku)
|
||||
DO UPDATE SET
|
||||
quantity = inventory.quantity + EXCLUDED.quantity,
|
||||
last_restock = EXCLUDED.last_restock
|
||||
WHERE inventory.quantity < 100; -- only restock if low
|
||||
```
|
||||
|
||||
### Upsert Returning
|
||||
|
||||
```sql
|
||||
INSERT INTO users (email, name)
|
||||
VALUES ('a@b.com', 'Alice')
|
||||
ON CONFLICT (email)
|
||||
DO UPDATE SET name = EXCLUDED.name
|
||||
RETURNING id;
|
||||
```
|
||||
|
||||
Do not use `(xmax = 0)` as an inserted-versus-updated contract; it relies on implementation details. On PG18+, use `OLD`/`NEW` in `RETURNING`. On earlier versions, distinguish outcomes in application logic or use separate statements when the distinction matters.
|
||||
|
||||
### OLD/NEW in RETURNING (PG18+)
|
||||
|
||||
All DML commands support `OLD` and `NEW` references in RETURNING, making it easy to see before/after values:
|
||||
|
||||
```sql
|
||||
UPDATE products SET price = price * 1.10
|
||||
WHERE category = 'electronics'
|
||||
RETURNING id, old.price AS old_price, new.price AS new_price;
|
||||
|
||||
DELETE FROM sessions WHERE expired_at < now()
|
||||
RETURNING old.*;
|
||||
```
|
||||
|
||||
## MERGE (PG15+)
|
||||
|
||||
SQL-standard command for conditional INSERT, UPDATE, or DELETE in a single statement. Replaces UPSERT for complex cases where you need different actions based on whether the row exists.
|
||||
|
||||
```sql
|
||||
MERGE INTO inventory AS target
|
||||
USING incoming_shipment AS source
|
||||
ON target.sku = source.sku
|
||||
WHEN MATCHED AND source.quantity = 0 THEN
|
||||
DELETE
|
||||
WHEN MATCHED THEN
|
||||
UPDATE SET quantity = target.quantity + source.quantity,
|
||||
last_restock = now()
|
||||
WHEN NOT MATCHED THEN
|
||||
INSERT (sku, quantity, last_restock)
|
||||
VALUES (source.sku, source.quantity, now());
|
||||
```
|
||||
|
||||
### MERGE RETURNING (PG17+)
|
||||
|
||||
```sql
|
||||
MERGE INTO inventory AS target
|
||||
USING incoming_shipment AS source
|
||||
ON target.sku = source.sku
|
||||
WHEN MATCHED THEN
|
||||
UPDATE SET quantity = target.quantity + source.quantity
|
||||
WHEN NOT MATCHED THEN
|
||||
INSERT (sku, quantity) VALUES (source.sku, source.quantity)
|
||||
RETURNING merge_action(), target.*;
|
||||
-- merge_action() returns 'INSERT', 'UPDATE', or 'DELETE'
|
||||
```
|
||||
|
||||
**When to use MERGE vs UPSERT**: Use `INSERT ON CONFLICT` for simple insert-or-update on a single table. Use `MERGE` when you need different actions based on conditions, when the source is another table/query, or when you need DELETE as one of the outcomes.
|
||||
|
||||
## JSON_TABLE (PG17+)
|
||||
|
||||
Convert JSON data to a relational table, usable in FROM:
|
||||
|
||||
```sql
|
||||
SELECT jt.*
|
||||
FROM events,
|
||||
JSON_TABLE(
|
||||
payload, '$'
|
||||
COLUMNS (
|
||||
event_type text PATH '$.type',
|
||||
city text PATH '$.address.city',
|
||||
score int PATH '$.score'
|
||||
)
|
||||
) AS jt
|
||||
WHERE jt.event_type = 'click';
|
||||
```
|
||||
|
||||
## Bulk Operations
|
||||
|
||||
### Bulk Insert from Values
|
||||
|
||||
```sql
|
||||
INSERT INTO products (name, price, category)
|
||||
VALUES
|
||||
('Widget A', 9.99, 'gadgets'),
|
||||
('Widget B', 14.99, 'gadgets'),
|
||||
('Widget C', 19.99, 'gadgets');
|
||||
```
|
||||
|
||||
### Bulk Update with FROM
|
||||
|
||||
```sql
|
||||
UPDATE products p
|
||||
SET price = new_prices.price
|
||||
FROM (VALUES
|
||||
(1, 12.99),
|
||||
(2, 15.99),
|
||||
(3, 22.99)
|
||||
) AS new_prices(id, price)
|
||||
WHERE p.id = new_prices.id;
|
||||
```
|
||||
|
||||
### Bulk Delete with Subquery
|
||||
|
||||
```sql
|
||||
DELETE FROM sessions
|
||||
WHERE id IN (
|
||||
SELECT id FROM sessions
|
||||
WHERE last_active < now() - interval '30 days'
|
||||
ORDER BY last_active
|
||||
LIMIT 10000 -- batch to avoid long locks
|
||||
);
|
||||
```
|
||||
|
||||
## JSONB Queries
|
||||
|
||||
### Querying JSONB
|
||||
|
||||
```sql
|
||||
-- Key access
|
||||
SELECT payload->>'name' AS name FROM events; -- text
|
||||
SELECT payload->'address'->'city' FROM events; -- jsonb
|
||||
SELECT payload#>>'{address,city}' FROM events; -- text via path
|
||||
|
||||
-- Containment (uses GIN index)
|
||||
SELECT * FROM events WHERE payload @> '{"type": "click"}';
|
||||
|
||||
-- Key existence
|
||||
SELECT * FROM events WHERE payload ? 'error_code';
|
||||
|
||||
-- Array element access
|
||||
SELECT payload->'tags'->>0 FROM events;
|
||||
```
|
||||
|
||||
### JSONB Aggregation
|
||||
|
||||
```sql
|
||||
-- Build JSON object from rows
|
||||
SELECT jsonb_object_agg(key, value) FROM settings WHERE user_id = 1;
|
||||
|
||||
-- Build JSON array from rows
|
||||
SELECT jsonb_agg(jsonb_build_object('id', id, 'name', name))
|
||||
FROM products
|
||||
WHERE category = 'gadgets';
|
||||
```
|
||||
|
||||
### JSONB Modification
|
||||
|
||||
```sql
|
||||
-- Set/overwrite a key
|
||||
UPDATE events SET payload = payload || '{"processed": true}';
|
||||
|
||||
-- Remove a key
|
||||
UPDATE events SET payload = payload - 'temp_field';
|
||||
|
||||
-- Set nested key
|
||||
UPDATE events SET payload = jsonb_set(payload, '{status,code}', '"200"');
|
||||
```
|
||||
|
||||
## Date/Time Patterns
|
||||
|
||||
### JSONB Subscripting (PG14+)
|
||||
|
||||
Simpler syntax for JSONB access and assignment:
|
||||
|
||||
```sql
|
||||
-- Read JSONB (equivalent to payload->'address'->'city')
|
||||
SELECT payload['address']['city'] FROM events;
|
||||
|
||||
-- Update (equivalent to jsonb_set)
|
||||
UPDATE events SET payload['processed'] = 'true' WHERE id = 1;
|
||||
UPDATE events SET payload['address']['zip'] = '"90210"' WHERE id = 1;
|
||||
```
|
||||
|
||||
### date_bin() (PG14+)
|
||||
|
||||
Bin timestamps into uniform intervals — more flexible than `date_trunc`:
|
||||
|
||||
```sql
|
||||
-- 15-minute bins (date_trunc can only do hour/day/etc.)
|
||||
SELECT
|
||||
date_bin('15 minutes', created_at, '2024-01-01') AS bin,
|
||||
count(*)
|
||||
FROM orders
|
||||
GROUP BY 1 ORDER BY 1;
|
||||
|
||||
-- 6-hour bins
|
||||
SELECT date_bin('6 hours', occurred_at, '2024-01-01') AS bin, count(*)
|
||||
FROM events GROUP BY 1;
|
||||
```
|
||||
|
||||
### Truncation and Grouping
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
date_trunc('month', created_at) AS month,
|
||||
count(*) AS orders
|
||||
FROM orders
|
||||
GROUP BY 1
|
||||
ORDER BY 1;
|
||||
```
|
||||
|
||||
### Interval Arithmetic
|
||||
|
||||
```sql
|
||||
SELECT * FROM subscriptions
|
||||
WHERE expires_at BETWEEN now() AND now() + interval '7 days';
|
||||
```
|
||||
|
||||
### Extract Components
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
extract(dow FROM created_at) AS day_of_week, -- 0=Sun, 6=Sat
|
||||
extract(hour FROM created_at) AS hour,
|
||||
count(*)
|
||||
FROM orders
|
||||
GROUP BY 1, 2;
|
||||
```
|
||||
|
||||
## Concurrency Patterns
|
||||
|
||||
### SKIP LOCKED (Queue Processing)
|
||||
|
||||
Multiple workers can process a queue table without blocking each other:
|
||||
|
||||
```sql
|
||||
-- Worker claims and processes next N items atomically
|
||||
WITH next_batch AS (
|
||||
SELECT id FROM job_queue
|
||||
WHERE status = 'pending'
|
||||
ORDER BY created_at
|
||||
LIMIT 10
|
||||
FOR UPDATE SKIP LOCKED
|
||||
)
|
||||
UPDATE job_queue SET status = 'processing', started_at = now()
|
||||
WHERE id IN (SELECT id FROM next_batch)
|
||||
RETURNING *;
|
||||
```
|
||||
|
||||
### Deadlock Prevention
|
||||
|
||||
Always acquire row locks in a consistent order (e.g., by ID):
|
||||
|
||||
```sql
|
||||
-- Bad: two transactions locking rows in different order → deadlock
|
||||
-- Good: always lock in ID order
|
||||
SELECT * FROM accounts
|
||||
WHERE id IN (5, 12, 3)
|
||||
ORDER BY id
|
||||
FOR UPDATE;
|
||||
```
|
||||
|
||||
### Transaction Discipline
|
||||
|
||||
```sql
|
||||
-- Set statement timeout to prevent runaway queries
|
||||
SET statement_timeout = '30s';
|
||||
|
||||
-- Kill idle-in-transaction sessions (holds locks, blocks VACUUM)
|
||||
SET idle_in_transaction_session_timeout = '60s';
|
||||
|
||||
-- Limit total transaction duration (PG17+)
|
||||
SET transaction_timeout = '5min';
|
||||
```
|
||||
|
||||
Keep transactions short: move external calls (HTTP, file I/O) outside the transaction boundary.
|
||||
|
||||
### N+1 Query Elimination
|
||||
|
||||
```sql
|
||||
-- Bad: one query per item (N+1 pattern)
|
||||
-- SELECT * FROM orders WHERE customer_id = 1;
|
||||
-- SELECT * FROM orders WHERE customer_id = 2;
|
||||
-- ...
|
||||
|
||||
-- Good: single query with ANY
|
||||
SELECT * FROM orders WHERE customer_id = ANY($1::bigint[]);
|
||||
-- Pass array of IDs: ARRAY[1, 2, 3, ...]
|
||||
```
|
||||
|
||||
## Useful PG18+ Functions
|
||||
|
||||
### array_sort() and array_reverse()
|
||||
|
||||
```sql
|
||||
SELECT array_sort(ARRAY[3, 1, 4, 1, 5]); -- {1, 1, 3, 4, 5}
|
||||
SELECT array_reverse(ARRAY[1, 2, 3]); -- {3, 2, 1}
|
||||
```
|
||||
|
||||
### casefold() — Unicode Case-Insensitive Matching
|
||||
|
||||
More robust than `lower()` for internationalized text:
|
||||
|
||||
```sql
|
||||
SELECT casefold('Hello World') = casefold('HELLO WORLD'); -- true
|
||||
```
|
||||
|
||||
## Anti-Patterns to Avoid
|
||||
|
||||
### SELECT * in Production Queries
|
||||
Use explicit column lists. `SELECT *` breaks when columns change and prevents index-only scans.
|
||||
|
||||
### NOT IN with NULLs
|
||||
`NOT IN (subquery)` returns no rows if any subquery result is NULL. Use `NOT EXISTS` instead:
|
||||
|
||||
```sql
|
||||
-- Bad: breaks if orders.customer_id has any NULL
|
||||
SELECT * FROM customers WHERE id NOT IN (SELECT customer_id FROM orders);
|
||||
|
||||
-- Good: always correct
|
||||
SELECT * FROM customers c
|
||||
WHERE NOT EXISTS (SELECT 1 FROM orders o WHERE o.customer_id = c.id);
|
||||
```
|
||||
|
||||
### Implicit Casts in WHERE Clauses
|
||||
Casting a column prevents index use:
|
||||
|
||||
```sql
|
||||
-- Bad: casts every row, no index
|
||||
WHERE created_at::date = '2024-01-15'
|
||||
|
||||
-- Good: range scan, uses index
|
||||
WHERE created_at >= '2024-01-15' AND created_at < '2024-01-16'
|
||||
```
|
||||
|
||||
### ORDER BY on Unindexed Large Result Sets
|
||||
If you ORDER BY + LIMIT on a large table, ensure there's an index on the ORDER BY columns to avoid a full sort.
|
||||
|
||||
### Using OFFSET for Pagination
|
||||
OFFSET scans and discards rows. Use keyset pagination instead:
|
||||
|
||||
```sql
|
||||
-- Bad: slow at high offsets
|
||||
SELECT * FROM products ORDER BY id LIMIT 20 OFFSET 10000;
|
||||
|
||||
-- Good: keyset pagination
|
||||
SELECT * FROM products WHERE id > $last_seen_id ORDER BY id LIMIT 20;
|
||||
```
|
||||
@@ -1,9 +1,362 @@
|
||||
# Schema Design
|
||||
# Schema Design Reference
|
||||
|
||||
Best practices for designing Postgres schemas.
|
||||
## Contents
|
||||
- Data type best practices
|
||||
- Primary keys
|
||||
- Foreign keys and referential integrity
|
||||
- Check and exclusion constraints
|
||||
- Normalization guidelines
|
||||
- Denormalization patterns
|
||||
- Partitioning
|
||||
- Multi-tenant patterns
|
||||
- Migration safety
|
||||
|
||||
## Choosing Data Types
|
||||
## Data Type Best Practices
|
||||
|
||||
- Prefer `text` over `varchar(n)` unless a length constraint is meaningful to the domain.
|
||||
- Use `timestamptz` instead of `timestamp` to always store timezone-aware timestamps.
|
||||
- Use `uuid` for primary keys when IDs may be exposed externally or generated client-side.
|
||||
### Identity Columns
|
||||
|
||||
```sql
|
||||
-- Preferred: GENERATED ALWAYS
|
||||
CREATE TABLE orders (
|
||||
id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||
...
|
||||
);
|
||||
|
||||
-- If you need to occasionally override:
|
||||
INSERT INTO orders OVERRIDING SYSTEM VALUE ...
|
||||
|
||||
-- UUID alternative (good for distributed systems)
|
||||
CREATE TABLE orders (
|
||||
id uuid DEFAULT gen_random_uuid() PRIMARY KEY,
|
||||
...
|
||||
);
|
||||
```
|
||||
|
||||
**UUID v4 vs v7**: `gen_random_uuid()` generates UUIDv4 (random), which causes B-tree index fragmentation on large tables because inserts scatter across the index. UUIDv7 (time-ordered) preserves insert locality. **PG18+** has built-in `uuidv7()` (and explicit `uuidv4()`). On older versions, use the `pg_uuidv7` extension. Prefer UUIDv7 for primary keys.
|
||||
|
||||
```sql
|
||||
-- PG18+: built-in UUIDv7
|
||||
CREATE TABLE events (
|
||||
id uuid DEFAULT uuidv7() PRIMARY KEY,
|
||||
...
|
||||
);
|
||||
```
|
||||
|
||||
Avoid `serial` / `bigserial` — they create an implicit sequence with looser ownership semantics.
|
||||
|
||||
### Timestamps
|
||||
|
||||
Always use `timestamptz`. PostgreSQL stores it as UTC internally and converts on display.
|
||||
|
||||
```sql
|
||||
CREATE TABLE events (
|
||||
id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||
occurred_at timestamptz NOT NULL DEFAULT now(),
|
||||
-- NOT: occurred_at timestamp (loses timezone context)
|
||||
);
|
||||
```
|
||||
|
||||
### Text Fields
|
||||
|
||||
Prefer `text` with a `CHECK` constraint over `varchar(n)`:
|
||||
|
||||
```sql
|
||||
CREATE TABLE users (
|
||||
email text NOT NULL CHECK (length(email) <= 254),
|
||||
username text NOT NULL CHECK (length(username) BETWEEN 3 AND 30)
|
||||
);
|
||||
```
|
||||
|
||||
`varchar(n)` provides no performance benefit and makes future changes harder.
|
||||
|
||||
### UNIQUE NULLS NOT DISTINCT (PG15+)
|
||||
|
||||
By default, NULLs are considered distinct in unique constraints (multiple NULLs allowed). PG15 adds an option to treat NULLs as equal:
|
||||
|
||||
```sql
|
||||
-- Standard: allows multiple rows with NULL in email
|
||||
CREATE UNIQUE INDEX idx_email_standard ON users(email);
|
||||
|
||||
-- PG15+: only one NULL allowed
|
||||
CREATE UNIQUE INDEX idx_email_nulls_not_distinct
|
||||
ON users(email) NULLS NOT DISTINCT;
|
||||
```
|
||||
|
||||
### Naming Conventions
|
||||
|
||||
Use `snake_case` without quotes for all identifiers. Unquoted identifiers fold to lowercase automatically. Quoted mixed-case identifiers (e.g., `"userId"`) require quotes everywhere and break many ORMs, tools, and AI assistants.
|
||||
|
||||
### Enums vs. Check Constraints vs. Lookup Tables
|
||||
|
||||
| Approach | Pros | Cons |
|
||||
|----------|------|------|
|
||||
| `CHECK (col IN (...))` | Simple, no DDL type | Requires ALTER TABLE to add values |
|
||||
| `CREATE TYPE ... AS ENUM` | Type safety, compact storage | Cannot remove values, ALTER TYPE needed |
|
||||
| Lookup/reference table | FK enforced, can add metadata | Extra join |
|
||||
|
||||
For small, stable sets (status, priority): CHECK or ENUM.
|
||||
For evolving sets or sets needing metadata: lookup table.
|
||||
|
||||
## Foreign Keys
|
||||
|
||||
### Basics
|
||||
|
||||
```sql
|
||||
CREATE TABLE orders (
|
||||
id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||
customer_id bigint NOT NULL REFERENCES customers(id),
|
||||
...
|
||||
);
|
||||
```
|
||||
|
||||
### ON DELETE / ON UPDATE Actions
|
||||
|
||||
| Action | Use when |
|
||||
|--------|----------|
|
||||
| `NO ACTION` (default) | Reject the update or deletion if a violation remains when the constraint is checked; the check can be deferred when the foreign key is deferrable and currently deferred |
|
||||
| `RESTRICT` | Reject the referenced operation immediately; the check cannot be deferred |
|
||||
| `CASCADE` | Child rows are meaningless without parent (e.g., order_items when order is deleted) |
|
||||
| `SET NULL` | Relationship is optional, preserve child row |
|
||||
| `SET DEFAULT` | Rare; reassign to a valid default parent |
|
||||
|
||||
### Deferrable Foreign Keys
|
||||
|
||||
Useful when you need to insert rows in both tables within a single transaction regardless of order:
|
||||
|
||||
```sql
|
||||
ALTER TABLE order_items
|
||||
ADD CONSTRAINT fk_order
|
||||
FOREIGN KEY (order_id) REFERENCES orders(id)
|
||||
DEFERRABLE INITIALLY DEFERRED;
|
||||
```
|
||||
|
||||
### Index the FK Column
|
||||
|
||||
PostgreSQL does NOT auto-create indexes on FK columns. Always add one:
|
||||
|
||||
```sql
|
||||
CREATE INDEX idx_orders_customer_id ON orders(customer_id);
|
||||
```
|
||||
|
||||
Without this index:
|
||||
- JOINs on the FK are slow (seq scan on child table)
|
||||
- DELETE on the parent table locks the child table and scans it fully
|
||||
|
||||
## Check Constraints
|
||||
|
||||
```sql
|
||||
CREATE TABLE products (
|
||||
price numeric(10,2) NOT NULL CHECK (price > 0),
|
||||
discount_pct numeric(3,2) CHECK (discount_pct BETWEEN 0 AND 1),
|
||||
start_date date NOT NULL,
|
||||
end_date date,
|
||||
CHECK (end_date IS NULL OR end_date > start_date)
|
||||
);
|
||||
```
|
||||
|
||||
### Exclusion Constraints (Prevent Overlaps)
|
||||
|
||||
```sql
|
||||
CREATE EXTENSION IF NOT EXISTS btree_gist;
|
||||
|
||||
CREATE TABLE room_bookings (
|
||||
room_id int NOT NULL,
|
||||
during tstzrange NOT NULL,
|
||||
EXCLUDE USING gist (room_id WITH =, during WITH &&)
|
||||
);
|
||||
```
|
||||
|
||||
This prevents overlapping bookings for the same room at the database level.
|
||||
|
||||
### Temporal Constraints — WITHOUT OVERLAPS (PG18+)
|
||||
|
||||
PG18 adds built-in temporal constraint support, simplifying the exclusion constraint pattern above:
|
||||
|
||||
```sql
|
||||
CREATE EXTENSION IF NOT EXISTS btree_gist;
|
||||
|
||||
CREATE TABLE room_bookings (
|
||||
room_id int NOT NULL,
|
||||
during tstzrange NOT NULL,
|
||||
PRIMARY KEY (room_id, during WITHOUT OVERLAPS)
|
||||
);
|
||||
```
|
||||
|
||||
Foreign keys can also reference temporal primary keys using `PERIOD`:
|
||||
|
||||
```sql
|
||||
CREATE TABLE reservations (
|
||||
id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||
room_id int NOT NULL,
|
||||
during tstzrange NOT NULL,
|
||||
FOREIGN KEY (room_id, PERIOD during)
|
||||
REFERENCES room_bookings (room_id, PERIOD during)
|
||||
);
|
||||
```
|
||||
|
||||
This eliminates the need to write the exclusion constraint manually. The temporal key is implemented with GiST; scalar key columns such as `room_id int` still need an appropriate GiST operator class, supplied by `btree_gist` for common scalar types.
|
||||
|
||||
### NOT ENFORCED Constraints (PG18+)
|
||||
|
||||
CHECK and foreign key constraints can be marked as informational only — the database trusts the data without enforcing the constraint. Useful for documenting intent or helping the planner without paying enforcement cost:
|
||||
|
||||
```sql
|
||||
ALTER TABLE orders ADD CONSTRAINT positive_total
|
||||
CHECK (total > 0) NOT ENFORCED;
|
||||
```
|
||||
|
||||
## Normalization Guidelines
|
||||
|
||||
### When to Normalize
|
||||
|
||||
- Data integrity is critical (financial, compliance)
|
||||
- Multiple writers update the same logical data
|
||||
- Storage is a concern (avoid data duplication)
|
||||
|
||||
### When to Denormalize
|
||||
|
||||
- Read-heavy workloads where join cost is measurable
|
||||
- Materialized views can serve as denormalized read models
|
||||
- JSONB columns for flexible, schema-less attributes that don't need FK integrity
|
||||
|
||||
### Materialized Views as Denormalization
|
||||
|
||||
```sql
|
||||
CREATE MATERIALIZED VIEW mv_order_summary AS
|
||||
SELECT
|
||||
o.id,
|
||||
o.created_at,
|
||||
c.name AS customer_name,
|
||||
sum(oi.quantity * oi.unit_price) AS total
|
||||
FROM orders o
|
||||
JOIN customers c ON c.id = o.customer_id
|
||||
JOIN order_items oi ON oi.order_id = o.id
|
||||
GROUP BY o.id, o.created_at, c.name;
|
||||
|
||||
CREATE UNIQUE INDEX ON mv_order_summary(id);
|
||||
|
||||
-- Refresh periodically or after batch writes:
|
||||
REFRESH MATERIALIZED VIEW CONCURRENTLY mv_order_summary;
|
||||
```
|
||||
|
||||
## Partitioning (Declarative)
|
||||
|
||||
### When to Partition
|
||||
|
||||
- Table exceeds tens of millions of rows
|
||||
- Queries consistently filter on the partition key
|
||||
- You need efficient bulk deletion (DROP partition vs. DELETE)
|
||||
- Maintenance operations (VACUUM, reindex) are slow on the full table
|
||||
|
||||
### Range Partitioning (most common)
|
||||
|
||||
```sql
|
||||
CREATE TABLE events (
|
||||
id bigint GENERATED ALWAYS AS IDENTITY,
|
||||
occurred_at timestamptz NOT NULL,
|
||||
payload jsonb
|
||||
) PARTITION BY RANGE (occurred_at);
|
||||
|
||||
CREATE TABLE events_2024_q1 PARTITION OF events
|
||||
FOR VALUES FROM ('2024-01-01') TO ('2024-04-01');
|
||||
CREATE TABLE events_2024_q2 PARTITION OF events
|
||||
FOR VALUES FROM ('2024-04-01') TO ('2024-07-01');
|
||||
```
|
||||
|
||||
### List Partitioning (multi-tenant)
|
||||
|
||||
```sql
|
||||
CREATE TABLE tenant_data (
|
||||
tenant_id int NOT NULL,
|
||||
data jsonb
|
||||
) PARTITION BY LIST (tenant_id);
|
||||
|
||||
CREATE TABLE tenant_data_1 PARTITION OF tenant_data FOR VALUES IN (1);
|
||||
CREATE TABLE tenant_data_2 PARTITION OF tenant_data FOR VALUES IN (2);
|
||||
```
|
||||
|
||||
### Detaching Partitions (PG14+)
|
||||
|
||||
```sql
|
||||
-- Non-blocking detach (PG14+): doesn't hold AccessExclusiveLock
|
||||
ALTER TABLE events DETACH PARTITION events_2024_q1 CONCURRENTLY;
|
||||
```
|
||||
|
||||
### Key Rules
|
||||
|
||||
- Partition key must be part of the primary key and all unique indexes
|
||||
- Indexes defined on parent are auto-created on child partitions
|
||||
- Foreign keys referencing partitioned tables are supported
|
||||
- Exclusion constraints on partitioned tables require PG17+ (equality on partition key only)
|
||||
|
||||
## Virtual Generated Columns (PG18+)
|
||||
|
||||
PG18 defaults generated columns to `VIRTUAL` (computed at read time, not stored on disk). Use `STORED` when you need to index the expression.
|
||||
|
||||
```sql
|
||||
CREATE TABLE orders (
|
||||
id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||
quantity int NOT NULL,
|
||||
unit_price numeric(10,2) NOT NULL,
|
||||
description text NOT NULL,
|
||||
-- Virtual: computed on read, no storage cost
|
||||
total numeric GENERATED ALWAYS AS (quantity * unit_price) VIRTUAL,
|
||||
-- Stored: persisted on disk, can be indexed
|
||||
search_text tsvector GENERATED ALWAYS AS (
|
||||
to_tsvector('english', description)
|
||||
) STORED
|
||||
);
|
||||
```
|
||||
|
||||
Virtual columns save storage and write I/O but cannot be indexed directly. Use `STORED` when you need an index on the expression.
|
||||
|
||||
## Multi-Tenant Patterns
|
||||
|
||||
### Row-Level Security (RLS)
|
||||
|
||||
```sql
|
||||
ALTER TABLE orders ENABLE ROW LEVEL SECURITY;
|
||||
|
||||
CREATE POLICY tenant_isolation ON orders
|
||||
USING (tenant_id = current_setting('app.current_tenant')::int);
|
||||
|
||||
-- Set per-connection:
|
||||
SET app.current_tenant = '42';
|
||||
```
|
||||
|
||||
### Shared Schema with tenant_id
|
||||
|
||||
- Add `tenant_id` to every table
|
||||
- Include `tenant_id` in all indexes (composite with other columns)
|
||||
- Use RLS or application-level filtering
|
||||
|
||||
## Migration Safety
|
||||
|
||||
### Safe Column Operations
|
||||
|
||||
| Operation | Safe online? | Notes |
|
||||
|-----------|-------------|-------|
|
||||
| `ADD COLUMN` (nullable, no default) | Yes | Instant metadata change |
|
||||
| `ADD COLUMN ... DEFAULT x` | Yes | Default stored in catalog, no table rewrite |
|
||||
| `DROP COLUMN` | Yes | Marks column as dropped, no rewrite |
|
||||
| `ALTER COLUMN SET NOT NULL` | Caution | Full table scan to validate (use CHECK first) |
|
||||
| `ALTER COLUMN TYPE` | No | Full table rewrite + exclusive lock |
|
||||
| `ADD CONSTRAINT ... NOT VALID` | Yes | Doesn't scan existing rows |
|
||||
| `VALIDATE CONSTRAINT` | Yes | ShareUpdateExclusiveLock (reads/writes allowed) |
|
||||
|
||||
### Safe NOT NULL Pattern
|
||||
|
||||
```sql
|
||||
-- Step 1: Add CHECK constraint without validating existing rows
|
||||
ALTER TABLE orders ADD CONSTRAINT orders_status_nn
|
||||
CHECK (status IS NOT NULL) NOT VALID;
|
||||
|
||||
-- Step 2: Validate in background (non-blocking)
|
||||
ALTER TABLE orders VALIDATE CONSTRAINT orders_status_nn;
|
||||
|
||||
-- Step 3: SET NOT NULL is instant if a valid CHECK exists
|
||||
ALTER TABLE orders ALTER COLUMN status SET NOT NULL;
|
||||
|
||||
-- Step 4: Drop the now-redundant CHECK
|
||||
ALTER TABLE orders DROP CONSTRAINT orders_status_nn;
|
||||
```
|
||||
|
||||
@@ -0,0 +1,489 @@
|
||||
# Security & Roles Reference
|
||||
|
||||
## Contents
|
||||
- Role management
|
||||
- Privilege system
|
||||
- Schema-based access control
|
||||
- Row-Level Security (RLS)
|
||||
- pg_hba.conf authentication
|
||||
- Password policies and authentication methods
|
||||
- Security functions and best practices
|
||||
|
||||
## Role Management
|
||||
|
||||
PostgreSQL uses **roles** for both users and groups. A role with `LOGIN` is a user; a role without is a group.
|
||||
|
||||
### Creating Roles
|
||||
|
||||
```sql
|
||||
-- User role (can log in)
|
||||
CREATE ROLE app_user WITH LOGIN PASSWORD 'strong_password_here';
|
||||
|
||||
-- Group role (cannot log in, used for privilege grouping)
|
||||
CREATE ROLE readonly;
|
||||
CREATE ROLE readwrite;
|
||||
CREATE ROLE admin;
|
||||
|
||||
-- Role with specific attributes
|
||||
CREATE ROLE backup_user WITH LOGIN REPLICATION PASSWORD '...';
|
||||
CREATE ROLE migrator WITH LOGIN CREATEDB PASSWORD '...';
|
||||
```
|
||||
|
||||
### Role Attributes
|
||||
|
||||
| Attribute | Meaning |
|
||||
|-----------|---------|
|
||||
| `LOGIN` | Can connect to the database |
|
||||
| `SUPERUSER` | Bypasses all permission checks (dangerous) |
|
||||
| `CREATEDB` | Can create databases |
|
||||
| `CREATEROLE` | Can create/alter/drop other roles |
|
||||
| `REPLICATION` | Can initiate streaming replication |
|
||||
| `BYPASSRLS` | Bypasses Row-Level Security policies |
|
||||
| `INHERIT` | Automatically inherits privileges of member roles (default) |
|
||||
| `CONNECTION LIMIT n` | Max concurrent connections for this role |
|
||||
| `VALID UNTIL 'timestamp'` | Password expiration |
|
||||
|
||||
```sql
|
||||
-- Modify attributes
|
||||
ALTER ROLE app_user WITH CONNECTION LIMIT 10;
|
||||
ALTER ROLE temp_user VALID UNTIL '2025-01-01';
|
||||
ALTER ROLE app_user WITH PASSWORD 'new_password';
|
||||
|
||||
-- Inspect roles
|
||||
SELECT rolname, rolsuper, rolcreatedb, rolcreaterole, rolcanlogin,
|
||||
rolreplication, rolbypassrls, rolconnlimit, rolvaliduntil
|
||||
FROM pg_roles
|
||||
WHERE rolname NOT LIKE 'pg_%'
|
||||
ORDER BY rolname;
|
||||
```
|
||||
|
||||
### Group Membership
|
||||
|
||||
```sql
|
||||
-- Add a user to a group
|
||||
GRANT readonly TO app_user;
|
||||
GRANT readwrite TO app_user;
|
||||
|
||||
-- Remove membership
|
||||
REVOKE readwrite FROM app_user;
|
||||
|
||||
-- Check memberships
|
||||
SELECT
|
||||
r.rolname AS role,
|
||||
m.rolname AS member
|
||||
FROM pg_auth_members am
|
||||
JOIN pg_roles r ON r.oid = am.roleid
|
||||
JOIN pg_roles m ON m.oid = am.member
|
||||
ORDER BY r.rolname, m.rolname;
|
||||
```
|
||||
|
||||
### Predefined Roles
|
||||
|
||||
PostgreSQL provides system-defined roles for common privileged capabilities. PostgreSQL 13 and earlier documentation called these "default roles"; PostgreSQL 14 renamed the category to "predefined roles." Individual roles were introduced in different releases.
|
||||
|
||||
| Role | Grants |
|
||||
|------|--------|
|
||||
| `pg_read_all_data` | SELECT on all tables, views, sequences in all schemas |
|
||||
| `pg_write_all_data` | INSERT, UPDATE, DELETE on all tables, sequences in all schemas |
|
||||
| `pg_read_all_settings` | Read all GUC settings (even superuser-only) |
|
||||
| `pg_read_all_stats` | Read all pg_stat_* views |
|
||||
| `pg_monitor` | Read monitoring views (`pg_stat_*`, `pg_locks`, etc.) |
|
||||
| `pg_signal_backend` | Send signals to other backends (cancel/terminate) |
|
||||
| `pg_checkpoint` (PG15+) | Run CHECKPOINT |
|
||||
| `pg_maintain` (PG17+) | Run VACUUM, ANALYZE, REINDEX, CLUSTER, REFRESH MATERIALIZED VIEW, and LOCK TABLE on all relations |
|
||||
|
||||
```sql
|
||||
-- Give a monitoring role read access to all stats
|
||||
GRANT pg_monitor TO monitoring_user;
|
||||
|
||||
-- Give an app role read access to all data without per-table grants
|
||||
GRANT pg_read_all_data TO reporting_user;
|
||||
```
|
||||
|
||||
## Privilege System
|
||||
|
||||
### Object Privileges
|
||||
|
||||
```sql
|
||||
-- Grant on tables
|
||||
GRANT SELECT ON orders TO readonly;
|
||||
GRANT SELECT, INSERT, UPDATE, DELETE ON orders TO readwrite;
|
||||
GRANT ALL PRIVILEGES ON orders TO admin;
|
||||
|
||||
-- Grant on all tables in a schema
|
||||
GRANT SELECT ON ALL TABLES IN SCHEMA public TO readonly;
|
||||
GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO readwrite;
|
||||
|
||||
-- Grant on sequences (needed for INSERT with serial/identity columns)
|
||||
GRANT USAGE ON ALL SEQUENCES IN SCHEMA public TO readwrite;
|
||||
|
||||
-- Grant on schema (required to access objects within it)
|
||||
GRANT USAGE ON SCHEMA public TO readonly;
|
||||
GRANT USAGE, CREATE ON SCHEMA public TO readwrite;
|
||||
```
|
||||
|
||||
### Default Privileges
|
||||
|
||||
Set privileges that automatically apply to future objects:
|
||||
|
||||
```sql
|
||||
-- As the object owner or superuser:
|
||||
ALTER DEFAULT PRIVILEGES IN SCHEMA public
|
||||
GRANT SELECT ON TABLES TO readonly;
|
||||
|
||||
ALTER DEFAULT PRIVILEGES IN SCHEMA public
|
||||
GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO readwrite;
|
||||
|
||||
ALTER DEFAULT PRIVILEGES IN SCHEMA public
|
||||
GRANT USAGE ON SEQUENCES TO readwrite;
|
||||
|
||||
-- For objects created by a specific role:
|
||||
ALTER DEFAULT PRIVILEGES FOR ROLE app_owner IN SCHEMA public
|
||||
GRANT SELECT ON TABLES TO readonly;
|
||||
```
|
||||
|
||||
**Important**: Default privileges only affect objects created **after** the `ALTER DEFAULT PRIVILEGES` command. Existing objects need explicit `GRANT`.
|
||||
|
||||
### Revoking Privileges
|
||||
|
||||
```sql
|
||||
-- Revoke specific privileges
|
||||
REVOKE INSERT, UPDATE, DELETE ON orders FROM readonly;
|
||||
|
||||
-- Revoke all privileges
|
||||
REVOKE ALL PRIVILEGES ON orders FROM some_role;
|
||||
|
||||
-- Revoke from public (default grant on new databases)
|
||||
REVOKE CREATE ON SCHEMA public FROM PUBLIC;
|
||||
REVOKE ALL ON DATABASE mydb FROM PUBLIC;
|
||||
```
|
||||
|
||||
### Inspecting Privileges
|
||||
|
||||
```sql
|
||||
-- Table privileges
|
||||
SELECT
|
||||
grantee,
|
||||
table_schema,
|
||||
table_name,
|
||||
privilege_type
|
||||
FROM information_schema.table_privileges
|
||||
WHERE table_schema = 'public'
|
||||
ORDER BY table_name, grantee;
|
||||
|
||||
-- Compact: privileges per table
|
||||
SELECT
|
||||
relname,
|
||||
relacl -- array of 'grantee=privileges/grantor'
|
||||
FROM pg_class
|
||||
WHERE relkind = 'r' AND relnamespace = 'public'::regnamespace;
|
||||
|
||||
-- Function privileges
|
||||
SELECT
|
||||
routine_name,
|
||||
grantee,
|
||||
privilege_type
|
||||
FROM information_schema.routine_privileges
|
||||
WHERE routine_schema = 'public';
|
||||
```
|
||||
|
||||
## Schema-Based Access Control
|
||||
|
||||
Use schemas to organize objects and control access:
|
||||
|
||||
```sql
|
||||
-- Create isolated schemas
|
||||
CREATE SCHEMA app;
|
||||
CREATE SCHEMA reporting;
|
||||
CREATE SCHEMA staging;
|
||||
|
||||
-- Grant access per schema
|
||||
GRANT USAGE ON SCHEMA app TO app_user;
|
||||
GRANT USAGE ON SCHEMA reporting TO reporting_user;
|
||||
GRANT USAGE ON SCHEMA staging TO etl_user;
|
||||
|
||||
-- Remove default public schema access
|
||||
REVOKE CREATE ON SCHEMA public FROM PUBLIC;
|
||||
```
|
||||
|
||||
### Search Path Security
|
||||
|
||||
The `search_path` determines which schemas are searched for unqualified table names. A malicious `search_path` can redirect queries to attacker-controlled tables.
|
||||
|
||||
```sql
|
||||
-- Set a safe search path (explicit, no reliance on public)
|
||||
ALTER ROLE app_user SET search_path = app, public;
|
||||
|
||||
-- For functions: use SECURITY DEFINER carefully
|
||||
CREATE FUNCTION get_balance(account_id int) RETURNS numeric
|
||||
SECURITY DEFINER
|
||||
SET search_path = pg_catalog, pg_temp
|
||||
AS $$
|
||||
SELECT balance FROM app.accounts WHERE id = account_id;
|
||||
$$ LANGUAGE sql;
|
||||
```
|
||||
|
||||
**Always give `SECURITY DEFINER` functions a fixed `search_path` containing only trusted schemas.** Explicitly list `pg_temp` last: if it is omitted, PostgreSQL searches the session's temporary schema first for relations and data types, which can allow temporary objects to shadow intended objects. When application objects are fully qualified, prefer `pg_catalog, pg_temp`; otherwise list only trusted application schemas followed by `pg_temp`.
|
||||
|
||||
## Row-Level Security (RLS)
|
||||
|
||||
RLS adds per-row access control enforced by the database, not the application.
|
||||
|
||||
### Basic Setup
|
||||
|
||||
```sql
|
||||
-- Enable RLS on a table
|
||||
ALTER TABLE orders ENABLE ROW LEVEL SECURITY;
|
||||
|
||||
-- Policy for tenant isolation
|
||||
CREATE POLICY tenant_isolation ON orders
|
||||
USING (tenant_id = current_setting('app.current_tenant')::int);
|
||||
|
||||
-- Test through a non-owner application role; table owners bypass RLS by default.
|
||||
SET ROLE app_user;
|
||||
SET app.current_tenant = '42';
|
||||
|
||||
-- All queries on orders now automatically filter by tenant_id = 42
|
||||
SELECT * FROM orders; -- only sees tenant 42's orders
|
||||
RESET ROLE;
|
||||
```
|
||||
|
||||
### Policy Types
|
||||
|
||||
```sql
|
||||
-- SELECT policy (restrict which rows can be read)
|
||||
CREATE POLICY read_own ON documents
|
||||
FOR SELECT
|
||||
USING (owner_id = current_setting('app.current_user_id')::bigint);
|
||||
|
||||
-- INSERT policy (restrict which rows can be inserted)
|
||||
CREATE POLICY insert_own ON documents
|
||||
FOR INSERT
|
||||
WITH CHECK (owner_id = current_setting('app.current_user_id')::bigint);
|
||||
|
||||
-- UPDATE policy (restrict which rows can be updated, and what values are allowed)
|
||||
CREATE POLICY update_own ON documents
|
||||
FOR UPDATE
|
||||
USING (owner_id = current_setting('app.current_user_id')::bigint) -- rows selectable for update
|
||||
WITH CHECK (owner_id = current_setting('app.current_user_id')::bigint); -- required post-update value
|
||||
|
||||
-- DELETE policy
|
||||
CREATE POLICY delete_own ON documents
|
||||
FOR DELETE
|
||||
USING (owner_id = current_setting('app.current_user_id')::bigint);
|
||||
|
||||
-- ALL (applies to all commands)
|
||||
CREATE POLICY full_access ON documents
|
||||
FOR ALL
|
||||
USING (owner_id = current_setting('app.current_user_id')::bigint)
|
||||
WITH CHECK (owner_id = current_setting('app.current_user_id')::bigint);
|
||||
```
|
||||
|
||||
### Multiple Policies
|
||||
|
||||
Multiple policies on the same table are combined with OR (by default). Use `RESTRICTIVE` for AND:
|
||||
|
||||
```sql
|
||||
-- Permissive (default): combined with OR
|
||||
CREATE POLICY see_active ON orders
|
||||
USING (status = 'active');
|
||||
|
||||
CREATE POLICY see_own ON orders
|
||||
USING (user_id = current_setting('app.current_user_id')::bigint);
|
||||
-- User sees rows that are active OR owned by them
|
||||
|
||||
-- Restrictive: combined with AND (with permissive policies)
|
||||
CREATE POLICY must_be_active ON orders AS RESTRICTIVE
|
||||
USING (status != 'deleted');
|
||||
-- Combined: (active OR own) AND not_deleted
|
||||
```
|
||||
|
||||
### RLS Caveats
|
||||
|
||||
- **Table owner bypasses RLS** by default. Use `ALTER TABLE ... FORCE ROW LEVEL SECURITY` to apply RLS to the owner too.
|
||||
- **Superusers and `BYPASSRLS` roles** bypass all RLS policies.
|
||||
- **No policies = deny all** when RLS is enabled (except for table owner).
|
||||
- **Performance**: RLS adds filter conditions to every query. Ensure the policy columns are indexed.
|
||||
- **Leaky views**: Functions in RLS policies that are not `LEAKPROOF` could theoretically leak data via error messages or side channels. Use `LEAKPROOF` functions where possible.
|
||||
|
||||
```sql
|
||||
-- Force RLS on table owner
|
||||
ALTER TABLE orders FORCE ROW LEVEL SECURITY;
|
||||
|
||||
-- Check which tables have RLS enabled
|
||||
SELECT relname, relrowsecurity, relforcerowsecurity
|
||||
FROM pg_class
|
||||
WHERE relkind = 'r' AND relnamespace = 'public'::regnamespace;
|
||||
```
|
||||
|
||||
## pg_hba.conf Authentication
|
||||
|
||||
`pg_hba.conf` controls who can connect, from where, and how they authenticate. Rules are evaluated top-to-bottom; first match wins.
|
||||
|
||||
### Format
|
||||
|
||||
```
|
||||
# TYPE DATABASE USER ADDRESS METHOD
|
||||
local all all scram-sha-256
|
||||
host all all 127.0.0.1/32 scram-sha-256
|
||||
host all all ::1/128 scram-sha-256
|
||||
host mydb app_user 10.0.0.0/8 scram-sha-256
|
||||
host all all 0.0.0.0/0 reject
|
||||
```
|
||||
|
||||
### Authentication Methods
|
||||
|
||||
| Method | Security | Use case |
|
||||
|--------|----------|----------|
|
||||
| `scram-sha-256` | Strong | **Recommended default** — salted challenge-response |
|
||||
| `md5` | Weak | Legacy — **deprecated in PG18** (emits warnings) |
|
||||
| `cert` | Strong | Client certificate (mutual TLS) |
|
||||
| `peer` | Strong | Local connections — maps OS user to PG role |
|
||||
| `ident` | Moderate | TCP — maps OS user via ident server |
|
||||
| `gss` | Strong | Kerberos/GSSAPI |
|
||||
| `ldap` | Moderate | LDAP directory authentication |
|
||||
| `trust` | None | **Never use in production** — no password required |
|
||||
| `reject` | N/A | Explicitly deny connections |
|
||||
|
||||
### Best Practices
|
||||
|
||||
```
|
||||
# 1. Use scram-sha-256 (not md5)
|
||||
# 2. Be specific — don't use 'all' for database/user in production
|
||||
# 3. Restrict by IP range
|
||||
# 4. Put reject rules at the bottom as a catch-all
|
||||
# 5. Use 'local peer' for admin access (no password over unix socket)
|
||||
|
||||
local all postgres peer
|
||||
host mydb app_user 10.0.1.0/24 scram-sha-256
|
||||
host replication repl_user 10.0.2.0/24 scram-sha-256
|
||||
hostssl mydb all 0.0.0.0/0 scram-sha-256
|
||||
host all all 0.0.0.0/0 reject
|
||||
```
|
||||
|
||||
After modifying `pg_hba.conf`, reload:
|
||||
|
||||
```sql
|
||||
SELECT pg_reload_conf();
|
||||
-- or from CLI: pg_ctl reload
|
||||
```
|
||||
|
||||
## Password Policies and Authentication
|
||||
|
||||
### Enforce scram-sha-256
|
||||
|
||||
```sql
|
||||
-- In postgresql.conf:
|
||||
-- password_encryption = scram-sha-256 (default in PG14+)
|
||||
|
||||
-- Verify
|
||||
SHOW password_encryption;
|
||||
|
||||
-- Check which roles still use md5
|
||||
SELECT rolname, rolpassword LIKE 'md5%' AS uses_md5
|
||||
FROM pg_authid
|
||||
WHERE rolcanlogin AND rolpassword IS NOT NULL;
|
||||
```
|
||||
|
||||
### Password Expiration
|
||||
|
||||
```sql
|
||||
-- Set expiration on a role
|
||||
ALTER ROLE app_user VALID UNTIL '2025-06-01';
|
||||
|
||||
-- Check expiration
|
||||
SELECT rolname, rolvaliduntil
|
||||
FROM pg_roles
|
||||
WHERE rolvaliduntil IS NOT NULL;
|
||||
```
|
||||
|
||||
PostgreSQL does not enforce password complexity natively. Use `passwordcheck` module or application-level validation:
|
||||
|
||||
```sql
|
||||
-- In postgresql.conf:
|
||||
-- shared_preload_libraries = 'passwordcheck'
|
||||
-- Enforces minimum length and basic complexity
|
||||
```
|
||||
|
||||
### SSL/TLS
|
||||
|
||||
```sql
|
||||
-- Check if SSL is enabled
|
||||
SHOW ssl;
|
||||
|
||||
-- Check current connection's SSL status
|
||||
SELECT ssl, version, cipher, bits
|
||||
FROM pg_stat_ssl
|
||||
WHERE pid = pg_backend_pid();
|
||||
|
||||
-- Require SSL for specific connections (in pg_hba.conf):
|
||||
-- hostssl mydb all 0.0.0.0/0 scram-sha-256
|
||||
-- hostnossl mydb all 0.0.0.0/0 reject
|
||||
```
|
||||
|
||||
## Security Best Practices
|
||||
|
||||
### Principle of Least Privilege
|
||||
|
||||
```sql
|
||||
-- 1. Revoke default public access
|
||||
REVOKE CREATE ON SCHEMA public FROM PUBLIC;
|
||||
REVOKE ALL ON DATABASE mydb FROM PUBLIC;
|
||||
|
||||
-- 2. Create role groups with specific privileges
|
||||
CREATE ROLE app_readonly;
|
||||
GRANT USAGE ON SCHEMA public TO app_readonly;
|
||||
GRANT SELECT ON ALL TABLES IN SCHEMA public TO app_readonly;
|
||||
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO app_readonly;
|
||||
|
||||
CREATE ROLE app_readwrite;
|
||||
GRANT USAGE ON SCHEMA public TO app_readwrite;
|
||||
GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO app_readwrite;
|
||||
GRANT USAGE ON ALL SEQUENCES IN SCHEMA public TO app_readwrite;
|
||||
ALTER DEFAULT PRIVILEGES IN SCHEMA public
|
||||
GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO app_readwrite;
|
||||
ALTER DEFAULT PRIVILEGES IN SCHEMA public
|
||||
GRANT USAGE ON SEQUENCES TO app_readwrite;
|
||||
|
||||
-- 3. Assign users to groups
|
||||
CREATE ROLE web_app WITH LOGIN PASSWORD '...';
|
||||
GRANT app_readwrite TO web_app;
|
||||
|
||||
CREATE ROLE analyst WITH LOGIN PASSWORD '...';
|
||||
GRANT app_readonly TO analyst;
|
||||
```
|
||||
|
||||
### Audit Who Has Access
|
||||
|
||||
```sql
|
||||
-- All role memberships
|
||||
SELECT
|
||||
r.rolname AS group_role,
|
||||
m.rolname AS member,
|
||||
am.admin_option
|
||||
FROM pg_auth_members am
|
||||
JOIN pg_roles r ON r.oid = am.roleid
|
||||
JOIN pg_roles m ON m.oid = am.member
|
||||
ORDER BY r.rolname, m.rolname;
|
||||
|
||||
-- Roles with superuser
|
||||
SELECT rolname FROM pg_roles WHERE rolsuper;
|
||||
|
||||
-- Roles with BYPASSRLS (can circumvent row-level security)
|
||||
SELECT rolname FROM pg_roles WHERE rolbypassrls;
|
||||
|
||||
-- Roles that can create databases or roles
|
||||
SELECT rolname, rolcreatedb, rolcreaterole
|
||||
FROM pg_roles
|
||||
WHERE rolcreatedb OR rolcreaterole;
|
||||
```
|
||||
|
||||
### Avoid Common Security Mistakes
|
||||
|
||||
1. **Never use `trust` authentication** in production (allows passwordless access)
|
||||
2. **Never run applications as superuser** — create dedicated roles with minimal privileges
|
||||
3. **Don't store passwords in connection strings** in code — use `.pgpass`, environment variables, or secret managers
|
||||
4. **Pin `search_path`** in `SECURITY DEFINER` functions
|
||||
5. **Enable SSL** for all non-local connections (`hostssl` in pg_hba.conf)
|
||||
6. **Audit superuser access** regularly — minimize the number of superuser roles
|
||||
7. **Use `scram-sha-256`** — md5 is deprecated (PG18 emits warnings on md5 password creation)
|
||||
@@ -0,0 +1,344 @@
|
||||
# Transaction Isolation Reference
|
||||
|
||||
## Contents
|
||||
- Isolation levels overview
|
||||
- READ COMMITTED (default)
|
||||
- REPEATABLE READ
|
||||
- SERIALIZABLE
|
||||
- Common pitfalls and surprises
|
||||
- Choosing the right level
|
||||
- Retry patterns for serialization failures
|
||||
|
||||
## Isolation Levels Overview
|
||||
|
||||
PostgreSQL implements three isolation levels (READ UNCOMMITTED maps to READ COMMITTED):
|
||||
|
||||
| Level | Dirty reads | Non-repeatable reads | Phantom reads | Serialization anomalies |
|
||||
|-------|-------------|---------------------|---------------|------------------------|
|
||||
| READ COMMITTED | No | Yes | Yes | Yes |
|
||||
| REPEATABLE READ | No | No | No | Yes |
|
||||
| SERIALIZABLE | No | No | No | No |
|
||||
|
||||
```sql
|
||||
-- Check current isolation level
|
||||
SHOW transaction_isolation; -- default: 'read committed'
|
||||
|
||||
-- Set for a single transaction
|
||||
BEGIN ISOLATION LEVEL REPEATABLE READ;
|
||||
-- ... work ...
|
||||
COMMIT;
|
||||
|
||||
-- Set for the session
|
||||
SET SESSION CHARACTERISTICS AS TRANSACTION ISOLATION LEVEL SERIALIZABLE;
|
||||
|
||||
-- Set per role or database (persistent)
|
||||
ALTER ROLE myapp SET default_transaction_isolation = 'repeatable read';
|
||||
ALTER DATABASE mydb SET default_transaction_isolation = 'serializable';
|
||||
```
|
||||
|
||||
## READ COMMITTED (Default)
|
||||
|
||||
Each statement within the transaction sees a **fresh snapshot** — it sees all data committed before that statement began, including commits by other transactions that happened after the transaction started.
|
||||
|
||||
### Behavior
|
||||
|
||||
```sql
|
||||
-- Transaction A -- Transaction B
|
||||
BEGIN;
|
||||
SELECT balance FROM accounts
|
||||
WHERE id = 1; -- sees: 1000
|
||||
BEGIN;
|
||||
UPDATE accounts SET balance = 500
|
||||
WHERE id = 1;
|
||||
COMMIT;
|
||||
SELECT balance FROM accounts
|
||||
WHERE id = 1; -- sees: 500 (!)
|
||||
-- The second SELECT sees B's commit
|
||||
COMMIT;
|
||||
```
|
||||
|
||||
### The Lost Update Problem
|
||||
|
||||
The most common surprise with READ COMMITTED — concurrent updates can silently overwrite each other:
|
||||
|
||||
```sql
|
||||
-- Transaction A -- Transaction B
|
||||
BEGIN; BEGIN;
|
||||
SELECT balance FROM accounts
|
||||
WHERE id = 1; -- 1000
|
||||
SELECT balance FROM accounts
|
||||
WHERE id = 1; -- 1000
|
||||
|
||||
UPDATE accounts
|
||||
SET balance = 1000 - 200 -- = 800
|
||||
WHERE id = 1;
|
||||
-- B blocks here until A commits
|
||||
COMMIT;
|
||||
UPDATE accounts
|
||||
SET balance = 1000 - 300 -- = 700 (!)
|
||||
WHERE id = 1;
|
||||
COMMIT;
|
||||
-- Final balance: 700
|
||||
-- Expected (if serial): 500
|
||||
-- A's withdrawal was lost!
|
||||
```
|
||||
|
||||
**Fixes for lost updates in READ COMMITTED:**
|
||||
|
||||
1. **Use atomic SQL** — avoid read-then-write patterns:
|
||||
```sql
|
||||
UPDATE accounts SET balance = balance - 200 WHERE id = 1;
|
||||
```
|
||||
This is safe because the UPDATE re-evaluates `balance` from the latest committed row.
|
||||
|
||||
2. **Use SELECT FOR UPDATE** — lock the row before reading:
|
||||
```sql
|
||||
BEGIN;
|
||||
SELECT balance FROM accounts WHERE id = 1 FOR UPDATE;
|
||||
-- Other transactions block here until we commit
|
||||
UPDATE accounts SET balance = balance - 200 WHERE id = 1;
|
||||
COMMIT;
|
||||
```
|
||||
|
||||
3. **Use REPEATABLE READ or SERIALIZABLE** — the second transaction gets a serialization error instead of silently overwriting.
|
||||
|
||||
### UPDATE Re-evaluation
|
||||
|
||||
When an UPDATE in READ COMMITTED encounters a row locked by another transaction, it **waits** for the lock to release, then **re-evaluates** the WHERE clause against the newly committed row. If the row still matches, it proceeds with the update. If not, it skips the row.
|
||||
|
||||
This means UPDATE with complex WHERE clauses can behave unexpectedly:
|
||||
|
||||
```sql
|
||||
-- Transaction A -- Transaction B
|
||||
BEGIN; BEGIN;
|
||||
UPDATE orders
|
||||
SET status = 'processing'
|
||||
WHERE status = 'pending'
|
||||
AND id = 42;
|
||||
UPDATE orders
|
||||
SET status = 'processing'
|
||||
WHERE status = 'pending'
|
||||
AND id = 42;
|
||||
-- blocks, waiting for A...
|
||||
COMMIT;
|
||||
-- A committed: row 42 is now 'processing'
|
||||
-- Re-evaluates: WHERE status = 'pending' → false
|
||||
-- UPDATE 0 (silently skips the row)
|
||||
COMMIT;
|
||||
```
|
||||
|
||||
## REPEATABLE READ
|
||||
|
||||
The entire transaction sees a **single snapshot** taken at the start of the first non-transaction-control statement. Subsequent statements see the same data regardless of concurrent commits.
|
||||
|
||||
### Behavior
|
||||
|
||||
```sql
|
||||
-- Transaction A -- Transaction B
|
||||
BEGIN ISOLATION LEVEL
|
||||
REPEATABLE READ;
|
||||
SELECT balance FROM accounts
|
||||
WHERE id = 1; -- sees: 1000
|
||||
BEGIN;
|
||||
UPDATE accounts SET balance = 500
|
||||
WHERE id = 1;
|
||||
COMMIT;
|
||||
SELECT balance FROM accounts
|
||||
WHERE id = 1; -- still sees: 1000
|
||||
COMMIT;
|
||||
```
|
||||
|
||||
### Serialization Errors
|
||||
|
||||
If a REPEATABLE READ transaction tries to update a row that was modified by a concurrent committed transaction, it fails:
|
||||
|
||||
```sql
|
||||
-- Transaction A -- Transaction B
|
||||
BEGIN ISOLATION LEVEL
|
||||
REPEATABLE READ;
|
||||
SELECT * FROM accounts
|
||||
WHERE id = 1;
|
||||
BEGIN;
|
||||
UPDATE accounts SET balance = 500
|
||||
WHERE id = 1;
|
||||
COMMIT;
|
||||
UPDATE accounts
|
||||
SET balance = balance - 200
|
||||
WHERE id = 1;
|
||||
-- ERROR: could not serialize access due to concurrent update
|
||||
```
|
||||
|
||||
This is **safer** than READ COMMITTED's silent re-evaluation — you know the operation failed and can retry.
|
||||
|
||||
### What REPEATABLE READ Does NOT Prevent
|
||||
|
||||
REPEATABLE READ prevents non-repeatable reads and phantoms, but does not prevent all serialization anomalies. Write skew is still possible:
|
||||
|
||||
```sql
|
||||
-- Constraint: at least one doctor must be on-call
|
||||
-- Doctor A and Doctor B are both on-call
|
||||
|
||||
-- Transaction A (REPEATABLE READ) -- Transaction B (REPEATABLE READ)
|
||||
BEGIN; BEGIN;
|
||||
SELECT count(*) FROM oncall
|
||||
WHERE on_duty = true; -- 2
|
||||
SELECT count(*) FROM oncall
|
||||
WHERE on_duty = true; -- 2
|
||||
UPDATE oncall
|
||||
SET on_duty = false
|
||||
WHERE doctor = 'A'; -- OK, 1 left
|
||||
UPDATE oncall
|
||||
SET on_duty = false
|
||||
WHERE doctor = 'B'; -- OK, 1 left
|
||||
COMMIT; COMMIT;
|
||||
-- Both committed! No one is on-call.
|
||||
```
|
||||
|
||||
Use SERIALIZABLE to prevent write skew.
|
||||
|
||||
## SERIALIZABLE
|
||||
|
||||
The strongest level. Transactions behave **as if they executed one at a time** (serially). PostgreSQL uses Serializable Snapshot Isolation (SSI) — an optimistic approach that detects conflicts during statement execution or at commit rather than acquiring heavy locks upfront.
|
||||
|
||||
### Behavior
|
||||
|
||||
All the anomalies prevented by REPEATABLE READ are prevented, plus serialization anomalies like write skew:
|
||||
|
||||
```sql
|
||||
-- Same on-call scenario with SERIALIZABLE:
|
||||
-- One of the two transactions will get:
|
||||
-- ERROR: could not serialize access due to read/write dependencies among transactions
|
||||
```
|
||||
|
||||
### Performance Characteristics
|
||||
|
||||
- Reads are not blocked — SSI tracks read/write dependencies optimistically
|
||||
- Slightly higher overhead per transaction (dependency tracking)
|
||||
- More serialization failures under contention (requires retry logic)
|
||||
- Rarely impacts throughput in practice for OLTP workloads
|
||||
|
||||
### When to Use SERIALIZABLE
|
||||
|
||||
- Financial calculations where correctness is paramount
|
||||
- Constraint enforcement that spans multiple rows or tables
|
||||
- Any case where "check then act" patterns must be atomic
|
||||
- When the alternative is complex application-level locking
|
||||
|
||||
## Common Pitfalls and Surprises
|
||||
|
||||
### Pitfall: Assuming READ COMMITTED Is "Safe Enough"
|
||||
|
||||
READ COMMITTED is safe for individual statements but not for multi-statement read-then-write patterns. Any time you SELECT a value and then use it in a subsequent UPDATE/INSERT, you have a potential race condition.
|
||||
|
||||
### Pitfall: Not Handling Serialization Failures
|
||||
|
||||
REPEATABLE READ and SERIALIZABLE can throw:
|
||||
```
|
||||
ERROR: could not serialize access due to concurrent update
|
||||
SQLSTATE: 40001
|
||||
```
|
||||
|
||||
**This is expected behavior, not a bug.** Applications must catch and retry.
|
||||
|
||||
### Pitfall: Long Transactions in REPEATABLE READ/SERIALIZABLE
|
||||
|
||||
The snapshot is held for the entire transaction. Long transactions:
|
||||
- Prevent VACUUM from cleaning dead rows visible to the snapshot
|
||||
- Increase the chance of serialization failures (more time for conflicts)
|
||||
- Hold SSI dependency information longer (memory overhead)
|
||||
|
||||
Keep transactions short regardless of isolation level.
|
||||
|
||||
### Pitfall: Mixing Isolation Levels
|
||||
|
||||
If Transaction A is SERIALIZABLE but Transaction B is READ COMMITTED, you only get SERIALIZABLE guarantees for A's view of the data. B never sees uncommitted intermediate states, but it can observe newer committed states between statements and produce non-serializable outcomes. For full serializable behavior, **all participating transactions** must use SERIALIZABLE.
|
||||
|
||||
### Pitfall: DDL and Isolation
|
||||
|
||||
DDL statements (ALTER TABLE, CREATE INDEX) always acquire strong locks regardless of isolation level. They can block and be blocked by other transactions normally.
|
||||
|
||||
## Choosing the Right Level
|
||||
|
||||
| Scenario | Recommended level | Why |
|
||||
|----------|------------------|-----|
|
||||
| Simple CRUD, web apps | READ COMMITTED | Default is fine; use atomic SQL for updates |
|
||||
| Balance transfers, inventory | READ COMMITTED + `SELECT FOR UPDATE` | Explicit row locking prevents lost updates |
|
||||
| Reports reading consistent data | REPEATABLE READ | Snapshot consistency across multiple queries |
|
||||
| Multi-row constraints, write skew | SERIALIZABLE | Only level that prevents all anomalies |
|
||||
| High-contention counters | READ COMMITTED + atomic UPDATE | `UPDATE x SET n = n + 1` is inherently safe |
|
||||
|
||||
**Default to READ COMMITTED** and escalate only when you identify a specific anomaly your application cannot tolerate.
|
||||
|
||||
## Retry Patterns for Serialization Failures
|
||||
|
||||
### Application-Level Retry (Recommended)
|
||||
|
||||
```python
|
||||
# Python example (psycopg)
|
||||
import random
|
||||
import time
|
||||
|
||||
import psycopg
|
||||
from psycopg.errors import DeadlockDetected, SerializationFailure
|
||||
|
||||
MAX_RETRIES = 5
|
||||
BASE_DELAY_SECONDS = 0.01
|
||||
MAX_DELAY_SECONDS = 0.5
|
||||
|
||||
for attempt in range(MAX_RETRIES):
|
||||
try:
|
||||
# This must be a top-level transaction, not a savepoint inside one.
|
||||
with conn.transaction():
|
||||
conn.execute("SET TRANSACTION ISOLATION LEVEL SERIALIZABLE")
|
||||
conn.execute("UPDATE accounts SET balance = balance - 100 WHERE id = 1")
|
||||
conn.execute("UPDATE accounts SET balance = balance + 100 WHERE id = 2")
|
||||
break # success
|
||||
except (SerializationFailure, DeadlockDetected):
|
||||
if attempt == MAX_RETRIES - 1:
|
||||
raise # give up after max retries
|
||||
|
||||
# Bounded exponential backoff with jitter reduces repeated collisions.
|
||||
delay = min(MAX_DELAY_SECONDS, BASE_DELAY_SECONDS * (2 ** attempt))
|
||||
time.sleep(random.uniform(delay / 2, delay))
|
||||
```
|
||||
|
||||
### Key Retry Rules
|
||||
|
||||
1. **Retry the entire transaction** — not just the failed statement
|
||||
2. **Re-run all reads and decision-making** — transaction inputs may no longer be valid after a conflict
|
||||
3. **Use a bounded retry count** — avoid infinite retry loops
|
||||
4. **Use bounded exponential backoff with jitter** — concurrent retriers should not repeatedly collide; tune the delays for the workload
|
||||
5. **Log retries and exhaustion** — frequent retries indicate contention or transaction-design problems
|
||||
6. **SQLSTATE 40001** is the error code to catch (`serialization_failure`)
|
||||
7. **SQLSTATE 40P01** is deadlock (`deadlock_detected`) — it can be retried, but also fix inconsistent lock ordering where possible
|
||||
8. **Do not publish external side effects before commit** — messages, emails, and API calls can otherwise be duplicated by a retry
|
||||
|
||||
PostgreSQL requires retrying the complete transaction and warns that multiple attempts may be needed. See [Serialization Failure Handling](https://www.postgresql.org/docs/current/mvcc-serialization-failure-handling.html).
|
||||
|
||||
### PL/pgSQL Procedure for the Transaction Body
|
||||
|
||||
A PL/pgSQL function cannot retry a complete transaction. An `EXCEPTION` block rolls back only its subtransaction; retrying the block remains inside the same top-level transaction and uses the same transaction snapshot at REPEATABLE READ or SERIALIZABLE.
|
||||
|
||||
A procedure can encapsulate the database work, but the application should still own the complete transaction and retry loop:
|
||||
|
||||
```sql
|
||||
CREATE OR REPLACE PROCEDURE transfer_funds(
|
||||
p_from_id int,
|
||||
p_to_id int,
|
||||
p_amount numeric
|
||||
)
|
||||
LANGUAGE plpgsql
|
||||
AS $$
|
||||
BEGIN
|
||||
UPDATE accounts
|
||||
SET balance = balance - p_amount
|
||||
WHERE id = p_from_id;
|
||||
|
||||
UPDATE accounts
|
||||
SET balance = balance + p_amount
|
||||
WHERE id = p_to_id;
|
||||
END;
|
||||
$$;
|
||||
```
|
||||
|
||||
Invoke `CALL transfer_funds(...)` inside the application-managed transaction and retry that entire transaction from the application. A serialization failure can occur at commit, so a procedure cannot reliably catch every failure and restart itself.
|
||||
Reference in New Issue
Block a user