mirror of https://github.com/magnus919/agent-skills.git synced 2026-09-15 13:36:35 +03:00

Files

T

Magnus HedemarkandGitHub 6fd559e903 feat: add site-reliability-engineering skill (#10 )

Signed-off-by: Magnus Hedemark <magnus919@pm.me>

2026-07-12 14:34:26 -04:00

32 KiB

Raw Permalink Blame History

Monitoring & Alerting Reference

Overview

Monitoring and alerting form the operational backbone of any production system. This reference covers the principles, patterns, and practical implementations that SREs use to observe system health, detect anomalies, and respond to incidents. It is designed as a standalone guide for site reliability engineers building or refining their observability stack.

1. The Four Golden Signals

Coined by Google's SRE team, the Four Golden Signals are the highest-level metrics every distributed system should track. They provide a minimal, universally applicable set of health indicators that cut across application domains.

1.1 Latency

Latency measures the time required to service a request. It must be tracked separately for successful and failed requests — a slow failure (e.g., a 500 that takes 30 seconds to return) can mask a real problem with success-path latency.

Key considerations:

Use percentiles, not averages. p50, p95, p99 tell a far richer story than mean latency.
Distinguish between latency (time to first byte / time to complete) and response time (round-trip from client perspective).
Instrument at every layer: load balancer, application, database, downstream dependency.

Example PromQL latency queries:

# p99 latency of HTTP requests (successful only), last 5m
histogram_quantile(0.99,
  rate(http_request_duration_seconds_bucket{status=~"2..|3.."}[5m])
)

# p50 latency
histogram_quantile(0.50,
  rate(http_request_duration_seconds_bucket[5m])
)

# Slow request ratio: requests exceeding 1 second
rate(http_request_duration_seconds_count{status=~"2..|3.."}[5m])
/
rate(http_request_duration_seconds_bucket{le="1.0", status=~"2..|3.."}[5m])

1.2 Traffic

Traffic describes the demand placed on the system. Its units depend on the service type:

Service Type	Traffic Metric	Example
HTTP API	Requests per second (RPS)	`rate(http_requests_total[5m])`
Database	Queries per second (QPS)	`rate(mysql_queries_total[5m])`
Message queue	Messages consumed per minute	`rate(kafka_messages_total[5m])`
CDN / file serving	Bytes per second	`rate(nginx_bytes_sent_total[5m])`
Web application	Active sessions / users	`active_users`

Traffic pattern alerts should distinguish organic growth (capacity planning) from sudden spikes (incidents):

# Sudden traffic spike: 2x over 5-minute baseline
(
  rate(http_requests_total[5m])
  /
  rate(http_requests_total[30m] offset 10m)
) > 2

1.3 Errors

Errors are requests that fail, explicitly (5xx) or implicitly (returning wrong data, meeting SLOs but returning stale results). Track both:

Explicit errors: HTTP 5xx, gRPC UNAVAILABLE, TCP connection resets.
Implicit errors: Success codes with semantically wrong responses (e.g., 200 OK with empty body), responses that exceed latency SLOs, degraded functionality.

Error ratio is often more useful than raw error count, especially during traffic changes:

# Error ratio over 5m window
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))

Alert on error budget burn rate for SLO-based alerting:

# Error budget burn rate over 1h: how fast we're consuming the SLO budget
(
  1 - (
    sum(rate(http_requests_total{status=~"2.."}[1h]))
    / sum(rate(http_requests_total[1h]))
  )
) / (1 - 0.99)  // 99% SLO target

1.4 Saturation

Saturation measures how "full" the service is. The most limited resource (CPU, memory, disk I/O, network bandwidth, connection pool, database connections, file descriptors) dominates.

Key principles:

Saturation often precedes performance degradation — it is a leading indicator.
Use utilization (percentage) but also track queue depth or request-dropping behavior.
The USE Method (Utilization, Saturation, Errors) maps directly here.

# CPU saturation: load average relative to core count
node_load1 / count(node_cpu_seconds_total{mode="idle"}) > 0.8

# Memory saturation: available to total ratio
node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes < 0.1

# Connection pool saturation: active connections vs max
pg_stat_activity_count / pg_settings_max_connections > 0.8

# Disk I/O saturation: queue depth
rate(node_disk_io_time_weighted_seconds_total[5m]) > 1000

2. Black-Box vs White-Box Monitoring

2.1 Black-Box Monitoring

Black-box monitoring observes the system from the outside as a user would. It tests behaviour, not implementation.

Characteristics:

Synthetic probes, health checks, external uptime monitors.
Validates what the system does, not how.
Catches issues invisible to internal metrics: DNS resolution failures, TLS certificate expiry, CDN misconfiguration, routing problems.
Example: "HTTP GET /health returns 200 within 5 seconds."

Tools: Prometheus blackbox exporter, synthetic monitoring (Checkly, Grafana Synthetics), external uptime services.

Alert rules:

# Probe failure over 5 minutes
avg_over_time(probe_success[5m]) < 0.5

# Latency from probe perspective
probe_duration_seconds > 3

2.2 White-Box Monitoring

White-box monitoring observes the system from the inside, exposing internal state, metrics, and performance characteristics.

Characteristics:

Application metrics (request durations, error rates, goroutine counts).
Database query performance, connection pool depth, cache hit ratios.
Runtime behaviour (GC pauses, thread contention, memory allocation).
Enables root-cause analysis and capacity planning.
Example: "JVM heap usage is at 85% and growing at 100 MB/min."

Tools: Application metrics instrumentation (Prometheus client libraries, OpenTelemetry), runtime debugging endpoints, structured logging.

The two approaches are complementary. A mature observability stack uses both: black-box for customer-facing SLIs, white-box for debugging and proactive capacity management.

3. Symptom-Based vs Cause-Based Alerting

3.1 Symptom-Based Alerting (Preferred)

Alert on symptoms — observable problems that affect users right now or will imminently.

Symptom	Example Alert
High latency	p99 latency > 2 seconds for 5 minutes
Elevated error rate	Error ratio > 5% for 10 minutes
Error budget exhaustion	2% of monthly error budget consumed in 1 hour
Degraded user experience	Page load time > 3 seconds for logged-in users

Rule of thumb: If the alert fires and no user-facing impact can be articulated, the alert is cause-based and should be re-evaluated or demoted to a diagnostic signal.

3.2 Cause-Based Alerting

Alert on causes — internal conditions that might lead to symptoms. These should generally be:

Log-only or warning-level notifications (dashboards, not pages).
Tied to a specific known failure mode with clear remediation.
Rate-limited to avoid noise storms.

Cause	Better Approach
CPU > 90%	Dashboard panel + capacity planning ticket
Disk > 80% full	Automated cleanup or low-severity notification
Specific error log line appears	Log-based metric + dashboard, not page
Single pod OOMKilled	Auto-restart (Kubernetes handles this)

4. Alert Design Principles

Every alert should pass three tests: actionable, urgent, and novel.

4.1 Actionable

The recipient must be able to do something about it.

Good: "API error ratio exceeds 5% — investigate upstream database connectivity."
Bad: "CPU is at 88% — no further context, no clear action."
Test: If the alert fires and the on-call engineer asks "so what?" it is not actionable.

4.2 Urgent

The alert must require immediate attention.

Good: "Error budget depletion rate will exhaust budget in 2 hours at current burn rate."
Bad: "Certificate expires in 30 days" (should be a scheduled task, not a page).
Test: Would it be acceptable to ignore this alert for 30 minutes? If yes, it is not urgent.

4.3 Novel

The alert should represent new information not already visible in dashboards or automated remediation.

Good: "This is the first occurrence of a database connection pool exhaustion event today."
Bad: "Pod restarted" — if Kubernetes auto-recovered and there's no systemic pattern.
Test: Is this the same incident-meets-recovery cycle firing repeatedly? Suppress or tune.

4.4 Additional Design Heuristics

One alert = one problem. Do not bundle multiple conditions into a single rule.
Include runbook links in alert annotations.
Use consistent naming conventions: Service/Component/Severity/Description.
Set appropriate for durations (e.g., 5 minutes) to avoid flapping on transient issues.
Define expected response times in the alert metadata (e.g., "Respond within 15 minutes").
Every page-level alert needs a documented escalation path.

5. Time-Series Monitoring Architecture (Prometheus-Inspired)

5.1 Core Architecture Pattern

[Instrumented Services] --scrape--> [Prometheus Server] --read--> [Alertmanager]
                                     |                        +--> [Dashboards (Grafana)]
                                     |                        +--> [Long-term storage (Thanos/Cortex)]
                                     v
                                  [Recording Rules] --> [Aggregated metrics]

5.2 Metric Types

Type	Description	Example
Counter	Monotonically increasing value (resets on restart)	`http_requests_total`, `errors_total`
Gauge	Point-in-time value that can go up and down	`memory_usage_bytes`, `queue_depth`
Histogram	Configurable buckets for observing distributions	`request_duration_seconds_bucket{le="0.5"}`
Summary	Pre-computed quantiles (cannot aggregate)	`request_duration_seconds{quantile="0.99"}`

5.3 Recording Rules

Pre-compute expensive or frequently queried expressions to reduce query load:

# Group: "service:request_error_ratio:5m"
record: namespace:http_errors:ratio_rate5m
expr: |
  sum(rate(http_requests_total{status=~"5.."}[5m])) by (namespace)
  /
  sum(rate(http_requests_total[5m])) by (namespace)

# Group: "instance:cpu_utilization:rate5m"
record: instance:node_cpu_utilization:rate5m
expr: |
  1 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) by (instance)

5.4 Alerting Rules Architecture

Separate evaluation tiers:

# Tier 1 — Direct metric threshold (immediate symptom)
alert: APIHighErrorRate
expr: |
  (sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
   /
   sum(rate(http_requests_total[5m])) by (service)) > 0.05
for: 5m
labels:
  severity: critical
annotations:
  summary: "{{ $labels.service }} error rate above 5%"
  runbook: "https://runbooks.example.com/high-error-rate"

# Tier 2 — Error-budget-based (burn rate)
alert: ErrorBudgetBurnedFast
expr: |
  (
    (1 - (sum(rate(http_requests_total{status=~"2.."}[1h])) by (service)
         / sum(rate(http_requests_total[1h])) by (service)))
    / (1 - 0.99)
  ) > 0.05
for: 2m
labels:
  severity: critical
annotations:
  summary: "Error budget burning at 5x safe rate"

# Tier 3 — Absent metric (service down)
alert: ServiceDown
expr: |
  absent(up{job="api-server"}) == 1
for: 30s
labels:
  severity: critical

6. Alert Routing and Severity

6.1 Severity Levels

Level	Label	Response SLA	Channel	Purpose
Critical	P0	< 15 minutes	Page (phone)	User-facing outage, data loss, breach
High	P1	< 30 minutes	Page (push)	Severe degradation, error budget risk
Medium	P2	< 4 hours	Chat (thread)	Degradation without immediate impact
Low	P3	Next business day	Ticket	Capacity warnings, cleanup reminders
Informational	P4	None	Log only	Dashboard annotations, debugs

6.2 Routing Matrix

Route based on service, severity, time of day, and team ownership:

[Alertmanager Configuration]
route:
  receiver: 'default'
  routes:
    - match:
        severity: critical
      receiver: 'page-oncall'
      repeat_interval: 1h
    - match:
        service: billing
      receiver: 'billing-team'
    - match_re:
        component: "^database"
      receiver: 'data-platform'
    - match:
        severity: warning
      receiver: 'chat-channel'
      group_wait: 30s
      group_interval: 5m

6.3 Grouping and Deduplication

Prevent alert storms by grouping related alerts:

group_by: ['alertname', 'service', 'cluster']
group_wait: 30s       # collect alerts for 30s before sending
group_interval: 5m    # wait 5m between repeat notifications
repeat_interval: 4h   # don't re-notify for 4h unless severity changes

7. Dashboard Design Principles

7.1 The Golden Rules

One dashboard per story. A dashboard should answer a single question or serve a single persona (e.g., "Is the API healthy?", "Database capacity planning").
Three-tier layout:
- Top row: Summary / health — the single number or status that tells you if everything is OK.
- Middle rows: Key time-series — latency, errors, traffic, saturation per component.
- Bottom rows: Drill-down details — per-shard, per-host, per-endpoint.
Consistent time ranges across panels on the same dashboard.
Use left Y-axis for the primary metric; avoid dual-axes unless essential.
Label clearly with units (ms, req/s, %). Include SLO targets as threshold lines.

7.2 Dashboard Types

Type	Audience	Refresh	Content Summary
Executive	Management	1 day	SLIs, error budget, cost, capacity trends
Service Health	On-call	30s	Golden signals per service
Capacity	Infrastructure	1 hour	Resource utilization, growth projections
Debug / Triage	Engineering	Real-time	Detailed traces, log rates, anomaly plots
SLO Burn Rate	SRE	1 min	Burn rate vectors, remaining budget

7.3 Anti-Patterns

Spaghetti dashboards: More than 20 panels on one dashboard. Split by concern.
Red/green everywhere: Red color should be reserved for failing the SLO.
No time context: Single-number stat panels without a sparkline or change indicator.
Orphan dashboards: No owner, no last-modified date, no description.

8. On-Call Alert Fatigue Prevention

Alert fatigue is the leading cause of pager burnout and missed real incidents.

8.1 Causes

Cause	Remedy
Too many low-severity alerts	Demote to P3/P4; page only on critical.
Flaky alerts (transient)	Increase `for` duration; use multi-window logic.
Duplicate alerts	Use alertmanager grouping; deduplicate rules.
Noisy auto-remediation cycles	Silence alerts for N minutes after auto-fix.
Undefined ownership	Every alert must route to exactly one team.
Missing runbooks	Add runbook links; if unclear, the alert is bad.

8.2 Fatigue Prevention Techniques

Weekly alert reviews: Every critical alert is reviewed for signal-to-noise ratio. If it fired "unnecessarily" more than once in the past week, tune or delete it.
Gradual escalation: Start with a 5-minute for duration, alert warning first, then escalate to critical if condition persists.
Suppression during maintenance: Tag maintenance windows; suppress alerts during known change windows.
Machine learning / dynamic thresholds: For seasonal systems, use anomaly detection instead of static thresholds.
Error budget alerts over raw threshold alerts: Raw threshold alerts (e.g., "p99 > 500ms") are rigid. Error-budget-based alerts account for the SLO's remaining headroom.
Self-healing integration: If a runbook step can be automated (restart, scale-up, traffic shed), add it. Only page if the automated action fails.
Psychologically safe culture: No blame for acknowledging an alert and determining it was not actionable — that becomes tuning feedback.

8.3 Alert Fatigue Metrics to Track

Alert-to-incident ratio: Number of pages that resulted in a documented incident vs. total pages. Target: > 50%.
Mean time to acknowledge (MTTA): Should be under the defined SLA. Spikes suggest routing issues or fatigue.
Mean time to resolve (MTTR): Tracks operational effectiveness.
False positive rate per rule: Identify the worst offenders in the weekly review.

9. The USE Method and RED Method for Metrics

9.1 USE Method (Utilization, Saturation, Errors)

Used for infrastructure resources (CPU, memory, disk, network). Ask: For every resource, what is:

Utilization: The average percent of time the resource is busy.
Saturation: The degree to which the resource has extra work queued (can't service more).
Errors: The count of error events.

USE Method Counterpart per Resource:

Resource	Utilization	Saturation	Errors
CPU	`avg by(instance) (rate(cpu_seconds_total{mode!="idle"}[5m]))`	`load1 / cpu_count`	`process_errors_total` (per process)
Memory	`used / total`	`swap_usage` / OOM events	`vmstat -s` page failures
Disk I/O	`avg by(device) (rate(disk_io_time_seconds_total[5m]))`	`rate(disk_io_time_weighted_seconds_total[5m])`	`disk_read_errors_total` / `disk_write_errors_total`
Network	`rate(bytes_total[5m]) / bandwidth`	`drop_count / segment_retransmits`	`interface_errors_total`
File handles	`filefd_allocated / filefd_max`	—	`failed_file_opens_total`

Example alert using USE:

# Saturation: CPU run queue > 4x core count
(
  node_load15
  /
  count(node_cpu_seconds_total{mode="idle"})
) > 4

9.2 RED Method (Rate, Errors, Duration)

Used for services and applications. Ask for every service:

Rate: Requests per second (throughput).
Errors: Number of requests that fail.
Duration: Time taken to process a request (latency distribution).

The RED Method is essentially the Four Golden Signals minus saturation — it focuses purely on user-facing service health rather than resource health.

Instrumentation pattern (per endpoint or per operation):

# Rate
rate(http_requests_total{job="api"}[5m])

# Errors (ratio)
sum(rate(http_requests_total{job="api", status=~"5.."}[5m]))
/
sum(rate(http_requests_total{job="api"}[5m]))

# Duration (p99)
histogram_quantile(0.99,
  rate(http_request_duration_seconds_bucket{job="api"}[5m])
)

9.3 When to Use Which

Layer	Best Fit	Rationale
Infrastructure (VM, container, disk, NIC)	USE	Resources have bounded capacity; saturation is meaningful.
Application (API, database, queue)	RED (or Golden Signals)	Behaviours (requests, errors, latency) are the concern.
Middleware / Proxy	Both	Resource concerns (connections, memory) AND service behaviours matter.
Network / Load balancer	RED + USE	Throughput and errors (RED) plus interface saturation (USE).

10. SLI-Based Alerting vs Threshold-Based Alerting

10.1 Threshold-Based Alerting (Classic)

How it works: Static thresholds on raw metrics (e.g., p95 latency > 500ms, CPU > 90%).

Pros:

Simple to understand and implement.
Works well for resource saturation alerts (disk full, OOM).
No SLO definition required.

Cons:

Requires manual tuning for each service.
Does not account for time remaining — a p99 of 600ms at 8am on a Monday might be fine, but at 3am it indicates something wrong.
Brittle: thresholds that are too tight generate noise, those too loose miss incidents.

10.2 SLI-Based Alerting

How it works: Define Service Level Indicators (SLIs) and Service Level Objectives (SLOs). Alert based on error budget burn rate — how fast the remaining allowable error budget is being consumed.

SLI definition pattern:

sli_name: "api_request_latency_sli"
description: "Fraction of requests served in under 500ms"
good_event: "http_request_duration_seconds <= 0.5"
valid_events: "http_requests_total"
measurement_window: "28d"

Burn rate alerting approach (Google SRE Workbook):

Burn Rate (x of target)	Duration	Severity	Meaning
>= 10x	5 minutes	Critical	Very severe, multi-9s breach imminent in minutes
>= 5x	30 minutes	Critical	Severe, will exhaust budget in hours
>= 2x	6 hours	High	Moderate burn, needs investigation
>= 1x	3 days	Low	Slow bleed, ticket for next business day

PromQL burn rate alert:

# 10x burn rate over 5 minutes
alert: HighBurnRate_10x_5m
expr: |
  (
    (
      1 - (
        sum(rate(http_requests_total{status=~"2..", job="api"}[5m]))
        / sum(rate(http_requests_total{job="api"}[5m]))
      )
    ) / (1 - 0.99)
  ) > 10
for: 2m
labels:
  severity: critical

10.3 Comparison

Aspect	Threshold-Based	SLI-Based (Burn Rate)
Tuning effort	Per-service, per-metric	Shared SLO framework
Sensitivity to traffic	Fixed threshold may be too tight at low traffic or too loose at high traffic	Proportional — error ratio, not absolute count
Leading vs lagging	Leading (catches impending resource saturation)	Lagging for resource issues, leading for SLO breaches
Business alignment	Weak	Direct: maps to customer-facing objectives
Complexity	Low	Moderate (requires SLO definitions)

Recommendation: Use SLI/burn rate alerting for user-facing symptoms. Use threshold-based alerting for resource saturation and capacity warnings. The two approaches are complementary.

11. Multi-Window Alerting Approaches

Multi-window, multi-burn-rate alerting is the gold standard for balancing sensitivity with noise reduction. It evaluates the same condition across multiple time windows before firing.

11.1 Why Multi-Window?

A single-window alert (e.g., error_ratio > 5% over 5m) can:

Fire on transient spikes that resolve before anyone acts (false positive).
Miss slow-burn degradation that accumulates over hours but never crosses a short-window threshold (false negative).

11.2 Multi-Window, Multi-Burn-Rate Design

Evaluate the metric across two windows — a short window (high sensitivity) and a long window (confirms sustained condition):

# Conditional: short-window burn rate is > 10x target
# AND long-window burn rate is > 10x target
# Both must be true — catches sustained high burn, rejects transient spikes

alert: MultiWindowErrorBudgetBurn
expr: |
  (
    # Short window: 5m burn rate > 10x
    (
      (1 - (sum(rate(http_requests_total{status=~"2.."}[5m]))
           / sum(rate(http_requests_total[5m]))))
      / (1 - 0.99)
    ) > 10
  )
  and
  (
    # Long window: 1h burn rate > 10x (confirms sustained)
    (
      (1 - (sum(rate(http_requests_total{status=~"2.."}[1h]))
           / sum(rate(http_requests_total[1h]))))
      / (1 - 0.99)
    ) > 10
  )
for: 2m
labels:
  severity: critical

11.3 The Google SRE Multi-Window Alerting Matrix

Condition	Fires When	Character
Short (5m) high burn AND Long (1h) high burn	Instant + sustained severe event	"House on fire"
Short (5m) low burn AND Long (1h) high burn	Slow bleed below burst sensitivity	"Radiator leak"
Short (5m) high burn AND Long (1h) low burn	Transient spike, already recovered	No alert (correct!)
Short (5m) low burn AND Long (1h) low burn	Everything nominal	No alert

11.4 Implementation Pattern with Recording Rules

# Recording rule: 5m burn rate ratio
record: slo:api_error_burn_rate_5m
expr: |
  (
    (1 - (sum(rate(http_requests_total{job="api", status=~"2.."}[5m]))
         / sum(rate(http_requests_total{job="api"}[5m]))))
    / (1 - 0.99)
  )

# Recording rule: 1h burn rate ratio
record: slo:api_error_burn_rate_1h
expr: |
  (
    (1 - (sum(rate(http_requests_total{job="api", status=~"2.."}[1h]))
         / sum(rate(http_requests_total{job="api"}[1h]))))
    / (1 - 0.99)
  )

# Alert: both windows agree on high burn
alert: APIErrorBudgetCriticalBurn
expr: |
  slo:api_error_burn_rate_5m > 10 and slo:api_error_burn_rate_1h > 10
for: 2m
labels:
  severity: critical
annotations:
  summary: "API error budget burning at critical rate"

# Alert: long window burn without short window (slow drain)
alert: APIErrorBudgetSlowBurn
expr: |
  slo:api_error_burn_rate_1h > 2 and slo:api_error_burn_rate_5m <= 10
for: 10m
labels:
  severity: warning
annotations:
  summary: "API error budget slowly draining"

11.5 Alternative: Sliding-Window with `min_over_time`

For environments where PromQL and semantics are unavailable, simulate multi-window with min_over_time:

alert: SlidingWindowHighErrorRate
expr: |
  min_over_time(
    (
      sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
      /
      sum(rate(http_requests_total[5m])) by (service)
    )[15m:]
  ) > 0.05
for: 2m

This requires the high error rate to be sustained across three consecutive 5-minute windows (15 minutes).

12. Practical PromQL Alert Rule Templates

12.1 Service Availability

# Service down — no metrics received
alert: ServiceDown
expr: absent(up{job="api-server"}) == 1
for: 1m
labels:
  severity: critical
annotations:
  summary: "{{ $labels.job }} is not sending metrics"

12.2 High Error Rate

# Per-path error rate
alert: PathHighErrorRate
expr: |
  (
    sum(rate(http_requests_total{status=~"5.."}[5m])) by (service, path)
    /
    sum(rate(http_requests_total[5m])) by (service, path)
  ) > 0.05
for: 5m
labels:
  severity: critical
annotations:
  summary: "5xx rate > 5% on {{ $labels.path }}"

12.3 High Latency

# Per-service p99 latency
alert: HighLatency
expr: |
  histogram_quantile(0.99,
    sum(rate(http_request_duration_seconds_bucket{status=~"2.."}[5m])) by (le, service)
  ) > 2.0
for: 5m
labels:
  severity: critical
annotations:
  summary: "p99 latency > 2s on {{ $labels.service }}"

12.4 Certificate Expiry

alert: TLSCertExpiringSoon
expr: |
  probe_ssl_earliest_cert_expiry - time() < 604800  # 7 days in seconds
for: 1h
labels:
  severity: warning
annotations:
  summary: "TLS certificate for {{ $labels.instance }} expires in less than 7 days"

12.5 Disk Space

alert: DiskSpaceCritical
expr: |
  (
    1 - (node_filesystem_avail_bytes{mountpoint="/", fstype!~"tmpfs|overlay"}
         / node_filesystem_size_bytes{mountpoint="/", fstype!~"tmpfs|overlay"})
  ) > 0.95
for: 5m
labels:
  severity: critical
annotations:
  summary: "Disk usage > 95% on {{ $labels.instance }}"

12.6 Connection Pool Saturation

alert: PgConnectionPoolExhausted
expr: |
  (
    sum by (instance) (pg_stat_activity_count{datname!~"template.*|postgres"})
    /
    avg by (instance) (pg_settings_max_connections)
  ) > 0.9
for: 5m
labels:
  severity: critical
annotations:
  summary: "PostgreSQL connection pool > 90% on {{ $labels.instance }}"

12.7 OOM Kill Detection

alert: OOMKilled
expr: |
  increase(kube_pod_container_status_restarts_total{reason="OOMKilled"}[15m]) > 0
for: 0m
labels:
  severity: critical
annotations:
  summary: "Pod {{ $labels.pod }} was OOMKilled"

12.8 Traffic Anomaly

alert: TrafficDrop
expr: |
  (
    rate(http_requests_total[5m])
    /
    rate(http_requests_total[5m] offset 1w at same day of week)
  ) < 0.5
for: 15m
labels:
  severity: warning
annotations:
  summary: "Traffic dropped > 50% compared to same time last week"

13. Closing Principles

Page on symptoms, investigate causes. Symptoms affect users; causes affect operators. One is urgent, the other is not.
Every alert needs an owner and a runbook. If the on-call cannot find both within 30 seconds, the alert is not production-ready.
Test your alerts. Use alert simulation tools (e.g., amtool, Prometheus alert unit tests with promtool) to validate logic before deploying.
Review alerts weekly. Each alert that fired is either a real incident (document it) or a false positive (tune it). No alert fires without producing an improvement action.
Prefer dashboards for diagnostic data, alerts for escalations. If a metric only says "look at the dashboard" it should be a panel, not a page.
Multi-window beats single-window. Short windows are noisy; long windows are slow. Use both for reliable detection.
Error budget burn rate alerts are the gold standard for user-facing services. They automatically account for the service's SLO, remaining budget, and traffic levels.
Instrumentation is an investment. Every metric you add has a storage and maintenance cost. Start with the Four Golden Signals, expand only when you have a concrete question that existing data cannot answer.

References

Google SRE Book — Chapter 6: Monitoring Distributed Systems
Google SRE Workbook — Chapter 5: Alerting on SLOs
The USE Method (Brendan Gregg) — https://www.brendangregg.com/usemethod.html
The RED Method (Tom Wilkie) — https://grafana.com/blog/2018/08/02/the-red-method-how-to-instrument-your-services/
Prometheus Documentation — Alerting Rules: https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/
Alertmanager Documentation: https://prometheus.io/docs/alerting/latest/alertmanager/
Grafana Dashboard Design Best Practices: https://grafana.com/docs/grafana/latest/dashboards/

32 KiB Raw Permalink Blame History