magnus919_agent-skills/telemetry/references/01-prometheus-operations.md

Prometheus Operations

Last Updated: 2026-08-03

Operational patterns for the Prometheus half of the telemetry stack: scrape configuration, recording and alerting rules, relabeling, retention, and high availability. Sources: Prometheus documentation (prometheus.io/docs, accessed 2026-08-03) and the rule format reference in the Prometheus source (reviewed 2026-08-03).

Scrape configuration

One scrape job is one scrape group: a job_name, a scrape_interval, a scrape_timeout strictly below the interval, a metrics_path, and a target source. The default metrics_path is /metrics; TLS and auth go in scheme, tls_config, and basic_authorization/authorization.

scrape_configs:
  - job_name: node
    scrape_interval: 30s
    scrape_timeout: 10s
    metrics_path: /metrics
    static_configs:
      - targets: ["node1:9100", "node2:9100"]

Verify the live state, not the file: /api/v1/status/config returns the effective config and /api/v1/targets?state=active returns per-target scrape state. A target that never appears in the target list is usually a relabeling or discovery problem, not a Prometheus outage.

Recording and alerting rules

Rules files are groups, each with a name, an optional interval, and a rules list. Every rule has exactly one of record or alert plus an expr; alerting rules may add for, keep_firing_for, labels, and annotations. Group names must be unique within a file; label and annotation names must be valid label names; label values must be strings.

groups:
  - name: api-slo
    interval: 1m
    rules:
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total{job="api"}[5m]))
      - alert: ApiHighErrorRate
        expr: job:http_errors:rate5m / job:http_requests:rate5m > 0.05
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "API error rate above 5%"

Rules that touch the same series belong in one group because groups evaluate sequentially; cross-group timing is undefined. Validate with promtool check rules (full PromQL parsing) and telemetry-check --rules (structural sanity, no external tools) before every reload, then promtool reload via POST /-/reload and confirm with /api/v1/rules?type=alert.

Recording rules are caching, not aggregation religion: name them with the conventional level:metric:operation style, keep them idempotent, and prefer sum/rate over count-style ratios that need division in every query. Alerting rules should be small and reviewable; a rule whose expression needs a comment to explain is a candidate for a recording rule instead.

Relabeling

relabel_configs run at target discovery time (before scraping) and metric_relabel_configs run after scraping (per metric). Use them to:

• enforce label naming and drop forbidden labels (__meta_*, job);
• add scrape metadata (__address__, __scheme__, __metrics_path__);
• drop high-cardinality labels from metric_relabel_configs before the TSDB.

Relabeling is the classic silent-breakage point: a dropped or renamed label changes series identity without an error. Verify with the target's effective labels in /api/v1/targets and a spot-check of /metrics on the endpoint. keep/drop/replace/labelmap are the operators you will actually use; regex capture groups feed replacement with $1-style references.

Retention

Local retention is --storage.tsdb.retention.time (age) and --storage.tsdb.retention.size (bytes); blocks are ~2h and compaction merges them. Set retention as a deliberate capacity decision (see references/04-stack-integration-and-retention.md), never leave the defaults for a long-running server. Watch prometheus_tsdb_head_series (cardinality), prometheus_tsdb_compactions_total and prometheus_tsdb_blocks_loaded for compaction health, and prometheus_tsdb_storage_blocks_bytes for the footprint. Retention is enforced lazily by compaction — a server under compaction pressure can exceed its retention window temporarily.

High availability (HA)

HA for Prometheus means two identical instances scraping the same targets, with the same rules, and consistent external labels, so a query layer can deduplicate or shard. The instances do not share state: each has its own TSDB, its own for-counter state, and its own alert evaluation. Practical rules:

• run both replicas with --web.external-url stable and identical rule files;
• give replicas distinct replica external labels so dedup can pick one;
• never add alert-specific noise that makes the two replicas fire different alert instances — dedup is by label sets;
• consider Thanos or Mimir for query federation and long-term retention, but only after the two-replica story is correct.

Rule evaluation correctness across replicas matters more than uptime: a failover that changes when alerts fire is worse than a brief scrape gap.