magnus919_agent-skills/traefik/references/observability.md

Observability Reference

Traefik provides comprehensive observability: metrics, access logs, tracing, and health checks.

Metrics

Configure metrics collection in static configuration:

metrics:
  addInternals: false         # Include metrics for Traefik's own services

  # --- Prometheus (most common) ---
  prometheus:
    buckets:
      - 0.1
      - 0.3
      - 1.2
      - 5.0                    # Request latency buckets (seconds)
    addEntryPointsLabels: true
    addRoutersLabels: true
    addServicesLabels: true
    entryPoint: "metrics"       # Dedicated entryPoint for metrics scraping
    manualRouting: false        # Set true to create your own router
    headerLabels:               # Additional labels from request headers
      X-Custom: "custom_label"

  # --- Datadog ---
  datadog:
    address: "localhost:8125"
    pushInterval: 10s
    addEntryPointsLabels: true
    addRoutersLabels: true
    addServicesLabels: true
    prefix: "traefik"

  # --- StatsD ---
  statsD:
    address: "localhost:8125"
    pushInterval: 10s
    addEntryPointsLabels: true
    addRoutersLabels: true
    addServicesLabels: true
    prefix: "traefik"

  # --- InfluxDB v2 ---
  influxDB2:
    address: "http://localhost:8086"
    token: "my-token"
    pushInterval: 10s
    org: "my-org"
    bucket: "traefik"
    addEntryPointsLabels: true
    addRoutersLabels: true
    addServicesLabels: true
    additionalLabels: {}

  # --- OpenTelemetry ---
  otlp:
    grpc:
      endpoint: "localhost:4317"
      insecure: true
    http:
      endpoint: "localhost:4318"
    pushInterval: 10s
    addEntryPointsLabels: true
    addRoutersLabels: true
    addServicesLabels: true
    explicitBoundaries:
      - 0.0

Prometheus Scraping Setup

# Static config — dedicated entryPoint for metrics
entryPoints:
  metrics:
    address: ":9100"

metrics:
  prometheus:
    entryPoint: "metrics"
    addRoutersLabels: true
    addServicesLabels: true

# Verify Prometheus metrics
curl http://localhost:9100/metrics | grep traefik

Key Prometheus Metrics

Metric	Type	Description
traefik_http_requests_total	Counter	Total request count
traefik_http_request_duration_seconds	Histogram	Request duration
traefik_http_requests_in_flight	Gauge	Current in-flight requests
traefik_backend_server_up	Gauge	Backend server health (0/1)
traefik_config_reloads_total	Counter	Config reload count
traefik_config_last_reload_success	Gauge	Last reload success (0/1)
traefik_tls_certs_not_after	Gauge	TLS cert expiry timestamp
traefik_entrypoint_open_connections	Gauge	Open connections per entryPoint
traefik_entrypoint_request_duration_seconds	Histogram	Per-entryPoint request duration

Access Logs

Traefik can log every request (similar to Apache/Nginx access logs). Configure in static config:

accessLog:
  filePath: "/var/log/traefik/access.log"   # File path (stdout if empty)
  format: "common"                          # common or json
  bufferingSize: 0                          # Buffer N lines (0=unbuffered)
  addInternals: false                       # Log Traefik's internal requests

  filters:
    statusCodes:
      - "200-299"
      - "400-499"
      - "500-599"                           # Only log these status code ranges
    retryAttempts: true                     # Only log requests that were retried
    minDuration: "10s"                      # Only log requests over this duration

  fields:
    defaultMode: "keep"                     # keep, drop, redact
    names:
      ClientHost: "keep"
      ClientPort: "drop"                    # Per-field override — drop client port
      RequestHost: "keep"
      RequestPath: "keep"
      RequestMethod: "keep"
      RequestProtocol: "keep"
      ResponseStatus: "keep"
      Duration: "keep"
      RetryAttempts: "keep"

    headers:
      defaultMode: "drop"                   # By default, don't log headers
      names:
        Authorization: "redact"             # Redact auth header values
        User-Agent: "keep"                  # But keep user-agent
        Referer: "keep"

Access Log Fields

Field Name	Description
ClientHost	Client IP address
ClientPort	Client port
ClientUsername	Authenticated username
RequestHost	Requested host
RequestPath	Request path
RequestMethod	HTTP method
RequestProtocol	HTTP protocol version
RequestContentSize	Request body size
RequestLine	Full request line
ResponseStatus	HTTP status code
ResponseContentSize	Response body size
Duration	Request duration
OriginDuration	Duration at origin
RouterName	Router that matched
ServiceName	Service that handled request
ServiceURL	Backend URL that handled request
DownstreamStatus	Downstream connection status
StartUTC	Request start time (UTC)
StartLocal	Request start time (local)
RetryAttempts	Number of retry attempts
FrontendName	(deprecated) Legacy router name

Common Log Format

<ClientHost> - <ClientUsername> [<StartUTC>] "<RequestLine>" <ResponseStatus> <ResponseContentSize> "<RequestRefererHeader>" "<RequestUserAgentHeader>" <Duration> <RequestCount>

Real example:

192.168.1.100 - - [05/Jul/2026:10:15:30 +0000] "GET /api/users HTTP/2" 200 1234 "-" "curl/8.0" 0.045 1

JSON Log Format

{
  "ClientHost": "192.168.1.100",
  "ClientPort": 54321,
  "StartUTC": "2026-07-05T10:15:30Z",
  "RequestMethod": "GET",
  "RequestPath": "/api/users",
  "RequestProtocol": "HTTP/2.0",
  "ResponseStatus": 200,
  "ResponseContentSize": 1234,
  "Duration": 45000000,
  "RouterName": "api-router",
  "ServiceName": "api-backend",
  "ServiceURL": "http://10.0.0.5:3000",
  "RetryAttempts": 0
}

Tracing

Traefik supports OpenTelemetry tracing:

tracing:
  serviceName: "traefik"
  sampleRate: 0.1                           # Sample 10% of requests (0.0 to 1.0)
  addInternals: false
  globalAttributes:
    environment: "production"

  # OpenTelemetry Protocol (OTLP)
  otlp:
    grpc:
      endpoint: "localhost:4317"
      insecure: true
    http:
      endpoint: "localhost:4318"

  # Headers to capture in spans
  capturedRequestHeaders:
    - "User-Agent"
    - "X-Request-Id"
  capturedResponseHeaders:
    - "Content-Type"
  safeQueryParams:                           # Query params NOT redacted
    - "id"
    - "page"

Tracing verbosity:

• minimal (default): One server span, one client span per request
• detailed: Additional spans for each middleware

entryPoints:
  websecure:
    address: ":443"
    observability:
      tracing: true
      traceVerbosity: detailed              # Or: minimal

Health Check (Ping)

ping:
  entryPoint: "web"                         # EntryPoint for /ping
  manualRouting: false
  terminatingStatusCode: 503                # Return this during graceful shutdown

The ping endpoint returns:

• 200 OK — Traefik is healthy and accepting requests
• 503 Service Unavailable — Traefik is shutting down (graceful termination)

# Health check
curl -s -o /dev/null -w "%{http_code}" http://localhost:80/ping

# With Docker healthcheck
healthcheck:
  test: ["CMD", "wget", "-q", "-O-", "http://localhost:80/ping"]
  interval: 30s
  timeout: 3s
  retries: 3

Per-Router Observability Control

Router-level overrides for observability:

http:
  routers:
    internal-only:
      rule: "Host(`internal.example.com`)"
      service: "internal-backend"
      observability:
        metrics: false                       # Don't emit metrics for this router
        accessLogs: true                     # Do log this router
        tracing: false                       # Don't trace this router