magnus919_agent-skills/traefik/references/community-patterns.md

Community Patterns & Production Wisdom

Real-world patterns, best practices, and troubleshooting knowledge gathered from community blogs, forums, and production deployments — supplementing the official Traefik documentation.

Middleware Execution Order

The order of middlewares in a chain is the single most impactful configuration decision. Incorrect ordering can bypass security controls.

Proven production chain order:

IP Whitelist → Rate Limit → Authentication → Header Injection → Path Rewriting

http:
  middlewares:
    prod-security-chain:
      chain:
        middlewares:
          - ip-whitelist
          - rate-limit
          - auth-middleware
          - secure-headers
          - strip-api-prefix

    ip-whitelist:
      ipAllowList:
        sourceRange:
          - "10.0.0.0/8"
          - "172.16.0.0/12"

    rate-limit:
      rateLimit:
        average: 100
        burst: 200
        sourceCriterion:
          requestHost: true

    auth-middleware:
      basicAuth:
        users:
          - "admin:$2y$10$..."
        realm: "Protected Service"

    secure-headers:
      headers:
        customResponseHeaders:
          X-Content-Type-Options: "nosniff"
          Strict-Transport-Security: "max-age=31536000; includeSubDomains; preload"

    strip-api-prefix:
      stripPrefix:
        prefixes:
          - "/api/v1"

Why this order matters: A misordered chain where rate limit comes before IP whitelist allows a blocked subnet to send 200 burst requests that hit the auth middleware unnecessarily. Testing showed 80% wasted auth invocations from the wrong order.

Rate Limiting Per-Service vs Global

Traefik's rate limiting is per-router, not global. A single misbehaving client on one route should not degrade others.

# Per-service rate limiting — isolates noisy neighbors
http:
  routers:
    api-v2:
      rule: "Host(`api.example.com`) && PathPrefix(`/v2`)"
      middlewares:
        - strict-rate-limit
      service: api-backend

  middlewares:
    strict-rate-limit:
      rateLimit:
        average: 100
        burst: 50
        sourceCriterion:
          ipStrategy:
            depth: 1          # Use X-Forwarded-For to get real client IP

Trade-off: Per-service limits prevent a noisy client from degrading all routes. A single compromised client can still exhaust its own backend's connection pool. Combine with circuit breakers for full protection.

Structured Logging & Metrics — Do This First

Enable before serving production traffic. Without structured logs, debugging takes 4x longer.

# Static config
log:
  level: INFO
  format: json                  # Essential for log aggregation
  filePath: "/var/log/traefik/traefik.log"

accessLog:
  format: json
  filePath: "/var/log/traefik/access.log"
  filters:
    statusCodes:
      - "200-499"               # Don't log successful responses
    minDuration: "500ms"        # Only log slow requests
  fields:
    headers:
      defaultMode: "drop"
      names:
        User-Agent: "keep"      # But keep user-agent for analysis

metrics:
  prometheus:
    addEntryPointsLabels: true
    addServicesLabels: true
    buckets:
      - 0.005
      - 0.01
      - 0.025
      - 0.05
      - 0.1
      - 0.25
      - 0.5
      - 1.0

ping:
  entryPoint: "web"

Real-world impact: Teams with structured logs and Grafana dashboards triaged incidents in 11 minutes vs 45 minutes without — a 75% reduction in mean-time-to-resolution.

Performance Tuning

Connection Timeouts

entryPoints:
  websecure:
    address: ":443"
    transport:
      respondingTimeouts:
        readTimeout: 30s
        writeTimeout: 30s
        idleTimeout: 180s
      keepAliveMaxRequests: 1000          # Max requests per keep-alive connection
      keepAliveMaxTime: 5m                # Max keep-alive connection lifetime

serversTransport:
  maxIdleConnsPerHost: 200               # Connection pool size
  forwardingTimeouts:
    dialTimeout: 30s
    responseHeaderTimeout: 15s
    idleConnTimeout: 90s

TLS Performance

# Dynamic config
tls:
  options:
    performance:
      minVersion: VersionTLS12
      cipherSuites:
        - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256    # Fast, hardware-accelerated
        - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
      preferServerCipherSuites: true
      curvePreferences:
        - CurveP256                    # Fastest P-256 curve, widely supported

HTTP/2 and HTTP/3

entryPoints:
  websecure:
    address: ":443"
    http2:
      maxConcurrentStreams: 250         # Default is fine for most workloads
    http3: {}                           # Enable HTTP/3 (UDP) — zero-config

Note: HTTP/3 requires a TCP entryPoint (starts as TCP then upgrades to UDP). Port 443 must be open for both TCP and UDP.

Request Buffering

http:
  middlewares:
    buf:
      buffering:
        maxRequestBodyBytes: 4194304         # 4MB — protects memory
        memRequestBodyBytes: 1048576          # 1MB in memory before disk spill
        maxResponseBodyBytes: 4194304
        memResponseBodyBytes: 1048576

Load Balancing Strategies

Strategy	Best For	Notes
wrr (default)	Equal-capacity backends	Simple round-robin with optional weights
p2c (Power of Two Choices)	Variable request durations	Picks 2 random servers, routes to the one with fewer active connections
hrw (Highest Random Weight)	Session affinity without cookies	Consistent hashing by client IP
leasttime	Latency-sensitive services	Routes to the server with lowest response time + fewest connections

http:
  services:
    latency-sensitive-svc:
      loadBalancer:
        strategy: "leasttime"
        servers:
          - url: "http://10.0.0.1:3000"
          - url: "http://10.0.0.2:3000"

Health Checks & Circuit Breakers

Always configure passive health checks — they catch latency spikes before they cascade into outages.

http:
  services:
    api-backend:
      loadBalancer:
        servers:
          - url: "http://10.0.0.1:8080"
          - url: "http://10.0.0.2:8080"
          - url: "http://10.0.0.3:8080"
        healthCheck:
          path: "/health"
          interval: "10s"
          timeout: "3s"
          followRedirects: false
        passiveHealthCheck:
          maxFailedAttempts: 3
          failureWindow: "60s"

  middlewares:
    circuit-breaker:
      circuitBreaker:
        expression: "LatencyAtQuantileMS(50.0) > 5000 || NetworkErrorRatio() > 0.1"
        checkPeriod: "500ms"
        fallbackDuration: "30s"
        recoveryDuration: "10s"

The circuit breaker expression catches two failure modes:

• LatencyAtQuantileMS(50.0) > 5000 — median latency over 5 seconds
• NetworkErrorRatio() > 0.1 — 10%+ of requests producing network errors

TLS Automation with Let's Encrypt

Traefik's built-in ACME is superior to cert-manager for Docker deployments — no external dependency.

certificatesResolvers:
  letsencrypt:
    acme:
      email: "admin@example.com"
      storage: "/letsencrypt/acme.json"
      # HTTP-01 — simplest, no DNS provider needed
      httpChallenge:
        entryPoint: "web"

When to use HTTP-01 vs DNS-01 vs TLS-ALPN-01:

Challenge	Wildcard	Port Needed	Complexity
HTTP-01	No	80	Simple
TLS-ALPN-01	No	443	Simple
DNS-01	Yes	None	Complex (DNS provider API)

Recommendation: Start with HTTP-01. Only switch to DNS-01 if you need wildcard certificates. HTTP-01 can coexist with HTTPS redirections — Traefik handles the challenge at the entryPoint level before the redirect.

Traefik Behind Cloudflare / CDN

When Traefik sits behind Cloudflare (or any CDN), the client IP must be correctly identified.

Step 1: Trust Cloudflare IPs at EntryPoint Level

This is the only correct way to get real client IPs in access logs. Middleware plugins cannot fix access log IPs.

entryPoints:
  http:
    address: :80
    forwardedHeaders:
      trustedIPs: &cloudflareIPs
        - 103.21.244.0/22
        - 103.22.200.0/22
        - 103.31.4.0/22
        - 104.16.0.0/13
        - 104.24.0.0/14
        - 108.162.192.0/18
        - 131.0.72.0/22
        - 141.101.64.0/18
        - 162.158.0.0/15
        - 172.64.0.0/13
        - 173.245.48.0/20
        - 188.114.96.0/20
        - 190.93.240.0/20
        - 197.234.240.0/22
        - 198.41.128.0/17
        - 2400:cb00::/32
        - 2606:4700::/32
        - 2803:f800::/32
        - 2405:b500::/32
        - 2405:8100::/32
        - 2a06:98c0::/29
        - 2c0f:f248::/32
    http:
      redirections:
        entryPoint:
          to: https
          scheme: https

  https:
    address: :443
    forwardedHeaders:
      trustedIPs: *cloudflareIPs        # YAML anchor reuses the list

Get the current Cloudflare IPs: https://www.cloudflare.com/ips/ — update these periodically as Cloudflare's ranges change.

Step 2: Use Plugin for Real-IP in Backend Headers (Optional)

The forwardedHeaders.trustedIPs setting correctly populates X-Forwarded-For and X-Real-IP headers. For additional control, use the Cloudflare plugin:

experimental:
  plugins:
    cloudflare:
      moduleName: "github.com/BetterCorp/cloudflarewarp"
      version: "v1.0.0"

Step 3: PROXY Protocol (Alternative to forwardedHeaders)

If your CDN supports PROXY protocol (Cloudflare does for some plans), use it instead:

entryPoints:
  https:
    address: :443
    proxyProtocol:
      trustedIPs:
        - 10.0.0.0/8          # Your CDN's egress IPs
    # No forwardedHeaders needed with PROXY protocol

Don't use both PROXY protocol and forwardedHeaders on the same entryPoint — they conflict.

CrowdSec Integration for Threat Intelligence

CrowdSec provides community-powered IP reputation filtering. Integrate as a ForwardAuth middleware.

Architecture

Request → Traefik → CrowdSec Bouncer (ForwardAuth) → Backend Service
                          ↓
                    CrowdSec Agent analyzes Traefik access logs
                          ↓
                    Block decision via LAPI

Docker Compose Setup

services:
  traefik:
    image: traefik:v3.7
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./logs:/var/log/traefik
    labels:
      - "traefik.http.middlewares.crowdsec-bouncer.forwardAuth.address=http://crowdsec-bouncer:8080/api/v1/forward-auth"
      - "traefik.http.middlewares.crowdsec-bouncer.forwardAuth.trustForwardHeader=true"
      # Apply middleware to routers
      - "traefik.http.routers.secured-app.middlewares=crowdsec-bouncer"

  crowdsec:
    image: crowdsecurity/crowdsec:latest
    environment:
      - COLLECTIONS=crowdsecurity/traefik crowdsecurity/http-cve crowdsecurity/linux
    volumes:
      - ./crowdsec:/etc/crowdsec
      - ./logs:/var/log/traefik:ro        # Read Traefik access logs
    restart: unless-stopped

  crowdsec-bouncer:
    image: crowdsecurity/crowdsec-traefik-bouncer:latest
    environment:
      - CROWDSEC_BOUNCER_API_URL=http://crowdsec:8080
      - CROWDSEC_BOUNCER_API_KEY=your-api-key
    restart: unless-stopped

Key points:

• CrowdSec agent reads Traefik's access logs (must be file-based, not stdout)
• The bouncer runs as a separate container implementing ForwardAuth
• Bouncer returns 403 for blocked IPs before requests reach the backend
• CrowdSec agent uses community blocklists for known malicious IPs

Authelia Integration (Single Sign-On)

Authelia provides SSO with 2FA, integrating via Traefik's ForwardAuth middleware.

services:
  authelia:
    image: authelia/authelia:4.38
    labels:
      # Expose Authelia itself through Traefik
      - "traefik.enable=true"
      - "traefik.http.routers.authelia.rule=Host(`auth.example.com`)"
      - "traefik.http.routers.authelia.entrypoints=websecure"
      - "traefik.http.routers.authelia.tls=true"
      - "traefik.http.routers.authelia.tls.certresolver=letsencrypt"
      # Define ForwardAuth middleware
      - "traefik.http.middlewares.authelia.forwardAuth.address=http://authelia:9091/api/authz/forward-auth"
      - "traefik.http.middlewares.authelia.forwardAuth.trustForwardHeader=true"
      - "traefik.http.middlewares.authelia.forwardAuth.authResponseHeaders=X-Forwarded-User"
      # Trusted proxies configuration for Authelia
      - "traefik.http.middlewares.authelia.forwardAuth.tls.insecureSkipVerify=true"

  # Protected service
  protected-app:
    image: nginx:alpine
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.protected-app.rule=Host(`app.example.com`)"
      - "traefik.http.routers.protected-app.entrypoints=websecure"
      - "traefik.http.routers.protected-app.tls=true"
      - "traefik.http.routers.protected-app.middlewares=authelia"
      - "traefik.http.services.protected-app.loadbalancer.server.port=80"

Authelia configuration considerations:

• The ForwardAuth endpoint is http://authelia:9091/api/authz/forward-auth
• trustForwardHeader must be true so Authelia sees the original request URL
• authResponseHeaders passes the authenticated user to the backend
• Authelia must be on the same Docker network as Traefik
• For production, configure TLS between Traefik and Authelia (serversTransport with client certificates)

Authentik Integration

Similar to Authelia but with more SSO/OAuth provider features:

labels:
  # Authentik ForwardAuth middleware
  - "traefik.http.middlewares.authentik.forwardAuth.address=http://authentik-proxy:9000/outpost.goauthentik.io/auth/traefik"
  - "traefik.http.middlewares.authentik.forwardAuth.trustForwardHeader=true"
  - "traefik.http.middlewares.authentik.forwardAuth.authResponseHeaders=X-authentik-username,X-authentik-groups,X-authentik-email,X-authentik-name,X-authentik-uid"
  - "traefik.http.middlewares.authentik.forwardAuth.authResponseHeadersRegex=X-authentik-.*"

  # Router for Authentik's embedded outpost
  - "traefik.http.routers.authentik.rule=Host(`auth.example.com`)"
  - "traefik.http.routers.authentik.service=authentik-proxy"
  - "traefik.http.services.authentik-proxy.loadbalancer.server.port=9000"

Authentik uses an "outpost" model — the proxy component runs alongside Authentik and handles ForwardAuth. The outpost URL differs based on whether you use the embedded or standalone outpost.

Known Limitations & Workarounds

1. Built-in Rate Limiting is Per-Instance

Traefik OSS rate limiting operates independently on each instance. With 3 Traefik replicas and a limit of 100 req/s, each instance allows 100 req/s = 300 total.

Workaround: Use Redis-backed distributed rate limiting (Traefik Hub/Enterprise feature). Or configure per-instance limits assuming worst-case single-instance load.

2. No Native Global Rate Limiting

Unlike NGINX's limit_req_zone, there's no built-in global rate limit store.

Workaround similar to above: Use the Redis rate limit plugin from the plugin catalog.

3. TCP Router Precedence Over HTTP

On shared entryPoints, TCP routers take precedence over HTTP routers. If a TCP router matches, HTTP routers never get to evaluate the request.

Fix: Use separate entryPoints for TCP and HTTP traffic wherever possible.

4. No URL Rewrite / Map (NGINX Equivalent)

Traefik lacks NGINX's rewrite ... break capability. Path manipulation is limited to prefix stripping, prefix adding, and regex replacement.

Workaround: Use replacePathRegex for most rewrite needs, or combine stripPrefix with addPrefix for mapping patterns.

5. Connection Draining

Traefik doesn't support connection draining on shutdown as gracefully as NGINX/HAPROXY. The lifeCycle.graceTimeOut helps but active connections may be dropped during rapid restarts.

Mitigation: Set lifeCycle.requestAcceptGraceTimeout to a reasonable value (5-10s) and use lifeCycle.graceTimeOut of at least 30s. Use reusePort: true for zero-downtime deployments.

6. Large Configuration = Higher Memory

Traefik's dynamic configuration scales with the number of routes. With 1000+ routes in Kubernetes, memory usage can exceed 2GB.

Mitigation: Use provider constraints, namespaces, and label selectors to limit the configuration scope.

7. No Native Fail2Ban Integration

Traefik doesn't have built-in fail2ban style rate-limiting-by-log-analysis.

Workaround: Use CrowdSec (recommended), or parse Traefik access logs with external fail2ban that updates iptables.

Troubleshooting Quick Reference

502 Bad Gateway

Likely Cause	Check	Fix
Backend container not running	docker ps	Start the container
Wrong port	traefik.http.services.X.loadbalancer.server.port	Set the correct exposed port
Backend on different network	Docker network connectivity	Ensure same network or external routing
Backend health check failing	traefik.http.services.X.loadbalancer.healthCheck	Fix backend health endpoint or remove health check

503 Service Unavailable

Likely Cause	Check	Fix
Circuit breaker open	Metrics show circuit breaker tripped	Check backend health, wait for recovery
All backends unhealthy	Passive health check counts	Fix backend or increase maxFailedAttempts
No servers in service	Provider didn't discover backends	Check labels/tags are correct

TLS / ACME Issues

Symptom	Likely Cause	Fix
Certificate not issued	ACME challenge can't reach Traefik	Check port 80/443 accessibility
Certificate expiry warning	DNS-01 propagation delay	Increase delayBeforeCheck
"acme.json" permission denied	File permissions	chmod 600 acme.json
Rate limited by Let's Encrypt	Too many cert requests	Use staging CA for testing, reduce cert churn

Real-IP Problems

Symptom	Likely Cause	Fix
Logs show Docker bridge IP	No forwardedHeaders configured	Set forwardedHeaders.trustedIPs on entryPoint
Backend sees only CDN IP	Cloudflare IPs not trusted	Add Cloudflare ranges to trustedIPs
Double IP in X-Forwarded-For	Normal with CDN — correct	Backend should use the first IP in the chain

Config Validation

# Check Traefik config (if using file provider)
traefik healthcheck --conf=/etc/traefik/traefik.yml

# Check dynamic config syntax with Traefik itself
docker exec traefik traefik healthcheck

# Verify routing is working
curl -v -H "Host: app.example.com" http://localhost/
curl -v -H "Host: app.example.com" https://localhost/ -k

# Check ACME certificate status
docker exec traefik sh -c 'cat /letsencrypt/acme.json' | python3 -m json.tool