magnus919_agent-skills/traefik/references/community-patterns.md

# Community Patterns & Production Wisdom

Real-world patterns, best practices, and troubleshooting knowledge gathered from community blogs, forums, and production deployments — supplementing the official Traefik documentation.

## Middleware Execution Order

The order of middlewares in a chain is the single most impactful configuration decision. Incorrect ordering can bypass security controls.

**Proven production chain order:**

```
IP Whitelist → Rate Limit → Authentication → Header Injection → Path Rewriting
```

```yaml
http:
  middlewares:
    prod-security-chain:
      chain:
        middlewares:
          - ip-whitelist
          - rate-limit
          - auth-middleware
          - secure-headers
          - strip-api-prefix

    ip-whitelist:
      ipAllowList:
        sourceRange:
          - "10.0.0.0/8"
          - "172.16.0.0/12"

    rate-limit:
      rateLimit:
        average: 100
        burst: 200
        sourceCriterion:
          requestHost: true

    auth-middleware:
      basicAuth:
        users:
          - "admin:$2y$10$..."
        realm: "Protected Service"

    secure-headers:
      headers:
        customResponseHeaders:
          X-Content-Type-Options: "nosniff"
          Strict-Transport-Security: "max-age=31536000; includeSubDomains; preload"

    strip-api-prefix:
      stripPrefix:
        prefixes:
          - "/api/v1"
```

**Why this order matters:** A misordered chain where rate limit comes before IP whitelist allows a blocked subnet to send 200 burst requests that hit the auth middleware unnecessarily. Testing showed 80% wasted auth invocations from the wrong order.

## Rate Limiting Per-Service vs Global

Traefik's rate limiting is per-router, not global. A single misbehaving client on one route should not degrade others.

```yaml
# Per-service rate limiting — isolates noisy neighbors
http:
  routers:
    api-v2:
      rule: "Host(`api.example.com`) && PathPrefix(`/v2`)"
      middlewares:
        - strict-rate-limit
      service: api-backend

  middlewares:
    strict-rate-limit:
      rateLimit:
        average: 100
        burst: 50
        sourceCriterion:
          ipStrategy:
            depth: 1          # Use X-Forwarded-For to get real client IP
```

**Trade-off:** Per-service limits prevent a noisy client from degrading all routes. A single compromised client can still exhaust its own backend's connection pool. Combine with circuit breakers for full protection.

## Structured Logging & Metrics — Do This First

Enable before serving production traffic. Without structured logs, debugging takes 4x longer.

```yaml
# Static config
log:
  level: INFO
  format: json                  # Essential for log aggregation
  filePath: "/var/log/traefik/traefik.log"

accessLog:
  format: json
  filePath: "/var/log/traefik/access.log"
  filters:
    statusCodes:
      - "200-499"               # Don't log successful responses
    minDuration: "500ms"        # Only log slow requests
  fields:
    headers:
      defaultMode: "drop"
      names:
        User-Agent: "keep"      # But keep user-agent for analysis

metrics:
  prometheus:
    addEntryPointsLabels: true
    addServicesLabels: true
    buckets:
      - 0.005
      - 0.01
      - 0.025
      - 0.05
      - 0.1
      - 0.25
      - 0.5
      - 1.0

ping:
  entryPoint: "web"
```

**Real-world impact:** Teams with structured logs and Grafana dashboards triaged incidents in 11 minutes vs 45 minutes without — a 75% reduction in mean-time-to-resolution.

## Performance Tuning

### Connection Timeouts

```yaml
entryPoints:
  websecure:
    address: ":443"
    transport:
      respondingTimeouts:
        readTimeout: 30s
        writeTimeout: 30s
        idleTimeout: 180s
      keepAliveMaxRequests: 1000          # Max requests per keep-alive connection
      keepAliveMaxTime: 5m                # Max keep-alive connection lifetime

serversTransport:
  maxIdleConnsPerHost: 200               # Connection pool size
  forwardingTimeouts:
    dialTimeout: 30s
    responseHeaderTimeout: 15s
    idleConnTimeout: 90s
```

### TLS Performance

```yaml
# Dynamic config
tls:
  options:
    performance:
      minVersion: VersionTLS12
      cipherSuites:
        - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256    # Fast, hardware-accelerated
        - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
      preferServerCipherSuites: true
      curvePreferences:
        - CurveP256                    # Fastest P-256 curve, widely supported
```

### HTTP/2 and HTTP/3

```yaml
entryPoints:
  websecure:
    address: ":443"
    http2:
      maxConcurrentStreams: 250         # Default is fine for most workloads
    http3: {}                           # Enable HTTP/3 (UDP) — zero-config
```

**Note:** HTTP/3 requires a TCP entryPoint (starts as TCP then upgrades to UDP). Port 443 must be open for both TCP and UDP.

### Request Buffering

```yaml
http:
  middlewares:
    buf:
      buffering:
        maxRequestBodyBytes: 4194304         # 4MB — protects memory
        memRequestBodyBytes: 1048576          # 1MB in memory before disk spill
        maxResponseBodyBytes: 4194304
        memResponseBodyBytes: 1048576
```

### Load Balancing Strategies

| Strategy | Best For | Notes |
|----------|----------|-------|
| `wrr` (default) | Equal-capacity backends | Simple round-robin with optional weights |
| `p2c` (Power of Two Choices) | Variable request durations | Picks 2 random servers, routes to the one with fewer active connections |
| `hrw` (Highest Random Weight) | Session affinity without cookies | Consistent hashing by client IP |
| `leasttime` | Latency-sensitive services | Routes to the server with lowest response time + fewest connections |

```yaml
http:
  services:
    latency-sensitive-svc:
      loadBalancer:
        strategy: "leasttime"
        servers:
          - url: "http://10.0.0.1:3000"
          - url: "http://10.0.0.2:3000"
```

## Health Checks & Circuit Breakers

Always configure passive health checks — they catch latency spikes before they cascade into outages.

```yaml
http:
  services:
    api-backend:
      loadBalancer:
        servers:
          - url: "http://10.0.0.1:8080"
          - url: "http://10.0.0.2:8080"
          - url: "http://10.0.0.3:8080"
        healthCheck:
          path: "/health"
          interval: "10s"
          timeout: "3s"
          followRedirects: false
        passiveHealthCheck:
          maxFailedAttempts: 3
          failureWindow: "60s"

  middlewares:
    circuit-breaker:
      circuitBreaker:
        expression: "LatencyAtQuantileMS(50.0) > 5000 || NetworkErrorRatio() > 0.1"
        checkPeriod: "500ms"
        fallbackDuration: "30s"
        recoveryDuration: "10s"
```

**The circuit breaker expression catches two failure modes:**
- `LatencyAtQuantileMS(50.0) > 5000` — median latency over 5 seconds
- `NetworkErrorRatio() > 0.1` — 10%+ of requests producing network errors

## TLS Automation with Let's Encrypt

Traefik's built-in ACME is superior to cert-manager for Docker deployments — no external dependency.

```yaml
certificatesResolvers:
  letsencrypt:
    acme:
      email: "admin@example.com"
      storage: "/letsencrypt/acme.json"
      # HTTP-01 — simplest, no DNS provider needed
      httpChallenge:
        entryPoint: "web"
```

**When to use HTTP-01 vs DNS-01 vs TLS-ALPN-01:**

| Challenge | Wildcard | Port Needed | Complexity |
|-----------|----------|-------------|------------|
| HTTP-01 | No | 80 | Simple |
| TLS-ALPN-01 | No | 443 | Simple |
| DNS-01 | Yes | None | Complex (DNS provider API) |

**Recommendation:** Start with HTTP-01. Only switch to DNS-01 if you need wildcard certificates. HTTP-01 can coexist with HTTPS redirections — Traefik handles the challenge at the entryPoint level before the redirect.

## Traefik Behind Cloudflare / CDN

When Traefik sits behind Cloudflare (or any CDN), the client IP must be correctly identified.

### Step 1: Trust Cloudflare IPs at EntryPoint Level

This is the **only** correct way to get real client IPs in access logs. Middleware plugins cannot fix access log IPs.

```yaml
entryPoints:
  http:
    address: :80
    forwardedHeaders:
      trustedIPs: &cloudflareIPs
        - 103.21.244.0/22
        - 103.22.200.0/22
        - 103.31.4.0/22
        - 104.16.0.0/13
        - 104.24.0.0/14
        - 108.162.192.0/18
        - 131.0.72.0/22
        - 141.101.64.0/18
        - 162.158.0.0/15
        - 172.64.0.0/13
        - 173.245.48.0/20
        - 188.114.96.0/20
        - 190.93.240.0/20
        - 197.234.240.0/22
        - 198.41.128.0/17
        - 2400:cb00::/32
        - 2606:4700::/32
        - 2803:f800::/32
        - 2405:b500::/32
        - 2405:8100::/32
        - 2a06:98c0::/29
        - 2c0f:f248::/32
    http:
      redirections:
        entryPoint:
          to: https
          scheme: https

  https:
    address: :443
    forwardedHeaders:
      trustedIPs: *cloudflareIPs        # YAML anchor reuses the list
```

**Get the current Cloudflare IPs:** https://www.cloudflare.com/ips/ — update these periodically as Cloudflare's ranges change.

### Step 2: Use Plugin for Real-IP in Backend Headers (Optional)

The `forwardedHeaders.trustedIPs` setting correctly populates `X-Forwarded-For` and `X-Real-IP` headers. For additional control, use the Cloudflare plugin:

```yaml
experimental:
  plugins:
    cloudflare:
      moduleName: "github.com/BetterCorp/cloudflarewarp"
      version: "v1.0.0"
```

### Step 3: PROXY Protocol (Alternative to forwardedHeaders)

If your CDN supports PROXY protocol (Cloudflare does for some plans), use it instead:

```yaml
entryPoints:
  https:
    address: :443
    proxyProtocol:
      trustedIPs:
        - 10.0.0.0/8          # Your CDN's egress IPs
    # No forwardedHeaders needed with PROXY protocol
```

**Don't use both PROXY protocol and forwardedHeaders on the same entryPoint — they conflict.**

## CrowdSec Integration for Threat Intelligence

CrowdSec provides community-powered IP reputation filtering. Integrate as a ForwardAuth middleware.

### Architecture

```
Request → Traefik → CrowdSec Bouncer (ForwardAuth) → Backend Service
                          ↓
                    CrowdSec Agent analyzes Traefik access logs
                          ↓
                    Block decision via LAPI
```

### Docker Compose Setup

```yaml
services:
  traefik:
    image: traefik:v3.7
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./logs:/var/log/traefik
    labels:
      - "traefik.http.middlewares.crowdsec-bouncer.forwardAuth.address=http://crowdsec-bouncer:8080/api/v1/forward-auth"
      - "traefik.http.middlewares.crowdsec-bouncer.forwardAuth.trustForwardHeader=true"
      # Apply middleware to routers
      - "traefik.http.routers.secured-app.middlewares=crowdsec-bouncer"

  crowdsec:
    image: crowdsecurity/crowdsec:latest
    environment:
      - COLLECTIONS=crowdsecurity/traefik crowdsecurity/http-cve crowdsecurity/linux
    volumes:
      - ./crowdsec:/etc/crowdsec
      - ./logs:/var/log/traefik:ro        # Read Traefik access logs
    restart: unless-stopped

  crowdsec-bouncer:
    image: crowdsecurity/crowdsec-traefik-bouncer:latest
    environment:
      - CROWDSEC_BOUNCER_API_URL=http://crowdsec:8080
      - CROWDSEC_BOUNCER_API_KEY=your-api-key
    restart: unless-stopped
```

**Key points:**
- CrowdSec agent reads Traefik's access logs (must be file-based, not stdout)
- The bouncer runs as a separate container implementing ForwardAuth
- Bouncer returns 403 for blocked IPs before requests reach the backend
- CrowdSec agent uses community blocklists for known malicious IPs

## Authelia Integration (Single Sign-On)

Authelia provides SSO with 2FA, integrating via Traefik's ForwardAuth middleware.

```yaml
services:
  authelia:
    image: authelia/authelia:4.38
    labels:
      # Expose Authelia itself through Traefik
      - "traefik.enable=true"
      - "traefik.http.routers.authelia.rule=Host(`auth.example.com`)"
      - "traefik.http.routers.authelia.entrypoints=websecure"
      - "traefik.http.routers.authelia.tls=true"
      - "traefik.http.routers.authelia.tls.certresolver=letsencrypt"
      # Define ForwardAuth middleware
      - "traefik.http.middlewares.authelia.forwardAuth.address=http://authelia:9091/api/authz/forward-auth"
      - "traefik.http.middlewares.authelia.forwardAuth.trustForwardHeader=true"
      - "traefik.http.middlewares.authelia.forwardAuth.authResponseHeaders=X-Forwarded-User"
      # Trusted proxies configuration for Authelia
      - "traefik.http.middlewares.authelia.forwardAuth.tls.insecureSkipVerify=true"

  # Protected service
  protected-app:
    image: nginx:alpine
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.protected-app.rule=Host(`app.example.com`)"
      - "traefik.http.routers.protected-app.entrypoints=websecure"
      - "traefik.http.routers.protected-app.tls=true"
      - "traefik.http.routers.protected-app.middlewares=authelia"
      - "traefik.http.services.protected-app.loadbalancer.server.port=80"
```

**Authelia configuration considerations:**
- The ForwardAuth endpoint is `http://authelia:9091/api/authz/forward-auth`
- `trustForwardHeader` must be `true` so Authelia sees the original request URL
- `authResponseHeaders` passes the authenticated user to the backend
- Authelia must be on the same Docker network as Traefik
- For production, configure TLS between Traefik and Authelia (serversTransport with client certificates)

## Authentik Integration

Similar to Authelia but with more SSO/OAuth provider features:

```yaml
labels:
  # Authentik ForwardAuth middleware
  - "traefik.http.middlewares.authentik.forwardAuth.address=http://authentik-proxy:9000/outpost.goauthentik.io/auth/traefik"
  - "traefik.http.middlewares.authentik.forwardAuth.trustForwardHeader=true"
  - "traefik.http.middlewares.authentik.forwardAuth.authResponseHeaders=X-authentik-username,X-authentik-groups,X-authentik-email,X-authentik-name,X-authentik-uid"
  - "traefik.http.middlewares.authentik.forwardAuth.authResponseHeadersRegex=X-authentik-.*"

  # Router for Authentik's embedded outpost
  - "traefik.http.routers.authentik.rule=Host(`auth.example.com`)"
  - "traefik.http.routers.authentik.service=authentik-proxy"
  - "traefik.http.services.authentik-proxy.loadbalancer.server.port=9000"
```

Authentik uses an "outpost" model — the proxy component runs alongside Authentik and handles ForwardAuth. The outpost URL differs based on whether you use the embedded or standalone outpost.

## Known Limitations & Workarounds

### 1. Built-in Rate Limiting is Per-Instance

Traefik OSS rate limiting operates independently on each instance. With 3 Traefik replicas and a limit of 100 req/s, each instance allows 100 req/s = 300 total.

**Workaround:** Use Redis-backed distributed rate limiting (Traefik Hub/Enterprise feature). Or configure per-instance limits assuming worst-case single-instance load.

### 2. No Native Global Rate Limiting

Unlike NGINX's `limit_req_zone`, there's no built-in global rate limit store.

**Workaround similar to above:** Use the Redis rate limit plugin from the plugin catalog.

### 3. TCP Router Precedence Over HTTP

On shared entryPoints, TCP routers take precedence over HTTP routers. If a TCP router matches, HTTP routers never get to evaluate the request.

**Fix:** Use separate entryPoints for TCP and HTTP traffic wherever possible.

### 4. No URL Rewrite / Map (NGINX Equivalent)

Traefik lacks NGINX's `rewrite ... break` capability. Path manipulation is limited to prefix stripping, prefix adding, and regex replacement.

**Workaround:** Use `replacePathRegex` for most rewrite needs, or combine `stripPrefix` with `addPrefix` for mapping patterns.

### 5. Connection Draining

Traefik doesn't support connection draining on shutdown as gracefully as NGINX/HAPROXY. The `lifeCycle.graceTimeOut` helps but active connections may be dropped during rapid restarts.

**Mitigation:** Set `lifeCycle.requestAcceptGraceTimeout` to a reasonable value (5-10s) and use `lifeCycle.graceTimeOut` of at least 30s. Use `reusePort: true` for zero-downtime deployments.

### 6. Large Configuration = Higher Memory

Traefik's dynamic configuration scales with the number of routes. With 1000+ routes in Kubernetes, memory usage can exceed 2GB.

**Mitigation:** Use provider constraints, namespaces, and label selectors to limit the configuration scope.

### 7. No Native Fail2Ban Integration

Traefik doesn't have built-in fail2ban style rate-limiting-by-log-analysis.

**Workaround:** Use CrowdSec (recommended), or parse Traefik access logs with external fail2ban that updates iptables.

## Troubleshooting Quick Reference

### 502 Bad Gateway

| Likely Cause | Check | Fix |
|-------------|-------|-----|
| Backend container not running | `docker ps` | Start the container |
| Wrong port | `traefik.http.services.X.loadbalancer.server.port` | Set the correct exposed port |
| Backend on different network | Docker network connectivity | Ensure same network or external routing |
| Backend health check failing | `traefik.http.services.X.loadbalancer.healthCheck` | Fix backend health endpoint or remove health check |

### 503 Service Unavailable

| Likely Cause | Check | Fix |
|-------------|-------|-----|
| Circuit breaker open | Metrics show circuit breaker tripped | Check backend health, wait for recovery |
| All backends unhealthy | Passive health check counts | Fix backend or increase maxFailedAttempts |
| No servers in service | Provider didn't discover backends | Check labels/tags are correct |

### TLS / ACME Issues

| Symptom | Likely Cause | Fix |
|---------|-------------|-----|
| Certificate not issued | ACME challenge can't reach Traefik | Check port 80/443 accessibility |
| Certificate expiry warning | DNS-01 propagation delay | Increase `delayBeforeCheck` |
| "acme.json" permission denied | File permissions | `chmod 600 acme.json` |
| Rate limited by Let's Encrypt | Too many cert requests | Use staging CA for testing, reduce cert churn |

### Real-IP Problems

| Symptom | Likely Cause | Fix |
|---------|-------------|-----|
| Logs show Docker bridge IP | No forwardedHeaders configured | Set `forwardedHeaders.trustedIPs` on entryPoint |
| Backend sees only CDN IP | Cloudflare IPs not trusted | Add Cloudflare ranges to `trustedIPs` |
| Double IP in X-Forwarded-For | Normal with CDN — correct | Backend should use the first IP in the chain |

### Config Validation

```bash
# Check Traefik config (if using file provider)
traefik healthcheck --conf=/etc/traefik/traefik.yml

# Check dynamic config syntax with Traefik itself
docker exec traefik traefik healthcheck

# Verify routing is working
curl -v -H "Host: app.example.com" http://localhost/
curl -v -H "Host: app.example.com" https://localhost/ -k

# Check ACME certificate status
docker exec traefik sh -c 'cat /letsencrypt/acme.json' | python3 -m json.tool
```