docs(traefik): add CSP/SPA debugging pitfall and reference (#86)

2026-09-18 06:56:59 +03:00 · 2026-07-21 10:18:43 -04:00
parent 051bf2a9c6
commit 2b9d5a765b
3 changed files with 135 additions and 1 deletions
@@ -20,7 +20,7 @@ When your agent loads this skill, it becomes a **Traefik infrastructure engineer
 | `SKILL.md` | Quick-start deployment, core concepts, reference index |
 | `templates/` | Production-ready Docker Compose template |
 | `scripts/` | Health check script with JSON output |
-| `references/` | 10 reference files: static config, dynamic config, all providers, routing, TLS/ACME, middlewares, observability, production, TCP/UDP, troubleshooting |
+| `references/` | Reference files: static config, all providers, routing, TLS/ACME, middlewares, observability, production, TCP/UDP, CSP/SPA debugging, troubleshooting |

 ## Triggers

@@ -80,6 +80,7 @@ The request flow: `EntryPoint → Router → (Middlewares) → Service → Backe
 | **Other Providers** | ECS, Nomad, Consul Catalog, KV stores, File, HTTP, REST providers | `references/other-providers.md` |
 | **Community Patterns** | Production wisdom — middleware ordering, performance tuning, CDN real-IP, CrowdSec, Authelia, troubleshooting | `references/community-patterns.md` |
 | **Operational Audit** | Full-stack audit methodology — surface inventory, config review, runtime state, log analysis, classification framework | `references/operational-audit.md` |
+| **CSP / SPA Debugging** | Entrypoint header overwrite silently breaks cross-origin SPAs — diagnostic flow, fix, CORS preflight interception | `references/csp-spa-debugging.md` |
 | **Plugins & Extending** | Yaegi and WASM plugins, plugin configuration, FastProxy | `references/plugins-extend.md` |

 ## Common Pitfalls
@@ -100,6 +101,7 @@ The request flow: `EntryPoint → Router → (Middlewares) → Service → Backe
 - **Entrypoint-level middleware + router-level declaration = double execution:** When an entryPoint applies a middleware (e.g., `http.middlewares: [default@file]`) and a router ALSO declares the same middleware, the middleware executes twice. Symptoms: doubled rate-limit counts, wasted CPU on duplicate compression, confusing debug logs. **Diagnose:** query the runtime API (`/api/rawdata`) and check the router's `middlewares` array for duplicates. **Fix:** remove the middleware from router-level declarations — the entryPoint already covers it. Services that need EXACTLY the entryPoint middleware (no additions) can omit the `middlewares` field entirely.
 - **YAML parse error drops entire file provider:** When a single file in `providers.file.directory` has a YAML parse error, Traefik discards the ENTIRE provider's configuration — every middleware, router, and service from all files in that directory disappears. The tell: a burst of `"middleware X does not exist"` errors at the same timestamp across every router. Python's `yaml.safe_load()` is not a sufficient validator — Traefik's parser can reject files that pass Python's parser (e.g., subtle indentation differences, trailing whitespace, or template-variable-like strings). **Recovery:** immediately restore the last-known-good file from backup (`docker cp /tmp/backup.yml traefik:/etc/traefik/dynamic/config.yml`). **Prevention:** always snapshot configs before editing, deploy dynamic config changes incrementally (one logical change → verify with smoke test → then next change), and keep a backup of every file you touch.
 - **Rate limiting breaks SPA page loads (429 Too Many Requests):** Modern SPAs fire 50–100+ JS chunk requests on initial page load. A rate limit of 400 req/s will 429 these requests, producing a black browser window. **Diagnose:** `docker logs traefik | grep "429" | grep "/assets/"` — if you see many 429s on JS/CSS assets within a single second, the rate limit is too low. **Fix:** raise limits. 1000 avg / 1500 burst (rate limit) and 100 concurrent (inFlightReq) are reasonable for homelab deployments with heavy web UIs. Note that entrypoint-applied middleware cannot be overridden per-service — if different services need different limits, you must either raise the global limit or move middleware from entrypoint to per-router application.
+- **Entrypoint-level `headers` middleware overwrites router-level CSP (silent SPA breakage):** Entrypoint middlewares run **last on the response path**, and the `headers` middleware overwrites existing headers with identical names ([docs](https://doc.traefik.io/traefik/reference/routing-configuration/http/middlewares/headers/)). When an entrypoint chain sets `contentSecurityPolicy`, it overwrites any router-level CSP — router overrides are impossible. If that CSP is generic (`default-src 'self'` with no `connect-src`), the browser blocks every cross-origin `fetch`/`XHR` the SPA makes. **The tell:** the SPA page and assets load (200s) but login/API calls do nothing, and the backend logs show **zero requests** from that client. **Diagnose:** `curl -D- -o /dev/null https://your-spa/ | grep content-security-policy` — if the SPA page carries a restrictive CSP, check whether the entrypoint middleware is the source. **Fix:** remove `contentSecurityPolicy` from the entrypoint default chain; let each service emit its own tailored CSP. A proxy-wide `default-src 'self'` is actively harmful for any SPA that talks to a different origin. See `references/csp-spa-debugging.md` for the full diagnostic flow and CORS preflight interception pattern.

 ## When NOT to Use This Skill

@@ -0,0 +1,132 @@
+# CSP / SPA Debugging — Entrypoint Header Overwrite
+
+Debugging guide for the failure mode where an entrypoint-level `headers` middleware
+silently breaks cross-origin SPAs by overwriting router-level
+Content-Security-Policy headers.
+
+## The Mechanism
+
+Traefik middleware execution order on the response path is the **reverse** of the
+request path:
+
+```
+Request:  entrypoint middlewares → router middlewares → service middlewares → backend
+Response: backend → service middlewares → router middlewares → entrypoint middlewares
+```
+
+Entrypoint-level middlewares run **last on the response path**. The `headers`
+middleware documentation states:
+
+> "Custom headers will overwrite existing headers if they have identical names."
+> — [Traefik Headers middleware docs](https://doc.traefik.io/traefik/reference/routing-configuration/http/middlewares/headers/)
+
+Therefore, when an entrypoint applies a chain containing `contentSecurityPolicy`
+(e.g., a `default@file` security-headers chain on `websecure`), it overwrites any
+router-level CSP on the response. **Router-level CSP overrides are impossible**
+when the entrypoint also sets CSP.
+
+## The Failure
+
+If the entrypoint CSP is generic — typically `default-src 'self'` with no
+`connect-src` directive — the browser enforces it and blocks every cross-origin
+`fetch`/`XHR` the SPA makes. The SPA's own page and static assets load normally
+(they are same-origin), but all API calls to a different origin silently fail.
+
+**The backend logs show zero requests from the SPA.** This is the key tell that
+distinguishes this failure from backend issues, network problems, or CORS
+misconfiguration on the backend.
+
+## Diagnostic Flow
+
+```
+SPA page loads (200) but API calls fail silently
+│
+├─ 1. Check CSP on the SPA page:
+│     curl -D- -o /dev/null https://your-spa.example.com/ | grep -i content-security-policy
+│
+│   If you see `default-src 'self'` with no `connect-src` → this is the problem.
+│   The browser blocks all cross-origin requests.
+│
+├─ 2. Confirm zero requests reach the backend:
+│     Check backend logs for the complete absence of requests from the SPA.
+│     (Not 403s, not CORS errors — nothing at all.)
+│
+├─ 3. Identify the source of the CSP:
+│     Check whether the CSP comes from the backend or from Traefik.
+│     curl -D- -o /dev/null https://your-backend-api.example.com/ | grep -i content-security-policy
+│
+│   If the backend emits its own (different) CSP but the SPA page shows a
+│   generic one, the entrypoint middleware is overwriting it.
+│
+└─ 4. Check the entrypoint middleware chain:
+      Look at the entrypoint's `http.middlewares` list in static config,
+      then trace the chain to find `contentSecurityPolicy` in a headers middleware.
+```
+
+## The Fix
+
+Remove `contentSecurityPolicy` from the entrypoint-level default middleware chain.
+Let each service emit its own tailored CSP. Many applications (GoToSocial,
+Mastodon, Nextcloud, etc.) ship their own CSP headers that are specific to their
+needs.
+
+A proxy-wide `default-src 'self'` is actively harmful for any SPA that
+communicates with a different origin. The proxy should not impose a CSP that
+overrides what the application itself intends.
+
+Other security headers (HSTS, X-Content-Type-Options, X-Frame-Options,
+Referrer-Policy) are safe to keep in the entrypoint chain — they do not interfere
+with cross-origin API calls.
+
+## CORS Preflight Interception
+
+The `headers` middleware also intercepts CORS preflight requests when CORS headers
+are configured:
+
+> "If CORS headers are set, then the middleware does not pass preflight requests
+> to any service, instead the response will be generated and sent back to the
+> client directly."
+> — [Traefik Headers middleware docs](https://doc.traefik.io/traefik/reference/routing-configuration/http/middlewares/headers/)
+
+This is useful when a backend does not handle `OPTIONS` preflight requests (returns
+405). Adding a `headers` middleware with `accessControlAllowMethods` and
+`accessControlAllowOriginList` to the router handles preflights at the proxy level.
+
+Example:
+
+```yaml
+http:
+  middlewares:
+    cors-preflight:
+      headers:
+        accessControlAllowMethods:
+          - GET
+          - POST
+          - PUT
+          - DELETE
+          - PATCH
+          - OPTIONS
+        accessControlAllowOriginList:
+          - "*"
+        accessControlAllowHeaders:
+          - "*"
+        accessControlMaxAge: 120
+        addVaryHeader: true
+```
+
+## Related but Distinct Failures
+
+| Failure | Symptom | Cause |
+|---------|---------|-------|
+| **CSP overwrite (this guide)** | SPA loads, zero API requests reach backend | Entrypoint `headers` middleware overwrites router CSP |
+| **Rate limiting** | SPA loads, 429 errors on asset/API requests | Entrypoint rate limit too low for SPA burst |
+| **Double middleware execution** | Doubled rate-limit counts, duplicate compression | Same middleware declared at both entrypoint and router level |
+
+## Sources
+
+- [Traefik Headers middleware](https://doc.traefik.io/traefik/reference/routing-configuration/http/middlewares/headers/) — header overwrite behavior, CORS preflight interception
+- [Traefik Middleware overview](https://doc.traefik.io/traefik/reference/routing-configuration/http/middlewares/overview/) — router vs. service middleware execution order
+- [Traefik Entrypoints](https://doc.traefik.io/traefik/reference/install-configuration/entrypoints/) — entrypoint-level `http.middlewares` configuration
+- [MDN Content-Security-Policy](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy) — `default-src`, `connect-src`, browser enforcement
+- [MDN CORS](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS) — preflight request mechanics
+- [unrolled/secure](https://github.com/unrolled/secure#available-options) — the library Traefik uses for security headers