From 2b9d5a765bf7cb1dccaaee8e9fe97defd84f53b1 Mon Sep 17 00:00:00 2001 From: Magnus Hedemark Date: Tue, 21 Jul 2026 10:18:43 -0400 Subject: [PATCH] docs(traefik): add CSP/SPA debugging pitfall and reference (#86) --- traefik/README.md | 2 +- traefik/SKILL.md | 2 + traefik/references/csp-spa-debugging.md | 132 ++++++++++++++++++++++++ 3 files changed, 135 insertions(+), 1 deletion(-) create mode 100644 traefik/references/csp-spa-debugging.md diff --git a/traefik/README.md b/traefik/README.md index 4ebc053..b15d563 100644 --- a/traefik/README.md +++ b/traefik/README.md @@ -20,7 +20,7 @@ When your agent loads this skill, it becomes a **Traefik infrastructure engineer | `SKILL.md` | Quick-start deployment, core concepts, reference index | | `templates/` | Production-ready Docker Compose template | | `scripts/` | Health check script with JSON output | -| `references/` | 10 reference files: static config, dynamic config, all providers, routing, TLS/ACME, middlewares, observability, production, TCP/UDP, troubleshooting | +| `references/` | Reference files: static config, all providers, routing, TLS/ACME, middlewares, observability, production, TCP/UDP, CSP/SPA debugging, troubleshooting | ## Triggers diff --git a/traefik/SKILL.md b/traefik/SKILL.md index 2f66678..576134b 100644 --- a/traefik/SKILL.md +++ b/traefik/SKILL.md @@ -80,6 +80,7 @@ The request flow: `EntryPoint → Router → (Middlewares) → Service → Backe | **Other Providers** | ECS, Nomad, Consul Catalog, KV stores, File, HTTP, REST providers | `references/other-providers.md` | | **Community Patterns** | Production wisdom — middleware ordering, performance tuning, CDN real-IP, CrowdSec, Authelia, troubleshooting | `references/community-patterns.md` | | **Operational Audit** | Full-stack audit methodology — surface inventory, config review, runtime state, log analysis, classification framework | `references/operational-audit.md` | +| **CSP / SPA Debugging** | Entrypoint header overwrite silently breaks cross-origin SPAs — diagnostic flow, fix, CORS preflight interception | `references/csp-spa-debugging.md` | | **Plugins & Extending** | Yaegi and WASM plugins, plugin configuration, FastProxy | `references/plugins-extend.md` | ## Common Pitfalls @@ -100,6 +101,7 @@ The request flow: `EntryPoint → Router → (Middlewares) → Service → Backe - **Entrypoint-level middleware + router-level declaration = double execution:** When an entryPoint applies a middleware (e.g., `http.middlewares: [default@file]`) and a router ALSO declares the same middleware, the middleware executes twice. Symptoms: doubled rate-limit counts, wasted CPU on duplicate compression, confusing debug logs. **Diagnose:** query the runtime API (`/api/rawdata`) and check the router's `middlewares` array for duplicates. **Fix:** remove the middleware from router-level declarations — the entryPoint already covers it. Services that need EXACTLY the entryPoint middleware (no additions) can omit the `middlewares` field entirely. - **YAML parse error drops entire file provider:** When a single file in `providers.file.directory` has a YAML parse error, Traefik discards the ENTIRE provider's configuration — every middleware, router, and service from all files in that directory disappears. The tell: a burst of `"middleware X does not exist"` errors at the same timestamp across every router. Python's `yaml.safe_load()` is not a sufficient validator — Traefik's parser can reject files that pass Python's parser (e.g., subtle indentation differences, trailing whitespace, or template-variable-like strings). **Recovery:** immediately restore the last-known-good file from backup (`docker cp /tmp/backup.yml traefik:/etc/traefik/dynamic/config.yml`). **Prevention:** always snapshot configs before editing, deploy dynamic config changes incrementally (one logical change → verify with smoke test → then next change), and keep a backup of every file you touch. - **Rate limiting breaks SPA page loads (429 Too Many Requests):** Modern SPAs fire 50–100+ JS chunk requests on initial page load. A rate limit of 400 req/s will 429 these requests, producing a black browser window. **Diagnose:** `docker logs traefik | grep "429" | grep "/assets/"` — if you see many 429s on JS/CSS assets within a single second, the rate limit is too low. **Fix:** raise limits. 1000 avg / 1500 burst (rate limit) and 100 concurrent (inFlightReq) are reasonable for homelab deployments with heavy web UIs. Note that entrypoint-applied middleware cannot be overridden per-service — if different services need different limits, you must either raise the global limit or move middleware from entrypoint to per-router application. +- **Entrypoint-level `headers` middleware overwrites router-level CSP (silent SPA breakage):** Entrypoint middlewares run **last on the response path**, and the `headers` middleware overwrites existing headers with identical names ([docs](https://doc.traefik.io/traefik/reference/routing-configuration/http/middlewares/headers/)). When an entrypoint chain sets `contentSecurityPolicy`, it overwrites any router-level CSP — router overrides are impossible. If that CSP is generic (`default-src 'self'` with no `connect-src`), the browser blocks every cross-origin `fetch`/`XHR` the SPA makes. **The tell:** the SPA page and assets load (200s) but login/API calls do nothing, and the backend logs show **zero requests** from that client. **Diagnose:** `curl -D- -o /dev/null https://your-spa/ | grep content-security-policy` — if the SPA page carries a restrictive CSP, check whether the entrypoint middleware is the source. **Fix:** remove `contentSecurityPolicy` from the entrypoint default chain; let each service emit its own tailored CSP. A proxy-wide `default-src 'self'` is actively harmful for any SPA that talks to a different origin. See `references/csp-spa-debugging.md` for the full diagnostic flow and CORS preflight interception pattern. ## When NOT to Use This Skill diff --git a/traefik/references/csp-spa-debugging.md b/traefik/references/csp-spa-debugging.md new file mode 100644 index 0000000..7dffcfc --- /dev/null +++ b/traefik/references/csp-spa-debugging.md @@ -0,0 +1,132 @@ +# CSP / SPA Debugging — Entrypoint Header Overwrite + +Debugging guide for the failure mode where an entrypoint-level `headers` middleware +silently breaks cross-origin SPAs by overwriting router-level +Content-Security-Policy headers. + +## The Mechanism + +Traefik middleware execution order on the response path is the **reverse** of the +request path: + +``` +Request: entrypoint middlewares → router middlewares → service middlewares → backend +Response: backend → service middlewares → router middlewares → entrypoint middlewares +``` + +Entrypoint-level middlewares run **last on the response path**. The `headers` +middleware documentation states: + +> "Custom headers will overwrite existing headers if they have identical names." +> — [Traefik Headers middleware docs](https://doc.traefik.io/traefik/reference/routing-configuration/http/middlewares/headers/) + +Therefore, when an entrypoint applies a chain containing `contentSecurityPolicy` +(e.g., a `default@file` security-headers chain on `websecure`), it overwrites any +router-level CSP on the response. **Router-level CSP overrides are impossible** +when the entrypoint also sets CSP. + +## The Failure + +If the entrypoint CSP is generic — typically `default-src 'self'` with no +`connect-src` directive — the browser enforces it and blocks every cross-origin +`fetch`/`XHR` the SPA makes. The SPA's own page and static assets load normally +(they are same-origin), but all API calls to a different origin silently fail. + +**The backend logs show zero requests from the SPA.** This is the key tell that +distinguishes this failure from backend issues, network problems, or CORS +misconfiguration on the backend. + +## Diagnostic Flow + +``` +SPA page loads (200) but API calls fail silently +│ +├─ 1. Check CSP on the SPA page: +│ curl -D- -o /dev/null https://your-spa.example.com/ | grep -i content-security-policy +│ +│ If you see `default-src 'self'` with no `connect-src` → this is the problem. +│ The browser blocks all cross-origin requests. +│ +├─ 2. Confirm zero requests reach the backend: +│ Check backend logs for the complete absence of requests from the SPA. +│ (Not 403s, not CORS errors — nothing at all.) +│ +├─ 3. Identify the source of the CSP: +│ Check whether the CSP comes from the backend or from Traefik. +│ curl -D- -o /dev/null https://your-backend-api.example.com/ | grep -i content-security-policy +│ +│ If the backend emits its own (different) CSP but the SPA page shows a +│ generic one, the entrypoint middleware is overwriting it. +│ +└─ 4. Check the entrypoint middleware chain: + Look at the entrypoint's `http.middlewares` list in static config, + then trace the chain to find `contentSecurityPolicy` in a headers middleware. +``` + +## The Fix + +Remove `contentSecurityPolicy` from the entrypoint-level default middleware chain. +Let each service emit its own tailored CSP. Many applications (GoToSocial, +Mastodon, Nextcloud, etc.) ship their own CSP headers that are specific to their +needs. + +A proxy-wide `default-src 'self'` is actively harmful for any SPA that +communicates with a different origin. The proxy should not impose a CSP that +overrides what the application itself intends. + +Other security headers (HSTS, X-Content-Type-Options, X-Frame-Options, +Referrer-Policy) are safe to keep in the entrypoint chain — they do not interfere +with cross-origin API calls. + +## CORS Preflight Interception + +The `headers` middleware also intercepts CORS preflight requests when CORS headers +are configured: + +> "If CORS headers are set, then the middleware does not pass preflight requests +> to any service, instead the response will be generated and sent back to the +> client directly." +> — [Traefik Headers middleware docs](https://doc.traefik.io/traefik/reference/routing-configuration/http/middlewares/headers/) + +This is useful when a backend does not handle `OPTIONS` preflight requests (returns +405). Adding a `headers` middleware with `accessControlAllowMethods` and +`accessControlAllowOriginList` to the router handles preflights at the proxy level. + +Example: + +```yaml +http: + middlewares: + cors-preflight: + headers: + accessControlAllowMethods: + - GET + - POST + - PUT + - DELETE + - PATCH + - OPTIONS + accessControlAllowOriginList: + - "*" + accessControlAllowHeaders: + - "*" + accessControlMaxAge: 120 + addVaryHeader: true +``` + +## Related but Distinct Failures + +| Failure | Symptom | Cause | +|---------|---------|-------| +| **CSP overwrite (this guide)** | SPA loads, zero API requests reach backend | Entrypoint `headers` middleware overwrites router CSP | +| **Rate limiting** | SPA loads, 429 errors on asset/API requests | Entrypoint rate limit too low for SPA burst | +| **Double middleware execution** | Doubled rate-limit counts, duplicate compression | Same middleware declared at both entrypoint and router level | + +## Sources + +- [Traefik Headers middleware](https://doc.traefik.io/traefik/reference/routing-configuration/http/middlewares/headers/) — header overwrite behavior, CORS preflight interception +- [Traefik Middleware overview](https://doc.traefik.io/traefik/reference/routing-configuration/http/middlewares/overview/) — router vs. service middleware execution order +- [Traefik Entrypoints](https://doc.traefik.io/traefik/reference/install-configuration/entrypoints/) — entrypoint-level `http.middlewares` configuration +- [MDN Content-Security-Policy](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy) — `default-src`, `connect-src`, browser enforcement +- [MDN CORS](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS) — preflight request mechanics +- [unrolled/secure](https://github.com/unrolled/secure#available-options) — the library Traefik uses for security headers