magnus919_agent-skills/secure-software-engineering/references/multi-tenant-isolation.md

Multi-Tenant Isolation

Security owns the threat model and enforceable controls for tenant boundaries. The end-to-end SaaS architecture, commercial tiers, placement strategy, and lifecycle authority belong to multi-tenant-saas-architecture.

Make The Decision Now

State the isolation promise per tenant tier and resource. For compute, storage, keys, queues, caches, search, logs, backups, exports, support paths, and recovery, record whether tenants share a pooled boundary, use a bridge boundary with a dedicated sensitive component, or receive a silo boundary. A tier name is not a control: identify the mechanism, the failure it prevents, and the evidence that supports the promise. Route demand distributions, quota sizing, fairness targets, and unit-cost evidence to capacity-and-cost-engineering.

Model The Two Planes

Treat the control plane and application plane as distinct trust boundaries even if they run in one deployment. The control plane may decide tenant identity, placement, entitlements, support grants, provisioning, suspension, export, and deletion; it must not become an unrestricted reader of application data by default. The application plane serves tenant work and must consume authenticated, versioned policy state without treating a client claim as authority.

For every crossing, document the actor and authority, credential audience, tenant and user context, policy version, freshness rule, failure behavior, idempotency key, and audit event. A stale or unavailable control-plane decision must fail in a defined safe mode, not silently widen access. Provisioning and deprovisioning commands need authenticated origin, replay protection, duplicate handling, and reconciliation evidence. Platform implementation of network, workload, secret, logging, and backup substrate belongs to platform-engineering.

Propagate Identity Without Trusting Labels

Derive tenant context from a verified user or workload identity and an authoritative membership/placement lookup. Bind it to the request, transaction, job, trace, export, and downstream call using a server-controlled context. Reject missing, stale, mismatched, or audience-confused context. A tenant ID in a header, URL, queue payload, callback, cache key, or JWT custom claim is an input to validate, not proof of authority.

At each hop preserve the subject, tenant, actor type, delegated purpose, resource scope, and correlation identifier. Downstream services and workers must authorize again within their own authority; gateway authentication or an upstream predicate is not sufficient. API and event contract details belong to api-design-and-evolution, while service middleware, data access, jobs, and integration tests belong to backend-engineering.

Protect Privileged Support Paths

Support access and impersonation are separate administrative capabilities, not a special case that bypasses tenant authorization. Require a named operator, an approved purpose, target tenant and resource scope, time limit, ticket or case reference where policy requires it, and a visible session indicator. Prefer read-only or customer-approved actions; require step-up approval for writes, exports, key access, deletion, or cross-tenant investigation. Prevent chaining impersonation into broader operator privileges, and make break-glass access expire automatically.

Record who initiated, who was impersonated, why, what policy allowed it, which objects were touched, what changed, and when the grant expired. Test direct API, bulk, background, and support-tool paths with expired, revoked, wrong-tenant, and cross-tenant grants. Audit-event design and sensitive-field minimization belong to the logging owner; the security requirement is that the action be attributable and reviewable.

Apply Controls To Every Tenant-Bearing Resource

• Data and keys: Scope queries and object stores by server-derived tenant context. Use separate accounts, databases, schemas, namespaces, predicates, RLS, workload boundaries, or key hierarchies as the promise requires. Per-tenant keys can reduce blast radius but do not replace authorization; define key access, rotation, disablement, recovery, and destruction evidence.
• Caches, search, and queues: Include an authoritative tenant and resource scope in cache identity and invalidation. Filter search at query authorization and index/write boundaries. Bind queue messages, worker credentials, callbacks, retries, dead letters, and replay tools to tenant scope. Never let a shared worker select a tenant solely from message data.
• Logs and traces: Keep tenant context for investigation without copying customer payloads into shared telemetry. Separate audit access from ordinary operators, restrict query scope, redact secrets and sensitive content, and define retention and deletion behavior. A trace ID is correlation, not an authorization token.
• Backups, restore, export, and deletion: Define whether backup snapshots are pooled, bridge, or siloed and who may restore them. Restore into a bounded quarantine before serving data; verify tenant identity, key availability, authorization, and integrity. Exports must be tenant-scoped, purpose-bound, time-limited, encrypted, and independently authorized. Deletion must cover primaries, replicas, indexes, caches, queues, logs, derived data, exports, and backups according to the adopted retention policy, with completion and gap evidence. Route lifecycle and privacy acceptance artifacts to privacy-engineering.
• Resource controls: Give each tenant and tier explicit concurrency, storage, request, queue, search, export, and job limits where exhaustion can cross a boundary. Protect shared pools with admission control, bounded work, fair scheduling, backpressure, and reserved capacity where justified. A quota is not a capacity model; route sizing, load/soak evidence, hot-tenant analysis, and cost tradeoffs to capacity-and-cost-engineering.

Threat Cases

Threat-model at least these abuse paths and add design-specific variants:

Case	Security question	Evidence to seek
Control-plane compromise or confused deputy	Can a provisioning, support, entitlement, or restore capability read or alter application data beyond its purpose?	Separate credentials and policy tests for each plane and privileged command.
Context substitution	Can a caller replace tenant context between authentication, lookup, queueing, storage, or downstream calls?	Direct, asynchronous, retry, and callback tests with mismatched context.
Support impersonation abuse	Can an operator hide, prolong, chain, or broaden an impersonation session?	Approval, expiry, scope, audit, revocation, and break-glass tests.
Shared-resource bleed	Can cache reuse, search ranking, logs, backups, exports, or dead letters reveal another tenant?	Distinct sentinel data and negative tests at every resource boundary.
Noisy neighbor or exhaustion	Can one tenant consume shared workers, connections, memory, queue depth, search capacity, storage, or export bandwidth and deny another?	Tenant-distributed load/soak evidence, quotas, fairness observations, and safe degradation.
Lifecycle race	Can a suspended, deleted, or deprovisioned tenant continue receiving jobs, tokens, exports, or restored data?	Provision/deprovision state machine tests, tombstone handling, retries, and reconciliation records.

Evidence And Verification

Build an authorization matrix and a resource-boundary inventory. Prove, with distinct tenants and tiers, allowed and denied reads/writes, direct and bulk API paths, worker retries and replays, cache hits, search results, logs, keys, backup restore, exports, deletion, support impersonation, and resource exhaustion. Verify that a new tenant is born with restrictive permissions, scoped credentials, tenant-aware observability, limits, and no unintended shared data.

For deprovisioning, show the initiating authority, state transition, revocation of sessions and jobs, blocking of new work, completion across every store, and reconciliation of failures. Record evidence location, test environment, policy version, owner, and residual gap. A passing database test does not prove that cache, search, logs, backups, queues, support tools, or capacity boundaries are safe.

Misuse To Avoid

• Treating a pooled, bridge, or silo label as proof without naming its actual isolation mechanism and failure blast radius.
• Filtering tenant IDs only in the UI or trusting client headers, URL segments, queue fields, trace fields, or support claims.
• Treating control-plane administrator access, database privilege, or restore access as permission to view every tenant's application data.
• Calling a database boundary complete while cache keys, logs, search indexes, asynchronous workers, exports, backups, or deletion jobs remain shared.
• Calling a rate limit or quota capacity evidence without representative load, distribution, fairness, and degradation measurements.
• Granting an AI agent tenant-wide capability when the user needs a document-scoped capability; prompts and retrieval content cannot grant access.