magnus919_agent-skills/backend-engineering/references/event-driven-service-implementation.md

# Event-Driven Service Implementation

Load this reference when a service publishes domain facts, consumes messages, or
coordinates durable state with asynchronous work. It covers implementation seams;
use `api-design-and-evolution` for the event contract and delivery agreement.

## Boundaries And Flow

Keep three responsibilities visible, even when the project uses different names:

| Boundary | Owns | Must not know |
|---|---|---|
| Domain | Invariants, state transitions, and meaningful past-tense business facts | ORM types, broker clients, HTTP, serialization, retry policy |
| Application | Use-case coordination, unit-of-work scope, ports, and event dispatch intent | Vendor-specific connection or broker details |
| Infrastructure | Database mappings, transaction implementation, outbox relay, broker adapter, inbox store | Business decisions and domain invariants |

The normal command path is:

1. Translate the request into an application command at the transport boundary.
2. Open one unit of work, load the aggregate through a domain-facing repository, and
   invoke behavior that protects its invariants.
3. Collect domain events produced by the successful state transition. Events are facts,
   not instructions to make setters run; do not emit one for every field change.
4. Persist aggregate changes and an outbox row in the same local transaction. The row
   contains a stable event identity, type, schema version, aggregate identity/version,
   occurred-at value, payload, and trace/correlation context appropriate for the
   service's data policy.
5. Commit before performing broker I/O. A relay publishes committed rows and records a
   delivery result or retry state. It must tolerate a crash after publish and before
   marking the row complete, so consumers still need deduplication.

The unit of work is a logical application operation, normally one aggregate's
immediate invariants plus its outbox records. A broader transaction needs a concrete
invariant justification. Do not turn it into a distributed transaction by calling an
external service while the local transaction is open.

## Inbox And Idempotent Handlers

Assume at-least-once delivery unless the contract proves a narrower guarantee. An
inbox record or equivalent deduplication mechanism should be keyed by the consumer's
identity and the producer event identity, not merely by a business field that may be
reused. This reference uses a single atomic transaction design: the inbox insert,
handler effects, and completed state commit together. It does not use leases or
separate claim/completion transactions.

1. Validate the envelope and schema version before domain mapping. Treat event input
   as untrusted: verify producer authenticity and authorization before trusting the
   producer identity or event identity. Route the protocol, key, credential, and
   authorization design to `secure-software-engineering`.
2. Parse with a safe, data-only deserializer; never use pickle, eval, unsafe YAML, or
   gadget-prone formats. Treat every deserialized field as untrusted at downstream
   boundaries and use parameterized queries plus safe command, template, and path
   handling. The quarantine path must use bounded, access-controlled storage for
   opaque or encrypted raw bytes and sanitized metadata, not blindly persist and
   reparse executable or injection-bearing content. Route detailed injection and
   deserialization controls to `secure-software-engineering`.
3. Insert the inbox identity with a uniqueness constraint in the same transaction as
   the handler effects.
4. If the identity already completed, acknowledge without repeating side effects or
   claiming to return an outcome that was not persisted. A competing transaction
   observes the unique-conflict result after the owner commits; it does not wait on a
   lease or stale claim.
5. Apply a handler whose state transition is safe to repeat. Use a natural idempotency
   key or a version/precondition check for effects outside the local store.
6. Persist the handler's changes and completed-inbox state atomically, then acknowledge.

Do not call a handler "idempotent" because the final row looks unchanged. Check
emails, payments, downstream commands, counters, notifications, and external writes
for duplicate effects. Where an external effect cannot be made idempotent, persist a
durable intent and reconcile its status rather than guessing after a timeout.

## Retry, Replay, And Failure

Classify failures before choosing the response:

| Failure | Default action | Evidence or stop condition |
|---|---|---|
| Temporary broker/database/network issue | Bounded retry with backoff and jitter | Attempt count, age, and queue lag remain within budget |
| Concurrency or serialization conflict | Roll back the unit of work and retry the whole operation | Finite attempts; preserve the original correlation context |
| Invalid or unauthorized event | Do not retry; quarantine and alert the owner | Payload, producer, and reason are available without leaking secrets |
| Poison message or deterministic handler bug | Move to a dead-letter/quarantine path after policy limits | Repair or replacement is tested before replay |
| Unknown schema version or gap | Pause or route to compatibility handling | Contract owner resolves version/ordering decision |
| Publish succeeded but acknowledgement was lost | Republish safely; consumer deduplication resolves the duplicate | Stable event identity and duplicate metrics |

Replay is a controlled operation, not a blind retry loop. Record the source position
or event IDs, selection criteria, handler version, destination, operator, and expected
side effects. Use the same consumer identity and inbox semantics by default. A fresh
consumer scope deliberately defeats inbox deduplication and must be restricted to
handlers proven side-effect-safe or routed through a repair/compensation path; it is
not safe for payments, refunds, or other irreversible effects. Make the stop condition
explicit. If handlers are not deterministic across versions,
use a versioned translator or a new repair command instead of replaying old payloads
through changed rules.

## Observability And Tests

Instrument the lifecycle with stable dimensions such as service, event type, producer,
consumer, schema version, result class, and deployment version. Useful signals include
outbox age and backlog, publish attempts and failures, inbox duplicate rate, handler
latency, retry counts, dead-letter volume, replay volume, and consumer lag. Propagate
trace/correlation identifiers, but redact payloads, credentials, and sensitive fields.

Test the smallest useful unit at each boundary:

- Unit-test aggregate invariants and the exact domain facts emitted for successful
  transitions; no broker or database is needed.
- Integration-test that state and outbox rows commit together, and that a rollback
  leaves neither durable effect.
- Test relay crashes and duplicate publication, then verify consumer deduplication.
- Test handler retry, permanent failure, quarantine, schema mismatch, replay, and
  acknowledgement ordering against a real or production-compatible broker/store.
- Assert observability fields and alert inputs for lag, age, duplicate, and poison paths.

## Ownership Boundary

`programming-principles` owns bounded contexts, aggregates, domain language, domain
events as DDD concepts, repositories, and anti-corruption guidance. This reference
only explains how those decisions become application and infrastructure behavior.
`api-design-and-evolution` owns event names, envelopes, compatibility, ordering scope,
retention, and consumer-facing delivery semantics. `site-reliability-engineering`
owns SLOs and paging policy; `secure-software-engineering` owns threat modeling,
authorization, and secure logging.