magnus919_agent-skills/agent-production-operations/references/runtime-control-plan.md

# Runtime Control Plan

Defines the versioning, rollout, and fallback strategy for an agent operating
under the [agent-production-contract.md](agent-production-contract.md).

## 1. Versioning

All five dimensions of an agent are independently versioned and recorded in the
production contract.

### 1.1 Model versioning

| Field | Description |
|---|---|
| `model_id` | Unique model identifier (provider + model name + version hash) |
| `model_version` | Provider-assigned version or deployment timestamp |
| `model_fallback` | Fallback model to use if the primary model is unavailable |
| `model_capability_baseline` | Reference eval scores against the current eval suite |
| `model_regression_threshold` | Maximum acceptable score drop before the model version is blocked (e.g., "no more than 5% drop on any eval case") |

**Change procedure**: a new model version must pass the full eval suite before
it can be referenced in a production contract. Model regression in production
(observed via trace-to-eval feedback) triggers an automatic eval re-run. If
regression exceeds the threshold, the runtime falls back to the previous model
version.

### 1.2 Prompt versioning

| Field | Description |
|---|---|
| `prompt_id` | Unique prompt identifier (hash of prompt template + system message) |
| `prompt_version` | Monotonically increasing version number |
| `prompt_change_summary` | Human-readable description of the change |
| `prompt_eval_baseline` | Reference eval scores for this prompt version |

**Change procedure**: prompt changes must be evaluated against the eval suite.
A prompt-only change that does not alter the tool surface or authority may use
a reduced eval subset. Prompt regression triggers fallback to the previous
prompt version (not model fallback, unless both regress).

### 1.3 Tool versioning

| Field | Description |
|---|---|
| `tool_id` | Unique tool identifier |
| `tool_version` | Semantic version of the tool implementation |
| `tool_api_version` | API version the tool exposes to the agent |
| `tool_health_check` | Endpoint or command that verifies the tool is operational |
| `tool_deprecation_date` | Date after which the tool version is unsupported |

**Change procedure**: a new tool or tool version must be registered in the
production contract with its health check and authority scope. Tools that
change their side-effect surface (new mutation capability) require a
production-readiness re-review. Tool version rollback is immediate if the new
version fails its health check.

### 1.4 Policy versioning

| Field | Description |
|---|---|
| `policy_id` | Unique policy identifier |
| `policy_version` | Monotonically increasing version number |
| `policy_rules` | The set of rules governing agent behavior (authority, escalation, cost, privacy) |
| `policy_change_log` | Ordered list of policy changes with dates and rationales |

**Change procedure**: policy changes that expand authority MUST be gated by a
production-readiness review. Policy changes that restrict authority (tightening
thresholds, adding constraints) may be applied immediately but must be recorded
in the tool-authority-health record.

### 1.5 Evaluator versioning

| Field | Description |
|---|---|
| `evaluator_id` | Unique evaluator identifier |
| `evaluator_version` | Semantic version of the evaluator |
| `evaluator_suite` | The set of eval cases this evaluator runs |
| `evaluator_gate` | Whether this evaluator is a release gate (`blocking` or `advisory`) |

**Change procedure**: evaluator changes that add or modify eval cases must be
reviewed for case quality. A blocking evaluator must pass before any
corresponding agent version can be promoted to production. Evaluator regression
(an evaluator that incorrectly passes or fails) is treated as a production
incident and routed through incident-learning.

## 2. Staged rollout

Rollout proceeds through four stages. Authority expands at each stage only when
the stage's exit criteria are satisfied.

### Stage 1 — Shadow / dry-run (0% traffic, read-only)

| Parameter | Value |
|---|---|
| Traffic | 0% of production traffic; replay of logged production requests |
| Authority | Read-only; side effects logged but not executed |
| Duration | Minimum 24 hours or until 1,000 requests processed, whichever is longer |
| Monitoring | Model latency, prompt compliance, uncertainty distribution |
| Exit criteria | p95 latency within 2x baseline; no authority breach attempts; uncertainty distribution within expected range |

### Stage 2 — Canary (1–5% traffic, read-only with logged side effects)

| Parameter | Value |
|---|---|
| Traffic | 1% of production, ramping to 5% over the observation window |
| Authority | Read-only in production; side effects logged but not executed (dual-write comparison against existing system output) |
| Duration | Minimum 48 hours |
| Monitoring | All stage-1 metrics plus: cost-per-request, dual-write divergence rate, user-satisfaction-equivalent signal |
| Exit criteria | Cost-per-request within budget; dual-write divergence < 2%; no escalation events |

### Stage 3 — Limited production (5–25% traffic, side-effect-gated)

| Parameter | Value |
|---|---|
| Traffic | 5% ramping to 25% in 5% increments each 24 hours |
| Authority | Side-effect-capable per the authority contract, with `side_effect_approval: gate` (human approval required per side-effect class until proven safe) |
| Duration | Minimum 72 hours; each increment gates on the previous increment's exit criteria |
| Monitoring | All stage-2 metrics plus: side-effect audit trail, authority breach counter, escalation rate, tool health |
| Exit criteria | Side-effect audit clean (no unexpected mutations); authority breach count = 0; escalation rate < 1%; all tools healthy (health-check pass rate > 99%) |

### Stage 4 — Full production (25–100% traffic)

| Parameter | Value |
|---|---|
| Traffic | 25% to 100% in 25% increments each 24 hours |
| Authority | Full per the authority contract; `side_effect_approval` may move from `gate` to `auto` after 7 days of clean audit |
| Duration | Continuous monitoring |
| Monitoring | All stage-3 metrics plus: cost-budget tracking, trace-to-eval feedback sampling, production-readiness re-review trigger |
| Exit criteria | N/A — continuous operation; triggers for fallback or disablement remain active |

### Rollout abort triggers (any stage)

If any of the following occurs during rollout, abort the current stage and fall
back to the previous stage's authority level:

- Any authority breach (stage 2+).
- Tool health-check failure for a critical tool (stage 3+).
- Cost-per-request > 150% of baseline for > 1 hour (stage 2+).
- p95 latency > 3x baseline for > 30 minutes (stage 2+).
- Escalation rate > 5% (stage 3+).
- Production-readiness review outcome changes to `defer` or `no-go`.

## 3. Fallback

Fallback is a predetermined safe behavior when the agent, model, or a tool
cannot operate at normal capability. Fallback is distinct from disablement:
fallback preserves reduced capability; disablement removes all capability.

### Fallback paths by trigger

| Trigger | Fallback action | Rollback to |
|---|---|---|
| Model endpoint 5xx > 30s | Switch to `model_fallback` from the production contract | Previous model version |
| Model regression detected (eval score drop > threshold) | Switch to `model_fallback`; trigger eval re-run | Previous model version |
| Prompt regression detected | Revert to previous `prompt_version` | Previous prompt version |
| Critical tool unhealthy > 2 min | Degrade authority: revoke that tool's actions; agent continues with remaining tools | Previous tool-set configuration |
| Cost budget 100% consumed | Degrade to read-only; queue mutations | Read-only mode |
| Cost budget 110% consumed | Disable agent entirely | Offline |
| Latency p95 > 5x baseline > 2 min | Degrade to read-only; switch to `model_fallback` if model is the latency source | Read-only mode |
| Authority breach detected | Immediate: block the action; escalate; if breach count ≥ 3 in 24h, disable | Disabled |
| Privacy breach (PII access outside scope) | Immediate: revoke all data access; escalate; mandatory security review | Read-only, no data access |

### Fallback verification

Every fallback path must be exercised in a non-production environment before
the agent enters Stage 2 (canary) of rollout. The fallback exercise record
includes:

- Trigger simulated.
- Fallback action executed.
- Time to fallback completion measured.
- Post-fallback agent behavior verified (reduced authority, correct degraded
  response).
- Recovery path tested (return to normal operation after trigger clears).

Fallback exercises are repeated at each production-readiness review cycle.