From 1291f9576e8c35451c440b06efab0ee9f70079aa Mon Sep 17 00:00:00 2001 From: Magnus Hedemark Date: Fri, 21 Aug 2026 16:21:23 -0400 Subject: [PATCH] fix: SkillOpt optimize AI operating economics (#369) * fix: SkillOpt epoch 1 for AI operating economics Promote cold-load entry points, quick-start reference routing, the minimum decision-record contract, and trigger-oriented progressive disclosure. Signed-off-by: Magnus Hedemark * fix: SkillOpt epoch 2 for AI operating economics Add review-depth selection, evidence-to-disposition guidance, and scenario-led routing across adjacent skills. Signed-off-by: Magnus Hedemark * fix: SkillOpt epoch 3 for AI operating economics Expose a minimum claim ledger and explicit closure conditions for every bounded disposition. Signed-off-by: Magnus Hedemark * fix: resolve SkillOpt review consistency findings Align entry-point paths, canonical step routing, claim-ledger fields, and triage disposition wording. Signed-off-by: Magnus Hedemark * fix: resolve final SkillOpt disposition wording Keep review-depth outputs inside the canonical disposition set and distinguish supported claims from permitted language. Signed-off-by: Magnus Hedemark * fix: complete SkillOpt routing correction Route triage through the outcome-map step and identify the evidence-classification step explicitly. Signed-off-by: Magnus Hedemark * fix: complete AI economics review template Add the minimum decision-record fields required by the optimized skill routing contract. Signed-off-by: Magnus Hedemark --------- Signed-off-by: Magnus Hedemark --- ai-operating-economics/SKILL.md | 87 +++++++++++++++---- .../references/evidence-method.md | 6 +- .../templates/ai-economics-review.md | 14 +++ .../ai-initiative-evidence-record.md | 6 +- 4 files changed, 92 insertions(+), 21 deletions(-) diff --git a/ai-operating-economics/SKILL.md b/ai-operating-economics/SKILL.md index 8f8eed8..79b1752 100644 --- a/ai-operating-economics/SKILL.md +++ b/ai-operating-economics/SKILL.md @@ -25,6 +25,16 @@ This skill provides the cross-domain decision spine for evaluating an AI-enabled The core question is not “Did the model make people faster?” It is: “What changed in this workflow, for whom, at what full cost, with what outcome and countermetric evidence, and what authority should the organization grant next?” +## Entry Points + +| Starting state | Start with | Primary artifact or route | +|---|---|---| +| Idea or proposed AI workflow | Steps 1–2 | `templates/ai-initiative-evidence-record.md` | +| Existing pilot or outcome data | Steps 3–7 | `references/evidence-method.md` plus the evidence record | +| Request for broader population or side-effect authority | Steps 7–8; load `references/evidence-method.md` section 7a for the governance packet | Governance evidence packet plus the evidence record | +| Executive, portfolio, launch, or lifecycle review | Steps 8–9 | `templates/ai-economics-review.md`; route launch/runtime details onward | +| Standalone financial, statistical, telemetry, runtime, or governance implementation task | When Not to Use | Named adjacent specialist skill | + ## When to Use Load this skill when the user needs to: @@ -39,15 +49,17 @@ Load this skill when the user needs to: ## When Not to Use -- **Standalone financial statements, pricing, CAC/LTV, runway, or SaaS metrics:** use [financial-modeling](../financial-modeling/SKILL.md). -- **Token, infrastructure, quota, capacity, or SLO-cost modeling:** use [capacity-and-cost-engineering](../capacity-and-cost-engineering/SKILL.md). -- **Metric trees, event schemas, instrumentation QA, or product dashboards:** use [product-analytics-and-measurement](../product-analytics-and-measurement/SKILL.md). -- **Experimental design, causal inference, statistical testing, or power analysis:** use [data-scientist](../data-scientist/SKILL.md). -- **Agent datasets, graders, traces, regression analysis, or telemetry implementation:** use [agent-evals-and-observability](../agent-evals-and-observability/SKILL.md). -- **Production rollout, runtime budgets, authority, fallback, escalation, or disablement:** use [agent-production-operations](../agent-production-operations/SKILL.md). -- **Organization-wide AI risk, policy, compliance, or governance operating models:** use [ai-governance](../ai-governance/SKILL.md). -- **A launch-readiness packet or production go/no-go decision:** use [production-readiness](../production-readiness/SKILL.md). -- **General product governance cadence without an AI-specific value question:** use [product-operations-and-governance](../product-operations-and-governance/SKILL.md). +| If the task is primarily... | Route to | This skill still contributes... | +|---|---|---| +| Financial statements, pricing, CAC/LTV, runway, or SaaS metrics | [financial-modeling](../financial-modeling/SKILL.md) | The AI workflow's outcome and cost evidence can feed the model | +| Token, infrastructure, quota, capacity, or SLO-cost modeling | [capacity-and-cost-engineering](../capacity-and-cost-engineering/SKILL.md) | The economic decision can consume the resulting cost boundary | +| Metric trees, event schemas, instrumentation QA, or product dashboards | [product-analytics-and-measurement](../product-analytics-and-measurement/SKILL.md) | The decision defines which outcome and countermetric evidence matters | +| Experimental design, causal inference, statistical testing, or power analysis | [data-scientist](../data-scientist/SKILL.md) | The decision specifies the claim and comparison it must support | +| Agent datasets, graders, traces, regression analysis, or telemetry implementation | [agent-evals-and-observability](../agent-evals-and-observability/SKILL.md) | The decision consumes verified evaluation and telemetry evidence | +| Production rollout, runtime budgets, authority, fallback, escalation, or disablement | [agent-production-operations](../agent-production-operations/SKILL.md) | The decision sets the evidence and authority boundary | +| Organization-wide AI risk, policy, compliance, or governance operating models | [ai-governance](../ai-governance/SKILL.md) | The initiative record supplies an operating case and unresolved gaps | +| Launch-readiness packet or production go/no-go decision | [production-readiness](../production-readiness/SKILL.md) | The initiative disposition becomes one readiness input | +| General product governance cadence without an AI-specific value question | [product-operations-and-governance](../product-operations-and-governance/SKILL.md) | Use this skill only for the AI-specific value and operating-economics question | ## Non-Negotiable Reasoning Rules @@ -66,6 +78,23 @@ Load this skill when the user needs to: Use this sequence for an AI initiative review. Load the detailed method and the evidence-record template when the task requires a durable artifact. +### Quick Start by Need + +| Need | First action | Load next | +|---|---|---| +| Triage a claim | Name the workflow, decision, and evidence class | Steps 1–3; evidence classes are defined in Step 7 | +| Build a durable record | Copy the initiative evidence record and complete the header first | `templates/ai-initiative-evidence-record.md` | +| Investigate uncertain evidence | Freeze the claim table before drafting conclusions | `references/evidence-method.md` | +| Prepare a review | Assemble evidence, slices, cost, gaps, and disposition | `templates/ai-economics-review.md` | + +### Choose Review Depth + +| Mode | Use when | Minimum evidence | Output | +|---|---|---|---| +| Triage | A claim or opportunity needs a bounded first decision | Workflow, value hypothesis, one outcome, one countermetric, known gaps | Hold, with a routing/evidence plan | +| Standard | A pilot or workflow decision can change population or investment | Comparison, outcome/countermetrics, slices, cost boundary, owner, reversal path | Scale, constrain, redesign, or hold | +| High-assurance | Authority, sensitive data, material user impact, or irreversible change is involved | Standard evidence plus governance packet, human oversight, incident/revalidation, and decommissioning evidence | Scale only within an explicit authority boundary, or Hold | + ### 1. Define the intervention and decision Name the workflow, population, task boundary, intervention mode, baseline, decision sought, and decision owner. State whether the AI assists, recommends, routes, executes, or replaces/removes work. Define what remains human-controlled. @@ -143,6 +172,25 @@ For every material claim, label it: Keep the source, access date, scope, version, caveat, and permitted interpretation with each claim. Load `references/source-index.md` for the research basis and evidence boundaries. +### Minimum Claim Ledger + +For each material claim, record: **claim**, **evidence class**, **source and scope**, **what it supports**, **what it does not support**, **open challenge**, and **permitted language**. Keep unknown claims visible; do not let a source URL or vendor report stand in for direct workflow evidence. + +### Minimum Decision Record + +Every completed review must expose, in one durable artifact: the intervention and population, value hypothesis, primary outcome, countermetrics, comparison and limitations, cost boundary, relevant slices, evidence classes, missing evidence with owner, disposition, authority limit, reversal path, and review trigger. + +### Disposition Quick Pick + +| Evidence state | Default disposition | Next control | +|---|---|---| +| Outcome and countermetrics support a bounded expansion; cost and slices are understood | Scale | Name the next population and authority slice | +| Value is plausible but a cost, quality, subgroup, or authority boundary remains unresolved | Constrain | Limit population, task, quota, or human review | +| The mechanism creates avoidable failure or burden | Redesign | Change the workflow or control and rerun the comparison | +| Required evidence is missing or conflicting | Hold | Assign the evidence owner and review trigger | +| Value is absent or countermetrics exceed bounds | Retire | Protect affected people, migrate, and record learning | +| A material gap is accepted temporarily by a named human | Exception | Set expiry, containment, approver, and revisit condition | + ### 8. Produce a bounded decision Choose exactly one primary disposition: @@ -154,6 +202,15 @@ Choose exactly one primary disposition: - **Retire:** observed value is absent or countermetrics exceed acceptable bounds, with a transition path. - **Exception:** proceed despite a named gap only with an accountable human approver, expiry or revisit trigger, and containment plan. +### Closure Conditions + +- **Scale:** next population, authority slice, owner, and review trigger are recorded. +- **Constrain:** the boundary, quota, human-review rule, and condition for expansion are recorded. +- **Redesign:** the changed mechanism, rerun comparison, and new acceptance boundary are recorded. +- **Hold:** the missing evidence, owner, method, and due trigger are recorded. +- **Retire:** transition, affected-person protection, decommissioning, and retained learning are recorded. +- **Exception:** named human approver, scope, expiry, containment, and revisit condition are recorded. + A decision is incomplete without an owner, review date or trigger, evidence gaps, and reversal path. Route launch or runtime consequences to the appropriate specialist skill. ### 9. Close the learning loop @@ -162,12 +219,12 @@ At the review date, compare expected versus observed outcomes, cost, quality, wo ## Load-on-Demand References -| Need | Load | -|---|---| -| Apply the full research and decision method, including comparison design and uncertainty | [references/evidence-method.md](references/evidence-method.md) | -| Review the sources and their permitted interpretations | [references/source-index.md](references/source-index.md) | -| Fill a durable initiative record before a review | [templates/ai-initiative-evidence-record.md](templates/ai-initiative-evidence-record.md) | -| Prepare an executive or lifecycle review from one or more initiative records | [templates/ai-economics-review.md](templates/ai-economics-review.md) | +| Need | Load when | File | +|---|---|---| +| Apply the full research and decision method, including comparison design and uncertainty | Evidence is incomplete, contested, or consequential | [references/evidence-method.md](references/evidence-method.md) | +| Review sources and permitted interpretations | A claim needs provenance or a source boundary | [references/source-index.md](references/source-index.md) | +| Fill a durable initiative record | Starting a new workflow review or pilot assessment | [templates/ai-initiative-evidence-record.md](templates/ai-initiative-evidence-record.md) | +| Prepare an executive or lifecycle review | Combining one or more initiative records for a decision | [templates/ai-economics-review.md](templates/ai-economics-review.md) | ## Common Pitfalls diff --git a/ai-operating-economics/references/evidence-method.md b/ai-operating-economics/references/evidence-method.md index 2fae95e..57b8e93 100644 --- a/ai-operating-economics/references/evidence-method.md +++ b/ai-operating-economics/references/evidence-method.md @@ -35,9 +35,9 @@ Benefits-realization planning can borrow from public-sector benefits-management Build a claim table before writing the conclusion: -| Claim | Evidence class | Source and scope | What it supports | What it does not support | Open challenge | -|---|---|---|---|---|---| -| [claim] | observed / causal / inferred / vendor-reported / asserted / normative | [citation, date, version] | [permitted interpretation] | [boundary] | [test or missing evidence] | +| Claim | Evidence class | Source and scope | What it supports | What it does not support | Open challenge | Permitted interpretation | +|---|---|---|---|---|---|---| +| [claim] | observed / causal / inferred / vendor-reported / asserted / normative | [citation, date, version] | [supported claim] | [boundary] | [test or missing evidence] | [calibrated language] | Use the strongest source appropriate to the claim: diff --git a/ai-operating-economics/templates/ai-economics-review.md b/ai-operating-economics/templates/ai-economics-review.md index d0a3235..da89660 100644 --- a/ai-operating-economics/templates/ai-economics-review.md +++ b/ai-operating-economics/templates/ai-economics-review.md @@ -11,6 +11,20 @@ Use this template for an executive, portfolio, experiment, launch, or lifecycle - Records reviewed: - Decision deadline or trigger: +## Minimum decision record + +- Intervention and population: +- Value hypothesis: +- Primary outcome: +- Countermetrics: +- Comparison and limitations: +- Cost boundary: +- Relevant slices: +- Evidence classes: +- Missing evidence, owner, and next trigger: +- Authority limit: +- Reversal path: + ## Executive decision - Disposition: scale / constrain / redesign / hold / retire / exception diff --git a/ai-operating-economics/templates/ai-initiative-evidence-record.md b/ai-operating-economics/templates/ai-initiative-evidence-record.md index 2f5b3ad..4206e50 100644 --- a/ai-operating-economics/templates/ai-initiative-evidence-record.md +++ b/ai-operating-economics/templates/ai-initiative-evidence-record.md @@ -36,9 +36,9 @@ - Quality measurement limitations: - Statistical or causal analysis owner: -| Claim | Class | Source | Result | Caveat | Permitted interpretation | -|---|---|---|---|---|---| -| | observed / causal / inferred / vendor-reported / asserted / normative | | | | | +| Claim | Evidence class | Source and scope | What it supports | What it does not support | Open challenge | Permitted interpretation | +|---|---|---|---|---|---|---| +| | observed / causal / inferred / vendor-reported / asserted / normative | | | | | | ## Outcome and countermetrics