fix: SkillOpt optimize AI operating economics (#369)

* fix: SkillOpt epoch 1 for AI operating economics Promote cold-load entry points, quick-start reference routing, the minimum decision-record contract, and trigger-oriented progressive disclosure. Signed-off-by: Magnus Hedemark <magnus919@pm.me> * fix: SkillOpt epoch 2 for AI operating economics Add review-depth selection, evidence-to-disposition guidance, and scenario-led routing across adjacent skills. Signed-off-by: Magnus Hedemark <magnus919@pm.me> * fix: SkillOpt epoch 3 for AI operating economics Expose a minimum claim ledger and explicit closure conditions for every bounded disposition. Signed-off-by: Magnus Hedemark <magnus919@pm.me> * fix: resolve SkillOpt review consistency findings Align entry-point paths, canonical step routing, claim-ledger fields, and triage disposition wording. Signed-off-by: Magnus Hedemark <magnus919@pm.me> * fix: resolve final SkillOpt disposition wording Keep review-depth outputs inside the canonical disposition set and distinguish supported claims from permitted language. Signed-off-by: Magnus Hedemark <magnus919@pm.me> * fix: complete SkillOpt routing correction Route triage through the outcome-map step and identify the evidence-classification step explicitly. Signed-off-by: Magnus Hedemark <magnus919@pm.me> * fix: complete AI economics review template Add the minimum decision-record fields required by the optimized skill routing contract. Signed-off-by: Magnus Hedemark <magnus919@pm.me> --------- Signed-off-by: Magnus Hedemark <magnus919@pm.me>
2026-09-11 19:47:12 +03:00 · 2026-08-21 16:21:23 -04:00
parent 94b7231147
commit 1291f9576e
4 changed files with 92 additions and 21 deletions
@@ -25,6 +25,16 @@ This skill provides the cross-domain decision spine for evaluating an AI-enabled

 The core question is not “Did the model make people faster?” It is: “What changed in this workflow, for whom, at what full cost, with what outcome and countermetric evidence, and what authority should the organization grant next?”

+## Entry Points
+
+| Starting state | Start with | Primary artifact or route |
+|---|---|---|
+| Idea or proposed AI workflow | Steps 1–2 | `templates/ai-initiative-evidence-record.md` |
+| Existing pilot or outcome data | Steps 3–7 | `references/evidence-method.md` plus the evidence record |
+| Request for broader population or side-effect authority | Steps 7–8; load `references/evidence-method.md` section 7a for the governance packet | Governance evidence packet plus the evidence record |
+| Executive, portfolio, launch, or lifecycle review | Steps 8–9 | `templates/ai-economics-review.md`; route launch/runtime details onward |
+| Standalone financial, statistical, telemetry, runtime, or governance implementation task | When Not to Use | Named adjacent specialist skill |
+
 ## When to Use

 Load this skill when the user needs to:
@@ -39,15 +49,17 @@ Load this skill when the user needs to:

 ## When Not to Use

- **Standalone financial statements, pricing, CAC/LTV, runway, or SaaS metrics:** use [financial-modeling](../financial-modeling/SKILL.md).
- **Token, infrastructure, quota, capacity, or SLO-cost modeling:** use [capacity-and-cost-engineering](../capacity-and-cost-engineering/SKILL.md).
- **Metric trees, event schemas, instrumentation QA, or product dashboards:** use [product-analytics-and-measurement](../product-analytics-and-measurement/SKILL.md).
- **Experimental design, causal inference, statistical testing, or power analysis:** use [data-scientist](../data-scientist/SKILL.md).
- **Agent datasets, graders, traces, regression analysis, or telemetry implementation:** use [agent-evals-and-observability](../agent-evals-and-observability/SKILL.md).
- **Production rollout, runtime budgets, authority, fallback, escalation, or disablement:** use [agent-production-operations](../agent-production-operations/SKILL.md).
- **Organization-wide AI risk, policy, compliance, or governance operating models:** use [ai-governance](../ai-governance/SKILL.md).
- **A launch-readiness packet or production go/no-go decision:** use [production-readiness](../production-readiness/SKILL.md).
- **General product governance cadence without an AI-specific value question:** use [product-operations-and-governance](../product-operations-and-governance/SKILL.md).
+| If the task is primarily... | Route to | This skill still contributes... |
+|---|---|---|
+| Financial statements, pricing, CAC/LTV, runway, or SaaS metrics | [financial-modeling](../financial-modeling/SKILL.md) | The AI workflow's outcome and cost evidence can feed the model |
+| Token, infrastructure, quota, capacity, or SLO-cost modeling | [capacity-and-cost-engineering](../capacity-and-cost-engineering/SKILL.md) | The economic decision can consume the resulting cost boundary |
+| Metric trees, event schemas, instrumentation QA, or product dashboards | [product-analytics-and-measurement](../product-analytics-and-measurement/SKILL.md) | The decision defines which outcome and countermetric evidence matters |
+| Experimental design, causal inference, statistical testing, or power analysis | [data-scientist](../data-scientist/SKILL.md) | The decision specifies the claim and comparison it must support |
+| Agent datasets, graders, traces, regression analysis, or telemetry implementation | [agent-evals-and-observability](../agent-evals-and-observability/SKILL.md) | The decision consumes verified evaluation and telemetry evidence |
+| Production rollout, runtime budgets, authority, fallback, escalation, or disablement | [agent-production-operations](../agent-production-operations/SKILL.md) | The decision sets the evidence and authority boundary |
+| Organization-wide AI risk, policy, compliance, or governance operating models | [ai-governance](../ai-governance/SKILL.md) | The initiative record supplies an operating case and unresolved gaps |
+| Launch-readiness packet or production go/no-go decision | [production-readiness](../production-readiness/SKILL.md) | The initiative disposition becomes one readiness input |
+| General product governance cadence without an AI-specific value question | [product-operations-and-governance](../product-operations-and-governance/SKILL.md) | Use this skill only for the AI-specific value and operating-economics question |

 ## Non-Negotiable Reasoning Rules

@@ -66,6 +78,23 @@ Load this skill when the user needs to:

 Use this sequence for an AI initiative review. Load the detailed method and the evidence-record template when the task requires a durable artifact.

+### Quick Start by Need
+
+| Need | First action | Load next |
+|---|---|---|
+| Triage a claim | Name the workflow, decision, and evidence class | Steps 1–3; evidence classes are defined in Step 7 |
+| Build a durable record | Copy the initiative evidence record and complete the header first | `templates/ai-initiative-evidence-record.md` |
+| Investigate uncertain evidence | Freeze the claim table before drafting conclusions | `references/evidence-method.md` |
+| Prepare a review | Assemble evidence, slices, cost, gaps, and disposition | `templates/ai-economics-review.md` |
+
+### Choose Review Depth
+
+| Mode | Use when | Minimum evidence | Output |
+|---|---|---|---|
+| Triage | A claim or opportunity needs a bounded first decision | Workflow, value hypothesis, one outcome, one countermetric, known gaps | Hold, with a routing/evidence plan |
+| Standard | A pilot or workflow decision can change population or investment | Comparison, outcome/countermetrics, slices, cost boundary, owner, reversal path | Scale, constrain, redesign, or hold |
+| High-assurance | Authority, sensitive data, material user impact, or irreversible change is involved | Standard evidence plus governance packet, human oversight, incident/revalidation, and decommissioning evidence | Scale only within an explicit authority boundary, or Hold |
+
 ### 1. Define the intervention and decision

 Name the workflow, population, task boundary, intervention mode, baseline, decision sought, and decision owner. State whether the AI assists, recommends, routes, executes, or replaces/removes work. Define what remains human-controlled.
@@ -143,6 +172,25 @@ For every material claim, label it:

 Keep the source, access date, scope, version, caveat, and permitted interpretation with each claim. Load `references/source-index.md` for the research basis and evidence boundaries.

+### Minimum Claim Ledger
+
+For each material claim, record: **claim**, **evidence class**, **source and scope**, **what it supports**, **what it does not support**, **open challenge**, and **permitted language**. Keep unknown claims visible; do not let a source URL or vendor report stand in for direct workflow evidence.
+
+### Minimum Decision Record
+
+Every completed review must expose, in one durable artifact: the intervention and population, value hypothesis, primary outcome, countermetrics, comparison and limitations, cost boundary, relevant slices, evidence classes, missing evidence with owner, disposition, authority limit, reversal path, and review trigger.
+
+### Disposition Quick Pick
+
+| Evidence state | Default disposition | Next control |
+|---|---|---|
+| Outcome and countermetrics support a bounded expansion; cost and slices are understood | Scale | Name the next population and authority slice |
+| Value is plausible but a cost, quality, subgroup, or authority boundary remains unresolved | Constrain | Limit population, task, quota, or human review |
+| The mechanism creates avoidable failure or burden | Redesign | Change the workflow or control and rerun the comparison |
+| Required evidence is missing or conflicting | Hold | Assign the evidence owner and review trigger |
+| Value is absent or countermetrics exceed bounds | Retire | Protect affected people, migrate, and record learning |
+| A material gap is accepted temporarily by a named human | Exception | Set expiry, containment, approver, and revisit condition |
+
 ### 8. Produce a bounded decision

 Choose exactly one primary disposition:
@@ -154,6 +202,15 @@ Choose exactly one primary disposition:
 - **Retire:** observed value is absent or countermetrics exceed acceptable bounds, with a transition path.
 - **Exception:** proceed despite a named gap only with an accountable human approver, expiry or revisit trigger, and containment plan.

+### Closure Conditions
+
+- **Scale:** next population, authority slice, owner, and review trigger are recorded.
+- **Constrain:** the boundary, quota, human-review rule, and condition for expansion are recorded.
+- **Redesign:** the changed mechanism, rerun comparison, and new acceptance boundary are recorded.
+- **Hold:** the missing evidence, owner, method, and due trigger are recorded.
+- **Retire:** transition, affected-person protection, decommissioning, and retained learning are recorded.
+- **Exception:** named human approver, scope, expiry, containment, and revisit condition are recorded.
+
 A decision is incomplete without an owner, review date or trigger, evidence gaps, and reversal path. Route launch or runtime consequences to the appropriate specialist skill.

 ### 9. Close the learning loop
@@ -162,12 +219,12 @@ At the review date, compare expected versus observed outcomes, cost, quality, wo

 ## Load-on-Demand References

-| Need | Load |
-|---|---|
-| Apply the full research and decision method, including comparison design and uncertainty | [references/evidence-method.md](references/evidence-method.md) |
-| Review the sources and their permitted interpretations | [references/source-index.md](references/source-index.md) |
-| Fill a durable initiative record before a review | [templates/ai-initiative-evidence-record.md](templates/ai-initiative-evidence-record.md) |
-| Prepare an executive or lifecycle review from one or more initiative records | [templates/ai-economics-review.md](templates/ai-economics-review.md) |
+| Need | Load when | File |
+|---|---|---|
+| Apply the full research and decision method, including comparison design and uncertainty | Evidence is incomplete, contested, or consequential | [references/evidence-method.md](references/evidence-method.md) |
+| Review sources and permitted interpretations | A claim needs provenance or a source boundary | [references/source-index.md](references/source-index.md) |
+| Fill a durable initiative record | Starting a new workflow review or pilot assessment | [templates/ai-initiative-evidence-record.md](templates/ai-initiative-evidence-record.md) |
+| Prepare an executive or lifecycle review | Combining one or more initiative records for a decision | [templates/ai-economics-review.md](templates/ai-economics-review.md) |

 ## Common Pitfalls

@@ -35,9 +35,9 @@ Benefits-realization planning can borrow from public-sector benefits-management

 Build a claim table before writing the conclusion:

-| Claim | Evidence class | Source and scope | What it supports | What it does not support | Open challenge |
-|---|---|---|---|---|---|
-| [claim] | observed / causal / inferred / vendor-reported / asserted / normative | [citation, date, version] | [permitted interpretation] | [boundary] | [test or missing evidence] |
+| Claim | Evidence class | Source and scope | What it supports | What it does not support | Open challenge | Permitted interpretation |
+|---|---|---|---|---|---|---|
+| [claim] | observed / causal / inferred / vendor-reported / asserted / normative | [citation, date, version] | [supported claim] | [boundary] | [test or missing evidence] | [calibrated language] |

 Use the strongest source appropriate to the claim:

@@ -11,6 +11,20 @@ Use this template for an executive, portfolio, experiment, launch, or lifecycle
 - Records reviewed:
 - Decision deadline or trigger:

+## Minimum decision record
+
+- Intervention and population:
+- Value hypothesis:
+- Primary outcome:
+- Countermetrics:
+- Comparison and limitations:
+- Cost boundary:
+- Relevant slices:
+- Evidence classes:
+- Missing evidence, owner, and next trigger:
+- Authority limit:
+- Reversal path:
+
 ## Executive decision

 - Disposition: scale / constrain / redesign / hold / retire / exception
@@ -36,9 +36,9 @@
 - Quality measurement limitations:
 - Statistical or causal analysis owner:

-| Claim | Class | Source | Result | Caveat | Permitted interpretation |
-|---|---|---|---|---|---|
-| | observed / causal / inferred / vendor-reported / asserted / normative | | | | |
+| Claim | Evidence class | Source and scope | What it supports | What it does not support | Open challenge | Permitted interpretation |
+|---|---|---|---|---|---|---|
+| | observed / causal / inferred / vendor-reported / asserted / normative | | | | | |

 ## Outcome and countermetrics