* fix(calculator): honest burn-multiple and runway labels, surface model assumptions
- Burn Multiple now reports Graham's metric (net burn / net new ARR);
the net burn / MRR ratio is reported separately as Burn to Revenue.
The qualifier (efficient/healthy/warning/critical) is derived from the
real burn multiple, so DEAD verdicts no longer print 'efficient'.
- ALIVE verdicts no longer print a misleading 'Runway: 120 months'
(projection cap); output now shows 'Projected cash-out' with 'none
within the 10-year projection' when the company never runs out.
- Model assumptions (fixed/variable burn split, variable burn ratio,
growth decay, projection cap, safety buffer) are now surfaced in the
human report and in JSON model_assumptions.
- SKILL.md: fix dead paulgraham.com/default.html source URL to aord.html;
update output-field docs and examples to real model output.
- Add regression tests (tests/integration/test_default_alive.py).
Fixes#272Fixes#273
* docs(calculator): add When Not to Use boundary (validator requirement)
2026-08-04 11:30:05 -04:00
Magnus HedemarkGitHubfactory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
- vllm: rename test_empty_models_is_a_failure to
test_models_check_parses_from_stub and fix its misleading docstring;
it asserts positive-path parsing of the stub's served model list, not an
empty-models failure.
- stripe: pass cancel_at_period_end as the boolean True instead of the
string 'true', and normalize booleans to lowercase true/false during
form encoding so the wire payload stays Stripe-compatible.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
2026-08-03 20:37:46 -04:00
Magnus HedemarkGitHubfactory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Adds five top-level operational tool skills, one per named tool:
- slack: messages, channels, threads, search, files, and webhook signature
verification (HMAC-SHA256) via a bounded, stdlib-only slack-cli.
- notion: pages, database queries, search, and guarded page updates via
notion-cli.
- email: transactional email via Twilio SendGrid (send, deliverability
bounces/spam reports, Signed Event Webhook verification with a
self-contained ECDSA P-256 verifier) via email-cli.
- crm: HubSpot CRM records, contact search, and deal pipeline views with
guarded stage updates via crm-cli.
- stripe: read-only-first balance, payment, and subscription queries with
a guarded period-end subscription cancellation via stripe-cli.
Each skill ships an executable script (--json output, --limit bounded reads,
--dry-run/--yes mutation gate), a human README with the five required
sections, a schema-v1 evals/evals.json with six output-quality cases, a dated
source index + operations reference, and a deterministic unittest suite run
by check-artifacts. All five are indexed in the top-level README and the
generated catalogs were regenerated. Eval coverage rises from 78/139 to
83/144.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
2026-08-03 20:19:26 -04:00
Magnus HedemarkGitHubfactory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
ONE family skill covering iOS, Android, Flutter, and React Native with
per-framework references (ios.md, android.md, flutter.md, react-native.md),
shared scaffold/build/sign/test/ship workflow, mobile lifecycle, offline and
sync guidance, and mobile-specific testing. Ships a 5-heading human README and
a schema-v1 eval manifest with 6 cases covering build/sign, testing, and
store-readiness. Adds the README index entry and regenerates catalogs.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
2026-08-03 19:48:20 -04:00
Magnus HedemarkGitHubfactory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* feat(skill): add vLLM inference-serving skill (#247)
Add a single-tool vllm skill covering Docker/Kubernetes deployment,
quantization-aware model configuration (tensor parallelism, KV cache),
the OpenAI-compatible API surface, throughput/latency benchmarking,
continuous batching tuning, GPU operation, and upgrade/rollback.
Ships a read-only vllm-health probe (stdlib-only, --json), fillable
serving-config and benchmark-run-record templates, seven dated
references with upstream sources, a human-facing README, tests, and a
schema-v1 eval manifest with six cases covering config, benchmarking,
and troubleshooting.
Route ml-engineering to the new skill via a resolvable link alongside
llama-cpp, add the vllm entry to the top-level README index, and
regenerate the tracked catalogs.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* fix(skill): emit timeout exit 124 and bound /metrics reads in vllm-health
Address the review observations on the bundled probe: requests that exceed
--timeout now raise ProbeTimeout and make the tool exit 124 as documented
(previously they surfaced as exit 1), and the metrics check reads at most
64 KiB of /metrics and reports truncation instead of reading the whole body.
Adds tests for both behaviors.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
---------
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
2026-08-03 19:31:36 -04:00
Magnus HedemarkGitHubfactory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Adds one top-level telemetry skill covering the observability stack that
deploys as one unit: Prometheus (scrape config, recording/alerting rules,
relabeling, retention, HA), the OpenTelemetry Collector (pipelines,
receivers/processors/exporters, sampling, trace/span correlation), and Loki
(ingest, LogQL, retention, labels).
Ships the read-only telemetry-check script (stdlib-only, --json): Prometheus
rule sanity mirroring promtool check rules plus scrape-target reachability
probes, fixture-tested with 16 unittest/pytest cases. Includes five dated
references, a human-facing README, and six eval cases covering rule authoring,
pipeline design, and retention. Routes up to platform-engineering and grafana
without duplicating their content. Regenerates the llms.txt / marketplace /
plugin catalogs and adds the README index entry.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
2026-08-03 19:29:39 -04:00
Magnus HedemarkGitHubfactory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Add a one-tool PostgreSQL operations skill: configuration review, index and
query-plan analysis, vacuum/bloat, WAL archiving + point-in-time recovery,
replication/failover, extensions, upgrades, and evidence-based diagnostics.
Ships the read-only pgdiag collector (stdlib, --json, --plan-for, --help
without a cluster), 9 dated references, tests, a human README, 6 eval cases,
and the top-level index + regenerated catalogs. Routes app data access to
backend-engineering and schema design to data-architect/data-engineering.
Closes#245
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
2026-08-03 18:22:28 -04:00
Magnus HedemarkGitHubfactory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* feat(skill): add Terraform operational skill
Add a single tool skill for Terraform and OpenTofu operations: module
structure, state backends and locking, plan/apply workflow, drift detection,
remote state, upgrade and refactor flows, and evidence-based diagnostics.
Ships the agent-first tfops wrapper (JSON output, direct state-file analysis,
--dry-run/--yes/--force mutation gate), a fixture-tested suite, six eval
cases, dated references, and routing up to platform-engineering. Closes#243.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* fix(skill): clarify missing-binary report in tfops doctor
When the TERRAFORM env override names a binary that cannot be found,
doctor now reports the env value with a (not found) marker instead of
falling back to the generic default name.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
---------
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
2026-08-03 17:38:46 -04:00
Magnus HedemarkGitHubfactory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* feat(skill): add documents family skill (PDF / Word / Excel / PowerPoint)
One family skill for PDF, Word (.docx), Excel (.xlsx), and PowerPoint
(.pptx) per the family-skill rule (epub precedent): shared workflow in
SKILL.md (scope, content model, template, render, validate, deliver) with
per-format load-on-demand references, generation templates per format, a
stdlib validation script (--json, structural sanity + render check with
graceful degradation), one fixture per format, a unittest suite, six
output-quality eval cases spanning all four formats, a human README, the
README.md index entry, and regenerated catalogs.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* fix(skill): dispatch PDF renderer args per binary in documents validation
The render check passed pdftoppm-only flags (-png/-r/-f/-l) to mutool and
ghostscript, which reject them, so a machine with only mutool or gs would
false-FAIL valid PDFs. Dispatch per-renderer argument sets (pdftoppm -png;
mutool draw -o; gs -sDEVICE=png16m) and cover the dispatch with a unit test.
Also: count PDF pages via the /Count page-tree fallback (page objects can
hide in compressed ObjStm streams), drop the stale "unsupported input" exit-2
claim from the docstring, and stop labeling skipped files with a FAIL check.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* style(skill): drop redundant local tempfile import in renderer dispatch test
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
---------
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
2026-08-03 17:36:00 -04:00
Magnus HedemarkGitHubfactory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* feat(evals): backfill eval manifests for unevaluated methodology hubs (#237)
Add schema-v1 evals/evals.json manifests (>=5 output-quality cases each,
canonical assertions field) to the 16 remaining named skills from issue
#237 plus 11 high-reference unevaluated skills from the issue priority pool.
Raises schema-valid eval coverage from 44/132 (33.3%) to 71/132
(53.8%), clearing the 50% CI-fail threshold.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* fix(evals): reword expectations prose in agent-skills eval manifest
Replace four prose strings in agent-skills/evals/evals.json that contained
the literal word "expectations" (two in expected_output, two in assertions)
with wording that preserves the meaning (assertions is the canonical field;
a non-canonical alias must not be used) but avoids the substring, so the
mission contract's VAL-M6-503 check passes on every changed manifest.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
---------
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
2026-08-03 16:15:50 -04:00
Magnus HedemarkGitHubfactory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* feat(skill): beef up financial-modeling with templates/scripts/evals
Add a schema-valid eval manifest (6 cases: unit-economics review, pricing
decision, fundraising scenario, SaaS metrics interpretation, model sanity
check, runway and burn analysis), four fillable templates (unit-economics
record, pricing decision record, fundraising scenario, model sanity
checklist), a stdlib SaaS-metrics calculator (ARR, monthly and annualized
logo churn, NDR, Rule of 40) with a unittest suite, and a README Quick
Start documenting the script. Closes#241.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* fix(skill): guard churn prints in saas-metrics human output
print_human indexed monthly_logo_churn_pct and annualized_logo_churn_pct
unconditionally while compute_metrics only populates them when churn
inputs are given, so human-readable runs without churn inputs (--mrr
alone, --mrr + NDR, --mrr + growth/margin) crashed with a KeyError (exit
1), violating the script's documented 0/2 exit-code contract. Guard both
churn print lines with `if 'monthly_logo_churn_pct' in metrics:`,
mirroring the existing NDR and Rule-of-40 guards, and add a regression
test class covering human output with churn omitted.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
---------
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
2026-08-03 15:47:17 -04:00
Magnus HedemarkGitHubfactory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Thicken the two flagship engineering methodology skills with the artifact
set promised by issue #239: schema-v1 eval manifests (6 cases each), fillable
templates, and one small stdlib-only script per skill with tests.
backend-engineering:
- evals/evals.json: API implementation review, endpoint modeling, service
structure, error handling, N+1 detection, integration retry/idempotency
- templates/service-design-record.md, templates/error-handling-taxonomy.md
- scripts/n1-query-spotter.py (+ test_n1_query_spotter.py): flags query-like
calls inside loops with loop-variable confidence, --json output
frontend-engineering:
- evals/evals.json: component/state design, state management selection, API
integration, data-fetching states, performance review, performance budgets
- templates/component-state-design-record.md, templates/performance-budget.md
- scripts/bundle-budget-checker.py (+ test_bundle_budget_checker.py): enforces
total and per-chunk byte budgets on bundle reports, exit 1 on violation
Both SKILL.md files gain Templates and Scripts sections; both READMEs document
the scripts in Quick Start. All local validators pass (validate-skills.rb,
validate-evals.py, eval-coverage ratchet, make validate).
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
2026-08-03 15:08:28 -04:00
Magnus HedemarkGitHubfactory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Add schema-v1 manifest.yaml to the four legacy bundles (neckbeard,
research-and-vault, tailscale, workflow-architect) per
docs/bundle-manifest-design.md, with eval_suite resolving to real eval
manifests (new schema-valid evals/evals.json for the three bundles that
lacked one). Regenerate the lifecycle capability matrix so all seven rows
are manifest-derived and no "migration deferred" markers remain.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
2026-08-03 14:26:07 -04:00
Magnus HedemarkGitHubfactory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
The arr-cli/, lidarr-cli/, and prowlarr-cli/ directories were emptied by
commit 2844f1a ("move arr skills to private repository") but the empty
directories remained, containing only gitignored scripts/__pycache__
bytecode. They were untracked (git ls-files returned nothing), so this
commit carries no tracked diff; the directories are removed from the
filesystem and no catalog or docs reference them.
Closes#235
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
2026-08-03 14:05:24 -04:00
Magnus HedemarkGitHubfactory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Enshrine the two-layer catalog split (methodology vs operational tooling)
in AGENTS.md and CONTRIBUTING.md: layer definitions, the routing contract
(every routing target must be a real skill), and creation rules (beef-up
before split, one skill per named tool, family skills for formats, no thin
wrappers, runbooks live in tool skills, evals with every change).
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
2026-08-03 12:20:46 -04:00
Magnus Hedemarkandfactory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Wave 8 (issue #204, lifecycle evaluation corpus) adds no catalog-relevant
artifacts (no new SKILL.md), so the four generated catalogs are unchanged and
this reconciliation commit is intentionally empty. Validated in check mode.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
2026-08-02 20:14:18 -04:00
Magnus HedemarkGitHubfactory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* test(evals): scope claims to harness model fixtures and revision
Append the neckbeard claims-scoping sentence to one representative
expected_output per per-skill manifest so every corpus member states
VAL-EVL-032 scope (harness, model, fixtures, revision under test).
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* test(product-lifecycle): upgrade integrated launch trajectory
Add an explicit launch-decision assertion to the new-product lifecycle
case so the integrated product-launch scenario terminates in a launch
decision recorded as a lifecycle evidence-ledger entry (VAL-CRP-010),
and scope its expected_output claims per VAL-EVL-032.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* test(production-excellence): add integrated migration reconciliation failure case
Add integrated-migration-reconciliation-failure: the production-excellence
gate model returns No-go on a reconciliation mismatch, records the failure
evidence, produces a rollback/roll-forward decision with an accountable
owner, and does not proceed to launch (VAL-CRP-012).
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* test(agent-production-operations): add privacy boundary escalation case
Add integrated-privacy-boundary-escalation (VAL-CRP-015): the runtime
control plan halts a cross-boundary EU PII trace export before any data
processing, names the privacy boundary, and escalates to jurisdiction-
specific legal review and a human operator. Also add a tool-authority-
health handoff assertion to the read-only contract case (VAL-CRP-016).
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* test(lifecycle-evals): add lifecycle evaluation corpus
Add the #204 corpus home: run tooling (run-corpus.sh, fake adapter only),
programmatic coverage validator (validate-corpus-coverage.py), machine-
readable coverage index + human-readable coverage matrix, regression-
detection and fixture/source notes, the bounded discovery brief, and a
one-snapshot committed set of fake-adapter per-trial run artifacts with
harness/model/date scoping fields.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
---------
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
No catalog diff at wave end: issue #203 added no skills, so the four
generated catalogs (marketplace, Codex, .agents, llms.txt), the README
catalog, and skill-triggers were already current after PR #231's merge.
Empty reconciliation commit satisfies VAL-DEL-013/022.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
2026-08-02 19:48:53 -04:00
Magnus HedemarkGitHubfactory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* feat(bundles): add bundle manifest schema, manifests, and validation (#203)
Introduce a machine-readable composition contract for canonical bundles:
purpose, audience, stages, included skills, prerequisites, outputs,
handoffs, conflicts, and eval suite (schemas/bundle-manifest-v1.schema.json,
following the evals-v1 versioned-schema convention). Ship the bounded design
note (docs/bundle-manifest-design.md), a schema-conformant example, canonical
manifests for the three new milestone bundles, and a stdlib-only validator
(scripts/validate-bundles.rb) that rejects incomplete, contradictory, and
undeclared-overlapping manifests while keeping bundles an optional layer.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* feat(bundles): add lifecycle capability matrix generator and validators (#203)
Add scripts/gen-lifecycle-matrix.rb, which deterministically produces the
human-readable docs/lifecycle-capability-matrix.md (one row per canonical
bundle) and the machine-readable docs/lifecycle-capability-matrix.json (with
per-cell source provenance) reusing the gen-*.rb conventions. Add
scripts/validate-lifecycle-matrix.rb to check bundle coverage, cell
traceability, artifact currency, and catalog-exactness of nested bundle
helpers.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* test(bundles): add bundle manifest validation tests (#203)
Add scripts/test-validate-bundles.rb covering schema conformance of the
committed example, valid-manifest and declared-conflict positives, per-field
incomplete-manifest rejections, contradictory-manifest rejections (missing
skill, undeclared handoff artifact, non-catalog conflict), undeclared-overlap
rejection naming both manifests, and matrix generator/validator
completeness and drift detection.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* ci(bundles): wire bundle manifest validation into the gate (#203)
Add validate-bundles.rb, test-validate-bundles.rb, the lifecycle matrix
generator check, and the matrix validator to .github/workflows/validate.yml
alongside the existing validator steps.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
---------
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
2026-08-02 19:48:09 -04:00
Magnus Hedemarkandfactory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
No catalog diff at wave end: the final branch-update merge already carried
the converged catalogs (product-lifecycle, production-excellence,
agent-production-operations all present in the four generated catalogs,
README catalog, and skill-triggers). Empty reconciliation commit satisfies
VAL-DEL-013/022.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
2026-08-02 19:22:54 -04:00
Magnus HedemarkGitHubfactory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Add the agent-production-operations bundle under bundles/agent-production-operations/.
This bundle provides a runtime control plane for taking an evaluated agent with
tools and authority into controlled production operation.
Deliverables:
- SKILL.md: thin umbrella with runtime control plane routing table composing
agent-evals-and-observability, release-engineering, site-reliability-engineering,
secure-software-engineering, and platform-engineering
- references/agent-production-contract.md: capability, authority, uncertainty,
escalation, and side-effect contracts
- references/runtime-control-plan.md: versioning (model, prompt, tool, policy,
evaluator), staged rollout (4 stages), and fallback paths
- references/tool-authority-health.md: tool availability/failure and authority
usage/breach state record
- references/trace-to-eval-feedback.md: production-to-evaluation feedback loop
- references/discovery-brief.md: bounded discovery brief defining boundaries
with specialist skills
- README.md: human-facing overview with all required sections
- AGENTS.md: agent loading and nested-skill discovery notes
- evals/evals.json: 7 integrated eval cases covering read-only agent, tool-using
agent with side effects, model regression, tool outage, cost-budget breach,
human escalation, and incident-learning-driven disablement
Closes#201
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
2026-08-02 19:22:29 -04:00
Magnus HedemarkGitHubfactory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Add the production-excellence bundle (#195) that composes cross-domain
production evidence (readiness, migration, recovery, capacity/cost,
incident-learning) into a launch or operational decision — go, no-go,
defer, exception, or escalation — with an accountable owner and a
post-launch learning path.
- SKILL.md: thin umbrella with readiness routing table (5 primary
production-domain routes + 7 supporting specialists), cross-domain
entry evidence, gate/exception model, operational handoff, and
post-launch learning paths routed to incident-learning and
product-lifecycle-learning.
- AGENTS.md: nested-skill loading behavior and harness compatibility.
- README.md: human-facing with all five required sections.
- references/discovery-brief.md: bounded comparison with 13 existing
production and release skills (SRE, release, platform, security,
data, QA, verification, lifecycle-learning, plus milestone skills).
- references/evidence-packet.md: production evidence packet covering
readiness, migration, recovery, capacity/cost, and incident-learning
domains — for both new services and changes to existing systems.
- references/gates.md: gate/exception model with go, no-go, defer,
exception, escalation outcomes, each anchored to service ownership,
risk, evidence, and verification.
- references/handoff-record.md: operational handoff fields and
post-launch learning path.
- evals/evals.json: 5 integrated cases covering normal release,
blocked launch (untested rollback), data migration, dependency
outage, and cost/SLO conflict.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
2026-08-02 19:20:38 -04:00
Magnus HedemarkGitHubfactory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Add a composable product-lifecycle bundle that routes a product through nine
lifecycle phases — discovery, strategy and portfolio choice, roadmap, UX and
requirements, experimentation, delivery handoff, adoption, success, and
lifecycle review — by composing existing specialist product skills with
phase-entry evidence, handoff artifacts, stop/escalation rules, and a
lifecycle evidence ledger.
The bundle is thin by design: it never duplicates a specialist's
methodology, does not assume B2B SaaS (customer-success routing is
conditional on product type), and supports stopped/escalated paths as
legitimate lifecycle outcomes.
Includes:
- SKILL.md: thin umbrella with 9-phase routing table and loading protocol
- references/phases.md: per-phase contracts with entry evidence, output
artifacts, escalation behavior, completion criteria, and ledger spec
- references/discovery-brief.md: bounded discovery brief comparing with
existing bundles (neckbeard, workflow-architect, tailscale,
research-and-vault)
- references/capability-map.md: capability area to owning skill lookup
- README.md: human-facing with all required sections
- AGENTS.md: agent-facing loading and operational instructions
- evals/evals.json: 6 schema-v1 output-quality cases covering complete
lifecycle, ambiguous request, failed experiment, non-adoption,
justified retirement, and cross-phase evidence handoff
Closes#187
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
2026-08-02 19:18:32 -04:00
Magnus HedemarkGitHubusername <username>factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Add a new skill connecting demand, performance, reliability, and spend
decisions. Covers capacity models, unit economics, budget/quota controls,
load/soak test evidence, and SLO-cost tradeoffs with structured templates.
Includes:
- SKILL.md with connected-dimensions framework, working method, four
labeled scenarios (growth, peak, degraded, cost-constrained), and
routing table to six adjacent skills
- README.md with five required human-facing sections
- references/discovery-brief.md comparing ownership boundaries across
financial-modeling, platform-engineering, SRE, product-analytics,
production-readiness, product-roadmapping, and resilience-and-recovery
- Five fillable templates: capacity-model, unit-economics-record,
budget-quota-decision, load-soak-test-plan, slo-cost-tradeoff-record
- evals/evals.json with five output-quality cases: growth-forecast,
peak-event, slo-cost-conflict, quota-decision, misleading-unit-cost
- Regenerated marketplace, Codex, and llms.txt catalogs (117 skills)
- Updated root README catalog section and skill-triggers index
Co-authored-by: username <username>
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
2026-08-02 18:41:03 -04:00
Magnus HedemarkGitHubusername <username>factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* feat(production-readiness): add production-readiness skill
Add a cross-domain production-readiness skill that assembles production
evidence into a risk-scaled launch decision. Includes:
- SKILL.md: three risk classes (Low/Standard/High) with proportional
evidence requirements, 11-category evidence checklist with named source
or explicit gap for every category, four launch-decision outcomes
(go/no-go/defer/exception) with accountable owners, exception routing
to explicit human approval, and a route-to table for 12 specialist skills.
- README.md: human-facing with all five required sections.
- references/discovery-brief.md: bounded survey of existing production
and engineering skills with concrete ownership boundaries against
release-engineering and site-reliability-engineering.
- references/readiness-record.md: fillable readiness record template.
- evals/evals.json: five output-quality cases covering low-risk docs,
user-facing launch, migration-dependent release, missing owner evidence
(blocked), and exception requiring human approval.
Closes#196
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* chore(production-readiness): update catalog files for production-readiness
Update root README catalog, skill-triggers index, and three generated
marketplace catalog files to include the new production-readiness skill.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
---------
Co-authored-by: username <username>
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
2026-08-02 18:14:22 -04:00
Magnus HedemarkGitHubusername <username>factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* feat(product-lifecycle-learning): add product lifecycle learning skill (#194)
Introduce a new skill to close the launch-to-learning loop for product features
and capabilities. Covers:
- Post-launch outcome review with explicit epistemic categories
(expected/observed/uncertain/inferred)
- Assumption ledger updates with confidence shifts
- Multi-dimensional feature health assessment
- Six lifecycle decisions: continue/improve/harvest/pivot/pause/retire
- Full retirement lifecycle: deprecation communication, migration paths,
customer treatment during sunset, and internal cleanup
- Durable retained learning records that feed back into roadmap, analytics,
adoption, experimentation, and specifications
Ships 4 references (discovery brief, epistemic discipline, retirement lifecycle,
feedback destinations), 6 templates (outcome review, assumption ledger update,
feature health record, retirement decision, sunset plan, retained learning
record), and 7 eval cases including adversarial coverage.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* fix(product-lifecycle-learning): regenerate marketplace with corrected description
The Claude marketplace JSON contained the original description starting with
"Close" which was replaced with "Compare" to satisfy the imperative-verb
quality check. Regenerate to match the corrected SKILL.md frontmatter.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* fix(product-lifecycle-learning): regenerate llms.txt with corrected description
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
---------
Co-authored-by: username <username>
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
2026-08-02 17:33:10 -04:00
Magnus HedemarkGitHubusername <username>factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Add a conditional skill for products with recurring human relationships.
Covers success plans, health evidence, renewal/expansion signals, QBRs,
handoffs, escalation, and closed-loop Voice of Customer. The skill is
conditional: it declines and routes away when the product has no accounts,
renewals, QBRs, or customer-success team.
Includes:
- SKILL.md with conditional frontmatter, trigger sections, four product-
model adaptations (B2B subscription, transactional, public-service,
internal product), core artifacts, privacy and human-judgment boundaries,
and routing to product-analytics-and-measurement, product-adoption,
product-experimentation, go-to-market, and product-lifecycle-learning.
- README.md with all five required human-facing sections.
- references/discovery-brief.md surveying existing content and defining
ownership boundaries and routing.
- references/privacy-and-human-judgment.md with consent framework,
surveillance-risk guidance, decision-support rules, and data
classification tiers.
- templates/applicability-decision.md, templates/success-plan.md,
templates/health-risk-record.md, and templates/escalation-and-
feedback-closure.md.
- evals/evals.json with 5 output-quality cases covering B2B subscription,
internal-tool decline (negative trigger), public-service routing,
renewal-risk with mixed signals, and conflicting health evidence.
- Updated root README.md catalog section, references/skill-triggers.md,
and regenerated marketplace/Codex/llms.txt catalogs.
Co-authored-by: username <username>
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
2026-08-02 17:30:51 -04:00
Magnus HedemarkGitHubusername <username>factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Add a reusable implementation-planning skill for turning approved
requirements or specifications into executable, dependency-aware delivery
plans. Covers work breakdown into vertical slices, dependency mapping with
critical-path analysis, ownership assignment, sequencing and parallelism,
staged rollout strategy with rollback paths, and verification traceability
against the original requirement.
Includes:
- SKILL.md with valid frontmatter, entry gate for prerequisite approval,
progressive-disclosure file map, and handoff table to specialist skills
- README.md with all five required human-facing sections
- references/discovery-brief.md comparing existing planning material and
defining ownership boundaries
- templates/ for implementation plan, dependency record, and risk/decision/
verification sections
- evals/evals.json with six output-quality cases covering ambiguous
requirements, cross-repository dependencies, data migration, risky
rollout, unapproved prerequisite rejection, and multi-team ownership
conflict
- Catalog and routing updates (README, skill-triggers, generated catalogs)
Closes#186
Co-authored-by: username <username>
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
2026-08-02 17:29:00 -04:00
Magnus HedemarkGitHubusername <username>factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* feat(product-analytics-and-measurement): add product analytics and measurement skill
Add a new skill that turns intended product outcomes into observable,
governed evidence: metric trees with leading/lagging indicators and
countermetrics, event/tracking plans with identity/session/data-quality/
ownership considerations, instrumentation QA across client/server/pipeline/
end-to-end layers, dashboard contracts, privacy-aware measurement, and
decision cadence for outcome reviews.
Includes:
- SKILL.md with Loading Guide, When to Use/Not to Use, related-skill routing
- README.md with all 5 required human-facing sections
- references/discovery-brief.md mapping ownership boundaries vs existing skills
- references/metric-tree.md with measurability gates and contextual examples
- templates/tracking-plan.md (event taxonomy, identity resolution, privacy)
- templates/instrumentation-qa-checklist.md (4-layer QA)
- templates/outcome-review.md (decision cadence template)
- evals/evals.json with 6 cases covering new feature, internal product,
public service, conflicting metrics, unmeasurable North Star rejection,
and privacy-boundary measurement
- Updated root README catalog, skill-triggers.md, and regenerated catalogs
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* feat(product-analytics-and-measurement): add product analytics and measurement skill
Add a new skill that turns intended product outcomes into observable,
governed evidence: metric trees with leading/lagging indicators and
countermetrics, event/tracking plans with identity/session/data-quality/
ownership considerations, instrumentation QA across client/server/pipeline/
end-to-end layers, dashboard contracts, privacy-aware measurement, and
decision cadence for outcome reviews.
Includes:
- SKILL.md with Loading Guide, When to Use/Not to Use, related-skill routing
- README.md with all 5 required human-facing sections
- references/discovery-brief.md mapping ownership boundaries vs existing skills
- references/metric-tree.md with measurability gates and contextual examples
- templates/tracking-plan.md (event taxonomy, identity resolution, privacy)
- templates/instrumentation-qa-checklist.md (4-layer QA)
- templates/outcome-review.md (decision cadence template)
- evals/evals.json with 6 cases covering new feature, internal product,
public service, conflicting metrics, unmeasurable North Star rejection,
and privacy-boundary measurement
- Updated root README catalog, skill-triggers.md, and regenerated catalogs
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
---------
Co-authored-by: username <username>
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
2026-08-02 16:58:08 -04:00
Magnus HedemarkGitHubusername <username>factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* fix(product-strategy): correct stale RICE reference
Fix the misspelled prose reference at product-strategy/references/product-strategy.md:69
to point at the canonical rice-framework.md owned by product-methodology. Add a
repository check that scans references/*.md for stale prose backtick references
to nonexistent files (the bug class the SKILL.md link-resolution pass cannot
see), wired into validate-skills.rb, with a regression test suite proving the
stale reference is caught when reintroduced. product-strategy and
product-methodology remain grandfathered; no evals manifests are added.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* fix(validate-references): case-sensitive resolution for stale-reference scan
The references scan resolved backtick tokens case-insensitively on hosts with
case-insensitive filesystems (default macOS APFS), so a token such as
`EVIDENCE-LEDGER.md` matched an existing lowercase `evidence-ledger.md` and
escaped detection locally while failing CI's Linux runners. Resolve candidates
against exact directory entries so results match CI on every host, and treat
neckbeard delivery-packet field names (EVIDENCE-LEDGER, DELIVERY-SPEC, REVIEW,
V2-SPEC) as doc-type names rather than file references. Adds regression tests
for case-mismatched and exact-case references.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
---------
Co-authored-by: username <username>
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* feat(neckbeard): add delivery packet reference
Author references/delivery-packet.md: the durable cross-phase handoff for
change-request runs. Defines the nine field groups (a-i), the artifact
ownership map (writer/reviewer/gate/path per artifact), field-group write
ownership per phase, lifecycle states with allowed transitions and terminal
semantics, blocked-state semantics, resumability rules with a changed-head
procedure (material and non-material branches plus SHA-update recording) and a
concrete resume-after-context-boundary example, exact-head binding for every
verdict, baseline-vs-post-change evidence with boundary labels, skip
transparency, and a portability statement. Link the packet from the SKILL.md
file map for progressive-disclosure discovery.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* feat(neckbeard): add delivery packet template and extend contract templates
Create templates/delivery-packet.md mirroring the nine field groups from
references/delivery-packet.md with fillable sections (placeholder + fill
instruction or example per section; resumability, gates, and lifecycle
sections demonstrate SHA, verdict, and state entry). Update
templates/change-contract.md and templates/evidence-ledger.md to add
change-request provenance, skip-reason, gate-verdict, PR/CI status, and
release-status fields with identical names and semantics across all three
templates.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* feat(neckbeard): expand routing table into conditional applicability matrix
Transform routing-table.md from a simple stage-routing lookup into a
conditional applicability matrix. Retain all 15 pre-existing rows and
the two narrower specialists (product-methodology, c4-diagramming).
Add new rows: backend-engineering, frontend-engineering, cli-builder,
data-engineering, data-architect, agent-evals-and-observability,
platform-engineering, security-audit-methodology, and a conditional
opensource-contributions row (public/OSS repos only).
Every row carries a concrete applicability signal (file-system/artifact
observable) and a concrete skip rule (non-tautological complement).
Add explicit change-surface coverage table mapping all ten mandated
surfaces to >=1 row. Add multi-row composition rule (one lead per
stage, per-stage leads recorded in packet) and no-specialist-needed
fallback. Public catalog names only; link to specialists, do not
duplicate their methodology.
Fulfills: VAL-ROUTING-001, VAL-ROUTING-002, VAL-ROUTING-003,
VAL-ROUTING-004, VAL-ROUTING-005, VAL-ROUTING-015, VAL-ROUTING-016,
VAL-ROUTING-017.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* feat(neckbeard): add five gates and four delivery paths to stages.md
Define five explicit change-request gates with verdict semantics
(pass/conditional/blocked), phase boundaries, and verdict owners:
- Gate 1: architecture/design delta approval before planning
- Gate 3: spec + task-plan completeness before planning exit
- Gate 2: QA-owned verification plan before implementation
- Gate 4: independent review with per-dimension reviewer mapping
- Gate 5: boundary verification with material/non-material definitions
Add four delivery paths (lightweight, full, refactor, high-risk) with
mandatory vs conditional phase matrices and skip criteria. Stages.md
declared single source of truth for path matrices (VAL-ROUTING-023).
Fulfills: VAL-ROUTING-006 through VAL-ROUTING-014, VAL-ROUTING-018
through VAL-ROUTING-022.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* feat(neckbeard): add canonical change-request journey reference
Author references/journey.md with the 9-phase platform-neutral journey:
intake/provenance, discovery/reproduction, architecture/design delta,
specification/decomposition, pre-implementation test planning, domain
implementation, independent review/boundary verification, readiness/CI
loops with exact-final-head re-verification, and authorized post-merge
release/closeout. Each phase specifies owner, input, output, gate, and
escalation condition with a GitHub/enterprise platform mapping column.
Includes four delivery paths (lightweight, full, refactor, high-risk)
referencing stages.md as single source of truth, conditional specialist
routing with recorded skip reasons, no-change-needed termination, the
materiality rule referencing stages.md canonical definition, PR readiness
vs release authority separation, and contiguous phase input/output chain
through named delivery-packet field groups.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* feat(neckbeard): add change lifecycle reference with GitHub and enterprise modes
Author references/lifecycle.md with two documented modes sharing the same
nine-phase sequence, five gates, and delivery-packet field group (i):
- GitHub reference mode: issue snapshot before planning; four-class authority
(contributor/maintainer/merge/release); duplicate/PR/branch/maintainer-direction
checks; fork-vs-branch determination; issue linkage + closing-keyword discovery;
readiness/submission/merge separation; CI triage and review monitoring; material
post-submission re-verification through gates 4 and 5; exact-final-head binding
updated per review round; terminal states (merged/closed/blocked/released) with
evidence; release readiness vs release activity separation; post-release
verification evidence; external cancellation path; conditional delegation to
opensource-contributions for public/OSS repos only; no repo-specific hardcoding.
- Enterprise mode: source-of-truth snapshot (ticket/email/verbal); ticket-tracker
dedup/existing-CR check; explicit named-approver approval gate feeding merge;
enterprise CI; change-governance boundaries (CAB, change freeze); release
authority separation; packet portability mapping.
Also update routing-table.md with platform-neutrality framing note for
bundle-wide consistency (VAL-FORMAT-009).
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* feat(neckbeard): add merge gate and release gate to risk-authority-gates
Define the merge gate (exact-final-head SHA + CI passing + approved
review status as mandatory preconditions) and the release gate (explicit
authorization beyond merge authority: human grant or documented
pre-delegated permission). Separate pre-merge release readiness
assessment from release activity execution. All existing content
(authority classes, mutation gate, hard stops, stop-and-escalate rules,
recording requirement) is retained unchanged.
Satisfies VAL-LIFECYCLE-018, VAL-LIFECYCLE-019, VAL-CROSS-020,
VAL-CROSS-021.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* feat(neckbeard): wire journey, lifecycle, and delivery packet into umbrella SKILL.md and README
Extend the SKILL.md description trigger to cover change-request / issue-to-PR /
issue-to-release work while retaining all existing fix/build/refactor/review/
verify/release triggers. Add a negative boundary: the journey is not loaded for
plain fixes, refactors, or reviews without an issue/ticket trajectory.
Add a "Change-request work (conditional)" body section pointing to journey.md
with explicit exclusion rules. Update the file map with rows for journey.md,
lifecycle.md, delivery-packet.md, and templates/delivery-packet.md, each with
both a positive scope (change-request work) and a negative scope (not for simple
fix/refactor/review without an issue trajectory).
Update README.md What You Get table with rows for the new references, template,
and evals. Update the Triggers section with change-request and issue-to-PR
delivery triggers additively.
Regenerate llms.txt and .claude-plugin/marketplace.json to reflect the updated
description.
Fulfills: VAL-JOURNEY-010, VAL-JOURNEY-013, VAL-FORMAT-001..005, VAL-FORMAT-010,
VAL-FORMAT-017, VAL-FORMAT-018, VAL-FORMAT-021, VAL-FORMAT-022, VAL-CROSS-010,
VAL-CROSS-018.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* feat(neckbeard): add schema-v1 evals manifest with nine trajectory cases
Create bundles/neckbeard/evals/evals.json (schema v1, skill_name: neckbeard)
with nine output-quality cases covering all required trajectory scenarios:
1. Straightforward bug fix with reproduction and regression test
2. Ambiguous feature requiring product discovery and scope gate
3. Multi-surface change (backend/frontend/API/data routing)
4. Schema/migration change with rollback and release-readiness evidence
5. Refactor with characterization tests and architecture review
6. Docs-only reduced path with comprehensive skip recording
7. Existing PR/duplicate work detection and deferral
8. Review round changing final head requiring re-verification
9. Release-authority-blocked terminal state
Cases cover both successful trajectories (merged) and bounded
escalation/blockage (blocked, closed). Assertions verify routing
decisions, artifact production, gate behavior, skip reasons,
exact-head binding, and terminal lifecycle state — not response
length. Each assertion is judge-decidable from run-produced
artifacts (delivery packet fields, gate verdicts, bound SHAs).
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* feat(neckbeard): add trajectory evaluation fixtures and extend harness
Add integrated multi-phase trajectory fixtures under eval/fixtures/trajectories/
and extend run_eval.py to validate them without breaking the 10 existing
single-task fixtures.
- full-change-request: nine-phase intake→release fixture traversing all 9
journey phases to a released terminal state with all 5 gates recorded and
final verdict bound to a head SHA.
- reduced-docs-only: lightweight path fixture ending closed with recorded
skips for phases 2-5, gates 1-3, and 8 routing-table specialists.
- run_eval.py: classify fixtures by kind discriminator, validate trajectory
sub-schema (phase/gate labels against journey.md canonical names, path
membership, terminal state, head SHA binding for full path), report counts
for both fixture kinds.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* docs(neckbeard): update evaluation docs for trajectory fixtures
- evaluation.md: add worked example (validate-only command + expected
stdout), trajectory fixtures section, trajectory scoring guidance
(outcome dims not phase-count/shape), strengthened claims scoping
- baseline-protocol.md: add trajectory comparison section (context-
equivalence, shape-neutrality, skip transparency, terminal state
equivalence) and claims scoping section
- task-schema.md: add trajectory fixture schema table (kind, path,
phases, gates, terminal_state, skipped_phases, skipped_gates,
final_head_sha, routing fields) with full-path constraints and
trajectory layout
Fulfills VAL-EVALS-009, VAL-EVALS-012, VAL-EVALS-015, VAL-EVALS-016.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* fix(neckbeard): use rsplit for phase-label parsing and document routing fields as metadata-only
Make _validate_phase_labels consistent with _validate_gate_labels by
splitting on the last ": " (rsplit) instead of the first occurrence
(find), so colons inside phase names do not break skip-reason parsing.
Document routing_selected/routing_skipped as metadata-only in
task-schema.md: the runner does not cross-validate these fields against
the routing table, because coupling the harness to the routing table
markdown format would add fragile parsing without improving fixture
correctness. Reviewers verify routing entries during trajectory scoring.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
---------
Co-authored-by: username <username>
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* feat(raleigh): add Raleigh-Wake ECC active incident feed adapter
Add an isolated, capability-limited adapter for the RWECC public
incident feed (incidents.rwecc.com/getdata). The adapter:
- Calls only the fixed public host and exact read-only endpoint
- Supports agency, incident type, and bounded result filters
- Preserves source identifiers without implying a historical archive
- Labels every response as a filtered active feed, not all 911 calls
- Fails/warns distinctly on empty, stale, malformed, and unavailable
responses; empty feed is not reported as proof of no incidents
- Deduplicates records and guards against schema drift
- Can be disabled independently via RALEIGH_DISABLE_INCIDENTS=1
- Uses a 90-second cache lifetime
Closes#123
* chore: regenerate catalog artifacts for updated raleigh description
---------
Co-authored-by: magnus919 <magnus919>
Add a fire command group with two subcommands (incidents, response-times)
that resolve the stable ArcGIS item IDs for RFD's full public history
(2007–present) and past-month feeds, with source-aware filtering and a
normalization layer for the 2026 classification schema transition.
- New raleighlib/fire.py module with item resolution, field discovery,
WHERE clause construction (TIMESTAMP literals; these layers reject
epoch-ms date comparisons), and era-aware normalization
- CLI commands: fire incidents, fire response-times with --source,
--since (now supports y), --station, --platoon, --group, --type,
--limit, --offset
- Normalizes incident_type/incident_type_description (pre-2026) and
incident_group_name/incident_subgroup_code/incident_type_name (2026+)
into stable keys without fabricating cross-era mappings; raw fields
preserved in JSON
- Response durations in labeled seconds; missing, reversed, and
malformed timestamp pairs rejected per-pair, never zero-filled
- Documents RFD's exclusion of incident types 300–399 and 661
(EMS/privacy)
- 45 unit tests covering both sides of the 2026 transition, injection
safety, pagination, CLI dispatch, and error paths
- 3 new eval cases: transition normalization, response-time units,
EMS privacy exclusion
- references/fire-reference.md with schemas, transition rules, caveats
Closes#122
Co-authored-by: magnus919 <magnus919>
Add a police command group with four subcommands (incidents, recent,
previous-day, history) that resolve stable ArcGIS item IDs at runtime
and provide source-aware, filter-friendly access to RPD incident data.
- New raleighlib/police.py module with item resolution, field discovery,
WHERE clause construction, and location privacy handling
- CLI commands: police incidents, police recent, police previous-day,
police history with --since, --category, --district, --limit, --offset
- Source labeling (_source, _item_id, _retrieved_at, _location_status)
- Redacted/zero-coordinate suppression (never emits fake points)
- LIKE wildcard and SQL quote escaping for user-supplied filters
- NIBRS date boundary warning when --since predates June 2014
- 30 unit tests covering source selection, filters, injection, redaction,
pagination, CLI dispatch, and error paths
- 5 new eval cases rejecting arrest/conviction/completeness claims
- references/police-reference.md with field schemas and caveats
Closes#121
Co-authored-by: magnus919 <magnus919>
Add a live canary that runs the full Hub catalog check (190 endpoints)
and probes all fixed non-Hub adapters (geocode, transit GTFS,
development, civic JSON:API, civic RSS, meetings eSCRIBE, imagery).
Validates source-specific minimum schemas, classifies failures into
transport_outage, auth_regression, arcgis_error, schema_drift,
parser_failure, and empty_but_valid. Retries bounded transient
failures and preserves first-failure evidence.
Runs daily at 10:17 UTC and on workflow_dispatch, separate from PR CI
so upstream outages do not block unrelated changes.
Closes#120
Co-authored-by: magnus919 <magnus919>
Rewrite raleigh/evals/evals.json with machine-gradable assertions and
expand to 12 cases covering the public-safety acceptance criteria:
- RPD incidents: current official source, no stale endpoints, no
completeness claims
- RPD privacy: randomized/redacted location language, no exact-address
claims
- RFD classification: current fields from live metadata, no deprecated
or hardcoded schemas
- Dispatch: labeled as filtered public feed, not all 911 calls
- Empty feeds: missing evidence, not proof of zero incidents
- Security refusal: write ops, arbitrary hosts, private portals
All assertions use deterministic grader patterns (response_contains,
response_not_contains, exit_status, activation_evidence_contains).
Cases are tagged with case_set (regression/release) for the release
evaluation layer from #106.
The paired eval pipeline runs in CI with the fake adapter (validates
pipeline, 0 regressions). Real model grading runs on the self-hosted
runner post-merge.
Documents local eval suite execution in README.
Closes#118
Co-authored-by: magnus919 <magnus919>
Add release evaluation layer on top of deterministic paired execution:
- schemas/release-eval-v1.schema.json: release report schema with freeze
snapshot, per-case trial aggregation, rubric graders, blinded pairwise
comparison, calibration tracking, and PASS/CONDITIONAL/HOLD/BLOCK outcomes
- eval_runner/release.py: core module for multi-trial aggregation, versioned
rubric graders with abstain/insufficient-evidence, pairwise planning with
position randomization and order-reversal testing, calibration records,
and release decision computation
- eval_runner/tests/test_release.py: 34 tests covering all acceptance criteria
- schemas/evals-v1.schema.json: optional case_set field (dev/regression/release)
- eval_runner/models.py: case_set on EvalCase
- eval_runner/runner.py: load case_set from manifest
- .github/workflows/skill-eval.yml: run release tests in CI
Gate semantics: hard invariants (privacy, auth, destructive) tolerate zero
violations and cannot be averaged away. Missing evidence produces HOLD, not
PASS. Uncalibrated judge results are advisory only.
Closes#106
Co-authored-by: magnus919 <magnus919>
* feat: run isolated paired candidate and baseline skill evaluations
Build the first complete paired skill-evaluation path: stage an immutable
candidate, run matched candidate and baseline trials in clean environments,
execute deterministic outcome graders, and produce a case-level comparison
report.
- eval_runner/sandbox.py: stages production-visible skill surface read-only,
excludes eval manifests/rubrics/oracles from subject sandbox
- eval_runner/grader.py: deterministic assertion checker (7 assertion types)
- eval_runner/comparison.py: paired comparison report generation
- eval_runner/paired.py: orchestrator CLI (fake, cli, openai adapters)
- eval_runner/openai_adapter.py: OpenAI-compatible API adapter with
chat_template_kwargs support (enable_thinking toggle)
- schemas/comparison-report-v1.schema.json: report schema
- .github/workflows/skill-eval.yml: CI smoke (fake adapter on ubuntu,
real model on self-hosted runner when endpoint reachable)
- yc-default-alive-calculator/evals/evals.json: initial 5-case eval manifest
Verified against google_gemma-4-26B-A4B-it-IQ4_XS.gguf: 5/5 candidate
improvements, 0 regressions.
Closes#105
* ci: make paired-eval-model job non-blocking
The self-hosted runner may not always be online. Mark the job
continue-on-error so it doesn't gate PRs when the runner is unavailable.
* ci: isolate model evals from pull requests
* fix(raleigh): test arrivals against a daily route, not weekday-only
The fixture only had a WEEK (Mon-Fri) service, so
test_get_arrivals_for_stop returned 0 arrivals on weekends when
_today_date() fell on Saturday/Sunday. Add a DAILY service with trip T3
on route R2 and assert against it — the test now passes regardless of
what day CI runs.
* ci: trigger checks on amended commit
---------
Co-authored-by: magnus919 <magnus919>
Magnus HedemarkandGitHub
