Evidence Ledger

Intent

Implement the complete Grafana subject-matter-expert Agent Skill requested by issue #152, including discovery, dashboard engineering, alerting/routing, provisioning/GitOps, troubleshooting, and security/change control without collapsing Grafana into broader SRE or infrastructure guidance.

Authority

The user granted modify, publish, and merge authority for this repository, including commit, push, PR creation, CI/review follow-through, and merge. Changes to the live Grafana instance were not requested.

Inspected artifacts

Issue magnus919/agent-skills#152, repository AGENTS.md, CONTRIBUTING.md, Agent Skills specification/guidance, eval schema, validators, catalogs, and CI integration.
Comparable llama-cpp, supabase, restic, and kubernetes skill structures.
Official Grafana provisioning, dashboard, alerting, API, as-code, service-account, permissions, security, troubleshooting, source, and release material listed in references/source-index.md.
Read-only runtime evidence from Grafana 11.6.14+security-04 on saru, including Compose topology, health, provisioning metadata, protected API statuses, and bounded logs.

Assumptions

The repository's schema-version-1 eval contract remains authoritative.
A reference-only first version is preferable to a wrapper CLI because API and authentication behavior is target/version-specific.
Runtime evidence from saru is reusable as a failure pattern but does not establish universal Grafana behavior.

Alternatives rejected

Reducing the six required capabilities to a narrower first version: rejected because it violates issue acceptance criteria.
Expanding SRE/platform skills instead: rejected because Grafana has coherent product-specific APIs, ownership, and failure modes.
Bundling Prometheus/Loki/Tempo operation or generic Compose/Kubernetes/Terraform guidance: rejected as overlap with specialist skills.
Directly fixing the duplicate providers on saru: outside granted authority and not required to implement the repository skill.

Files changed

Added grafana/SKILL.md, README.md, this ledger, seven focused references, and an eight-case eval manifest.
Updated root README.md and references/skill-triggers.md.
Regenerated .claude-plugin/marketplace.json, .codex-plugin/plugin.json, and llms.txt; .agents/plugins/marketplace.json remained unchanged.

Commands / checks run

ruby scripts/validate-skills.rb
ruby scripts/test-validate-skill-quality.rb
ruby scripts/validate-skill-quality.rb --base origin/main
python3 scripts/test-eval-validation.py
python3 scripts/validate-evals.py
python3 scripts/test-eval-coverage.py
python3 scripts/eval-coverage.py --modified-from origin/main
python3 scripts/check-artifacts.py
Claude, Codex, and llms.txt generator write/check modes plus ruby scripts/test-gen-llms-txt.rb
python3 -m eval_runner.paired grafana/evals/evals.json --adapter fake --output-dir /var/folders/gn/gpr8z9bn72z5kqm_fmjndj180000gn/T/opencode/grafana-eval-fake
git diff --check
skills-ref validate ./grafana was attempted but skills-ref is not installed.
A broad python3 -m unittest discover -s tests -p 'test*.py' was attempted but imported unrelated /Volumes/tank01/magnus/git/hermes-cashew tests and failed on missing external agent/cron modules; the repository's canonical check-artifacts.py test discovery passed.

Observed outputs

Live Grafana health and duplicate-provider evidence are recorded in references/source-index.md and references/troubleshooting.md.
Structural validation accepted 111 canonical skills.
Changed-skill quality checked grafana with 0 errors and 0 warnings; its 19-test validator suite passed with 159 assertions.
All 12 present eval manifests passed schema-v1 and semantic validation; 27 eval-validation tests and 25 coverage tests passed.
Artifact checks ran 368 repository tests successfully; generated catalogs are current at 101 public skills/plugins.
The fake paired runner exercised all eight Grafana cases with 40 manual assertions per arm. This proves runner plumbing only; it reported no measurable candidate/baseline delta because the fake adapter cannot grade the manual assertions.
Independent scope review confirmed all six issue capability areas. Independent factual review findings about multiple policy trees, direct-contact-point routes, deepest-match semantics, OSS RBAC, bounded live claims, and matching eval assertions were corrected.

Verification boundary

Research: official primary sources plus one read-only live Grafana target.
Component/integration: Agent Skills structure, links, README, eval schema, quality rules, artifact checks, and generated catalogs passed repository validation.
Behavioral: portable eval cases and fake-adapter runner plumbing passed; real-model grading is not established.
Live: read-only host/container discovery was exercised; authenticated API and notification paths were not.

Unverified boundaries

Authenticated dashboard/data-source/alert/RBAC inventory and representative queries on saru.
Alert firing, policy selection, receiver delivery, and resolved notification behavior.
Real-model paired eval improvement and release-gated evidence.
Publication, CI, and merge.

Rollback / follow-up triggers

Revert if repository validation cannot pass without weakening issue scope or safety gates.
Refresh on Grafana API/schema, provisioning ownership, alerting, RBAC, or as-code lifecycle changes.
Add executable discovery tooling only if repeated eval traces demonstrate a stable, error-prone procedure worth maintaining.

Status

Local implementation and repository integration verification passed. Real-model eval evidence and authenticated live Grafana checks remain unverified. Publication, CI, and merge are the active delivery stage. No live Grafana state was changed.

5.8 KiB Raw Blame History