mirror of
https://github.com/magnus919/agent-skills.git
synced 2026-09-11 19:47:12 +03:00
docs(telemetry): deepen bounded PromQL and LogQL workflows
Squash-merge verified telemetry query depth at exact head 02d16c58a4. Required validate and paired evaluation checks passed; advisory droid review had no blocking findings.
This commit is contained in:
Magnus Hedemark
@@ -1245,7 +1245,7 @@
|
||||
"./telemetry"
|
||||
],
|
||||
"strict": false,
|
||||
"description": "Operate the observability stack that deploys as one unit: Prometheus scrape configuration, recording and alerting rules, relabeling, retention, and high availability; OpenTelemetry Collector pipelines (receivers, processors, exporters, sampling, trace/span correlation); and Loki ingest, LogQL, retention, and label design — with a bundled read-only telemetry-check script for Prometheus rule sanity and scrape-target reachability. Use when running, tuning, or troubleshooting a Prometheus, OpenTelemetry Collector, or Loki deployment, or reviewing the collection/ingest/retention layer. Do not use for observability strategy, SLI/SLO design, or paging policy (that is platform-engineering) or Grafana dashboards, panels, and Grafana-side alerting (that is grafana)."
|
||||
"description": "Operate the observability stack that deploys as one unit: Prometheus scrape configuration, recording and alerting rules, relabeling, retention, and high availability; OpenTelemetry Collector pipelines (receivers, processors, exporters, sampling, trace/span correlation); and Loki ingest, LogQL, retention, and label design — with a bundled read-only telemetry-check script for Prometheus rule sanity and scrape-target reachability. Use when running, tuning, or troubleshooting a Prometheus, OpenTelemetry Collector, or Loki deployment, or reviewing the collection/ingest/retention layer, including bounded PromQL and LogQL query construction, semantic review, and no-data diagnosis. Do not use for observability strategy, SLI/SLO design, or paging policy (that is platform-engineering), Grafana dashboards, panels, and Grafana-side alerting (that is grafana), or Tempo/tracing operations, which remain deferred to a future named-tool skill."
|
||||
},
|
||||
{
|
||||
"name": "tempest",
|
||||
|
||||
@@ -139,7 +139,7 @@
|
||||
- [tailscale](tailscale/SKILL.md): Deploy and manage the self-hosted Tailscale/Headscale ecosystem: a Headscale control server, tailscale clients, ACL policies, node lifecycle, subnet routing, DERP relays, and backup/migration. Use when the user mentions Tailscale, Headscale, tailnet, mesh VPN, WireGuard mesh, or self-hosted VPN infrastructure. Do not use this skill for unrelated requests; route to the nearest named specialist.
|
||||
- [technical-documentation](technical-documentation/SKILL.md): Create and review technical documentation, including READMEs, agent-facing instructions, API references, and CLI help. Use when documentation must help someone complete real work. Do not use for marketing copy, brand messaging, or long-form editorial content.
|
||||
- [technology-radar](technology-radar/SKILL.md): Build and maintain technology radars for adoption, trial, assessment, and hold decisions, and choose proportionate architecture-governance paths for technology portfolios. Use when governing technology choices, build-versus-buy decisions, architecture standards, exceptions, or engineering portfolio risk. Do not use for enterprise capability or target-state architecture, writing ADRs, implementing systems, security engineering, or operational incident/runbook work.
|
||||
- [telemetry](telemetry/SKILL.md): Operate the observability stack that deploys as one unit: Prometheus scrape configuration, recording and alerting rules, relabeling, retention, and high availability; OpenTelemetry Collector pipelines (receivers, processors, exporters, sampling, trace/span correlation); and Loki ingest, LogQL, retention, and label design — with a bundled read-only telemetry-check script for Prometheus rule sanity and scrape-target reachability. Use when running, tuning, or troubleshooting a Prometheus, OpenTelemetry Collector, or Loki deployment, or reviewing the collection/ingest/retention layer. Do not use for observability strategy, SLI/SLO design, or paging policy (that is platform-engineering) or Grafana dashboards, panels, and Grafana-side alerting (that is grafana).
|
||||
- [telemetry](telemetry/SKILL.md): Operate the observability stack that deploys as one unit: Prometheus scrape configuration, recording and alerting rules, relabeling, retention, and high availability; OpenTelemetry Collector pipelines (receivers, processors, exporters, sampling, trace/span correlation); and Loki ingest, LogQL, retention, and label design — with a bundled read-only telemetry-check script for Prometheus rule sanity and scrape-target reachability. Use when running, tuning, or troubleshooting a Prometheus, OpenTelemetry Collector, or Loki deployment, or reviewing the collection/ingest/retention layer, including bounded PromQL and LogQL query construction, semantic review, and no-data diagnosis. Do not use for observability strategy, SLI/SLO design, or paging policy (that is platform-engineering), Grafana dashboards, panels, and Grafana-side alerting (that is grafana), or Tempo/tracing operations, which remain deferred to a future named-tool skill.
|
||||
- [tempest](tempest/SKILL.md): Query hyper-local weather from a WeatherFlow Tempest station over its REST API and the hub's local UDP broadcast: current conditions, forecast, historical observations, and real-time decoded datagrams (obs_st, rapid_wind, evt_precip, evt_strike, hub_status). Use when the user asks about weather, temperature, rain, wind, humidity, or forecast data from their own Tempest/WeatherFlow station, or wants to parse the hub's UDP port 50222 broadcast. Do not use this skill for generic or city forecasts without a Tempest station (public weather services serve those), for Shakespeare's play The Tempest or other literature questions, or for weather hardware from other vendors - the REST endpoints require a personal-use token and the UDP broadcast only exists on a Tempest hub's LAN.
|
||||
- [terraform](terraform/SKILL.md): Operate Terraform and OpenTofu across the whole infrastructure lifecycle: module structure, state backends and locking, plan/apply workflow, drift detection, remote state, upgrade and refactor flows, and evidence-based diagnostics. Use when running or inspecting terraform plans, applies, state files, imports, or state surgery, or when the bundled tfops script should handle the task. Do not use for IaC methodology or cloud design decisions - those route up to platform-engineering.
|
||||
- [three](three/SKILL.md): Build browser-based Three.js and WebGL scenes, animations, and interactive 3D visualizations with a small vanilla JavaScript starting point. Do not use this skill for unrelated requests; route to the nearest named specialist.
|
||||
|
||||
+2
-2
@@ -4,7 +4,7 @@ Operate the observability stack that deploys as one unit: Prometheus scrape conf
|
||||
|
||||
## Why Install This Skill
|
||||
|
||||
Your agent can run the collection/ingest/retention layer of observability instead of guessing: review and fix Prometheus scrape configs, author and sanity-check recording and alerting rules, design OpenTelemetry Collector pipelines with deliberate sampling, tune Loki ingest and retention, and diagnose the classic failure modes — missing series, silent ingest loss, and exploding label cardinality — in a fixed evidence order.
|
||||
Your agent can run the collection/ingest/retention layer of observability instead of guessing: review and fix Prometheus scrape configs, author and sanity-check recording and alerting rules, design OpenTelemetry Collector pipelines with deliberate sampling, tune Loki ingest and retention, and diagnose the classic failure modes — missing series, silent ingest loss, and exploding label cardinality — in a fixed evidence order. It can also construct bounded PromQL and LogQL investigations, separate syntax from semantic validation, and explain empty or partial results without turning missing data into zero.
|
||||
|
||||
It ships a read-only checker (`telemetry-check`) that parses Prometheus rules files with a bundled stdlib YAML reader and runs sanity checks mirroring `promtool check rules`, then probes scrape-target reachability with TCP connects. It cannot mutate anything: no config writes, no reloads, no data sent anywhere. That makes it safe for an agent to run during discovery.
|
||||
|
||||
@@ -15,7 +15,7 @@ The references are distilled from the official Prometheus, OpenTelemetry Collect
|
||||
| Directory | Purpose |
|
||||
|---|---|
|
||||
| `SKILL.md` | Agent-facing operating loop, mutation gates, and verification boundaries |
|
||||
| `references/` | Five dated references: source index, Prometheus operations, OpenTelemetry Collector, Loki operations, stack integration and retention |
|
||||
| `references/` | Six dated references: source index, Prometheus operations, OpenTelemetry Collector, Loki operations, bounded PromQL/LogQL query workflows, stack integration and retention |
|
||||
| `scripts/telemetry-check` | Read-only rule sanity + scrape-target reachability checker: stdlib-only Python, `--json`, `--rules`/`--scrape`/`--targets`, `--help` with no server |
|
||||
| `fixtures/` | Valid `prometheus-rules.yml` and `scrape-config.yml` used by the tests and as starting points |
|
||||
| `tests/` | Deterministic tests against the fixture configs, including the read-only contract |
|
||||
|
||||
+12
-8
@@ -8,10 +8,12 @@ description: >-
|
||||
retention, and label design — with a bundled read-only telemetry-check script
|
||||
for Prometheus rule sanity and scrape-target reachability. Use when running,
|
||||
tuning, or troubleshooting a Prometheus, OpenTelemetry Collector, or Loki
|
||||
deployment, or reviewing the collection/ingest/retention layer. Do not use
|
||||
for observability strategy, SLI/SLO design, or paging policy (that is
|
||||
platform-engineering) or Grafana dashboards, panels, and Grafana-side
|
||||
alerting (that is grafana).
|
||||
deployment, or reviewing the collection/ingest/retention layer, including
|
||||
bounded PromQL and LogQL query construction, semantic review, and no-data
|
||||
diagnosis. Do not use for observability strategy, SLI/SLO design, or paging
|
||||
policy (that is platform-engineering), Grafana dashboards, panels, and
|
||||
Grafana-side alerting (that is grafana), or Tempo/tracing operations, which
|
||||
remain deferred to a future named-tool skill.
|
||||
license: MIT
|
||||
compatibility: >-
|
||||
The bundled telemetry-check script runs on Python 3.9+ and needs no
|
||||
@@ -53,9 +55,10 @@ Exit codes: 0 all checks passed, 1 issues found or a fatal error, 2 usage error.
|
||||
|
||||
1. **Identify the deployment**: which components are in scope (Prometheus, OTel Collector, Loki), how they are deployed (binary, container, operator), where configs live, and who owns them.
|
||||
2. **Collect evidence**: run `telemetry-check --rules` and `--scrape` on the configs, then check the live status endpoints (`/-/healthy`, `/api/v1/targets`, collector health, Loki ready) where access exists.
|
||||
3. **Triage against the symptom**: map the reported problem to the evidence (missing series → scrape or relabeling; alert not firing → rule or retention; logs missing → ingest or label cardinality).
|
||||
4. **Act with confirmation**: bounded, scoped mutations after a human directive, with a rollback path named first.
|
||||
5. **Verify**: re-run the relevant check and confirm the observable at the delivery boundary.
|
||||
3. **For a query investigation**, load `references/05-query-workflows.md`. Define the signal, selector, UTC time window, step/limit, and expected unit; validate syntax separately from semantics; execute read-only instant then bounded range/log queries; and capture status, scope, time, limits, warnings, and cardinality evidence.
|
||||
4. **Triage against the symptom**: map the reported problem to the evidence (missing series → scrape or relabeling; alert not firing → rule or retention; logs missing → ingest or label cardinality). Treat an empty result as unknown, never as numeric zero, and distinguish stale, partial, expired, absent-label, and query-error states.
|
||||
5. **Act with confirmation**: bounded, scoped mutations after a human directive, with a rollback path named first.
|
||||
6. **Verify**: re-run the relevant check and confirm the observable at the delivery boundary.
|
||||
|
||||
## Prometheus: scrape, rules, relabeling, retention, HA
|
||||
|
||||
@@ -91,6 +94,7 @@ Retention is a stack-wide decision: Prometheus blocks (raw samples), OTel Collec
|
||||
| Scrape config, recording/alerting rules, relabeling, retention, HA | `references/01-prometheus-operations.md` |
|
||||
| Collector pipelines, receivers/processors/exporters, sampling, correlation | `references/02-opentelemetry-collector.md` |
|
||||
| Ingest, LogQL, retention, label design | `references/03-loki-operations.md` |
|
||||
| PromQL/LogQL construction, semantic review, bounded cost, and no-data diagnosis | `references/05-query-workflows.md` |
|
||||
| Cross-component retention decisions and stack integration | `references/04-stack-integration-and-retention.md` |
|
||||
|
||||
## Included artifacts
|
||||
@@ -98,7 +102,7 @@ Retention is a stack-wide decision: Prometheus blocks (raw samples), OTel Collec
|
||||
- `scripts/telemetry-check`: read-only rule sanity + scrape-target reachability checker (stdlib-only, `--json`, `--rules`/`--scrape`/`--targets`, `--help` without a server).
|
||||
- `tests/test_telemetry_check.py`: deterministic tests against fixture configs, including the read-only contract.
|
||||
- `fixtures/`: `prometheus-rules.yml` (valid rules) and `scrape-config.yml` (valid scrape config) used by the tests and as starting points.
|
||||
- `references/`: four dated, source-indexed references plus the source index.
|
||||
- `references/`: five dated, source-indexed references plus the source index, including the bounded PromQL/LogQL workflow.
|
||||
- `evals/evals.json`: six output-quality evaluation cases for agent runs.
|
||||
|
||||
## Verification boundary
|
||||
|
||||
@@ -67,6 +67,39 @@
|
||||
"The diagnosis is kept read-only with the bundled checker and read-only API queries, and changes require confirmation with a rollback path",
|
||||
"Correlation is not presented as causation and fixes are verified by re-running the checks"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "bounded-promql-logql-investigation",
|
||||
"prompt": "Design a read-only PromQL and LogQL investigation for elevated API latency. Include safe selectors, a time range and step, syntax versus semantic validation, aggregation or joins, cost limits, and the evidence to record.",
|
||||
"expected_output": "A bounded workflow that defines the service, UTC start/end, expected units, and step before querying; uses a selective PromQL matcher and a Loki stream selector before line parsing; validates syntax with promtool or the target parser separately from semantic checks for metric type, labels, aggregation, and join cardinality; starts with an instant existence query before a short range query; and records HTTP status, query scope, time window, step or limit, cardinality, latency, warnings, and interpretation. It avoids broad selectors, unbounded regex, high-cardinality grouping, and unconstrained joins, with explicit timeout, lookback, series/stream, and concurrency limits.",
|
||||
"assertions": [
|
||||
"PromQL and LogQL examples use selective selectors and explicitly bounded time and resolution",
|
||||
"Syntax parsing is distinguished from semantic validation of names, labels, types, grouping, and join cardinality",
|
||||
"The workflow records query status, exact scope and time, limits, cardinality, warnings, and interpretation",
|
||||
"Broad selectors, expensive regex/parsing, high-cardinality grouping, and unconstrained joins are rejected"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "empty-and-partial-query-diagnosis",
|
||||
"prompt": "A valid PromQL request returns an empty vector for a ratio, while a Loki range query returns a partial response with warnings. Explain how to report these results and distinguish wrong labels, retention expiry, stale producers, query errors, and incomplete evidence.",
|
||||
"expected_output": "The response says an empty vector is unknown and never numeric zero, so a missing numerator or denominator must not be substituted with zero. It distinguishes parser or non-2xx query errors, absent labels or wrong selectors, expired retention, failed targets or dropped series, stale timestamps or stalled producers, and partial responses caused by limits or shards. It checks the exact UTC window, metric and label existence, target and ingest health, retention, response status and warnings, then narrows the query without mutation; any eventual config change requires confirmation and rollback evidence.",
|
||||
"assertions": [
|
||||
"No data is explicitly reported as unknown rather than zero, including for ratios",
|
||||
"Absent labels, retention expiry, failed targets, stale data, query errors, and partial responses are separate diagnoses",
|
||||
"Response status and warnings plus time window and backend health are checked before interpretation",
|
||||
"The investigation remains read-only and incomplete evidence is not presented as a complete measurement"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "telemetry-routing-and-tracing-scope",
|
||||
"prompt": "An engineer asks whether the telemetry skill should also teach Grafana dashboards, SLO paging policy, and Tempo trace queries. Explain the ownership boundaries and what tracing coverage is intentionally deferred.",
|
||||
"expected_output": "The response routes SLI/SLO definitions, error budgets, alert thresholds, and paging strategy to platform-engineering; routes Grafana dashboards, panels, data sources, Grafana alert rules, contact points, and notification policies to grafana; and keeps telemetry focused on Prometheus/Loki/Collector backend query evidence. It explicitly says Tempo/tracing query operations are deferred to a future named-tool decision, without inventing Tempo commands or claiming trace-query semantics; application context propagation remains backend-engineering work.",
|
||||
"assertions": [
|
||||
"Platform-engineering is named for SLI/SLO, strategy, and paging ownership",
|
||||
"Grafana is named for dashboards, panels, and Grafana-side alerting/contact policy ownership",
|
||||
"Tempo/tracing operations are explicitly deferred without unsupported commands or behavioral claims",
|
||||
"The response preserves a bounded telemetry backend scope instead of creating an observability mega-skill"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
@@ -25,6 +25,9 @@ procedure for keeping it current.
|
||||
| Loki storage and retention | https://grafana.com/docs/loki/latest/operations/storage/retention/ |
|
||||
| LogQL | https://grafana.com/docs/loki/latest/query/ |
|
||||
| Loki label design guidance | https://grafana.com/docs/loki/latest/get-started/labels/ |
|
||||
| Prometheus HTTP API and querying | https://prometheus.io/docs/prometheus/latest/querying/api/ |
|
||||
| PromQL operators and functions | https://prometheus.io/docs/prometheus/latest/querying/operators/ |
|
||||
| Loki query HTTP API | https://grafana.com/docs/loki/latest/reference/loki-http-api/ |
|
||||
|
||||
## Version observations (as of this refresh)
|
||||
|
||||
|
||||
@@ -0,0 +1,71 @@
|
||||
# PromQL and LogQL Query Workflows
|
||||
|
||||
> **Last Updated:** 2026-09-01
|
||||
|
||||
Use this reference for an offline query review or a live, read-only query investigation. It covers query shape and evidence; it does not replace `promtool` or Loki's parser.
|
||||
|
||||
## 1. Define a bounded question
|
||||
|
||||
Write down the signal, service/job, environment, exact UTC start and end, resolution, and expected unit before writing the expression. Prefer a narrow selector with equality matchers:
|
||||
|
||||
```promql
|
||||
sum by (job, route) (rate(http_requests_total{job="api", environment="prod"}[5m]))
|
||||
```
|
||||
|
||||
For logs, select streams first and filter lines second:
|
||||
|
||||
```logql
|
||||
{app="api", environment="prod"} |= "timeout" | json | duration_ms > 1000
|
||||
```
|
||||
|
||||
Do not begin with `{}` or `metric{label=~".*"}`. Avoid unbounded regex, arbitrary joins, and multi-day ranges while exploring. A range query must state its step; choose a step no finer than the evidence needs. Bound concurrency and request timeouts at the client/query gateway.
|
||||
|
||||
## 2. Validate in two distinct passes
|
||||
|
||||
Syntax and semantics are different claims:
|
||||
|
||||
1. **Syntax:** parse the expression with `promtool check rules` (PromQL) or the target Loki/Grafana parser. Check balanced delimiters, operators, durations, and LogQL pipeline syntax. A HTTP 200 only proves the endpoint accepted the request, not that the question is correct.
|
||||
2. **Semantics:** confirm metric/log names, label keys and value types, counter-versus-gauge intent, compatible label sets for binary operators, and aggregation dimensions. For joins, make the matching labels explicit (`on(...)`/`ignoring(...)`) and use `group_left`/`group_right` only when the cardinality relationship is known. For LogQL, verify parsed fields exist and that aggregation is applied to the intended streams.
|
||||
|
||||
Record both results separately. A syntactically valid query can return an empty vector because the label was renamed, the metric was never scraped, or retention has expired.
|
||||
|
||||
## 3. Capture bounded evidence
|
||||
|
||||
For every query record:
|
||||
|
||||
- backend and endpoint, HTTP status, and response `status`/error type;
|
||||
- exact query text or a redacted query identifier;
|
||||
- UTC `start`, `end`, and Prometheus `step` (or Loki limit/direction);
|
||||
- selector and grouping labels, estimated series/stream count, and duration;
|
||||
- whether the result is instant, range, logs, or a derived metric;
|
||||
- parser result, semantic checks, warnings, and a link/request ID if available.
|
||||
|
||||
Use instant queries to test existence, then a short range query to establish behavior. Keep result samples bounded; do not paste raw logs or credentials.
|
||||
|
||||
## 4. Control cost and cardinality
|
||||
|
||||
Start with a narrow equality selector and a short range. Expand one dimension at a time, checking series count and latency after each change. Prefer recording rules for repeatedly used expensive PromQL expressions and pre-extracted low-cardinality Loki labels for common filters. Keep request limits explicit: maximum lookback, step floor, series/stream limit, bytes/line limit, timeout, and query concurrency. A gateway or tenant limit is a safety net, not permission to issue a broad query.
|
||||
|
||||
Avoid `count by` over unbounded labels, grouping by request/user/trace IDs, regex over all streams, JSON/regexp parsing before a selective matcher, and joins where both sides are high cardinality. A label whose value changes per event belongs in the log body or structured metadata, not Loki's index.
|
||||
|
||||
## 5. Diagnose empty and partial results without inventing zeros
|
||||
|
||||
An empty result is **unknown**, never numeric zero. Distinguish:
|
||||
|
||||
| Observation | Next read-only check | Interpretation |
|
||||
|---|---|---|
|
||||
| Query parser error / non-2xx | response error type and expression | Query error; fix syntax or request shape |
|
||||
| Empty with healthy backend | instant existence query, label/series API, exact time window | Could be absent label, wrong selector, or no events |
|
||||
| Empty only for old time | retention bounds and backend clock | Data expired or outside retention |
|
||||
| Empty for one target | scrape/ingest target state and relabel output | Failed target, dropped series, or missing stream |
|
||||
| Stale/flat value | sample timestamps, scrape freshness, exporter metrics | Stale data, stalled producer, or timestamp issue |
|
||||
| Partial response/warnings | HTTP status, response warnings, shard/limit metrics | Incomplete evidence; do not aggregate as complete |
|
||||
| Backend timeout/limit | query duration, series/stream limit, concurrency | Cost or capacity rejection; narrow query |
|
||||
|
||||
For ratios, do not substitute zero for a missing numerator or denominator. Report `no data`, preserve `NaN`/absence semantics, and state what evidence is missing. A target can be `up` while a particular metric is absent because relabeling or instrumentation changed.
|
||||
|
||||
## Ownership and deferral
|
||||
|
||||
Route SLI/SLO definitions, error budgets, alert thresholds, and paging strategy to [platform-engineering](../../platform-engineering/SKILL.md). Route Grafana panels, data-source configuration, Grafana alert rules, contact points, and notification policies to [grafana](../../grafana/SKILL.md). This skill supplies backend query evidence those owners consume.
|
||||
|
||||
Tempo/tracing query workflows are intentionally deferred to a future named-tool decision. This skill may preserve trace/span IDs for correlation in logs and metrics, but it does not claim Tempo commands, APIs, or trace-query semantics. Route application propagation to [backend-engineering](../../backend-engineering/SKILL.md) and revisit a dedicated Tempo skill only when a concrete operational surface and non-overlapping trigger are established.
|
||||
Reference in New Issue
Block a user