mirror of
https://github.com/magnus919/agent-skills.git
synced 2026-09-21 08:36:33 +03:00
f83d48ba53
Adds one top-level telemetry skill covering the observability stack that deploys as one unit: Prometheus (scrape config, recording/alerting rules, relabeling, retention, HA), the OpenTelemetry Collector (pipelines, receivers/processors/exporters, sampling, trace/span correlation), and Loki (ingest, LogQL, retention, labels). Ships the read-only telemetry-check script (stdlib-only, --json): Prometheus rule sanity mirroring promtool check rules plus scrape-target reachability probes, fixture-tested with 16 unittest/pytest cases. Includes five dated references, a human-facing README, and six eval cases covering rule authoring, pipeline design, and retention. Routes up to platform-engineering and grafana without duplicating their content. Regenerates the llms.txt / marketplace / plugin catalogs and adds the README index entry. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
98 lines
4.8 KiB
Markdown
98 lines
4.8 KiB
Markdown
# Stack Integration and Retention
|
|
|
|
> **Last Updated:** 2026-08-03
|
|
|
|
How the three components of the telemetry stack fit together, and how to make
|
|
retention decisions across them instead of per-component by accident. Sources:
|
|
Prometheus storage documentation, OpenTelemetry Collector documentation, and
|
|
Loki retention documentation (all accessed 2026-08-03).
|
|
|
|
## The stack as one unit
|
|
|
|
The components share a data flow — instrumented services emit metrics, logs,
|
|
and traces; the OpenTelemetry Collector (or direct exporters) delivers them;
|
|
Prometheus stores metrics and evaluates rules; Loki stores logs; Grafana
|
|
queries both and displays the result. The stack is operated as one unit
|
|
because a change in any layer changes the behavior of every layer above it:
|
|
|
|
| Layer | Component | Owns |
|
|
|---|---|---|
|
|
| Collection | OTel Collector receivers, Prometheus scrape jobs | What data enters the stack |
|
|
| Processing | Collector processors (batch, sample, resource) | What the data looks like when stored |
|
|
| Storage | Prometheus TSDB, Loki chunks/index | How long data lives and how fast queries are |
|
|
| Evaluation | Prometheus rules | What the data means (alerts, recording rules) |
|
|
| Presentation | Grafana | What humans see (dashboards, Grafana-side alerting) |
|
|
|
|
Operational decisions therefore cross component boundaries: a label added in
|
|
relabeling is a label PromQL sees; a `trace_id` carried by the collector is a
|
|
LogQL matcher; a retention window chosen for Prometheus is a gap in history a
|
|
dashboards-as-code change cannot recover.
|
|
|
|
## Retention as a stack decision
|
|
|
|
Each component has independent retention, and the combined footprint is what
|
|
the team pays for:
|
|
|
|
| Component | Setting | Applies to | Default behavior |
|
|
|---|---|---|---|
|
|
| Prometheus | `--storage.tsdb.retention.time` / `.size` | Raw samples (blocks) | 15 days; lazily enforced by compaction |
|
|
| OTel Collector | exporter queue + `memory_limiter` | In-flight data | Queue backs up then drops oldest on pressure |
|
|
| Loki | `retention_enabled`, `retention_period`, `retention_size` (per tenant) | Indexed log streams | No retention unless enabled; compactor enforces it |
|
|
|
|
Decide per component by the question the data answers:
|
|
|
|
- hot metrics for alerting: keep enough history to evaluate every rule window
|
|
plus headroom (a `for: 10m` rule needs at least that much history);
|
|
- samples for trends: retention length is a capacity trade-off, not a
|
|
correctness requirement — long histories belong in a separate store
|
|
(Thanos/Mimir) with its own owner;
|
|
- logs for debugging and audit: retention is a compliance decision with an
|
|
owner; short retention on logs destroys the only evidence an incident
|
|
post-mortem can use.
|
|
|
|
The failure mode is defaulting every component and discovering the cost when
|
|
storage is the incident. Write the decision down: `retention_period`,
|
|
retention flags, and the RPO/RTO framing belong in the deployment config and
|
|
its review checklist, not in tribal memory.
|
|
|
|
## Cross-component consistency checks
|
|
|
|
- Scrape config and collector receivers must agree on the metrics endpoint:
|
|
a `metric_relabel_configs` drop on Prometheus side does not stop the
|
|
collector from exporting the metric elsewhere.
|
|
- Rule expressions and recording rules must not depend on labels the scrape
|
|
config does not produce — validate with `telemetry-check --rules` plus a
|
|
live query, not by reading the config.
|
|
- LogQL matchers must match labels the `loki` exporter actually attaches —
|
|
a matcher on `app` fails silently when the collector maps it to
|
|
`service.name` only.
|
|
- Trace correlation needs `trace_id`/`span_id` in both the OTLP log records
|
|
and the metrics exemplars; verify one pivot query end-to-end after any
|
|
collector pipeline change.
|
|
|
|
## Alerting on the stack itself
|
|
|
|
The stack needs its own health rules (in a file you also validate with
|
|
`telemetry-check --rules`):
|
|
|
|
- `up{job=~"prometheus|otel-collector|loki"}` == 0 for component loss;
|
|
- `prometheus_tsdb_head_series` growth vs a budget for cardinality;
|
|
- `rate(loki_distributor_lines_received_total[5m])` vs a floor to catch
|
|
silent ingest loss;
|
|
- `otelcol_exporter_send_failed_ratio` above a threshold for delivery loss;
|
|
- retention drift: `prometheus_tsdb_storage_blocks_bytes` and compactor
|
|
delete counters vs the decided policy.
|
|
|
|
These rules live in this skill's scope (the Prometheus rules file); the
|
|
dashboards and Grafana-side alert routes for them are `grafana` territory.
|
|
|
|
## Routing
|
|
|
|
- Observability strategy — what to instrument, SLI/SLO design, error budgets —
|
|
is [platform-engineering](../platform-engineering/SKILL.md).
|
|
- Dashboards, panels, Grafana alert rules, contact points, and notification
|
|
policies are [grafana](../grafana/SKILL.md).
|
|
- This skill owns the collection/ingest/retention layer and the Prometheus
|
|
rules files those layers consume. When a task crosses into those skills,
|
|
route there instead of duplicating their content.
|