magnus919_agent-skills/telemetry/references/03-loki-operations.md

Loki Operations

Last Updated: 2026-08-03

Operational patterns for the Loki half of the telemetry stack: ingest, LogQL, retention, and label design. Sources: Grafana Loki documentation (grafana.com/docs/loki, accessed 2026-08-03).

Ingest

Loki ingests via the push API (POST /loki/api/v1/push) from Promtail, the OpenTelemetry Collector loki exporter, Grafana Alloy/Agent, or the SDKs. Ingest health is distributor-side:

• loki_distributor_bytes_received_total and loki_distributor_lines_received_total must advance per tenant;
• loki_ingester_streams shows active streams — a flat line here while producers push means a config or network problem;
• the ready endpoint (/ready) must return 200; 429 Too Many Requests from the distributor means rate limits — the producer retries, but a sustained backlog is a capacity signal.

The OTel Collector loki exporter maps log records to streams: the loki.tenant and loki.format attributes plus configured labels control stream cardinality (see Labels below). Promtail is the file-tailer option; choose one producer per source and document it.

LogQL

LogQL has two layers: stream selectors and pipeline expressions.

• Selectors choose streams by label: {app="api", env="prod"}. Label matchers are evaluated against the inverted index — this is the cheap part, and it is why label design matters.
• Pipeline expressions filter and transform lines: |= "error", |~ "5[0-9][0-9]", | json, | regexp "(?P<field>...)", | line_format. These run per line after selection — the expensive part.

Metric queries wrap the pipeline in rate, count_over_time, etc.: sum by (app) (rate({job="api"} |~ "error"[5m])). When a query is slow, the cause is almost always a too-broad selector (too many streams) or regexp/JSON parsing on every line; fix the selector or pre-extract fields at ingest, not by writing a faster regexp.

Retention

Loki retention is per-tenant, enforced by the compactor:

• retention_enabled: true in the limits config enables period-based retention; retention_period sets the age limit and retention_size the byte limit per tenant;
• the compactor deletes expired chunks and merges index shards; watch loki_compactor_delete_requests_total and loki_compactor_compactor_running to confirm it is actually running;
• retention applies at query and delete time, so expired data can still be counted until compaction finishes — size-based limits are the practical control for runaway log volume.

Set retention before rollout and treat it as a compliance decision with an owner (see references/04-stack-integration-and-retention.md). A Loki with default retention and no owner will silently grow until storage is the incident.

Labels

Loki labels are an inverted index — every distinct label value adds index entries and stream overhead. The guidance is deliberately simple:

• keep labels to tenant, app, environment, job, and a handful of service-defined dimensions;
• never index high-cardinality fields: request IDs, user IDs, trace IDs, IPs, timestamps, or any field whose values change per log line;
• put high-cardinality data in the log line itself and extract it with LogQL | json/| regexp, or as OTel structured metadata, when you need it;
• a good rule of thumb: if a label's values exceed the low hundreds of distinct values, it is a line field, not a label.

Cardinality damage is silent and cumulative: loki_ingester_streams climbing while label changes are "small" is the leading signal. Stream sharding and chunk_target_size tune large-stream handling, but they do not make a bad label design good.