magnus919_agent-skills/data-engineering/references/time-series-databases.md

Time-Series Database Patterns for Data Engineering

A thorough reference on time-series databases with a focus on InfluxDB, written for the data engineering methodology skill.

---

Table of Contents

1. When to Choose a Time-Series DB Over a Relational DB
2. InfluxDB Data Model
3. Schema Design for Time-Series
4. Flux Query Language Fundamentals
5. InfluxQL vs Flux vs SQL
6. Downsampling and Continuous Queries / Tasks
7. Data Lifecycle Management
8. Ingest Patterns
9. Integration with Data Engineering Pipelines
10. Comparison: InfluxDB vs TimescaleDB vs Prometheus vs QuestDB

---

1. When to Choose a Time-Series DB Over a Relational DB

Time-Series Data Characteristics

Not all timestamped data is time-series data. True time-series data has these properties:

• Append-heavy: New data points arrive continuously; updates/upserts are rare.
• Time-ordered: Write order closely follows the timestamp order (recent data is hot).
• Time-centric queries: Analysts almost always filter, aggregate, and slice by time ranges.
• Downsampling pattern: Old data is routinely summarized into lower-resolution rollups.
• Immutable by nature: Historical records are almost never modified.

Decision Matrix

Factor	Choose Relational (PostgreSQL/MySQL)	Choose Time-Series DB (InfluxDB/TimescaleDB)
Write pattern	Mixed read-write, UPDATE-heavy	Append-only streaming writes
Data volume	Millions of rows	Billions to trillions of data points
Query pattern	OLTP: single-row lookups, JOINs, transactions	Time-bucket aggregates, range scans
Retention	Keep everything indefinitely, DELETE rare	Auto-expire raw data after N days
Schema	Frequently evolving, normalized	Stable, denormalized per measurement
Consistency	ACID strongly required	Eventual or tunable consistency acceptable
Cardinality	Low (user IDs, order IDs)	Can range from low to very high (device IDs, container IDs)

When to Use a General-Purpose TSDB (like InfluxDB)

• Metrics infrastructure: Server/container CPU, memory, disk, network (DevOps/SRE).
• IoT/IIoT sensor data: Temperature, pressure, vibration readings at high frequency.
• Application telemetry: Request latencies, error rates, user counts.
• Industrial/historian workloads: Replacing PI System, OSIsoft, or other historians.
• Financial tick data: Stock trades, order book snapshots (though QuestDB may be better here).

When to Stick with a Relational DB + Time-Series Extension

• You already have a PostgreSQL ecosystem and want to avoid another infrastructure stack.
• Your time-series data has complex relational joins (e.g., sensor metadata normalized across 5 tables).
• You need full SQL with window functions, CTEs, and transactional guarantees.
• Solution: TimescaleDB hypertables on PostgreSQL.

When to Use an Analytical Columnar DB (ClickHouse) Instead

• You run large ad-hoc analytical queries on time-series data (OLAP-style).
• You need sub-second aggregation over billions of rows across many dimensions.
• Your query patterns are more "GROUP BY time, dimension" than "latest value per series."

---

2. InfluxDB Data Model

InfluxDB v1 and v2 share a four-component data model. InfluxDB 3 (the current recommended version) retains compatibility with this model but adds SQL-on-Parquet support.

The Four Components

Measurement   ->  Logical table name (e.g., "cpu", "sensor_temp")
Tag set       ->  Indexed metadata key=value pairs (e.g., host=server01, region=us-east)
Field set     ->  Actual data values (e.g., temperature=98.6, cpu_usage=0.85)
Timestamp     ->  Nanosecond-precision Unix timestamp

Line Protocol Format

This is the canonical way to write data into InfluxDB (all versions):

<measurement>[,<tag_key>=<tag_value>[,<tag_key>=<tag_value>...]] <field_key>=<field_value>[,<field_key>=<field_value>...] [<timestamp>]

Concrete example:

sensor_temp,host=server01,region=us-east temperature=98.6,humidity=0.45 1717610400000000000

Or in InfluxDB 3 SQL terms, the CREATE TABLE equivalent would be:

-- InfluxDB 3 uses SQL for schema management
CREATE TABLE sensor_temp (
    time TIMESTAMP,
    host STRING,      -- tag
    region STRING,    -- tag
    temperature DOUBLE,  -- field
    humidity DOUBLE      -- field
);

How InfluxDB Stores Data

• Tags are indexed — they form the series key. Every unique combination of measurement + tag set defines a time series.
• Fields are not indexed — querying by field values requires a full scan.
• Timestamp is the primary sort key within each series.
• In InfluxDB 3 (IOx engine), data is stored in Parquet files on object storage, with an in-memory catalog for indexing.
• In InfluxDB 1.x/2.x (TSM engine), data is stored in TSM (Time-Structured Merge Tree) files with an in-memory index.

Series Cardinality

series cardinality = number of unique (measurement, tag set) combinations

Example: If you have measurement cpu with tags host (1000 values) and region (5 values), you have up to 5,000 series.

High cardinality is the #1 performance killer in InfluxDB v1/v2 (TSM engine). InfluxDB 3 (IOx) largely solves this by using a columnar storage engine, but high cardinality still affects memory for the catalog.

Values that cause high cardinality and should NEVER be tags:

• Request IDs
• Session IDs
• User IDs (if unique per user and high volume)
• Timestamps as strings
• Email addresses
• Any value with millions of unique values

---

3. Schema Design for Time-Series

Measurement Design

Rule of thumb: One measurement per logical data source type.

GOOD:  measurement="cpu"     fields={usage_user, usage_system, usage_idle}
GOOD:  measurement="memory"  fields={used_bytes, free_bytes, total_bytes}
BAD:   measurement="metrics" fields={cpu_user, cpu_system, mem_used, mem_free, disk_read, disk_write}

Tag vs Field Decision Guide

Put in TAGS if...	Put in FIELDS if...
Low to moderate cardinality (< 100K unique values)	The actual measured numeric value
You filter or GROUP BY this attribute	High cardinality (request IDs, UUIDs)
It's static/reusable metadata (host, region, data_center)	It's the payload/metric value itself
You need fast indexed lookups	It changes on every data point

Tag Cardinality Management

For InfluxDB v1/v2 (TSM):

• Keep total series cardinality under 10 million per node (hard limit ~10-20M).
• Keep per-measurement cardinality under 1 million for good performance.
• Use TSI (Time Series Index) for higher cardinality in v1.7+ but expect memory pressure.

For InfluxDB 3 (IOx/columnar):

• Catalog memory scales with number of unique tag values, not combinations.
• Can handle 100M+ unique series; watch catalog memory (~2-4 GB per 100M series).

Anti-patterns to avoid:

• Putting timestamps, dates, or high-entropy strings as tags.
• Using tags for values that change on every write.
• Over-tagging (10+ tags per measurement when 3-4 would suffice).
• Tag values that grow unboundedly (e.g., container IDs in Kubernetes).

Retention Policies (v1) vs Buckets (v2)

InfluxDB v1 — Retention Policies (RPs):

-- Create a retention policy: keep data for 30 days, 1 replica
CREATE RETENTION POLICY "thirty_days" ON "mydb" DURATION 30d REPLICATION 1 DEFAULT;

InfluxDB v2 — Buckets:

# A bucket combines a database + retention policy from v1
influx bucket create --name "sensor_data_30d" --retention 30d

InfluxDB 3 — Retention at Database Level:

# InfluxDB 3 Core: retention set at database level
# Core OSS enforces a 72-hour default; Cloud Dedicated and Enterprise allow custom retention
influxdb3 create database iot_sensors_prod --retention-period 90d

Best practices:

• Use separate buckets for different retention durations.
• For long-term storage, downsample raw data into a separate measurement with longer retention.
• In InfluxDB 3, the retention period is set per database and defines how long data is kept before automatic deletion.

---

4. Flux Query Language Fundamentals

Note: Flux was introduced with InfluxDB v2.x. InfluxDB 3 now recommends SQL as the primary query language. Flux is still supported in InfluxDB 2.x and for backward compatibility, but new development on InfluxDB 3 should favor SQL. This section is retained for teams maintaining v2.x workloads.

Basic Structure

Flux is a functional, piped-data language. Every query is a chain of transformations with data flowing left-to-right through pipes (|>).

data_source
    |> transformation_1()
    |> transformation_2()
    |> transformation_3()

Core Functions

// 1. Define the data source and time range
from(bucket: "sensor_data")
    |> range(start: -1h)                    // last hour of data
    |> filter(fn: (r) => r._measurement == "cpu")
    |> filter(fn: (r) => r._field == "usage_user")
    |> filter(fn: (r) => r.host == "server01")
    |> yield(name: "cpu_usage")

Common Flux Patterns

Aggregation with windowing (downsampling):

from(bucket: "sensor_data")
    |> range(start: -7d)
    |> filter(fn: (r) => r._measurement == "sensor_temp")
    |> aggregateWindow(every: 1h, fn: mean)
    |> yield(name: "hourly_mean")

Multiple aggregations in one query:

from(bucket: "sensor_data")
    |> range(start: -24h)
    |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_user")
    |> aggregateWindow(every: 15m, fn: mean)
    |> duplicate(column: "_stop", as: "_time")
    |> drop(columns: ["_start", "_stop"])
    |> set(key: "_field", value: "usage_user_mean")
    |> to(bucket: "downsampled_cpu")

Pivoting to wide format (useful for Grafana):

from(bucket: "sensor_data")
    |> range(start: -1h)
    |> filter(fn: (r) => r._measurement == "cpu")
    |> pivot(rowKey: ["_time"], columnKey: ["_field"], valueColumn: "_value")
    |> yield(name: "wide")

Joining data streams:

cpu = from(bucket: "sensor_data")
    |> range(start: -1h)
    |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_user")

mem = from(bucket: "sensor_data")
    |> range(start: -1h)
    |> filter(fn: (r) => r._measurement == "mem" and r._field == "used_percent")

join(tables: {cpu: cpu, mem: mem}, on: ["_time", "host"])
    |> yield(name: "joined")

Flux Task (recurring script)

// Run every hour
option task = {
    name: "downsample_cpu_hourly",
    every: 1h,
    offset: 5m
}

from(bucket: "raw_sensor_data")
    |> range(start: -2h)
    |> filter(fn: (r) => r._measurement == "cpu")
    |> aggregateWindow(every: 1h, fn: mean)
    |> to(bucket: "downsampled_cpu")

---

5. InfluxQL vs Flux vs SQL

Overview

Feature	InfluxQL	Flux	SQL (InfluxDB 3)
Era	InfluxDB 1.x	InfluxDB 2.x	InfluxDB 3 (current)
Style	SQL-like	Functional / piped	Standard SQL
Complexity	Low	Medium-High	Low
Learning curve	Easy (if you know SQL)	Steep	Easy (if you know SQL)
Multi-bucket queries	No	Yes	Yes
Joins	Limited (subqueries only)	Native	Full SQL JOINs
Scripting	No	Yes (variables, conditionals, functions)	Via SQL functions
Window functions	Limited	Native	Yes
Performance	Good	Medium (interpreted)	Best (compiled)
Status in InfluxDB 3	Read-only compatibility	Supported for compatibility	Recommended

When to Use Which

Use SQL  (InfluxDB 3):  Default for ALL new projects on InfluxDB 3.
Use Flux (v2.x only):   Existing v2.x deployments, complex transformation pipelines.
Use InfluxQL (v1.x):    Existing v1.x deployments, minimal migration path.

InfluxQL vs Flux: Equivalent Queries

InfluxQL:

SELECT mean("usage_user")
FROM "cpu"
WHERE time > now() - 1h
AND "host" = 'server01'
GROUP BY time(15m)

Flux:

from(bucket: "mydb/autogen")
    |> range(start: -1h)
    |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_user" and r.host == "server01")
    |> aggregateWindow(every: 15m, fn: mean)

SQL (InfluxDB 3):

SELECT DATE_BIN(INTERVAL '15 minutes', time) AS bucket,
       AVG(usage_user) AS avg_usage
FROM cpu
WHERE time > now() - INTERVAL '1 hour'
  AND host = 'server01'
GROUP BY bucket
ORDER BY bucket;

Key Migration Notes

• InfluxQL GROUP BY time(interval) → Flux aggregateWindow(every: interval, fn: ...)
• InfluxQL INTO (downsample+write) → Flux ... |> to(bucket: "...")
• Continuous Queries (InfluxQL) → Tasks (Flux)
• InfluxDB 3: Rewrite InfluxQL CQs as SQL scheduled queries or use external orchestrators.

---

6. Downsampling and Continuous Queries / Tasks

The Downsampling Pattern

Downsampling is the most critical data engineering pattern for time-series:

RAW DATA (1-second resolution, keep 30 days)
    |
    v  [downsample every hour]
HOURLY ROLLUPS (1-minute aggregates, keep 1 year)
    |
    v  [downsample every day]
DAILY ROLLUPS (1-hour aggregates, keep 5 years)

Each tier provides exponentially smaller storage footprint while preserving analytical value.

InfluxDB 1.x: Continuous Queries (CQs)

CREATE CONTINUOUS QUERY "cq_cpu_hourly" ON "mydb"
BEGIN
  SELECT mean("usage_user") AS "mean_usage"
  INTO "mydb"."downsampled"."cpu_hourly"
  FROM "cpu"
  GROUP BY time(1h), "host"
END;

CQs run automatically at the end of each time window. They are simple but limited: only one aggregation function per CQ, no chaining.

InfluxDB 2.x: Tasks (Flux-based)

// downsample_cpu_hourly
option task = {
    name: "downsample_cpu_hourly",
    every: 1h,
    offset: 10m
}

from(bucket: "raw_data")
    |> range(start: -task.every)
    |> filter(fn: (r) => r._measurement == "cpu")
    |> aggregateWindow(every: 1h, fn: mean)
    |> set(key: "_measurement", value: "cpu_hourly")
    |> to(bucket: "downsampled_data")

Chained (hierarchical) downsampling with tasks:

Task 1: raw_1s -> hourly (runs every hour)
Task 2: hourly -> daily   (runs every day, queries the hourly bucket)
Task 3: daily -> monthly  (runs monthly, queries the daily bucket)

InfluxDB 3: Scheduled Queries / External Orchestration

InfluxDB 3 Core does not have built-in continuous aggregates. Recommended approaches:

1. External scheduler (Airflow, cron, Prefect):

   -- Run hourly via Airflow
   INSERT INTO cpu_hourly
   SELECT DATE_BIN(INTERVAL '1 hour', time) AS bucket,
          host,
          AVG(usage_user) AS avg_usage,
          MIN(usage_user) AS min_usage,
          MAX(usage_user) AS max_usage,
          COUNT(*) AS sample_count
   FROM cpu
   WHERE time > now() - INTERVAL '2 hours'
   GROUP BY bucket, host;
   

2. InfluxDB 3 processing engine plugins for real-time transformations on write.

3. Write-time processing via Telegraf aggregator plugins:

   [[processors.aggregate]]
   period = "60s"
   
   [[processors.aggregate.config]]
   measurement = "cpu"
   columns = ["usage_user", "usage_system"]
   functions = ["mean", "max", "min"]
   

Downsampling Best Practices

• Store raw data at full resolution for the shortest practical window.
• Always include COUNT(*) in aggregates to track sample density.
• Use hierarchical aggregation (aggregate hourly data into daily, not raw into daily).
• Align your downsampling schedule with your retention policies.
• Consider two-phase downsampling: real-time via Telegraf aggregators for the first 1-5 minutes, then batch tasks for correction/backfill.

---

7. Data Lifecycle Management

Storage Engine Architecture

InfluxDB 1.x / 2.x (TSM Engine):

• Data organized into shards by time range (typically 7 days).
• Each shard is a set of TSM files + Write-Ahead Log (WAL).
• Compaction merges smaller TSM files into larger ones, removing deleted/overwritten data.
• Shard management: Default 7-day shard duration; configurable.
• Memory index (in-memory) maps series keys to TSM file locations.

InfluxDB 3 (IOx Engine):

• Data stored as Parquet files in object storage (S3, local FS).
• Catalog (SQLite or PostgreSQL) tracks table/column metadata.
• Compaction merges Parquet files for read efficiency.
• No shards per se — data is partitioned by time but managed at the file level.

Retention Lifecycle Strategy

Example: 3-tier retention for IoT sensor data

Tier 1: Raw (1-second resolution)   -> 30 days   -> bucket "raw_30d"
Tier 2: Hourly aggregates           -> 1 year    -> bucket "hourly_1y"
Tier 3: Daily aggregates            -> 5 years   -> bucket "daily_5y"

InfluxDB v2/v3 implementation:

1. Create three buckets with different retention periods.
2. Run downsampling tasks from raw -> hourly -> daily.
3. InfluxDB automatically deletes data older than each bucket's retention period.

Compaction

TSM compaction stages:

• Level 1 (snapshot): WAL -> TSM file (when WAL reaches threshold).
• Level 2 (merge): 2-4 small TSM files -> 1 larger TSM file.
• Level 3 (full): Multiple TSM files -> 1 optimized TSM file (deduplicates, removes tombstones).
• Compaction runs automatically; tune cache-snapshot-write-cold-duration and compact-full-write-cold-duration for write-heavy workloads.

IOx/Parquet compaction:

• Merges small Parquet files (< 100 MB) into larger ones (~100-500 MB).
• Runs automatically but can be triggered manually via API.
• Compaction also applies retention deletion.

Shard Management (InfluxDB v1/v2)

Command:             Effect:
ALTER RETENTION      Change shard duration (default 7d)
  POLICY ... DURATION
DROP SHARD           Force-delete a specific shard and all its data
SHOW SHARDS          List all shards with durations, sizes, status
influx_inspect       Low-level TSM inspection and recovery tools

Choosing shard duration:

• Short shard duration (1-7d): More granular retention, easier to drop old data, more overhead.
• Long shard duration (1-4w): Less overhead, faster range queries, slower retention enforcement.
• Rule: shard duration should be ≤ 1/2 of your retention period for efficient expiry.

Cold / Tiered Storage

• InfluxDB Cloud Serverless: Automatically tiers data to object storage.
• InfluxDB Cloud Dedicated: Configurable cold storage with Parquet.
• TimescaleDB: Native tiering to S3 via tiering policies.
• InfluxDB OSS: No built-in tiering; manage via external scripts or data migration.

---

8. Ingest Patterns

Line Protocol

The core ingestion format for all InfluxDB versions.

Format:

measurement,tag1=val1,tag2=val2 field1=val1,field2=val2 timestamp

Data types:

• Tags: strings only (no quoting needed if no special chars).
• Fields: floats (default), integers (trailing i), strings ("quoted"), booleans (t/f/true/false).
• Timestamp: nanosecond epoch (default); configurable precision (s, ms, us, ns).

Examples:

# Float fields (default)
weather,location=us-midwest temperature=82.0 1465839830100400200

# Integer field (trailing i)
weather,location=us-midwest wind_speed=15i 1465839830100400200

# String field (double-quoted)
weather,location=us-midwest conditions="partly cloudy" 1465839830100400200

# Boolean field
weather,location=us-midwest is_raining=t 1465839830100400200

# Multiple fields
weather,location=us-midwest temperature=82.0,humidity=71.2 1465839830100400200

Write via HTTP API:

curl -X POST \
  "http://localhost:8086/write?db=mydb&precision=s" \
  --data-raw "weather,location=us-midwest temperature=82.0 1465839830"

Batch vs Streaming Writes

Factor	Batch	Streaming
Frequency	Every N seconds or N points	Every point as it arrives
Overhead	Low (HTTP overhead amortized)	High (per-request overhead)
Throughput	High (10K-100K points/s per node)	Low (1K-10K points/s per node)
Latency	Seconds to minutes	Sub-second
Use case	Backfill, batch ETL	Real-time monitoring, Telegraf

Best practice: Always batch writes — send 1,000-10,000 points per HTTP request. Never send single points.

Telegraf Agent

Telegraf is InfluxData's plugin-driven collection agent.

Architecture:

Input Plugins  ->  Aggregator/Processor Plugins  ->  Output Plugins
    |                       |                           |
  (CPU, disk,             (aggregate,                  (InfluxDB,
   MQTT, Kafka,            transform,                   Prometheus,
   Prometheus,              enrich,                      file, Kafka,
   syslog, SNMP,            filter)                      CloudWatch)
   Docker, k8s...)

Example Telegraf config (telegraf.conf):

# Global settings
[agent]
  interval = "10s"
  flush_interval = "10s"
  metric_batch_size = 5000

# Input: CPU metrics
[[inputs.cpu]]
  percpu = true
  totalcpu = true

# Input: MQTT subscriber
[[inputs.mqtt_consumer]]
  servers = ["tcp://broker.local:1883"]
  topics = ["sensors/#"]
  data_format = "json"
  json_time_key = "timestamp"
  json_time_format = "unix_ms"
  tag_keys = ["device_id", "location"]

# Processor: apply transformation
[[processors.enum]]
  [[processors.enum.mapping]]
    tag = "status"
    value_mappings = {online = 1, offline = 0}

# Aggregator: downsample in real-time
[[processors.aggregate]]
  period = "60s"
  [[processors.aggregate.config]]
    measurement = "cpu"
    columns = ["usage_idle", "usage_user"]
    functions = ["mean", "min", "max"]

# Output: InfluxDB v2
[[outputs.influxdb_v2]]
  urls = ["http://localhost:8086"]
  token = "${INFLUX_TOKEN}"
  organization = "myorg"
  bucket = "sensor_data"

# Output: backup to file (for audit trail)
[[outputs.file]]
  files = ["/var/log/telegraf_audit.log"]
  data_format = "json"

Telegraf best practices:

• Use metric_batch_size (5,000-10,000) and flush_interval (5-10s) for efficient batching.
• Tag inputs with consistent metadata (data center, region, environment).
• Use processor and aggregator plugins rather than sending raw data and downsampling later.
• Set fieldpass/fielddrop on inputs to avoid collecting unused metrics.
• File-based logging output for auditability and data recovery.

Ingest Performance Tuning

Parameter	InfluxDB v1/v2 (TSM)	InfluxDB 3 (IOx)
Write batch size	5,000-10,000 points	10,000-50,000 points
HTTP workers	8-16	16-64
WAL flush interval	1-10s (lower = safer)	N/A (no WAL)
Max points per second (single node)	500K-1M	3M-10M
Bottleneck	CPU (index updates)	Network I/O (Parquet writes)

Kafka Integration

Telegraf as Kafka consumer:

[[inputs.kafka_consumer]]
  brokers = ["kafka:9092"]
  topics = ["sensor_events"]
  group_id = "telegraf_ingest"
  data_format = "json"
  consumer_fetch_min = "100KB"
  consumer_fetch_default = "1MB"
  max_undelivered_messages = 10000

Telegraf as Kafka producer:

[[outputs.kafka]]
  brokers = ["kafka:9092"]
  topic = "aggregated_metrics"
  data_format = "json"
  compression_codec = 2  # snappy
  required_acks = -1      # all

---

9. Integration with Data Engineering Pipelines

Apache Airflow DAG Example

from airflow import DAG
from airflow.providers.http.operators.http import HttpOperator
from airflow.operators.python import PythonOperator
from datetime import datetime, timedelta

default_args = {
    'owner': 'data_engineering',
    'retries': 2,
    'retry_delay': timedelta(minutes=5),
}

dag = DAG(
    'influxdb_downsample_hourly',
    schedule='0 * * * *',   # Every hour
    start_date=datetime(2025, 1, 1),
    catchup=False,
)

def generate_line_protocol(**context):
    """Generate downsampled data from raw and write hourly rollup."""
    import requests
    import json

    # Query raw hourly aggregates via SQL (InfluxDB 3)
    raw_data = requests.get(
        f"{INFLUX_HOST}/api/v3/query",
        params={"db": "sensor_raw"},
        headers={"Authorization": f"Bearer {INFLUX_TOKEN}"},
        data={
            "query": """
                SELECT DATE_BIN(INTERVAL '1 hour', time) AS bucket,
                       host, region, AVG(temperature) AS avg_temp,
                       MIN(temperature) AS min_temp, MAX(temperature) AS max_temp,
                       COUNT(*) AS sample_count
                FROM sensor_readings
                WHERE time >= now() - INTERVAL '2 hours'
                  AND time < now() - INTERVAL '1 hour'
                GROUP BY bucket, host, region
            """
        }
    )

    # Convert to line protocol and write to hourly bucket
    points = []
    for row in raw_data.json():
        lp = (
            f"sensor_hourly,host={row['host']},region={row['region']} "
            f"avg_temp={row['avg_temp']},min_temp={row['min_temp']},"
            f"max_temp={row['max_temp']},sample_count={row['sample_count']}i "
            f"{row['bucket']}"
        )
        points.append(lp)

    # Batch write
    requests.post(
        f"{INFLUX_HOST}/api/v2/write",
        params={"bucket": "sensor_hourly", "precision": "ms"},
        headers={"Authorization": f"Bearer {INFLUX_TOKEN}"},
        data="\n".join(points),
    )

Apache Spark / PySpark Integration

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("influxdb_etl").getOrCreate()

# Read from InfluxDB v3 using JDBC (PostgreSQL-compatible driver)
df = spark.read \
    .format("jdbc") \
    .option("url", f"jdbc:postgresql://{INFLUX_HOST}:5432/mydb") \
    .option("query", """
        SELECT time, host, region, temperature
        FROM sensor_readings
        WHERE time >= now() - INTERVAL '24 hours'
    """) \
    .option("user", INFLUX_USER) \
    .option("password", INFLUX_PASS) \
    .option("driver", "org.postgresql.Driver") \
    .load()

# Transform
hourly_agg = df.groupBy(
    F.window("time", "1 hour").alias("bucket"),
    "host", "region"
).agg(
    F.avg("temperature").alias("avg_temp"),
    F.min("temperature").alias("min_temp"),
    F.max("temperature").alias("max_temp"),
    F.count("*").alias("sample_count")
)

# Write back to InfluxDB
hourly_agg.write \
    .format("jdbc") \
    .option("url", f"jdbc:postgresql://{INFLUX_HOST}:5432/downsampled_db") \
    .option("dbtable", "sensor_hourly") \
    .option("user", INFLUX_USER) \
    .option("password", INFLUX_PASS) \
    .mode("append") \
    .save()

Grafana Integration

Grafana is the most common visualization layer for InfluxDB:

Grafana Data Source:
  Type: InfluxDB
  URL: http://influxdb:8086
  Database: mydb     (v1) / Organization: myorg, Bucket: sensor_data (v2/v3)
  Min time interval: 10s (matches collection interval)
  Version: InfluxQL (v1) / Flux (v2) / SQL (v3)

ETL Pipeline Patterns

Pattern 1: Telegraf -> InfluxDB -> Grafana
  (simplest, real-time monitoring)

Pattern 2: Sensors -> MQTT/Kafka -> Telegraf -> InfluxDB -> Grafana
  (Kafka for buffering, backpressure handling)

Pattern 3: Sensors -> Kafka -> Flink/Spark -> InfluxDB -> Airflow (downsample) -> InfluxDB
  (heavy stream processing + scheduled downsampling)

Pattern 4: Application -> InfluxDB -> Airflow/Spark -> Parquet -> S3/Data Lake
  (time-series data lake architecture)

---

10. Comparison: InfluxDB vs TimescaleDB vs Prometheus vs QuestDB

At a Glance

Feature	InfluxDB 3	TimescaleDB	Prometheus	QuestDB
Type	Purpose-built TSDB	PostgreSQL extension	Monitoring + TSDB	Purpose-built TSDB
Engine	Columnar (Parquet/IOx)	Hybrid row-columnar (Hypercore)	Custom TSDB engine	Columnar (custom)
Query Language	SQL + Flux + InfluxQL	Full SQL + PG extensions	PromQL	SQL
Ingest Protocol	Line Protocol, SQL, InfluxDB v2 API	PostgreSQL INSERT, COPY	Push via remote write	Line Protocol, InfluxDB, PostgreSQL wire
Storage	Parquet files on object storage	PostgreSQL on local/cloud disk	Local TSDB blocks	Memory-mapped files + disk
Compression	Good (Parquet)	Excellent (up to 95%)	Good (snappy)	Good
High Availability	Enterprise / Cloud (multi-node)	Streaming replication (PG built-in)	Sidecar (Thanos/Cortex)	Enterprise (pending)
Clustering	Enterprise only	PG-based	Built-in with Thanos	Enterprise
Retention	Per-database config	Per-hypertable via policies	Configurable (local)	Partition-based
Continuous Aggregation	External only (Core)	Built-in (continuous aggregates)	Recording rules	Materialized views
Downsampling	Tasks / external orchestration	Continuous aggregates	Recording rules + federation	SAMPLE BY + scheduled queries
Real-time performance	Excellent (3M+ points/s per node)	Very good (1M+ points/s)	Good (1M samples/s)	Excellent (5M+ points/s)
SQL compatibility	High (PostgreSQL-like)	Complete (PostgreSQL)	None (PromQL only)	High (custom SQL)
Data lake export	Native Parquet output	Via PG tools	Remote write / Thanos	Native Parquet output
Best for...	General TSDB, IoT, app metrics, edge	Teams already on PostgreSQL, need full SQL	Kubernetes monitoring, site reliability	Lowest-latency, financial tick data, HFT

Detailed Comparison

InfluxDB 3

• Strengths: Mature ecosystem (Telegraf, 300+ plugins), multiple deployment options (edge to cloud), native line protocol (de facto standard), good for heterogeneous data sources.
• Weaknesses: Flux deprecation path creates migration friction; Core OSS has 72h retention limit; no built-in continuous aggregates in Core; clustering only in Enterprise.
• Best fit: General-purpose time-series, DevOps/SRE monitoring, IoT/IIoT, replacing legacy historians.

TimescaleDB (Tiger Data)

• Strengths: Full PostgreSQL compatibility (all SQL, all PG extensions, all tools), continuous aggregates with incremental refresh, hierarchical aggregation, excellent compression (up to 95%), hypertables with automatic partitioning, mature replication/PITR/HA from PostgreSQL.
• Weaknesses: Slightly lower raw ingest throughput than InfluxDB 3 or QuestDB; requires PostgreSQL knowledge; not as lightweight for edge deployments; parent company rebranded to Tiger Data (some confusion).
• Best fit: Data teams already on PostgreSQL; workloads needing complex JOINs, transactions, or full SQL analytics; long-term historical storage.

Prometheus

• Strengths: Cloud-native standard for Kubernetes monitoring, simple operational model (single binary), PromQL is excellent for alerting and service-level metrics, pull-based model works well for dynamic infra.
• Weaknesses: Not a general-purpose TSDB (no SQL, no complex aggregations), limited retention (default 15d), single-node, no native HA (requires Thanos/Cortex), poor for IoT or high-cardinality label sets.
• Best fit: Kubernetes and container monitoring, service-level dashboards, alerting (PagerDuty/AlertManager), infra metrics.

QuestDB

• Strengths: Highest raw ingest throughput (claimed 5M+ points/s on single node), lowest query latency for time-bucket aggregations, designed for capital markets/finance, native InfluxDB line protocol and PostgreSQL wire protocol support, SQL-compatible, non-blocking ingestion (immutable append), parallelized and vectorized query execution.
• Weaknesses: Smaller ecosystem (fewer integrations, fewer client libraries), newer project (less mature), clustering is Enterprise-only, less tooling for alerting/monitoring out of the box.
• Best fit: Financial tick data, high-frequency trading, real-time dashboards demanding microsecond query latency, capital markets infrastructure.

Decision Flowchart

Q: What infrastructure are you already running?
  |
  +-- PostgreSQL everywhere?
  |     |--> TimescaleDB (stay in PG ecosystem)
  |
  +-- Kubernetes / containers?
  |     |--> Prometheus for infra monitoring
  |     +--> InfluxDB for app metrics and IoT
  |
  +-- Need lowest possible latency (< 1ms queries)?
  |     |--> QuestDB (financial, HFT)
  |
  +-- Heterogeneous environment, edge devices, IoT?
        |--> InfluxDB (Telegraf ecosystem, multiple deployment options)

Q: What query language do you need?
  |--> Full SQL with JOINs, CTEs, window functions?    -> TimescaleDB or InfluxDB 3
  |--> Time-series specific: PromQL?                   -> Prometheus
  |--> Time-series specific: Flux?                     -> InfluxDB 2.x
  |--> DevOps dashboards: Grafana + search?            -> Any (Grafana supports all)

Q: How much data are you ingesting?
  |--> < 100K points/s                                 -> Any
  |--> 100K-1M points/s                                -> InfluxDB or TimescaleDB
  |--> > 1M points/s                                   -> QuestDB or InfluxDB 3
  |--> > 5M points/s                                   -> QuestDB (benchmark leader)

Version Guidance (InfluxDB Specific)

Deployment	Best For	Query Language
InfluxDB 3 Cloud Serverless	Rapid prototyping, variable workloads	SQL (recommended)
InfluxDB 3 Cloud Dedicated	Predictable production workloads	SQL (recommended)
InfluxDB 3 Enterprise	Self-managed HA production	SQL (recommended)
InfluxDB 3 Core	Edge, dev, prototypes	SQL (recommended)
InfluxDB OSS v2	Existing v2.x deployments	Flux (migrate to SQL when moving to v3)
InfluxDB OSS v1	Legacy, no migration budget yet	InfluxQL (migrate to SQL when possible)

---

Appendix: Quick Reference

Line Protocol Cheatsheet

# Measurement name
weather
# Tags (comma-separated after measurement, space before fields)
weather,location=us-midwest,station=A
# Fields (comma-separated, space before timestamp)
weather temperature=82.0,humidity=71.2
# Timestamp (nanoseconds since epoch; space after fields)
weather temperature=82.0 1465839830100400200

# Data type suffixes:
#   Integer:    value=42i
#   Float:      value=3.14   (default)
#   String:     value="hello world"
#   Boolean:    value=t   (t/true/True/TRUE)
#               value=f   (f/false/False/FALSE)
#   Timestamp:  value=1465839830100400200   (nanosecond)

# Escaping:
#   Commas in tag values:    tag=hello\,world
#   Spaces in tag values:    tag=hello\ world
#   Equals in tag values:    tag=hello\=world
#   Double-quotes in string fields: field="say \"hello\""
#   Backslashes:             tag=path\\to\\dir

Key InfluxDB v2/v3 CLI Commands

# InfluxDB v2
influx bucket create --name my_bucket --retention 30d
influx task create --file downsample.flux
influx query --file my_query.flux

# InfluxDB 3 Core
influxdb3 create database my_db --retention-period 90d
influxdb3 write --db my_db --file data.lp
influxdb3 query --db my_db "SELECT * FROM cpu WHERE time > now() - INTERVAL '1 hour'"

Key TimescaleDB SQL Commands

-- Create hypertable
SELECT create_hypertable('sensor_readings', 'time');

-- Add compression
ALTER TABLE sensor_readings SET (
  timescaledb.compress,
  timescaledb.compress_segmentby = 'sensor_id'
);
SELECT add_compression_policy('sensor_readings', INTERVAL '7 days');

-- Continuous aggregate
CREATE MATERIALIZED VIEW sensor_hourly
WITH (timescaledb.continuous) AS
SELECT time_bucket('1 hour', time) AS bucket,
       sensor_id,
       AVG(value) AS avg_value
FROM sensor_readings
GROUP BY bucket, sensor_id;

---

Generated: 2025-06-05 | InfluxDB 3 (IOx/columnar engine) is the current recommended version. Flux is supported for v2.x compatibility; new projects should prefer SQL.