mirror of
https://github.com/magnus919/agent-skills.git
synced 2026-09-20 08:06:35 +03:00
6f67a34ef1
* feat(skill): cross-pollinate the new tool wave into catalog routing Wire the recent tool skill wave into the two-layer routing graph so the new tool skills are reachable from the methodology skills that own their domains, and vice versa: - methodology -> tool down-routes: platform-engineering -> kubernetes, terraform, telemetry, postgres, grafana; site-reliability-engineering -> telemetry, grafana; data-engineering and backend-engineering -> postgres; frontend-engineering -> mobile-development; verification-methodology -> playwright, documents; technical-documentation -> documents - neckbeard: add mobile-development and documents routing rows plus change-surface coverage entries, and cross-link the lightweight test-hardening path to qa-methodology's bounded mutation-review material - collaboration layer: chief-of-staff-methodology -> slack/notion/email, go-to-market -> crm, conditional-customer-success -> crm; fix the dead seo-content-optimization reference in go-to-market (now seo-audit) - references/skill-triggers.md: add trigger rows for the 14 new skills - bring go-to-market's description up to the quality validator's imperative-verb + negative-boundary requirement and regenerate catalogs Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> * fix(skill): add eval manifest for technical-documentation The eval-coverage ratchet fails on modified skills without a schema-valid manifest once coverage passes 50%. technical-documentation was modified by the routing cross-pollination change and lacked one; add six output-quality cases covering README authorship, API reference generation, CLI help design, agent-facing docs, documentation-site IA, and troubleshooting sections. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> --------- Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
60 lines
5.2 KiB
Markdown
60 lines
5.2 KiB
Markdown
---
|
|
name: data-engineering
|
|
description: Design and operate data infrastructure — database operations (vector,
|
|
relational, graph, time-series), ETL/ELT pipeline design (dbt patterns, incremental
|
|
loading), SQL analytical patterns, data quality monitoring, schema migration, and
|
|
storage infrastructure management. Do not use for statistical analysis or ML model
|
|
development.
|
|
license: MIT
|
|
metadata:
|
|
tags: data-engineering, etl, dbt, sql, database, graph-db, time-series, vector-db,
|
|
migration, data-quality, storage, influxdb, neo4j
|
|
source_repo: https://github.com/magnus919/hermes-profiles
|
|
---
|
|
|
|
# Data Engineering Methodology
|
|
|
|
Data engineering is the operational backbone of data-driven systems. This methodology covers running, maintaining, and evolving data infrastructure — from relational databases and vector stores to graph databases, time-series stores, and the transformation pipelines that move data between them.
|
|
|
|
## The Data Engineer's Domain
|
|
|
|
| You own | You don't own |
|
|
|---------|--------------|
|
|
| Database operations — schema management, indexing, backup/recovery, migration across relational, vector, graph, and time-series stores | Data modeling and schema design — that's the data architect |
|
|
| Data transformation pipelines — dbt models, ETL/ELT patterns, incremental loading, incremental strategies | Statistical analysis and experiments — that's the data scientist |
|
|
| Analytical SQL — window functions, CTEs, query optimization, execution plan analysis, star schema queries | Training infrastructure and model deployment — that's the ML engineer |
|
|
| Graph database operations — Neo4j data modeling, Cypher queries, graph algorithms, import/export | Application-level data access patterns — that's the developer |
|
|
| Time-series database operations — InfluxDB schema design, downsampling, retention policies, Telegraf | Infrastructure provisioning — that's the platform engineer |
|
|
| Data quality monitoring — integrity checks, deduplication, anomaly detection, freshness validation | Visual dashboard design — that's the analyst / product-design-and-ux |
|
|
| Storage infrastructure — capacity planning, performance tuning, archival strategies |
|
|
|
|
## Reference Files
|
|
|
|
| Reference | When to load |
|
|
|-----------|-------------|
|
|
| `references/sql-analytical-patterns.md` | Writing analytical SQL — window functions, CTEs, execution plan reading, star schema queries, engine-specific optimization (PostgreSQL, DuckDB, ClickHouse, BigQuery, Snowflake) |
|
|
| `references/dbt-patterns.md` | Designing data transformation pipelines with dbt — project structure, modeling layers (staging/intermediate/facts/dimensions), materializations, tests, snapshots, Jinja macros, CI/CD, dbt Mesh |
|
|
| `references/etl-pipeline-design.md` | Building reliable data pipelines — extraction strategies (full, incremental, CDC), transformation layers, validation gates, error handling, idempotency |
|
|
| `references/data-quality.md` | Monitoring data integrity — quality dimensions, validation rule types, anomaly detection, deduplication strategies, pipeline health signals |
|
|
| `references/graph-databases.md` | Working with graph databases — Neo4j data modeling, Cypher query patterns (traversal, aggregation, pathfinding), import strategies, graph algorithms, pipeline integration |
|
|
| `references/time-series-databases.md` | Working with time-series databases — InfluxDB data model (measurements, tags, fields), schema design (cardinality), downsampling, retention, Telegraf ingest, comparison with TimescaleDB/QuestDB/Prometheus |
|
|
| `references/vector-db-operations.md` | Managing vector databases — Milvus, Qdrant, Chroma — index types, collection lifecycle, dimension migrations, backup strategies |
|
|
| `references/database-migrations.md` | Schema evolution — zero-downtime migration patterns, rollback planning, versioned schemas, test-first migrations |
|
|
| `references/backup-and-recovery.md` | Backup strategies per data store type, RPO/RTO planning, WAL archiving, snapshot management, recovery plan template |
|
|
|
|
## Related Skills
|
|
|
|
- [postgres](../postgres/SKILL.md) — operating a PostgreSQL server itself: configuration review, index and query-plan diagnosis, vacuum/bloat management, WAL archiving and point-in-time recovery, replication and failover, upgrades. This skill owns the engine-specific runbooks; data-engineering owns the engine-neutral methodology.
|
|
|
|
## Core Principles
|
|
|
|
**Data without integrity is noise** — No pipeline, model, or dashboard is worth more than the quality of the data feeding it. Validate at every boundary.
|
|
|
|
**Design for operability** — Every database, pipeline, and store needs monitoring, backup, and recovery procedures defined before it goes to production. If you can't detect failure, you can't recover from it.
|
|
|
|
**Idempotency is a requirement** — Every pipeline should produce the same result whether it runs once or twice. Duplicate handling is not optional.
|
|
|
|
**Schema changes are code changes** — Every migration needs review, testing, and a rollback plan. Schema drift is technical debt with compounding interest.
|
|
|
|
**Know your storage characteristics** — Access patterns, retention requirements, growth rates, and consistency guarantees determine the right storage architecture. Choose based on data, not familiarity.
|