diff --git a/.gitignore b/.gitignore index bb6d537..e1c1fca 100644 --- a/.gitignore +++ b/.gitignore @@ -95,3 +95,4 @@ Thumbs.db Desktop.ini $RECYCLE.BIN/ *.lnk +graphify-out/ diff --git a/README.md b/README.md index 1d677c8..175d926 100644 --- a/README.md +++ b/README.md @@ -34,6 +34,14 @@ Structure durable research work so people and agents can start with conclusions, An expert-level skill for building **conversational multi-agent systems** with Microsoft's AutoGen framework. Unlike graph-based or role-based orchestration, AutoGen uses **agent-to-agent conversations** as the orchestration primitive. +### [backend-engineering](backend-engineering/SKILL.md) + +Backend engineering methodology — API implementation patterns (REST, gRPC, GraphQL), service architecture (clean/hexagonal/layered), database access patterns, integration and middleware design, error handling, and service-level testing. Language and framework agnostic. + +```bash +cp -r backend-engineering ~/.hermes/skills/ +``` + ### [brand-designer](brand-designer/SKILL.md) Create comprehensive brand identity documentation for any brand. Guides you through documenting strategy, visual identity (logo, color, typography, imagery), voice and tone, application guidelines, governance, and asset inventory. Produces markdown specs, compiled brand books, and brand-compliant images via reference-image-aware generation. Ships 7 templates, a brand-book CLI for validation/compilation, and a generate script for brand card and mockup imagery. @@ -74,6 +82,14 @@ Guide a consent-based conversation that helps people discover how an AI agent co Act as a virtual data architect. Discover data assets, assess maturity, evaluate platforms, design architectures, establish governance, and create migration plans. Covers modern data patterns (data mesh, data lakehouse, streaming, real-time analytics) with vendor evaluation frameworks and maturity models. +### [data-engineering](data-engineering/SKILL.md) + +Data engineering methodology — database operations (vector, relational, graph, time-series), ETL/ELT pipeline design (dbt patterns, incremental loading), SQL analytical patterns, data quality monitoring, schema migration, and storage infrastructure management. Grounded in operational patterns for production data systems. + +```bash +cp -r data-engineering ~/.hermes/skills/ +``` + ### [data-scientist](data-scientist/SKILL.md) PhD-level expertise in data science, statistics, and machine learning. Rigorous statistical methodology, experimental design, causal inference, Bayesian analysis, model selection and diagnostics, and research-grade communication. Ships five analysis scripts (power analysis, assumption diagnostics, model comparison, effect size calculator, experimental design generator) with Python + R dual-language support. @@ -122,6 +138,14 @@ Use a small FlareSolverr JSON CLI for browser-backed GET and POST requests, read Safe Forgejo API v1 CLI for issues, pull requests, repositories, file contents, metadata, webhooks, and user settings. Includes a guarded generic `/api/v1/` route for version-specific endpoints such as Actions and admin APIs. +### [frontend-engineering](frontend-engineering/SKILL.md) + +Frontend engineering methodology — component architecture, state management, API integration, responsive layout, client-side performance, and frontend testing patterns. Framework agnostic, focused on web frontend implementation. + +```bash +cp -r frontend-engineering ~/.hermes/skills/ +``` + ### [ghost-cli](ghost-cli/SKILL.md) Ghost CMS from the terminal. Manage posts and pages, list tags, and check site info. Admin API key from Ghost Integrations. JWT authentication handled automatically. @@ -130,6 +154,14 @@ Ghost CMS from the terminal. Manage posts and pages, list tags, and check site i Deploy, manage, and troubleshoot self-hosted GitHub Actions runners. Covers systemd, Docker, Kubernetes (ARC), and Scale Set Client deployments. +### [go-to-market](go-to-market/SKILL.md) + +CMO methodology — positioning and messaging frameworks (April Dunford's positioning, message hierarchy), customer acquisition strategy (paid, organic, PLG, SLG), brand architecture (brand house vs house of brands), growth modeling (CAC/LTV by channel, cohort analysis), market entry strategy (beachhead, land-and-expand), competitive response (pricing wars, feature races, brand defense). + +```bash +cp -r go-to-market ~/.hermes/skills/ +``` + ### [gutenberg](gutenberg/SKILL.md) Search, download, and extract public-domain books from Project Gutenberg. Look up books by ID or keyword via gutendex, download plain-text and EPUB editions, strip licensing boilerplate, and classify fiction vs non-fiction. Ships a portable Python CLI with zero external dependencies. @@ -174,6 +206,14 @@ Build multi-agent AI systems with LangGraph — the low-level orchestration fram Last.fm music data API from the terminal. Lookup user listening history, get artist/album/track metadata, discover similar music via collaborative filtering, explore global and per-country charts, search, manage tags, and scrobble listening events. API key from last.fm/api/account/create (free). Includes a music discovery pipeline for turning liked tracks into recommendations. +### [legal-strategy](legal-strategy/SKILL.md) + +CLO/General Counsel methodology — regulatory landscape analysis (GDPR, CCPA, AI Act, sector-specific), IP strategy (patent, trademark, trade secret, open source licensing), contract risk assessment (indemnification, liability caps, force majeure), data privacy frameworks (privacy-by-design, DPIAs, data mapping), corporate governance (board responsibilities, fiduciary duties, shareholder rights), employment law (classification, IP assignment, non-competes). + +```bash +cp -r legal-strategy ~/.hermes/skills/ +``` + ### [linear](linear/SKILL.md) Work with Linear teams, projects, cycles, issues, comments, workflow transitions, and documents using a small, dependency-free GraphQL CLI with bounded reads, dry-run previews, and focused reference guidance. @@ -190,6 +230,14 @@ Create maintainable diagrams that render reliably in the documentation surfaces Operate MeshCore Companion radio packet capture over BLE, serial, or TCP. Covers MQTT publishing, TOML and environment configuration, token authentication, Docker, systemd, launchd, NixOS, and evidence-first troubleshooting. +### [ml-engineering](ml-engineering/SKILL.md) + +Machine learning engineering methodology — model training, fine-tuning (LoRA/QLoRA), evaluation, quantization, deployment, and MLOps pipeline design. Grounded in practical engineering patterns for production ML systems. + +```bash +cp -r ml-engineering ~/.hermes/skills/ +``` + ### [nous-branding](nous-branding/SKILL.md) Generate images and content consistent with the Nous Research brand identity. @@ -213,10 +261,34 @@ Open Library book metadata from the terminal. Search books and authors, get work Comprehensive open source contribution guidance — from reading CONTRIBUTING.md and filing good bug reports through branching, committing, PR creation, and the release cycle. Covers both contributor and maintainer workflows with progressive disclosure: a concise orchestrator SKILL.md loads detailed phase references on demand. Includes a portable PR template compliance checker script. Agent disclosure template for AI-assisted contributions. +### [operational-design](operational-design/SKILL.md) + +COO methodology for process design, organizational scaling, operational metrics, compliance and audit, vendor management, and team topology. Covers value stream mapping, BPMN, bottleneck analysis, scaling from 10 to 100 to 1000 people, KPI design, balanced scorecard, SOC 2, ISO 27001, GDPR readiness, RFP processes, SLA design, vendor scorecards, team topologies, Conway's Law, and Dunbar's Number. + +```bash +cp -r operational-design ~/.hermes/skills/ +``` + +### [org-design](org-design/SKILL.md) + +CHRO methodology — organizational design (team topologies, span of control, reporting structures), talent strategy (make-vs-buy, skill taxonomies, succession planning), compensation frameworks (market benchmarking, equity design, leveling), culture architecture (values codification, rituals, psychological safety), organizational health metrics (eNPS, retention risk, engagement surveys), DEI strategy (inclusive design, equitable systems, belonging). + +```bash +cp -r org-design ~/.hermes/skills/ +``` + ### [peertube](peertube/SKILL.md) PeerTube federated video platform from the terminal. Browse videos and channels, search across instances, view server info. OAuth2 login with token persistence. Set PEERTUBE_SERVER to point at any instance. +### [platform-engineering](platform-engineering/SKILL.md) + +Infrastructure as code, CI/CD, container orchestration, service networking — methodology and reference patterns for building and operating internal developer platforms. + +```bash +cp -r platform-engineering ~/.hermes/skills/ +``` + ### [product-design-and-ux](product-design-and-ux/SKILL.md) Turn validated evidence and approved product scope into traceable user-facing behavior: information architecture, plain-language content, task flows, applicable state and recovery models, interface contracts, authorized usability evidence, and observable engineering handoffs. Portable and framework-neutral; routes WCAG/ARIA depth to web-accessibility and formal software acceptance to spec-driven-development. Ships 10 focused references and 6 fillable templates. @@ -229,10 +301,26 @@ Discover product requirements from human stakeholders — map who to talk to, as Product management frameworks for translating validated evidence into prioritized backlogs, documented decisions, specifications, and stakeholder communications. Covers RICE scoring, MoSCoW prioritization, opportunity solution trees, decision logs, spec drafting, and audience-specific stakeholder communication. Ships 7 reference files covering each framework and a source index, plus 2 fillable templates (decision log and spec). Picks up where product-discovery ends. +### [product-strategy](product-strategy/SKILL.md) + +CPO methodology — product vision and strategy (North Star, product principles), competitive analysis and positioning, roadmap prioritization (RICE, Kano, OST), product-market fit frameworks (Sean Ellis test, retention curves), market sizing (TAM/SAM/SOM), platform strategy, product lifecycle management. + +```bash +cp -r product-strategy ~/.hermes/skills/ +``` + ### [pydanticai](pydanticai/SKILL.md) Build production-grade AI agents and graph-based state machines with PydanticAI and PydanticGraph. Covers agent creation, function tools with RunContext dependencies, structured output validation, streaming (text/events/graph nodes), a 20+ capability plugin system with on-demand loading (Thinking, WebSearch, MCP, Hooks, etc.), 16 model providers with FallbackModel and concurrency limiting, multi-agent delegation and programmatic hand-off, comprehensive testing with TestModel/FunctionModel, and the PydanticEvals evaluation framework. Includes the full PydanticGraph API — both BaseNode (class-based) and GraphBuilder (function-based) with parallel map/broadcast operations, joins with reducers, decisions, Mermaid rendering, and step-by-step execution. Ships 8 reference files covering core agents, capabilities/hooks, graph, models/output, patterns/integrations, testing/evals, worked examples, and an API surface quick reference. +### [qa-methodology](qa-methodology/SKILL.md) + +Quality assurance methodology — test strategy design, test automation patterns, regression testing, CI quality gates, test data management, and quality metrics. Grounded in practical patterns for teams that want confident shipping. + +```bash +cp -r qa-methodology ~/.hermes/skills/ +``` + ### [raleigh](raleigh/SKILL.md) Query, search, and download public datasets from the City of Raleigh Open Data portal. Wraps the ArcGIS REST API to access 170+ datasets — crime reports, food inspections, building permits, bike lanes, parks, zoning, traffic, budgets, and more. No API key needed. Ships a Python CLI with catalog, search, info, query, download, and categories commands. diff --git a/backend-engineering/README.md b/backend-engineering/README.md new file mode 100644 index 0000000..c34cf2b --- /dev/null +++ b/backend-engineering/README.md @@ -0,0 +1,26 @@ +# Backend Engineering + +Backend engineering methodology — API implementation patterns (REST, gRPC, GraphQL), service architecture (clean/hexagonal/layered), database access patterns, integration and middleware design, error handling, and service-level testing. Language and framework agnostic. + +## Why Install This Skill + +Your agent gains structured patterns for API design, service architecture, database access, error handling, and integration — instead of improvising each time. + +## What You Get + +| Directory | Purpose | +|-----------|---------| +| `SKILL.md` | Core methodology, trigger conditions, reference index | +| `references/` | Deep-dive reference files loaded on demand | + +## Triggers + +Building or reviewing APIs, designing service layers, implementing database access patterns, adding error handling, or integrating external services. + +## Requirements + +Platform-agnostic. Applicable to any language/framework stack. + +## Quick Start + +Load SKILL.md for the methodology overview and reference table, then load specific references as needed for the task at hand. diff --git a/backend-engineering/SKILL.md b/backend-engineering/SKILL.md new file mode 100644 index 0000000..57d614b --- /dev/null +++ b/backend-engineering/SKILL.md @@ -0,0 +1,49 @@ +--- +name: backend-engineering +description: Backend engineering methodology — API implementation patterns (REST, + gRPC, GraphQL), service architecture (clean/hexagonal/layered), database access + patterns, integration and middleware design, error handling, and service-level testing. + Language and framework agnostic. +license: MIT +metadata: + tags: backend, api, services, server, database, integration, middleware, query-optimization, + testing + source_repo: https://github.com/magnus919/hermes-profiles +--- + +# Backend Engineering Methodology + +Backend engineering is the craft of building the server-side systems that power applications — APIs, services, data access, integrations, and the runtime behavior that makes the architecture real. This methodology covers the implementation patterns between architecture design (technical-architect) and quality validation (reviewer). + +## The Backend Engineer's Domain + +| You own | You don't own | +|---------|--------------| +| API implementation — REST/gRPC/GraphQL endpoints, request validation, response formatting, error handling, middleware chains | API contract and service boundary design — that's the technical-architect | +| Service logic — business rules, workflow orchestration, state management, background job processing | Deployment pipeline and infrastructure — that's the platform-engineer | +| Database access patterns — query design, connection management, transaction boundaries, N+1 detection, pagination | Schema design and migrations — that's the data-architect / data-engineer | +| Integration code — third-party API clients, webhook handlers, message queue consumers/producers | Code review and quality gates — that's the reviewer | +| Observability instrumentation at the service level — structured logging, metrics, tracing hooks | Observability infrastructure — that's the SRE / platform-engineer | +| Service-level tests — unit tests for business logic, integration tests for API contracts | Test strategy and automation — that's the QA-engineer | + +## Reference Files + +| Reference | When to load | +|-----------|-------------| +| `references/api-patterns.md` | Designing or implementing API endpoints — resource modeling, versioning, pagination, error response formats, request validation | +| `references/service-patterns.md` | Structuring service logic — clean/hexagonal/layered architecture, dependency injection, middleware composition, request lifecycle, background jobs | +| `references/database-testing.md` | Database access patterns (connection pooling, query optimization, N+1 detection, pagination strategies, transaction boundaries, read/write splitting, replication lag) and service-level testing (unit testing business logic, integration testing API contracts with test containers/WireMock, contract testing with Pact, test fixtures, CI integration) | +| `references/integration-patterns.md` | Integrating with external systems — retry with backoff, circuit breakers, idempotency keys, webhook verification, message queue consumers | +| `references/error-handling.md` | Handling errors systematically — classification (client vs server), structured responses, exception handling patterns, observability correlation | + +## Core Principles + +**The interface is the contract** — API boundaries are service-level contracts. Every endpoint signature, request schema, response format, and error code is a promise to consumers. Breaking changes are coordination problems, not version bumps. + +**Business logic is the center of gravity** — Keep business rules isolated from framework concerns, transport protocols, and infrastructure details. A well-structured service can survive changes to its HTTP library, database driver, and deployment platform. + +**Handle errors where they make sense** — Catch errors at the boundary where you have enough context to handle them meaningfully. Catch too early and you lose context. Catch too late and you can't recover. + +**Design for failure, not just success** — Every external call can fail. Every database connection can drop. Every message can be duplicated. Idempotency, retry, and graceful degradation are not optimizations — they're requirements. + +**Test at the right level** — Business logic gets unit tests. API contracts get integration tests. Service boundaries get contract tests. Each level catches a different class of failure. diff --git a/backend-engineering/references/api-patterns.md b/backend-engineering/references/api-patterns.md new file mode 100644 index 0000000..ab8d02d --- /dev/null +++ b/backend-engineering/references/api-patterns.md @@ -0,0 +1,51 @@ +# API Patterns + +## Endpoint Design + +| Aspect | REST | gRPC | GraphQL | +|--------|------|------|---------| +| Resource modeling | Nouns as resources, verbs as methods | Services with RPC methods | Schema-defined types and queries | +| Request structure | Path params, query params, headers, body | Protobuf messages | Query/mutation with variables | +| Response structure | JSON with envelope | Protobuf messages | Shape matches query structure | +| Error reporting | HTTP status codes + error body | gRPC status codes + details | Errors in `errors` array | +| Versioning | URL path or header | Package version in proto | Schema evolution with deprecation | +| Pagination | Cursor-based preferred | Token-based in proto args | Connection/edges pattern (Relay) | + +## Versioning Strategies + +| Strategy | Mechanism | Breaking change handling | +|----------|-----------|------------------------| +| URL path | `/v1/resources`, `/v2/resources` | New path, old path maintained | +| Header | `Accept: application/vnd.api+json; version=2` | New accept header value | +| Query param | `?version=2` | New param value, old default maintained | +| No versioning | Evolve in place with additive changes | Only additive changes permitted | + +Prefer URL path versioning for public APIs — it's the most visible and least ambiguous. + +## Pagination + +| Strategy | Pros | Cons | Best for | +|----------|------|------|----------| +| Cursor-based | Stable under writes, no offset drift | Opaque cursors, can't jump to page N | Real-time data, feeds | +| Offset-based | Simple, can jump to any page | Skips/duplicates on writes | Static datasets, admin UIs | +| Keyset | Fast, stable | Requires sort key, complex multi-column | Large datasets, ordered data | + +Always include pagination metadata: `{data: [...], next_cursor: "...", has_more: true}`. + +## Error Response Format + +```json +{ + "error": { + "code": "VALIDATION_ERROR", + "message": "The request body is malformed.", + "details": [ + {"field": "email", "reason": "must be a valid email address"}, + {"field": "age", "reason": "must be a positive integer"} + ], + "request_id": "req_abc123" + } +} +``` + +Every error response should include: machine-readable code, human-readable message, request ID for tracing, and structured details for programmatic handling. diff --git a/backend-engineering/references/database-testing.md b/backend-engineering/references/database-testing.md new file mode 100644 index 0000000..71445f4 --- /dev/null +++ b/backend-engineering/references/database-testing.md @@ -0,0 +1,1084 @@ +# Backend Engineering Methodology Reference + +> Database access patterns & service-level testing — a comprehensive reference for backend engineering teams. +> Compiled: 2026-06-05 + +--- + +## Table of Contents + +1. [Connection Pooling Configuration & Sizing](#1-connection-pooling-configuration--sizing) +2. [Query Optimization — Index Usage, Query Plans, EXPLAIN](#2-query-optimization--index-usage-query-plans-explain) +3. [N+1 Detection & Mitigation](#3-n1-detection--mitigation) +4. [Pagination Strategies — Cursor vs Offset vs Keyset](#4-pagination-strategies--cursor-vs-offset-vs-keyset) +5. [Transaction Boundary Design](#5-transaction-boundary-design) +6. [Read/Write Splitting](#6-readwrite-splitting) +7. [Replication Lag Handling](#7-replication-lag-handling) +8. [Service-Level Testing Overview](#8-service-level-testing-overview) +9. [Unit Testing Business Logic](#9-unit-testing-business-logic) +10. [Integration Testing — API Contracts, Testcontainers, WireMock](#10-integration-testing--api-contracts-testcontainers-wiremock) +11. [Contract Testing — Pact](#11-contract-testing--pact) +12. [Test Fixtures](#12-test-fixtures) +13. [CI Integration](#13-ci-integration) + +--- + +## 1. Connection Pooling Configuration & Sizing + +### The Problem + +Creating a new TCP connection per request does not scale. At 10K+ RPS, the database is overwhelmed. PostgreSQL defaults to 100 simultaneous connections; exceeding that produces "sorry, too many clients already." Each new connection setup adds 20-50 ms of latency. + +### The Solution + +Connection pooling pre-establishes a fixed set of connections at application startup. Threads borrow a connection, execute queries, and return it to the pool. + +``` +┌──────────────┐ borrow ┌──────────────────┐ +│ App Thread │ ─────────────→ │ Connection │ +│ (request) │ │ Pool │ +│ │ ←───────────── │ (HikariCP/ │ +│ │ return │ pgBouncer) │ +└──────────────┘ └────────┬─────────┘ + │ + ┌─────────▼─────────┐ + │ Database Server │ + │ (PostgreSQL/MySQL)│ + └───────────────────┘ +``` + +### Pool Sizing Formula + +The most commonly cited rule of thumb: **pool size = 2x (number of CPU cores)**. + +However, the correct approach is empirical: + +1. **Start small** — 20-30 connections for most services. +2. **Run load tests** with real traffic patterns. Monitor DB CPU, memory, connection wait times, and query latency. +3. **Add a 15-20% buffer** above measured peak usage. +4. **Consider multiple pools** for distinct workload patterns (e.g., small pool for admin queries, larger for user-facing traffic). + +### Key Configuration Parameters + +| Parameter | Description | Common Default | +|-----------|-------------|---------------| +| `maximumPoolSize` | Max connections in the pool | 10-30 | +| `minimumIdle` | Min idle connections to maintain | same as maxPoolSize | +| `connectionTimeout` | Max wait time for a connection (ms) | 30000 | +| `idleTimeout` | Max time a connection stays idle (ms) | 600000 (10 min) | +| `maxLifetime` | Max lifetime of a connection in pool (ms) | 1800000 (30 min) | + +### Recommended Libraries + +| Language | Library | Notes | +|----------|---------|-------| +| Java/Kotlin | **HikariCP** | Industry standard — fastest, lightest | +| Python | **psycopg2.pool / SQLAlchemy pool** | Built-in; tune pool_size and max_overflow | +| Node.js | **pg-pool** | Default pool for node-postgres | +| Go | **pgxpool** (/jackc/pgx) | High-performance Postgres driver | +| Ruby | **connection_pool** | Used by ActiveRecord internally | +| Rust | **deadpool-postgres** | Async pool for tokio-postgres | +| .NET | **Npgsql pooling (built-in)** | Connection pooling enabled by default | + +### Proxy-Based Pooling (pgBouncer / PgCat) + +For microservices or serverless, use a database proxy instead of app-level pooling: + +```ini +[databases] +mydb = host=localhost port=5432 dbname=mydb + +[pgbouncer] +listen_addr = 127.0.0.1 +listen_port = 6432 +pool_mode = transaction # transaction-level pooling +max_client_conn = 100 +default_pool_size = 20 +``` + +- **Transaction pooling** — connections are returned to pool after each transaction (most common). +- **Session pooling** — connection held for entire session (useful for prepared statements). +- **Statement pooling** — connection returned after each statement (rarest). + +### Serverless Considerations + +Serverless functions are short-lived and cannot maintain persistent pools. Use proxy-based solutions: + +- **AWS RDS Proxy** (managed, IAM auth) +- **Cloudflare Hyperdrive** +- **Supabase Supavisor** +- **PgCat** (open-source proxy) + +--- + +## 2. Query Optimization — Index Usage, Query Plans, EXPLAIN + +### Index Types (PostgreSQL-focused) + +| Index Type | Best For | Considerations | +|------------|----------|---------------| +| **B-Tree** (default) | Equality & range queries, ORDER BY, foreign keys | General-purpose; works for most cases | +| **Hash** | Equality lookups only | Single-column; not WAL-logged in older versions | +| **GIN** (Generalized Inverted Index) | Full-text search, arrays, JSONB containment | Larger than B-tree; slower to build | +| **GiST** (Generalized Search Tree) | Geometric data, full-text search (ranking) | Lossy; supports nearest-neighbor | +| **BRIN** (Block Range INdex) | Very large tables with naturally ordered data (time-series, logs) | Extremely compact; only good for correlated data | +| **SP-GiST** | Space-partitioned data (maps, network trees) | Niche; for clustered data | +| **Covering Index** (`INCLUDE` columns) | Index-only scans | Adds payload columns without affecting key sort order | + +### Composite Index Guidelines + +- **Order matters**: column(s) for equality first, then range/ORDER BY columns. +- **Leftmost prefix rule**: a query must use the leftmost columns in the index to benefit from it. +- Example: `CREATE INDEX idx_users_org_status ON users (organization_id, status, created_at);` + - Helps `WHERE org_id = ? AND status = ? ORDER BY created_at` + - Does NOT help `WHERE status = ?` alone. + +### EXPLAIN Fundamentals + +```sql +EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON) SELECT * FROM orders WHERE user_id = 42; +``` + +**Key plan node types:** + +| Node | Meaning | +|------|---------| +| **Seq Scan** | Full table scan — expensive on large tables | +| **Index Scan** | Index lookup + heap fetch | +| **Index Only Scan** | All needed columns in the index itself (fastest) | +| **Bitmap Heap Scan** | Multiple index matches combined into a bitmap | +| **Nested Loop** | For each row in outer, scan inner (good for small joins) | +| **Hash Join** | Build hash table on one side, probe with other | +| **Merge Join** | Sort both sides, merge (good for large sorted sets) | + +### What to Look For in a Query Plan + +1. **Sequential scans on large tables** — missing index. +2. **High `rows` vs `actual rows` discrepancy** — planner has stale statistics; run `ANALYZE`. +3. **`Sort` nodes with large memory** — consider pre-sorted index or increased `work_mem`. +4. **`Nested Loop` joining large row sets** — might need a different join strategy. +5. **`Bitmap Heap Scan` with many row versions** — vacuum might be needed. +6. **`Filter` after index scan** — index is missing a column used in WHERE. + +### Index Maintenance + +```sql +-- Check index usage +SELECT schemaname, tablename, indexname, idx_scan, idx_tup_read, idx_tup_fetch +FROM pg_stat_user_indexes +WHERE idx_scan = 0; -- unused indexes (candidates for removal) + +-- Rebuild bloated indexes +REINDEX INDEX CONCURRENTLY idx_name; -- non-blocking in PG 12+ +``` + +### Common Anti-Patterns + +- Indexing low-cardinality columns (e.g., boolean) alone — not selective enough. +- Over-indexing — each index adds write overhead (INSERT/UPDATE/DELETE slower). +- Missing composite indexes for common query patterns. +- `SELECT *` pulling columns not covered by the index, forcing heap lookups. +- Function calls on indexed columns (`WHERE LOWER(email) = 'x'`) unless using a functional index. + +--- + +## 3. N+1 Detection & Mitigation + +### What Is N+1? + +The N+1 selects problem occurs when an application issues 1 query to fetch N parent rows, then issues N additional queries to fetch related data for each parent — N+1 total queries instead of a single efficient query. + +### Example (ORM-level Pseudocode) + +```python +# N+1: 1 query for users + N queries for orders +users = User.query.all() # 1 query → 100 users +for user in users: + orders = user.orders # 100 queries! + ... +``` + +```sql +-- Queries generated: +SELECT * FROM users; -- 1 +SELECT * FROM orders WHERE user_id = 1; -- 2 +SELECT * FROM orders WHERE user_id = 2; -- ... +SELECT * FROM orders WHERE user_id = 100; -- 101 +``` + +### Detection Techniques + +1. **ORM query logging** — enable SQL logging and watch for repeated similar queries. +2. **APM tools** — Scout, New Relic, Datadog highlight N+1 patterns automatically. +3. **Manual EXPLAIN** — detect many identical queries in a short window. +4. **Static analysis** — Rails' `bullet` gem, Django's `nplusone`, Java's `jpa-nplusone`. +5. **Database-side analysis** — `pg_stat_statements` showing high call counts. + +### Mitigation Strategies + +| Strategy | ORM | How | +|----------|-----|-----| +| **Eager loading (JOIN)** | Django `select_related` / Rails `includes` / Hibernate `JOIN FETCH` | Single query with JOIN | +| **Batch loading** | Django `prefetch_related` / Rails `preload` / Hibernate `@BatchSize` | Separate query per table, batched with `WHERE IN` | +| **GraphQL DataLoader** | Any GraphQL stack | Per-request batching & deduplication | +| **Lazy + batch** | Common in ORMs | Delay execution until accessed, then batch | + +```python +# Fix with eager loading (Django) +users = User.objects.select_related('profile').prefetch_related('orders').all() + +# Fix with DataLoader (GraphQL) +from promise import Promise +from promise.dataloader import DataLoader + +class OrderLoader(DataLoader): + def batch_load_fn(self, user_ids): + orders = Order.objects.filter(user_id__in=user_ids) + return Promise.resolve([list(orders.filter(user_id=uid)) for uid in user_ids]) +``` + +### When N+1 Is Acceptable + +- Small, fixed N (e.g., < 10 related items). +- Admin panels or reports where latency is not critical. +- Cached results with low cache-miss volume. + +--- + +## 4. Pagination Strategies — Cursor vs Offset vs Keyset + +### Offset/Limit (Most Common, Least Scalable) + +```sql +SELECT * FROM orders ORDER BY created_at DESC LIMIT 20 OFFSET 0; +SELECT * FROM orders ORDER BY created_at DESC LIMIT 20 OFFSET 20; +``` + +**Pros:** +- Simple to implement. +- Supports arbitrary page jumps (page 1, page 5, page 100). +- Intuitive for developers. + +**Cons:** +- **Performance degrades with page depth** — OFFSET 100000 must scan/skip 100K rows. +- **Phantom reads / missing rows** — if rows are inserted/deleted between requests, items may appear on multiple pages or be skipped entirely. +- **Inconsistent under write load** — `OFFSET` changes meaning as data shifts. + +### Cursor-Based Pagination (Most Scalable, API-First) + +```sql +-- First page: no cursor +SELECT * FROM orders ORDER BY created_at DESC LIMIT 20; + +-- Next page: use the last item's cursor value +SELECT * FROM orders +WHERE created_at < '2026-06-04T12:00:00Z' -- cursor value +ORDER BY created_at DESC LIMIT 20; +``` + +```json +// API response shape +{ + "data": [...], + "pagination": { + "next_cursor": "eyJpZCI6MTIzNDUsImNyZWF0ZWRfYXQiOiIyMDI2LTA2LTA0VDEyOjAwOjAwWiJ9", + "has_more": true + } +} +``` + +**Pros:** +- **O(1) performance at any depth** — uses index seek, not scan+skip. +- **Consistent** — no phantom reads or missed rows; cursor marks a fixed position. +- **Resilient to write load** — insertion/deletion doesn't shift cursor position. + +**Cons:** +- No arbitrary page jumping (only next/prev). +- Requires a unique, sortable column (usually an ID or timestamp). +- Cursor encoding/decoding overhead (base64, opaque tokens). + +### Keyset Pagination (Seek Method) + +```sql +-- Composite pagination on (created_at, id) +SELECT * FROM orders +WHERE (created_at, id) < ('2026-06-04T12:00:00Z', 12345) +ORDER BY created_at DESC, id DESC +LIMIT 20; +``` + +- Uses a composite index on `(created_at, id)`. +- Similar performance to cursor-based — index seek on the tuple. +- Requires a compound comparison and a multi-column index. + +### Comparison Table + +| Aspect | Offset/Limit | Cursor | Keyset | +|--------|-------------|--------|--------| +| **Performance at depth** | O(n) — degrades | O(1) — constant | O(1) — constant | +| **Random page access** | Yes | No | No | +| **Phantom reads** | Yes | No | No | +| **Consistency** | Unstable | Stable | Stable | +| **Implementation complexity** | Trivial | Medium | Low-Medium | +| **Requires sortable unique column** | No | Yes | Yes | +| **Write-aware** | No | Yes | Yes | + +### Recommendation + +| Use Case | Strategy | +|----------|----------| +| **Admin panels, small datasets** | Offset/Limit (fine for < 10K rows) | +| **Public APIs, infinite scroll** | Cursor (REST/GraphQL best practice) | +| **Time-series, logs, audit trails** | Cursor or Keyset on timestamp + ID | +| **Internal tools with DB pagination** | Keyset (lowest complexity, no cursor encoding) | + +--- + +## 5. Transaction Boundary Design + +### ACID Properties + +| Property | Meaning | +|----------|---------| +| **Atomicity** | All or nothing — transaction either completes fully or has no effect | +| **Consistency** | Transaction leaves DB in a valid state (constraints preserved) | +| **Isolation** | Concurrent transactions do not interfere with each other | +| **Durability** | Committed changes persist through failures | + +### Isolation Levels + +| Level | Dirty Read | Non-Repeatable Read | Phantom Read | Serialization Anomaly | +|-------|-----------|--------------------|-------------|---------------------| +| **Read Uncommitted** | Possible | Possible | Possible | Possible | +| **Read Committed** (default in PostgreSQL, SQL Server, Oracle) | Safe | Possible | Possible | Possible | +| **Repeatable Read** | Safe | Safe | Possible (PG: safe) | Possible | +| **Serializable** | Safe | Safe | Safe | Safe | + +**PostgreSQL specifics:** +- Default is **Read Committed**. +- Repeatable Read also prevents phantom reads (uses snapshot isolation). +- Serializable uses Serializable Snapshot Isolation (SSI) — detects serialization conflicts and aborts. + +### Choosing an Isolation Level + +```sql +SET TRANSACTION ISOLATION LEVEL READ COMMITTED; +-- or for the session: +SET default_transaction_isolation = 'repeatable read'; +``` + +| Level | When to Use | +|-------|------------| +| **Read Committed** | Default for most workloads. Good balance of consistency and performance. | +| **Repeatable Read** | Financial calculations, reporting — when you need consistent snapshots. | +| **Serializable** | Critical data integrity (ledgers, inventory allocation). Higher abort rate. | + +### Transaction Retry Patterns + +**Optimistic retry (for Serializable / Repeatable Read conflicts):** + +``` +RETRY_COUNT = 0 +MAX_RETRIES = 3 +BACKOFF = [50ms, 150ms, 500ms] + +WHILE RETRY_COUNT <= MAX_RETRIES: + BEGIN TRANSACTION + TRY: + -- business logic + COMMIT + BREAK + CATCH serialization_failure: + ROLLBACK + SLEEP(BACKOFF[RETRY_COUNT]) + RETRY_COUNT += 1 + CATCH deadlock: + ROLLBACK + SLEEP(random 0-100ms) + RETRY_COUNT += 1 + +IF RETRY_COUNT > MAX_RETRIES: + RAISE "Transaction failed after retries" +``` + +**Best practices:** +- Use **exponential backoff** with jitter to avoid thundering herd. +- Keep transactions **short** — minimize lock duration. +- **Read before write** — detect conflicts early inside the transaction. +- Use **optimistic locking** (version column) for entity-level concurrency instead of pessimistic locks when possible. + +### Distributed Transactions + +| Pattern | Description | When to Use | +|---------|-------------|------------| +| **Two-Phase Commit (2PC)** | Coordinator prepares all participants, then commits | Within a single database system only | +| **Saga (Choreography)** | Each service publishes events; compensating actions roll back | Microservices, async boundaries | +| **Saga (Orchestration)** | Central orchestrator sends commands and handles compensation | Complex multi-service workflows | +| **Outbox Pattern** | Write events to an outbox table in the same DB transaction, then async publish | Event-driven architecture with exactly-once guarantees | +| **Idempotency Keys** | Unique key per operation prevents duplicate processing | Payment handling, any external API call | + +### Transaction Anti-Patterns + +- **Long-running transactions** that hold locks — split into smaller units. +- **Nested transactions** across service boundaries — use Sagas instead. +- **Transaction inside a loop** — batch the work into a single transaction. +- **Mixing heavy I/O inside a transaction** — external API calls should happen before or after. +- **Not handling retries** for serialization failures — every Serializable workload needs retry logic. + +--- + +## 6. Read/Write Splitting + +### Architecture + +``` + ┌─────────────────┐ + │ Application │ + │ (ORM / Client) │ + └────┬────────┬───┘ + │ │ + Writes Reads + │ │ + ┌────▼──┐ ┌──▼────┐ + │Primary│ │Replica│ ──→ (more replicas) + │(Write)│ │(Read) │ + └───────┘ └───────┘ + │ ↑ + │ Async │ + │ Repl. │ + └──────────┘ +``` + +### Implementation Approaches + +| Approach | Mechanism | Pros | Cons | +|----------|-----------|------|------| +| **ORM-level** (`read_from=replica`) | Config in ORM (Django `DATABASES`, Rails `config`) | Simple; no infra change | Every service must configure manually | +| **Database Proxy** (ProxySQL, PgBouncer, PgCat) | Route based on query type | Centralized; no app changes | Extra hop; proxy becomes SPOF | +| **Middleware** (e.g., Spring `@Transactional(readOnly=true)`) | Annotation-driven routing | Fine-grained control; declarative | Framework-specific | +| **Client-side** (multi-DB driver config) | Connection string per role | Minimal infra | Deploy-time configuration | + +### Query Routing Rules + +``` +Writes → Primary: + - INSERT, UPDATE, DELETE, MERGE + - DDL (CREATE TABLE, ALTER) + - SELECT ... FOR UPDATE (needs primary) + - SELECT inside a read-write transaction + +Reads → Replica: + - SELECT (no locking) + - Read-only transactions (@Transactional(readOnly=true)) + - Reporting queries, analytics +``` + +### When NOT to Read from Replicas + +- **Read-after-write** queries — data may not have replicated yet. +- **Strong consistency** requirements (ledgers, inventory). +- **Tightly coupled** workflows where the next read depends on the previous write. + +### Spring Boot Example (ReadWriteSplit Routing) + +```java +@Transactional(readOnly = true) +public OrderDTO getOrder(Long id) { ... } // routed to replica + +@Transactional +public OrderDTO createOrder(OrderDTO dto) { ... } // routed to primary +``` + +Configure `AbstractRoutingDataSource` with a `@ReadOnlyRepository` annotation or AOP advice to switch between primary and replica `DataSource`. + +--- + +## 7. Replication Lag Handling + +### The Problem + +Asynchronously replicated databases always have some lag between write on the primary and visibility on replicas. This causes: + +- **Read-after-write inconsistency** — user creates a resource, then immediately gets a 404 reading from a stale replica. +- **Monotonic read violation** — user sees a newer version of data, then an older version (from a different replica). +- **Causality violations** — entity A's state depends on entity B, but B's update hasn't arrived yet. + +### Handling Strategies + +| Strategy | Description | Complexity | +|----------|-------------|------------| +| **Read-your-writes (RYW)** | Route reads for recently-written data to the primary | Low | +| **Monotonic reads** | Route a session's reads to the same replica | Low | +| **Bounded staleness** | Reject reads from replicas lagging beyond a threshold | Medium | +| **Causal consistency (GTID)** | Track which transaction IDs the client has seen; ensure replica applies those before serving reads | Medium | +| **Wait-for-replication** | After write, wait for replica to catch up before serving reads | Medium | +| **Synchronous replication** | Primary waits for N replicas before committing | High (latency cost) | + +### Read-Your-Writes (RYW) Pattern + +```python +class DatabaseRouter: + def __init__(self): + self.recent_writes = {} # user_id → timestamp + + def execute_write(self, user_id, query, params): + result = primary.execute(query, params) + self.recent_writes[user_id] = time.now() + return result + + def execute_read(self, user_id, query, params): + last_write = self.recent_writes.get(user_id, 0) + if time.now() - last_write < 5: # 5-second window + return primary.execute(query, params) # use primary + else: + return replica.execute(query, params) # use replica +``` + +### Monotonic Read Consistency (Shopify Pattern) + +Route all related reads to the **same replica** using a hash-based sticky selection: + +```sql +/* consistent_read_id:user_42 */ SELECT * FROM orders WHERE user_id = 42; +``` + +``` +Hash("user_42") % NUM_REPLICAS = replica_index → always hits the same server +``` + +**Trade-off:** Simple and low-overhead; occasional inconsistency if that replica goes down. + +### Wait-for-Replication + +```python +def write_and_wait(data): + primary.execute("INSERT INTO ...", data) + # Wait for the write to arrive on at least one replica + primary.execute("SELECT pg_current_wal_lsn()") # Postgres + # or use pg_stat_replication + +def read_with_consistency(key): + # Check that replica has caught up to a known LSN + replica_lsn = replica.execute("SELECT pg_last_wal_replay_lsn()") + if replica_lsn >= required_lsn: + return replica.read(key) + else: + return primary.read(key) # fallback to primary +``` + +### Strategies by Use Case + +| Use Case | Recommended Strategy | +|----------|---------------------| +| **User-facing web app after form submit** | Read-your-writes (route to primary for 5-30s) | +| **Social feed, timeline** | Monotonic reads (sessions stick to one replica) | +| **Analytics, reporting** | Bounded staleness acceptable; lag of minutes is fine | +| **Inventory, financial ledger** | Always read from primary (strong consistency) | +| **Notifications** | Accept eventual consistency; timestamp-driven dedup | + +--- + +## 8. Service-Level Testing Overview + +``` + Coverage ▲ + │ + ┌─────────┤ + │ E2E │ Few, slow, expensive + ┌───┤ Tests │ + │ └─────────┤ + ┌───┤ │ + │ │ Service │ Medium count, medium speed + ┌───┤ │ (Integ.) │ + │ │ └───────────┤ + ┌───┤ │ │ + │ │ │ Unit │ Many, fast, cheap + │ │ │ Tests │ + └───┴───┴──────────────┘ +``` + +The **Test Pyramid** recommends: +- **Unit tests**: ~70% — fast, deterministic, test business logic in isolation. +- **Integration tests**: ~20% — test boundaries (DB, external APIs). +- **Contract tests**: ~5% — verify API agreements between services. +- **E2E tests**: ~5% — happy-path critical flows. + +--- + +## 9. Unit Testing Business Logic + +### Principles + +- **Test in isolation** — mock/stub all collaborators (DB, file system, network). +- **Focus on logic** — test business rules, transformations, validations, and state changes. +- **Deterministic** — no flaky tests. No external dependencies. +- **Fast** — individual tests complete in milliseconds. + +### What to Unit Test + +```python +# GOOD: Pure business logic — test this +class OrderService: + def calculate_discount(self, order_total, customer_tier): + if customer_tier == 'vip': + return order_total * 0.20 + elif order_total > 1000: + return order_total * 0.10 + else: + return 0 + +# BAD: Impure — involves I/O, mock the boundary instead +class OrderController: + def create_order(self, request): + order = Order(...) + db.save(order) # this is an integration concern + notification.send(order) # mock this in unit tests + return order +``` + +### Repository/Data Layer Abstraction + +Use the **Repository Pattern** to make business logic testable: + +```java +// Business logic — unit testable with mock repository +public class OrderFulfillmentService { + private final OrderRepository orderRepo; + private final InventoryClient inventoryClient; + + public FulfillmentResult fulfillOrder(String orderId) { + Order order = orderRepo.findById(orderId); + if (order == null) return FulfillmentResult.notFound(); + + boolean inStock = inventoryClient.checkStock(order.getSku(), order.getQuantity()); + if (!inStock) return FulfillmentResult.outOfStock(); + + order.setStatus(OrderStatus.FULFILLED); + orderRepo.save(order); + return FulfillmentResult.success(); + } +} +// Unit test: Mock orderRepo and inventoryClient, test all branches +``` + +### Testing Patterns + +| Pattern | Description | +|---------|-------------| +| **Given-When-Then** | Arrange → Act → Assert structure | +| **Parameterized tests** | Test many input combinations with one test method | +| **Property-based testing** | Generate random inputs, assert invariants hold | +| **State-based vs Interaction-based** | Prefer state assertions over verifying mock interactions | + +--- + +## 10. Integration Testing — API Contracts, Testcontainers, WireMock + +### Testcontainers + +**What:** Library that provides lightweight, disposable containers for testing (PostgreSQL, Redis, Kafka, etc.) as JUnit `@Rule` / `@Container`. + +**Why real containers instead of in-memory:** + +| Approach | Issues | +|----------|--------| +| **H2 (in-memory)** | Different SQL dialect, missing features, different behavior under load | +| **SQLite** | No JSONB, no PostGIS, no full-text search, different type coercion | +| **Testcontainers** | Real PostgreSQL/MySQL — 100% behavior match | + +**Example (Java / Spring Boot + Testcontainers):** + +```java +@SpringBootTest +@Testcontainers +class UserRepositoryIntegrationTest { + + @Container + static PostgreSQLContainer postgres = new PostgreSQLContainer<>("postgres:16") + .withDatabaseName("testdb") + .withUsername("test") + .withPassword("test"); + + @DynamicPropertySource + static void configureProperties(DynamicPropertyRegistry registry) { + registry.add("spring.datasource.url", postgres::getJdbcUrl); + registry.add("spring.datasource.username", postgres::getUsername); + registry.add("spring.datasource.password", postgres::getPassword); + } + + @Autowired + private UserRepository userRepository; + + @Test + void shouldPersistAndRetrieveUser() { + User user = new User("alice@example.com", "Alice"); + User saved = userRepository.save(user); + + Optional found = userRepository.findByEmail("alice@example.com"); + assertThat(found).isPresent(); + assertThat(found.get().getName()).isEqualTo("Alice"); + } +} +``` + +**Testcontainers in other languages:** + +| Language | Library | +|----------|---------| +| Python | `testcontainers` (pip) | +| Node.js | `testcontainers` (npm) | +| Go | `testcontainers-go` | +| .NET | `Testcontainers for .NET` | +| Rust | `testcontainers` (crate) | + +### WireMock + +**What:** HTTP-based API mock server. Stub external HTTP services during integration tests. + +```java +@SpringBootTest +@WireMockTest(httpPort = 8089) +class PaymentServiceIntegrationTest { + + @Test + void shouldProcessPaymentWhenGatewayRespondsSuccess() { + // Arrange: stub the external payment gateway + stubFor(post(urlEqualTo("/gateway/charge")) + .willReturn(aResponse() + .withStatus(200) + .withHeader("Content-Type", "application/json") + .withBody(""" + { "status": "success", "transaction_id": "txn_123" } + """))); + + // Act + PaymentResult result = paymentService.charge(new Payment("user_1", 50.00)); + + // Assert + assertThat(result.isSuccess()).isTrue(); + assertThat(result.getTransactionId()).isEqualTo("txn_123"); + } + + @Test + void shouldHandleGatewayTimeoutGracefully() { + stubFor(post(urlEqualTo("/gateway/charge")) + .willReturn(aResponse() + .withStatus(504))); + + assertThrows(PaymentGatewayTimeoutException.class, () -> { + paymentService.charge(new Payment("user_1", 50.00)); + }); + } +} +``` + +**WireMock capabilities:** +- Stub based on URL, HTTP method, headers, body. +- Simulate delays, timeouts, and network failures. +- Record/playback (proxying real APIs during development). +- Verify requests were made (assert on expected interactions). +- Fault injection (malformed responses, connection resets). + +### Integration Test Best Practices + +1. **Test the boundary** — Repository tests with Testcontainers, external API tests with WireMock. +2. **Keep tests independent** — each test gets its own transaction or container state. +3. **Clean up between tests** — truncate tables or use transactional rollback. +4. **Use realistic data** — edge cases that trigger unique constraints, nulls, long strings. +5. **Don't test the framework** — you don't need to test that Hibernate/JPA/ActiveRecord works. +6. **Name tests by behavior** — `shouldRejectOrderWhenInventoryExhausted()`, never `testOrder1()`. + +--- + +## 11. Contract Testing — Pact + +### What Is Contract Testing? + +Contract testing verifies that two services (consumer and provider) can communicate correctly by testing each side independently against a shared contract — without deploying both services. + +### Pact Workflow + +``` +1. Consumer writes expectations (Pact file) + ┌──────────┐ ┌──────────┐ + │ Consumer │ ── generates ──→│ Pact │ + │ Tests │ │ File │ + └──────────┘ └────┬─────┘ + │ +2. Provider verifies against Pact │ + ┌──────────┐ │ + │ Provider │ ←── verifies ────────│ + │ Tests │ │ + └──────────┘ │ + │ +3. Pact Broker stores & diff │ + ┌──────────────┐ │ + │ Pact Broker │ ←── stores ──────│ + │ (versioned) │ │ + └──────┬───────┘ │ + │ │ +4. Can-I-Deploy checks versions │ + ┌──────────┐ │ + │ CI/CD │ ←── compatibility ───│ + └──────────┘ │ +``` + +### Consumer-Side Test (Pact) + +```java +@ExtendWith(PactConsumerTestExt.class) +@PactTestFor(providerName = "PaymentProvider", port = "8080") +class OrderServiceConsumerPactTest { + + @Pact(consumer = "OrderService") + public V4Pact createPact(PactDslWithProvider builder) { + return builder + .given("a payment method exists with ID 'pm_1'") + .uponReceiving("a request to charge a payment") + .path("/gateway/charge") + .method("POST") + .headers("Content-Type", "application/json") + .body(new PactDslJsonBody() + .stringType("payment_method_id", "pm_1") + .decimalType("amount", 49.99) + ) + .willRespondWith() + .status(200) + .headers("Content-Type", "application/json") + .body(new PactDslJsonBody() + .stringType("status", "success") + .stringType("transaction_id", "txn_abc123") + ) + .toPact(); + } + + @Test + @PactTestFor(pactMethod = "createPact") + void shouldChargePaymentSuccessfully(MockServer mockServer) { + PaymentClient client = new PaymentClient(mockServer.getUrl()); + PaymentResponse response = client.charge("pm_1", 49.99); + assertThat(response.getStatus()).isEqualTo("success"); + } +} +``` + +### Provider-Side Verification + +```java +@Provider("PaymentProvider") +@PactBroker(url = "${pactbroker.url}") +@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT) +class PaymentProviderPactVerificationTest { + + @LocalServerPort + int port; + + @BeforeEach + void setup(PactVerificationContext context) { + context.setTarget(new HttpTestTarget("localhost", port)); + } + + @TestTemplate + @ExtendWith(PactVerificationInvocationContextProvider.class) + void pactVerificationTestTemplate(PactVerificationContext context) { + context.verifyInteraction(); + } + + @State("a payment method exists with ID 'pm_1'") + void setupPaymentMethod() { + // Set up test data — this runs before the provider is called + paymentMethodRepository.save(new PaymentMethod("pm_1", ...)); + } +} +``` + +### Pact Best Practices + +- **Version both consumer and provider** — Pact Broker tracks compatibility matrix. +- **Use `can-i-deploy`** — the `pact-broker can-i-deploy` command checks if two versions are compatible before deploying. +- **Don't over-specify** — use matchers (`stringType`, `decimalType`) instead of exact values for most fields. Exact values should only be for fields where the value matters (e.g., status enums). +- **Tag pacts by environment** — tag pact versions with "prod", "staging" to gate deployments. +- **Run provider verification in CI** — not just locally. Break the build if a provider change breaks a consumer contract. + +--- + +## 12. Test Fixtures + +### What Are Test Fixtures? + +Test fixtures are predefined data setups that provide a known baseline state before tests run. They reduce duplication and make tests readable. + +### Fixture Strategies + +| Strategy | Description | Best For | +|----------|-------------|----------| +| **Inline (test-local)** | Create data directly in the test method | Simple, focused tests | +| **Factory methods** | Helper functions that create objects with sensible defaults | Most cases — flexible, composable | +| **Factory Boy / build()** | Use a library to generate test objects | Complex object graphs | +| **Seed SQL files** | Pre-populated SQL inserts loaded before test suite | Integration + E2E tests | +| **JSON/YAML snapshots** | Load test data from fixture files | When data is complex and nested | + +### Example: Factory Pattern (Python) + +```python +# factories.py +class UserFactory: + @staticmethod + def create( + email="test@example.com", + name="Test User", + tier="standard", + balance=Decimal("100.00") + ): + return User( + email=email, + name=name, + tier=tier, + balance=balance + ) + +# test_discount.py +def test_vip_discount(): + vip = UserFactory.create(tier="vip", balance=Decimal("500.00")) + result = discount_service.calculate(vip, 200) + assert result == Decimal("40.00") # 20% VIP discount +``` + +### Factory Boy (Python) / Builders (Java) + +```python +import factory + +class OrderFactory(factory.Factory): + class Meta: + model = Order + + id = factory.Sequence(lambda n: n) + user = factory.SubFactory(UserFactory) + total = Decimal("100.00") + status = OrderStatus.PENDING + created_at = factory.Faker("date_time_this_year") + +# Usage — only override what matters +order = OrderFactory.create(status=OrderStatus.FULFILLED) +assert order.user.email == "test@example.com" # default from UserFactory +``` + +### Fixture Anti-Patterns + +- **Shared mutable fixtures** — tests that mutate shared state cause flaky ordering dependencies. +- **Too much data** — loading 1000 rows for every test is slow; use the minimum needed. +- **Copy-paste fixtures** — leads to drift; use factories with default values. +- **Magic numbers** — use named constants: `UNIT_PRICE = Decimal("10.00")` instead of bare `10.00`. + +--- + +## 13. CI Integration + +### Test Execution in CI + +``` +┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌──────────────┐ +│ Lint & │ │ Unit │ │ Integration │ │ Contract │ +│ Static │ ──→ │ Tests │ ──→ │ Tests │ ──→ │ Tests / │ +│ Analysis │ │ (fast, par) │ │ (slower) │ │ E2E Tests │ +└─────────────┘ └─────────────┘ └─────────────┘ └──────────────┘ + < 2 min < 5 min < 15 min < 30 min +``` + +### Parallelization + +- **Unit tests** — run in parallel across CPU cores (pytest-xdist, JUnit parallel). +- **Integration tests** — parallel by service/module; isolate with Testcontainers per test class. +- **Contract tests** — consumer tests in parallel; provider tests sequentially per pact file. + +### CI Pipeline Example (GitHub Actions) + +```yaml +name: CI +on: [push, pull_request] + +jobs: + unit-tests: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-java@v4 + with: { java-version: '21', distribution: 'temurin' } + - run: ./gradlew test --parallel # unit tests only + + integration-tests: + needs: unit-tests + runs-on: ubuntu-latest + services: + postgres: + image: postgres:16 + env: + POSTGRES_PASSWORD: test + options: >- + --health-cmd pg_isready + --health-interval 10s + --health-timeout 5s + --health-retries 5 + steps: + - uses: actions/checkout@v4 + - run: ./gradlew integrationTest --tests *IntegrationTest + # Alternatively, use Testcontainers which starts containers in-test + + contract-tests: + needs: unit-tests + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - run: ./gradlew pactVerify # provider-side verification + - run: ./gradlew pactPublish # publish to Pact Broker + + pact-can-i-deploy: + needs: contract-tests + runs-on: ubuntu-latest + steps: + - run: pact-broker can-i-deploy + --pacticipant OrderService + --version $(cat version.txt) + --to-environment production + + e2e-tests: + needs: [integration-tests, pact-can-i-deploy] + runs-on: ubuntu-latest + steps: + - run: docker compose -f docker-compose.e2e.yml up --abort-on-container-exit +``` + +### CI Best Practices + +| Practice | Rationale | +|----------|-----------| +| **Fail fast** | Run fastest tests first (unit → integration → E2E). | +| **Cache dependencies** | Maven/Gradle/npm/pip caches speed up repeat builds. | +| **Cache Docker layers** | Testcontainers pulls — pre-warm image caches. | +| **Isolate flaky tests** | Quarantine flaky tests; don't let them block the pipeline. | +| **Test against production-like databases** | Use Testcontainers with the same DB version as production. | +| **Run pact verification as a required check** | Never deploy a provider that breaks a consumer contract. | +| **Use test reports as artifacts** | Publish JUnit XML / HTML reports for debugging. | + +### Test Run Optimization + +- **Selective test execution** — only run tests for changed modules (gradle `--changed-latest`, `pytest --last-failed`). +- **Test splitting** — split integration tests across multiple CI runners (`--shard` flags). +- **Docker layer reuse** — Dockerfile changes cause full rebuilds; keep rarely-changed layers early. +- **Database migrations in CI** — run migrations once, snapshot the DB, restore for each test runner. + +--- + +## References & Further Reading + +- **PostgreSQL Documentation** — [EXPLAIN](https://www.postgresql.org/docs/current/using-explain.html), [Transaction Isolation](https://www.postgresql.org/docs/current/transaction-iso.html) +- **HikariCP** — [GitHub](https://github.com/brettwooldridge/HikariCP) (connection pool sizing) +- **PgBouncer** — [Official docs](https://www.pgbouncer.org/) (transaction pooling) +- **Pact** — [Documentation](https://docs.pact.io/) (contract testing) +- **Testcontainers** — [Official site](https://testcontainers.com/) (integration testing) +- **WireMock** — [Official site](https://wiremock.org/) (API mocking) +- **Shopify Engineering** — [Read Consistency with Database Replicas](https://shopify.engineering/read-consistency-database-replicas) +- **Crunchy Data** — [Postgres Indexes for Newbies](https://www.crunchydata.com/blog/postgres-indexes-for-newbies) +- **Scout APM** — [Understanding N+1 Database Queries](https://www.scoutapm.com/blog/understanding-n1-database-queries) +- **AWS** — [RDS Proxy](https://aws.amazon.com/rds/proxy/) (serverless connection pooling) diff --git a/backend-engineering/references/error-handling.md b/backend-engineering/references/error-handling.md new file mode 100644 index 0000000..c63b924 --- /dev/null +++ b/backend-engineering/references/error-handling.md @@ -0,0 +1,46 @@ +# Error Handling + +## Error Classification + +| Category | HTTP analogue | What it means | Example | +|----------|--------------|---------------|---------| +| Validation | 400 | The client sent something wrong | Missing required field | +| AuthN/AuthZ | 401/403 | The caller can't do this | Expired token, insufficient permissions | +| Not Found | 404 | The resource doesn't exist | Invalid ID, deleted entity | +| Conflict | 409 | The operation can't complete due to state | Duplicate, stale version | +| Rate Limited | 429 | Too many requests | Quota exceeded | +| Internal | 500 | Something went wrong on the server | DB down, unhandled exception | +| Unavailable | 503 | The service can't handle the request right now | Circuit open, overloaded | + +## Exception Handling Strategy + +| Catch location | What to do | Example | +|---------------|------------|---------| +| Repository | Wrap DB errors in domain exceptions | `UserNotFoundException`, `DuplicateEmailError` | +| Service | Handle domain exceptions, orchestrate recovery | Retry on conflict, fallback on unavailable | +| Controller boundary | Map domain exceptions to error responses | `UserNotFoundException` → 404 with error body | +| Middleware boundary | Catch unhandled exceptions, log, return 500 | Global error handler, structured log + trace | + +## Structured Logging Fields + +Every log entry at service level should include: + +- `request_id` — correlation ID from request header or generated at middleware +- `service` — service name +- `operation` — what operation was being performed +- `duration_ms` — how long it took +- `error_code` — if error, machine-readable code +- `caller` — function/module that produced the log + +## Error Response Body + +```json +{ + "error": { + "code": "RATE_LIMITED", + "message": "Too many requests. Please retry after the specified time.", + "retry_after_seconds": 30, + "request_id": "req_abc123" + } +} +``` diff --git a/backend-engineering/references/integration-patterns.md b/backend-engineering/references/integration-patterns.md new file mode 100644 index 0000000..79a9fbf --- /dev/null +++ b/backend-engineering/references/integration-patterns.md @@ -0,0 +1,46 @@ +# Integration Patterns + +## External Call Resilience + +| Pattern | What it protects against | Implementation | +|---------|------------------------|----------------| +| Retry with backoff | Transient failures | Exponential backoff + jitter, max retries | +| Circuit breaker | Sustained failures | Open after N failures, half-open after timeout | +| Timeout | Hanging connections | Connect + read + write timeouts per operation | +| Bulkhead | Cascading failures | Separate thread pool / connection pool per dependency | +| Idempotency key | Duplicate requests | Client-generated key, server deduplicates | + +## Idempotency Key Pattern + +```http +POST /api/payments +Idempotency-Key: 7c3d5e8f-a1b2-4c3d-8e5f-6a7b8c9d0e1f +``` + +| Aspect | Design | +|--------|--------| +| Key generation | Client UUID v4 | +| Storage | Key-value store with TTL (24h) | +| First request | Process normally, store result keyed by idempotency key | +| Duplicate request | Return stored result, no side effects | +| Expired key | Process as new request | +| In-flight request | Return 409 Conflict | + +## Webhook Verification + +| Mechanism | What it verifies | Implementation | +|-----------|-----------------|----------------| +| HMAC signature | Payload integrity, sender authenticity | Shared secret → HMAC of body → compare header | +| Webhook secret | Sender identity | Pre-shared secret, rotated periodically | +| Timestamp freshness | Replay prevention | Reject webhooks older than N minutes | + +## Message Queue Consumer Patterns + +| Pattern | When to use | +|---------|-------------| +| At-least-once delivery | Default — requires idempotent processing | +| Exactly-once (dedup) | When duplicates are unacceptable — requires dedup store | +| Batch processing | High throughput, latency-tolerant | +| Dead letter queue | Messages that can't be processed after max retries | + +Every consumer should: acknowledge after processing, retry on transient failure, DLQ on permanent failure, and log at every stage. diff --git a/backend-engineering/references/service-patterns.md b/backend-engineering/references/service-patterns.md new file mode 100644 index 0000000..4d537dd --- /dev/null +++ b/backend-engineering/references/service-patterns.md @@ -0,0 +1,38 @@ +# Service Patterns + +## Architecture Styles + +| Style | Separation axis | Best for | Tradeoff | +|-------|----------------|----------|----------| +| Layered | Technical layer (controller → service → repository) | Simple CRUD services, convention-based frameworks | Business logic leaks across layers | +| Clean Architecture | Dependency direction (outer → inner) | Complex business logic, long-lived projects | Boilerplate for interfaces | +| Hexagonal (Ports & Adapters) | External vs internal (ports as boundaries) | Services with multiple I/O sources | More interfaces upfront | +| Pipeline | Request flow through stages | Data processing, middleware-heavy services | Composable but hard to trace | + +## Request Lifecycle + +``` +Request → Middleware 1 → Middleware N → Router → Controller → Service → Repository → Database + ↓ + Response ← Middleware N ← Middleware 1 ← +``` + +Each layer has a distinct responsibility: + +| Layer | Responsibility | Doesn't do | +|-------|---------------|------------| +| Middleware | Auth, logging, rate limiting, CORS, tracing | Business logic, data access | +| Controller | Request parsing, validation, response formatting | Business decisions, database queries | +| Service | Business rules, workflow orchestration, state mgmt | HTTP concerns, direct database access | +| Repository | Data access, query construction, result mapping | Business rules, request parsing | + +## Background Job Processing + +| Pattern | When to use | Concerns | +|---------|-------------|----------| +| In-process worker | Lightweight, no external deps | Memory, process lifecycle, scaling | +| Message queue | Reliable async processing | Queue management, retry, DLQ | +| Scheduled cron | Periodic batch work | Timing guarantees, overlap | +| Event-driven streaming | Real-time event processing | State management, ordering | + +Every background job should be: idempotent, retryable, and have a defined failure path (dead letter or alert). diff --git a/data-engineering/README.md b/data-engineering/README.md new file mode 100644 index 0000000..558f2f3 --- /dev/null +++ b/data-engineering/README.md @@ -0,0 +1,26 @@ +# Data Engineering + +Data engineering methodology — database operations (vector, relational, graph, time-series), ETL/ELT pipeline design (dbt patterns, incremental loading), SQL analytical patterns, data quality monitoring, schema migration, and storage infrastructure management. Grounded in operational patterns for production data systems. + +## Why Install This Skill + +Your agent gets operational patterns for production data systems — real SQL, dbt models, backup commands, and migration strategies instead of textbook theory. + +## What You Get + +| Directory | Purpose | +|-----------|---------| +| `SKILL.md` | Core methodology, trigger conditions, reference index | +| `references/` | Deep-dive reference files loaded on demand | + +## Triggers + +Designing ETL/ELT pipelines, writing analytical SQL, operating vector/graph/time-series databases, planning migrations, or setting up data quality monitoring. + +## Requirements + +Platform-agnostic. References cover PostgreSQL, DuckDB, ClickHouse, BigQuery, Snowflake, Neo4j, InfluxDB, TimescaleDB, and dbt. + +## Quick Start + +Load SKILL.md for the methodology overview and reference table, then load specific references as needed for the task at hand. diff --git a/data-engineering/SKILL.md b/data-engineering/SKILL.md new file mode 100644 index 0000000..658b195 --- /dev/null +++ b/data-engineering/SKILL.md @@ -0,0 +1,55 @@ +--- +name: data-engineering +description: Data engineering methodology — database operations (vector, relational, + graph, time-series), ETL/ELT pipeline design (dbt patterns, incremental loading), + SQL analytical patterns, data quality monitoring, schema migration, and storage + infrastructure management. Grounded in operational patterns for production data + systems. +license: MIT +metadata: + tags: data-engineering, etl, dbt, sql, database, graph-db, time-series, vector-db, + migration, data-quality, storage, influxdb, neo4j + source_repo: https://github.com/magnus919/hermes-profiles +--- + +# Data Engineering Methodology + +Data engineering is the operational backbone of data-driven systems. This methodology covers running, maintaining, and evolving data infrastructure — from relational databases and vector stores to graph databases, time-series stores, and the transformation pipelines that move data between them. + +## The Data Engineer's Domain + +| You own | You don't own | +|---------|--------------| +| Database operations — schema management, indexing, backup/recovery, migration across relational, vector, graph, and time-series stores | Data modeling and schema design — that's the data architect | +| Data transformation pipelines — dbt models, ETL/ELT patterns, incremental loading, incremental strategies | Statistical analysis and experiments — that's the data scientist | +| Analytical SQL — window functions, CTEs, query optimization, execution plan analysis, star schema queries | Training infrastructure and model deployment — that's the ML engineer | +| Graph database operations — Neo4j data modeling, Cypher queries, graph algorithms, import/export | Application-level data access patterns — that's the developer | +| Time-series database operations — InfluxDB schema design, downsampling, retention policies, Telegraf | Infrastructure provisioning — that's the platform engineer | +| Data quality monitoring — integrity checks, deduplication, anomaly detection, freshness validation | Visual dashboard design — that's the analyst or UX designer | +| Storage infrastructure — capacity planning, performance tuning, archival strategies | + +## Reference Files + +| Reference | When to load | +|-----------|-------------| +| `references/sql-analytical-patterns.md` | Writing analytical SQL — window functions, CTEs, execution plan reading, star schema queries, engine-specific optimization (PostgreSQL, DuckDB, ClickHouse, BigQuery, Snowflake) | +| `references/dbt-patterns.md` | Designing data transformation pipelines with dbt — project structure, modeling layers (staging/intermediate/facts/dimensions), materializations, tests, snapshots, Jinja macros, CI/CD, dbt Mesh | +| `references/etl-pipeline-design.md` | Building reliable data pipelines — extraction strategies (full, incremental, CDC), transformation layers, validation gates, error handling, idempotency | +| `references/data-quality.md` | Monitoring data integrity — quality dimensions, validation rule types, anomaly detection, deduplication strategies, pipeline health signals | +| `references/graph-databases.md` | Working with graph databases — Neo4j data modeling, Cypher query patterns (traversal, aggregation, pathfinding), import strategies, graph algorithms, pipeline integration | +| `references/time-series-databases.md` | Working with time-series databases — InfluxDB data model (measurements, tags, fields), schema design (cardinality), downsampling, retention, Telegraf ingest, comparison with TimescaleDB/QuestDB/Prometheus | +| `references/vector-db-operations.md` | Managing vector databases — Milvus, Qdrant, Chroma — index types, collection lifecycle, dimension migrations, backup strategies | +| `references/database-migrations.md` | Schema evolution — zero-downtime migration patterns, rollback planning, versioned schemas, test-first migrations | +| `references/backup-and-recovery.md` | Backup strategies per data store type, RPO/RTO planning, WAL archiving, snapshot management, recovery plan template | + +## Core Principles + +**Data without integrity is noise** — No pipeline, model, or dashboard is worth more than the quality of the data feeding it. Validate at every boundary. + +**Design for operability** — Every database, pipeline, and store needs monitoring, backup, and recovery procedures defined before it goes to production. If you can't detect failure, you can't recover from it. + +**Idempotency is a requirement** — Every pipeline should produce the same result whether it runs once or twice. Duplicate handling is not optional. + +**Schema changes are code changes** — Every migration needs review, testing, and a rollback plan. Schema drift is technical debt with compounding interest. + +**Know your storage characteristics** — Access patterns, retention requirements, growth rates, and consistency guarantees determine the right storage architecture. Choose based on data, not familiarity. diff --git a/data-engineering/references/backup-and-recovery.md b/data-engineering/references/backup-and-recovery.md new file mode 100644 index 0000000..ed2cb5c --- /dev/null +++ b/data-engineering/references/backup-and-recovery.md @@ -0,0 +1,75 @@ +# Backup and Recovery + +## RPO and RTO + +| Term | Definition | How to set | +|------|------------|------------| +| **RPO** (Recovery Point Objective) | Maximum acceptable data loss in time | How much data can you afford to lose? 1 hour? 1 day? 1 week? | +| **RTO** (Recovery Time Objective) | Maximum acceptable downtime | How long can the system be unavailable? 5 minutes? 1 hour? 1 day? | + +## Backup Strategies by Data Store Type + +### Relational Databases (PostgreSQL, MySQL) + +| Method | RPO | RTO | Storage | Best for | +|--------|-----|-----|---------|----------| +| Logical dump (pg_dump) | Point-in-time | Slow for large DBs | Large (SQL text) | Small DBs, schema-only backups | +| Physical backup (pg_basebackup) | Point-in-time if WAL archived | Fast | Moderate + WAL | Production deployments | +| WAL archiving + PITR | Continuous (every WAL segment) | Fast (base + replay WAL) | Moderate + continuous WAL | Maximum data protection | +| Replica-based (standby) | Zero (async) or near-zero (sync) | Minutes (promote replica) | Full replica storage | HA + backup combined | + +### Vector Databases (Milvus, Qdrant, Chroma) + +| Method | Considerations | +|--------|---------------| +| Milvus backup | `milvus-backup` tool for collection-level backup. Backup index + data separately. Restore requires matching index type. | +| Qdrant snapshot | Built-in `POST /collections/{name}/snapshots`. Snapshot full collection. Restore creates new collection. | +| Chroma | `chroma export` for collection export. File-based storage can use filesystem snapshots. | +| General vector DB | Backup embedding dimension must match target. Index rebuild required after restore. Always verify row count and sample queries. | + +### Graph Databases (Neo4j) + +| Method | Command | Frequency | +|--------|---------|-----------| +| Online backup (Enterprise) | `neo4j-admin backup` | Daily | +| Dump (Cypher-based) | `neo4j-admin dump --database=neo4j --to=backup.dump` | Daily/weekly | +| Causal cluster | Built-in replication across cluster members | Continuous | +| Offline copy | Stop DB → copy data directory → restart | Maintenance windows only | + +### Time-Series Databases (InfluxDB) + +| Method | Notes | +|--------|-------| +| InfluxDB backup | `influx backup` CLI for bucket-level backup. Includes data + metadata. | +| Downsample + retain | For time-series, consider downsampled archives vs raw data retention. Raw data may not need point-in-time recovery if derivable from sources. | + +### Embedded / File-Based (SQLite, DuckDB) + +| Method | Best practice | +|--------|---------------| +| WAL mode | Enable WAL journaling for crash recovery | +| File copy (with checkpoint) | `PRAGMA wal_checkpoint(TRUNCATE);` then copy file | +| `.backup` command | `sqlite3 db.sqlite '.backup /backup/db.sqlite'` | +| Replication (SQLite) | Litestream, rqlite for continuous backup | + +## Backup Testing Cadence + +| Type | Frequency | What to verify | +|------|-----------|----------------| +| Automated restore test | Weekly | Restore from latest backup, run integrity checks | +| Full DR drill | Quarterly | Complete recovery from scratch, measure RTO | +| RPO validation | Monthly | Verify WAL/snapshot frequency meets RPO targets | +| Corruption check | Daily | `pg_amcheck`, `sqlite3 db.sqlite 'PRAGMA integrity_check'` | + +## Recovery Plan Template + +``` +1. **Assess** — What failed? Data loss? Schema corruption? Infrastructure failure? +2. **Select backup** — Which backup to restore from (latest clean, T+1, T-1)? +3. **Restore** — Restore data to recovery environment +4. **Verify** — Run integrity checks, sample queries, row count validation +5. **Replay** — Apply WAL/logs to reach target point-in-time +6. **Cut over** — Point applications to restored instance +7. **Validate** — Application smoke test, data freshness check +8. **Communicate** — RTO met? Data loss within RPO? Root cause? +``` diff --git a/data-engineering/references/data-quality.md b/data-engineering/references/data-quality.md new file mode 100644 index 0000000..509f61e --- /dev/null +++ b/data-engineering/references/data-quality.md @@ -0,0 +1,56 @@ +# Data Quality Monitoring + +## Quality Dimensions + +| Dimension | What it measures | Example violation | +|-----------|-----------------|-------------------| +| **Completeness** | Are all required values present? | Null in a required field | +| **Uniqueness** | Are there duplicate records? | Same primary key appearing twice | +| **Consistency** | Are values coherent across systems? | Customer name differs between CRM and billing | +| **Accuracy** | Do values reflect reality? | Wrong currency code, stale address | +| **Timeliness** | Is data current enough? | Batch pipeline 4 hours behind schedule | +| **Validity** | Do values conform to expected format? | Email address missing `@` | +| **Integrity** | Are referential relationships intact? | Order references a deleted customer | + +## Validation Rule Types + +| Rule type | What it does | SQL example | +|-----------|-------------|-------------| +| Not null | Field must have a value | `COUNT(*) WHERE email IS NULL` | +| Uniqueness | No duplicate values | `COUNT(*) vs COUNT(DISTINCT id)` | +| Referential integrity | Foreign key exists | `LEFT JOIN WHERE fk IS NULL` | +| Accepted values | Field in allowed set | `WHERE status NOT IN ('active','inactive','pending')` | +| Range check | Value within bounds | `WHERE age < 0 OR age > 150` | +| Freshness | Data is recent enough | `WHERE MAX(updated_at) < NOW() - INTERVAL '1 day'` | +| Row count | Volume in expected range | `ABS(COUNT(*) - historical_avg) / historical_avg > threshold` | +| Distribution | Value distribution hasn't drifted | Compare histogram to historical baseline | + +## Anomaly Detection Strategies + +| Strategy | What it detects | Best for | +|----------|----------------|----------| +| Fixed threshold | Values outside absolute bounds | Age, price, quantity ranges | +| Statistical (z-score) | Values far from mean | Transaction amounts, latencies | +| Moving average | Trends over time | Daily active users, revenue | +| Seasonality-adjusted | Expected patterns by time | Hourly traffic, weekly sales | +| ML-based | Complex multi-dimensional anomalies | Fraud detection, system health | + +## Deduplication Strategies + +| Strategy | When to use | SQL pattern | +|----------|-------------|-------------| +| Exact dedup | Exact row duplicates | `DELETE USING ... WHERE ctid < (SELECT MAX(ctid) FROM ...)` | +| Key-based dedup | Same natural key, keep latest | `ROW_NUMBER() OVER (PARTITION BY id ORDER BY updated_at DESC) = 1` | +| Fuzzy dedup | Similar but not identical records | `pg_trgm` similarity, Levenshtein distance, ML matching | +| Merge/consolidate | Multiple records for same entity | Survive best values per field, create golden record | + +## Pipeline Health Monitoring + +| Signal | What to check | Action on failure | +|--------|---------------|-------------------| +| Pipeline freshness | Last successful run time | Alert if > expected interval * 2 | +| Row counts | Source vs target volume | Investigate if delta > 10% | +| Null rates | % null in critical fields | Alert if above threshold (configurable per field) | +| Duplicate rates | % duplicate keys | Investigate if > 0% on unique fields | +| Latency | Time from source event to target | Alert if exceeds SLA | +| Schema drift | Column count/type changes | Log and alert for review | diff --git a/data-engineering/references/database-migrations.md b/data-engineering/references/database-migrations.md new file mode 100644 index 0000000..ea763ec --- /dev/null +++ b/data-engineering/references/database-migrations.md @@ -0,0 +1,24 @@ +# Database Migrations + +## Migration Types + +| Type | Risk | Rollback | Example | +|------|------|----------|---------| +| Add column (nullable) | Low | Trivial | `ALTER TABLE ADD COLUMN x TEXT` | +| Add column (NOT NULL) | Medium | Trivial if default provided | `ALTER TABLE ADD COLUMN x INT NOT NULL DEFAULT 0` | +| Rename column | High | Requires migration | Two-phase: add new, dual-write, backfill, drop old | +| Drop column | High | Requires restore | Verify no readers first, soft-delete before hard-drop | +| Create table | Low | Trivial | `CREATE TABLE` | +| Drop table | Critical | Requires restore | Verify no readers, triply confirm | +| Data migration | High | Requires rollback script | Transform values, update references | + +## Migration Checklist + +- [ ] Migration reviewed by another engineer +- [ ] Rollback script exists and is tested +- [ ] Downtime window confirmed (if required) +- [ ] Read replicas considered (replication lag) +- [ ] Foreign key constraints handled +- [ ] Indexes created after data load (not before) +- [ ] Migration tested against copy of production data +- [ ] Query performance verified post-migration diff --git a/data-engineering/references/dbt-patterns.md b/data-engineering/references/dbt-patterns.md new file mode 100644 index 0000000..d77cdfd --- /dev/null +++ b/data-engineering/references/dbt-patterns.md @@ -0,0 +1,1169 @@ +# dbt (Data Build Tool) — Comprehensive Reference Guide + +> A methodology reference for data-engineering teams adopting dbt as the transformation layer in the modern data stack. + +--- + +## Table of Contents + +1. [What Is dbt and What Problem Does It Solve?](#1-what-is-dbt-and-what-problem-does-it-solve) +2. [dbt Core vs dbt Cloud](#2-dbt-core-vs-dbt-cloud) +3. [dbt Project Structure](#3-dbt-project-structure) +4. [dbt Modeling Concepts (Kimball Star Schema)](#4-dbt-modeling-concepts-kimball-star-schema) +5. [dbt Materializations](#5-dbt-materializations) +6. [dbt Tests](#6-dbt-tests) +7. [dbt Sources and Source Freshness](#7-dbt-sources-and-source-freshness) +8. [dbt Snapshots (Slowly Changing Dimensions)](#8-dbt-snapshots-slowly-changing-dimensions) +9. [dbt Documentation Generation](#9-dbt-documentation-generation) +10. [dbt Jinja/SQL Templating and Macros](#10-dbt-jinjasql-templating-and-macros) +11. [dbt Packages (dbt_utils, dbt_expectations)](#11-dbt-packages) +12. [dbt CI/CD Integration Patterns](#12-dbt-cicd-integration-patterns) +13. [dbt Mesh / Multi-Project Deployments](#13-dbt-mesh--multi-project-deployments) + +--- + +## 1. What Is dbt and What Problem Does It Solve? + +**dbt (data build tool)** is an open-source command-line tool and platform that enables analytics engineers and data analysts to transform data in their warehouse using SQL `SELECT` statements. It applies software-engineering best practices — version control, modularity, testing, CI/CD, documentation — to the data transformation layer. + +### The Core Problem + +Before dbt, the typical data workflow looked like: + +1. Raw data lands in a warehouse via EL(E) tools (Fivetran, Airbyte, Stitch). +2. Transformations are written as arbitrary Python scripts, stored procedures, or tangled SQL in BI tools. +3. There is no lineage tracking, no testing, no documentation, and no repeatable deployment process. +4. Collaboration is hard because transformations are ad-hoc, not modular. + +**dbt solves this by:** + +- Moving the **T** (transform) from ETL to ELT — transformations happen *inside* the warehouse after data is loaded. +- Providing a **declarative, modular** framework: you write SQL `SELECT` statements, and dbt handles DDL (`CREATE TABLE`, `CREATE VIEW`, `INSERT`, `MERGE`) automatically. +- **Inferring a DAG** (directed acyclic graph) from `ref()` calls between models, enabling automatic dependency resolution and execution ordering. +- Bringing **software engineering to data**: version control (git), testing, documentation, CI/CD, package management. + +### Key Concepts + +| Concept | Description | +|---|---| +| **Models** | SQL files that `SELECT` from sources or other models; dbt materializes them as views/tables/incremental builds | +| **Tests** | Assertions on data quality — uniqueness, not-null, referential integrity, custom logic | +| **Sources** | Declarations of raw database tables loaded by EL tools; enables lineage, freshness checks | +| **Snapshots** | Type-2 slowly changing dimension (SCD) recording | +| **Seeds** | CSV files loaded into the warehouse as tables (for small reference/lookup data) | +| **Exposures** | Declarations of downstream consumers (dashboards, apps, ML models) | +| **Metrics** | Business metric definitions used by the dbt Semantic Layer | + +> dbt is *not* an EL tool — it does not extract or load data. It assumes data already exists in a data warehouse (Snowflake, BigQuery, Redshift, Databricks, Postgres, etc.). + +--- + +## 2. dbt Core vs dbt Cloud + +### dbt Core + +- **Free and open-source** (Apache 2.0 license). +- Command-line tool: `pip install dbt-core` + adapter for your warehouse (`dbt-snowflake`, `dbt-bigquery`, etc.). +- Requires you to manage your own orchestration (Airflow, Dagster, cron, GitHub Actions, etc.). +- No web UI — all development happens in a code editor + CLI. +- Community-driven; no official scheduling, logging, or collaboration features. + +### dbt Cloud + +- **Managed SaaS platform** by dbt Labs. +- Includes a web-based IDE, job scheduler, run history, and alerting. +- Built-in CI/CD via "Compare Changes" and environment promotion. +- **dbt Semantic Layer** with GraphQL and JDBC APIs for BI tool integration. +- **dbt Mesh** support for cross-project collaboration (multi-project `ref`). +- Role-based access control (RBAC), audit logs, SSO (Enterprise). +- **Pricing** is usage-based (by model runs/credits); free Developer tier available. + +### Decision Matrix + +| Criteria | dbt Core | dbt Cloud | +|---|---|---| +| Cost | Free | Paid (metered) | +| Orchestration | External (Airflow, Dagster, etc.) | Built-in scheduler | +| UI | CLI only | Web IDE + CLI | +| CI/CD | Manual setup (CI runner) | Built-in (Compare Changes) | +| Semantic Layer | Not available | Included (all paid tiers) | +| dbt Mesh | Limited (dbt-loom, manual) | Native support | +| Multi-user Dev | Git-based only | Managed environments + RBAC | +| Support | Community | Vendor support tiers | + +**Typical pattern:** Teams using dbt Core locally for development and dbt Cloud (or a self-hosted orchestration tool) for production execution. Some teams use Core exclusively with Airflow/Dagster. + +--- + +## 3. dbt Project Structure + +A standard dbt project created via `dbt init ` has this layout: + +``` +my_dbt_project/ + ├── .gitignore + ├── README.md + ├── dbt_project.yml # Project config (name, profile, model paths, etc.) + ├── profiles.yml # (outside project dir, ~/.dbt/) — DB connection config + │ + ├── models/ # SQL models (the core of the project) + │ ├── staging/ # Raw → cleaned, one-to-one with source tables + │ │ ├── _stg__models.yml # schema/docs for staging models + │ │ ├── stg_customers.sql + │ │ └── stg_orders.sql + │ ├── intermediate/ # Business-logic transformations between staging and marts + │ │ ├── int_order_items.sql + │ │ └── ... + │ └── marts/ # Business-facing models (facts + dimensions) + │ ├── marketing/ + │ ├── finance/ + │ └── ... + │ + ├── tests/ # Singular tests (ad-hoc SQL assertions) + │ ├── assert_total_revenue_positive.sql + │ └── ... + │ + ├── macros/ # Jinja macros for reusable SQL logic + │ ├── generate_schema_name.sql + │ └── ... + │ + ├── snapshots/ # Type-2 SCD snapshots + │ ├── scd_customers.sql + │ └── ... + │ + ├── seeds/ # CSV files loaded as tables + │ ├── country_codes.csv + │ └── ... + │ + ├── analyses/ # Ad-hoc queries (not materialized) + │ └── ... + │ + └── data/ # (deprecated in favor of seeds/) +``` + +### Key Files + +**`dbt_project.yml`** — The project manifest: + +```yaml +name: my_project +version: "1.0.0" +config-version: 2 +profile: my_project_profile # references profiles.yml + +model-paths: ["models"] +seed-paths: ["seeds"] +test-paths: ["tests"] +macro-paths: ["macros"] +snapshot-paths: ["snapshots"] + +clean-targets: + - "target" + - "dbt_packages" + +models: + my_project: + staging: + +materialized: view + intermediate: + +materialized: view + marts: + +materialized: table +``` + +### Node Types in Detail + +| Node Type | Directory | Description | +|---|---|---| +| **Models** | `models/` | SQL `SELECT` statements materialized as views/tables | +| **Sources** | Defined in YAML (inside `models/`) | Declare upstream raw tables for lineage and freshness | +| **Tests** | `tests/` (singular) + YAML `tests:` blocks (generic) | Data quality assertions | +| **Snapshots** | `snapshots/` | SCD Type-2 tracking | +| **Seeds** | `seeds/` | Small CSV lookup tables | +| **Exposures** | Defined in YAML | Declare downstream consumers (dashboard URLs, etc.) | +| **Metrics** | Defined in YAML | Business metric definitions for the Semantic Layer | +| **Analyses** | `analyses/` | SQL that is *not* materialized (ad-hoc exploration) | + +--- + +## 4. dbt Modeling Concepts (Kimball Star Schema) + +The gold standard for dbt projects is the **Kimball dimensional modeling** approach organized into a **layered architecture**: + +``` +┌─────────────────────────────────────────────────┐ +│ Raw Data (EL layer — Fivetran, Airbyte, etc.) │ +│ Tables in warehouse: order_db.orders, etc. │ +└────────────────────┬────────────────────────────┘ + │ source() + ▼ +┌─────────────────────────────────────────────────┐ +│ Staging Layer (stg_*) │ +│ - One model per source table │ +│ - Light cleaning: rename, cast, deduplicate │ +│ - No joins — 1:1 with source │ +│ - Materialized as VIEW │ +└────────────────────┬────────────────────────────┘ + │ ref() + ▼ +┌─────────────────────────────────────────────────┐ +│ Intermediate Layer (int_*) │ +│ - Business-logic transformations │ +│ - Joins across staging models │ +│ - Pivot/unpivot, aggregations, filtering │ +│ - Usually VIEW (or ephemeral CTE) │ +└────────────────────┬────────────────────────────┘ + │ ref() + ▼ +┌─────────────────────────────────────────────────┐ +│ Mart Layer (fct_*, dim_*) │ +│ - Facts: measures, foreign keys, grain-defining │ +│ - Dimensions: descriptive attributes, conformed │ +│ - Materialized as TABLE or INCREMENTAL │ +└────────────────────┬────────────────────────────┘ + │ + ▼ + Dashboards / Exposures +``` + +### Staging Models (`stg_*`) + +Purpose: clean, type, and rename raw data. Always 1:1 with a source table. + +```sql +-- models/staging/stg_orders.sql +WITH source AS ( + SELECT * FROM {{ source('source_name', 'orders') }} +), +renamed AS ( + SELECT + id AS order_id, + customer_id AS customer_id, + order_date AS order_date, + status AS order_status, + amount AS order_amount, + -- standard timestamp + _loaded_at AS loaded_at + FROM source + WHERE id IS NOT NULL +) +SELECT * FROM renamed +``` + +### Intermediate Models (`int_*`) + +Purpose: bridge staging → marts. Common patterns: +- **Pivots**: `int_orders_pivoted` — pivot order statuses into columns +- **Aggregations**: `int_customer_orders` — aggregate orders per customer +- **Joins**: `int_order_items_joined` — join orders to line items + +```sql +-- models/intermediate/int_customer_orders.sql +SELECT + customer_id, + MIN(order_date) AS first_order_date, + MAX(order_date) AS most_recent_order_date, + COUNT(order_id) AS number_of_orders, + SUM(order_amount) AS lifetime_value +FROM {{ ref('stg_orders') }} +GROUP BY customer_id +``` + +### Fact Models (`fct_*`) + +- Represent business processes/events (sales, orders, clicks, shipments). +- Contain measures (numeric, additive) and foreign keys to dimensions. +- Grain must be explicitly stated in YAML documentation. + +```sql +-- models/marts/fct_orders.sql +SELECT + order_id, + customer_id, + order_date, + order_amount, + order_status +FROM {{ ref('stg_orders') }} +``` + +### Dimension Models (`dim_*`) + +- Represent business entities (customer, product, date, store). +- Contain descriptive attributes. +- Are *conformed* (same attributes mean the same thing across facts). + +```sql +-- models/marts/dim_customers.sql +SELECT + customer_id, + first_name || ' ' || last_name AS customer_name, + email, + city, + country, + first_order_date, + most_recent_order_date, + number_of_orders, + lifetime_value +FROM {{ ref('int_customer_orders') }} +``` + +### Best Practice: Directory Layout Inside `marts/` + +``` +models/ + marts/ + marketing/ + dim_customers.sql + fct_customer_attribution.sql + finance/ + fct_orders.sql + dim_products.sql + product/ + fct_sessions.sql + dim_products.sql # shared (conformed) +``` + +Each mart subdirectory gets its own `_models.yml` file for schema/documentation. + +--- + +## 5. dbt Materializations + +Materializations determine *how* a model is physically built in the warehouse. + +### View (default) + +```sql +{{ config(materialized='view') }} +SELECT ... +``` + +- Creates a `CREATE VIEW AS ...`. +- **Pros**: always up-to-date, no storage cost, fast to create. +- **Cons**: slower to query (especially with nested views), can't add indexes/partitions. +- **Use for**: staging and intermediate models. + +### Table + +```sql +{{ config(materialized='table') }} +SELECT ... +``` + +- Creates `CREATE TABLE AS SELECT` (full refresh every run). +- **Pros**: fast queries, can be indexed/clustered. +- **Cons**: expensive to rebuild fully each run, requires storage. +- **Use for**: small-to-medium marts, dimensions. + +### Incremental + +```sql +{{ config( + materialized='incremental', + unique_key='order_id', + incremental_strategy='merge' -- or 'insert_overwrite', 'delete+insert' +) }} +SELECT ... +{% if is_incremental() %} + WHERE updated_at > (SELECT MAX(updated_at) FROM {{ this }}) +{% endif %} +``` + +- Only processes new/changed rows since the last run. +- Strategies: `merge` (Snowflake, Databricks, BigQuery), `insert_overwrite` (BigQuery partitions), `delete+insert` (Redshift, Postgres). +- **Pros**: efficient for large-volume append or upsert workloads. +- **Cons**: more complex, risk of data drift if `unique_key` is wrong or source data is mutated outside incremental window. +- **Use for**: large fact tables, event logs, transaction tables. + +### Ephemeral + +```sql +{{ config(materialized='ephemeral') }} +SELECT ... +``` + +- Not materialized at all — becomes a CTE (common table expression) wherever it's `ref()`'d. +- **Pros**: no storage, zero maintenance. +- **Cons**: can't be directly queried, can cause deeply nested CTEs. +- **Use for**: lightweight intermediate transformations that are only used once. + +### Comparison + +| Materialization | DDL | Storage | Query Speed | Refresh | +|---|---|---|---|---| +| **view** | `CREATE VIEW` | None | Slow | Always live | +| **table** | `CREATE TABLE AS` | Full | Fast | Full refresh | +| **incremental** | `MERGE` / `INSERT` | Full | Fast | Incremental | +| **ephemeral** | None | None | Depends | N/A (CTE) | + +--- + +## 6. dbt Tests + +dbt provides a testing framework to assert data quality. Tests are run via `dbt test`. + +### Generic Tests (schema tests) + +Defined in YAML — reusable assertions against columns: + +```yaml +# models/marts/_models.yml +models: + - name: dim_customers + columns: + - name: customer_id + tests: + - unique + - not_null + - name: email + tests: + - unique + - not_null + - name: country + tests: + - accepted_values: + values: ['US', 'UK', 'DE', 'FR', 'CA'] + - name: fct_orders + columns: + - name: customer_id + tests: + - not_null + - relationships: + to: ref('dim_customers') + field: customer_id +``` + +**Built-in generic tests:** +- `unique` — no duplicate values in column +- `not_null` — no NULL values +- `accepted_values` — column values come from a defined list +- `relationships` — referential integrity (foreign key check) +- Custom ones from packages: `dbt_utils.expression_is_true`, `dbt_expectations.expect_column_values_to_match_regex`, etc. + +### Singular Tests (data tests) + +Standalone SQL files in `tests/` that return failing rows. Any returned row == test failure. + +```sql +-- tests/assert_positive_revenue.sql +SELECT + order_id, + order_amount +FROM {{ ref('fct_orders') }} +WHERE order_amount < 0 +``` + +### Custom Generic Tests (test macros) + +Create reusable test macros in `macros/tests/`: + +```sql +{% test assert_positive(model, column_name) %} +SELECT * +FROM {{ model }} +WHERE {{ column_name }} < 0 +{% endtest %} +``` + +Then use it in YAML: + +```yaml +tests: + - assert_positive +``` + +### Running Tests + +```bash +dbt test # run all tests +dbt test --select dim_customers # test a single model +dbt test --select tag:nightly # tests tagged 'nightly' +``` + +**Store test failures:** + +```bash +dbt test --store-failures # persists failures as tables for review +``` + +### Test Severity (dbt v1.5+) + +```yaml +tests: + - not_null: + severity: warn # non-blocking; reported but doesn't fail the run +``` + +--- + +## 7. dbt Sources and Source Freshness + +### Declaring Sources + +Sources define which raw database tables your pipeline starts from. + +```yaml +# models/staging/_sources.yml +version: 2 + +sources: + - name: jaffle_shop # logical name + database: raw_db + schema: public + tables: + - name: orders + description: "Raw orders from the jaffle_shop transactional system" + loaded_at_field: _etl_loaded_at + freshness: + warn_after: { count: 12, period: hour } + error_after: { count: 24, period: hour } + columns: + - name: id + description: Primary key + tests: + - unique + - not_null + - name: customers + loaded_at_field: _etl_loaded_at + freshness: + warn_after: { count: 24, period: hour } +``` + +### Using Sources in Models + +```sql +-- models/staging/stg_orders.sql +SELECT * +FROM {{ source('jaffle_shop', 'orders') }} +``` + +Using `source()` instead of raw table names gives you: +- **Lineage**: dbt tracks dependencies from models back to source tables. +- **Freshness**: `dbt source freshness` runs timestamp-based checks to detect stale data. + +### Source Freshness Command + +```bash +dbt source freshness +``` + +Output: a JSON file `target/sources.json` with per-source freshness results. This can be integrated into monitoring/alerting pipelines. Failures can be flagged as warnings or errors. + +### Snapshotting Source Config + +In `dbt_project.yml` you can set a blanket source freshness policy: + +```yaml +sources: + jaffle_shop: + freshness: + warn_after: { count: 6, period: hour } + loaded_at_field: _loaded_at +``` + +--- + +## 8. dbt Snapshots (Slowly Changing Dimensions) + +Snapshots implement **Type 2 Slowly Changing Dimensions (SCD)** — they track historical changes to dimension attributes. + +### How Snapshots Work + +1. You define a snapshot SQL file in `snapshots/` that `SELECT`s the source data. +2. dbt compares the current source data against the existing snapshot table. +3. If any tracked column changed, dbt **closes** the old row (sets `dbt_valid_to`) and **inserts** a new row (sets `dbt_valid_from`). + +### Snapshot Configuration + +```sql +-- snapshots/scd_customers.sql +{% snapshot scd_customers %} + +{{ + config( + target_schema='snapshots', + unique_key='customer_id', + strategy='check', + check_cols='all' -- or ['email', 'city', 'country'] + ) +}} + +SELECT * FROM {{ source('jaffle_shop', 'customers') }} + +{% endsnapshot %} +``` + +### Snapshot Strategies + +| Strategy | Description | +|---|---| +| **`timestamp`** | Uses a `updated_at` column to detect changes (more efficient). Requires `updated_at` column. | +| **`check`** | Compares specified columns (or all columns) for changes. No timestamp needed. | + +**Timestamp strategy (preferred when possible):** + +```sql +{{ + config( + target_schema='snapshots', + unique_key='customer_id', + strategy='timestamp', + updated_at='updated_at', + invalidate_hard_deletes=True + ) +}} +``` + +### Snapshot Metadata Columns + +Every snapshot row gets these columns automatically: + +| Column | Meaning | +|---|---| +| `dbt_scd_id` | Surrogate key for the SCD record | +| `dbt_updated_at` | When the row was updated (the `updated_at` value or snapshot run time) | +| `dbt_valid_from` | Start date/time of this version | +| `dbt_valid_to` | End date/time (NULL = current version) | +| `dbt_is_contaminated` | Flag if multiple changes happened between snapshot runs (unusual) | + +### Querying Snapshot Tables + +```sql +-- Get current customers +SELECT * FROM snapshots.scd_customers WHERE dbt_valid_to IS NULL + +-- Get customers as of a specific date +SELECT * FROM snapshots.scd_customers +WHERE '2024-06-01' BETWEEN dbt_valid_from AND COALESCE(dbt_valid_to, '9999-12-31') + +-- Full history for a specific customer +SELECT * FROM snapshots.scd_customers +WHERE customer_id = 42 +ORDER BY dbt_valid_from +``` + +--- + +## 9. dbt Documentation Generation + +dbt can auto-generate a static documentation site from your project using `dbt docs generate`. + +### What Gets Generated + +- **Model lineage** (DAG visualization) via `dbt docs serve` (interactive web UI). +- **Schema/datatype info** from the warehouse (via `dbt docs generate` which runs `dbt run` + `dbt test` + catalog collection). +- **Descriptions** from YAML schema files. +- **Test results** and sources information. + +### Adding Documentation + +```yaml +# models/marts/_models.yml +version: 2 + +models: + - name: dim_customers + description: > + Customer dimension table. One row per customer with current attributes + and aggregated lifetime metrics. + columns: + - name: customer_id + description: "Primary key from the source CRM system" + tests: + - unique + - not_null + - name: lifetime_value + description: "Total revenue from this customer (all orders)" +``` + +### Docs Blocks (reusable markdown) + +```sql +-- models/docs.md +{% docs dim_customers_description %} +The **customer dimension** contains one row per customer. +It includes: +- Demographics (name, email, location) +- Behavioral metrics (first/last order date, lifetime value) +{% enddocs %} +``` + +Referenced in YAML: + +```yaml +models: + - name: dim_customers + description: "{{ doc('dim_customers_description') }}" +``` + +### Generating and Serving + +```bash +dbt docs generate # produces target/catalog.json + target/manifest.json +dbt docs serve # serves docs at http://localhost:8080 +dbt docs serve --port 8081 +``` + +### CI Integration + +Many teams upload the generated docs to a static hosting service (S3, Netlify, GitHub Pages) as part of CI/CD, so the documentation is always up-to-date with production. + +--- + +## 10. dbt Jinja/SQL Templating and Macros + +dbt uses **Jinja** (Python templating engine) to make SQL programmable. + +### Basic Jinja in dbt + +```sql +SELECT + order_id, + {% if include_customer_name %} + customer_name, + {% endif %} + order_amount * {{ multiplier }} AS adjusted_amount +FROM {{ ref('fct_orders') }} +``` + +### Built-in Jinja Functions + +| Function | Purpose | +|---|---| +| `{{ ref('model_name') }}` | Reference another model (creates DAG edge) | +| `{{ source('source_name', 'table') }}` | Reference a declared source | +| `{{ config(...) }}` | Set model-level configuration | +| `{{ this }}` | Current model's database object reference | +| `{{ is_incremental() }}` | Returns `True` if the model is doing an incremental run | +| `{{ var('variable_name') }}` | Access user-defined variables | +| `{{ env_var('ENV_NAME') }}` | Access environment variables | + +### Macros + +Macros are reusable Jinja-SQL snippets, stored in `macros/`. They are like functions. + +**Creating a macro:** + +```sql +{# macros/cents_to_dollars.sql #} +{% macro cents_to_dollars(column_name, precision=2) -%} + ({{ column_name }} / 100.0)::numeric(16, {{ precision }}) +{%- endmacro %} +``` + +**Using a macro:** + +```sql +SELECT + {{ cents_to_dollars('order_amount_cents') }} AS order_amount_dollars +FROM {{ ref('stg_orders') }} +``` + +### Control Flow + +```sql +{% if target.name == 'prod' %} + -- only run in production + AND status IN ('shipped', 'delivered') +{% elif target.name == 'dev' %} + -- sample for development + LIMIT 1000 +{% endif %} +``` + +### Loops + +```sql +{% for column in var('payment_methods') %} + SUM(CASE WHEN payment_method = '{{ column }}' THEN amount ELSE 0 END) AS {{ column }}_amount + {%- if not loop.last %},{% endif %} +{% endfor %} +``` + +### DBT_UTILS Macro Example + +```sql +{% set payment_methods = dbt_utils.get_column_values( + table=ref('stg_payments'), + column='payment_method' +) %} +``` + +### Materialized Macro (Advanced) + +dbt also provides *dispatcher macros* for adapter-specific SQL: + +```sql +{% macro my_custom_merge() %} + {% if target.type == 'snowflake' %} + -- Snowflake MERGE syntax + {% elif target.type == 'bigquery' %} + -- BigQuery MERGE syntax + {% endif %} +{% endmacro %} +``` + +### Best Practices for Macros + +- Keep macros in `macros/`, organized by domain (`macros/pricing/`, `macros/logging/`). +- Prefix macros with a package name when distributing (`my_package::macro_name`). +- Document macro arguments with `{% docs %}` blocks. +- Avoid excessive Jinja complexity — it makes SQL harder to read and debug. + +--- + +## 11. dbt Packages + +dbt packages are reusable libraries of models, macros, and tests. They are managed via a `packages.yml` file. + +### Installing Packages + +```yaml +# packages.yml +packages: + - package: dbt-labs/dbt_utils + version: 1.1.1 + - package: calogica/dbt_expectations + version: 0.9.0 + - package: dbt-labs/spark_utils + version: 0.3.0 + - git: "https://github.com/dbt-labs/dbt-utils.git" + revision: 0.9.2 # optional +``` + +Install with: + +```bash +dbt deps +``` + +Packages are installed into the `dbt_packages/` directory. + +### dbt_utils (dbt-labs/dbt_utils) + +The most widely used dbt package. Key capabilities: + +**Cross-database macros:** + +| Macro | Purpose | +|---|---| +| `dbt_utils.surrogate_key('col1', 'col2')` | Create a hash-based surrogate key | +| `dbt_utils.datediff('start', 'end', 'day')` | Cross-database date difference | +| `dbt_utils.date_trunc('month', 'date_col')` | Cross-database date truncation | +| `dbt_utils.hash('col')` | Cross-database hash function | +| `dbt_utils.concat(['col1', 'col2'])` | Cross-database concatenation | + +**Testing macros:** + +| Test | Purpose | +|---|---| +| `dbt_utils.expression_is_true` | Assert that an expression is true | +| `dbt_utils.unique_combination_of_columns` | Composite uniqueness | +| `dbt_utils.mutually_exclusive_ranges` | No overlapping ranges | +| `dbt_utils.cardinality_equality` | Two sources have same set of values | +| `dbt_utils.recency` | Max timestamp is recent enough | + +**Schema/table utilities:** + +| Macro | Purpose | +|---|---| +| `dbt_utils.get_column_values()` | Return list of column values | +| `dbt_utils.get_tables_by_pattern()` | Find tables matching a pattern | +| `dbt_utils.get_query_results_as_dict()` | Run any SQL return results | + +**Schema tests (YAML):** + +```yaml +tests: + - dbt_utils.expression_is_true: + expression: "order_amount >= 0" + - dbt_utils.unique_combination_of_columns: + combination_of_columns: + - order_id + - line_item_id +``` + +### dbt_expectations (calogica/dbt_expectations) + +Inspired by the Python `great_expectations` library. Provides dozens of data-quality tests. + +**Common tests:** + +| Test | Purpose | +|---|---| +| `expect_column_values_to_match_regex` | Regex validation | +| `expect_column_values_to_be_between` | Range check | +| `expect_column_values_to_be_in_set` | Set membership | +| `expect_column_distinct_count_to_equal` | Exact distinct count | +| `expect_column_values_to_not_be_null` | Not-null (adds threshold support) | +| `expect_table_row_count_to_be_between` | Row count range | +| `expect_column_pair_values_A_to_be_greater_than_B` | Cross-column comparison | +| `expect_queried_row_count_to_be_between` | Dynamic SQL row count | +| `expect_queried_column_value_frequency_to_be_between` | Value frequency checks | +| `expect_table_columns_to_match_ordered_list` | Schema validation | + +### Other Notable Packages + +| Package | Purpose | +|---|---| +| `dbt-labs/audit_helper` | Compare row counts and values between two inputs | +| `dbt-labs/dbt-artifacts` | Parse dbt artifacts into warehouse tables | +| `dbt-labs/date_spine` | Generate date spines for calendar dimensions | +| `dbt-labs/codegen` | Auto-generate base models and YAML from source tables | +| `elementary-data/elementary` | Data monitoring, alerting, and observability | +| `re-data/re_data` | Data reliability and anomaly detection | +| `infinitelambda/dbt_ml` | ML preprocessing utilities in dbt | + +--- + +## 12. dbt CI/CD Integration Patterns + +### Pattern 1: dbt Cloud CI + +1. Create a **Merge Request / Pull Request** on GitHub/GitLab. +2. dbt Cloud's CI job fires automatically. +3. It creates a **temporary schema** with the PR's changes. +4. Runs `dbt build --select state:modified+` to run only changed models and their downstream tests. +5. Reports results back as a PR check. + +**Key commands:** + +```bash +# Compare against production manifest +dbt build --select state:modified+ --defer --state target-prod/ +``` + +Where `--defer` means "use production tables for unmodified models" and `state:modified+` selects changed models plus everything downstream. + +### Pattern 2: dbt Core + GitHub Actions + +```yaml +# .github/workflows/dbt-ci.yml +name: dbt CI +on: + pull_request: + branches: [main] + +jobs: + dbt-ci: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: "3.11" + - name: Install dependencies + run: | + pip install dbt-snowflake dbt-utils + dbt deps + - name: dbt build (CI) + env: + DBT_USER: ${{ vars.DBT_USER }} + DBT_PASSWORD: ${{ secrets.DBT_PASSWORD }} + run: | + dbt build --target ci --select state:modified+ --defer +``` + +### Pattern 3: dbt + Airflow + +Use the `Cosmos` library (by Astronomer) to run dbt inside Airflow DAGs: + +```python +from cosmos import DbtDag, ProjectConfig, ProfileConfig +from pendulum import datetime + +dbt_dag = DbtDag( + project_config=ProjectConfig("/path/to/dbt_project"), + profile_config=ProfileConfig( + profile_name="my_project", + target_name="prod", + profiles_yml_filepath="/path/to/profiles.yml", + ), + start_date=datetime(2024, 1, 1), + schedule="@daily", + catchup=False, + default_args={"retries": 2}, + tags=["dbt"], +) +``` + +### Pattern 4: Slim CI (state-based) + +```bash +# In production: upload manifest.json as an artifact +dbt run # full run +dbt docs generate +cp target/manifest.json target-prod/manifest.json + +# In CI: download production manifest, run slim CI +dbt build --select state:modified+ --defer --state target-prod/ +``` + +### Environment Strategy + +```yaml +# dbt_project.yml +models: + +post-hook: + - "GRANT SELECT ON {{ this }} TO ROLE ANALYST_ROLE" # only in prod + - "{{ 'GRANT SELECT ON {{ this }} TO ROLE DEV_ROLE' if target.name == 'dev' else '' }}" +``` + +**Recommended targets:** + +| Target | Purpose | Schema Suffix | +|---|---|---| +| `dev` | Individual developer | `_dev_` | +| `ci` | PR validation | `_pr_` | +| `staging` | Pre-production | `_staging` | +| `prod` | Production | (none) | + +--- + +## 13. dbt Mesh / Multi-Project Deployments + +dbt Mesh is dbt Labs' solution for scaling dbt across multiple teams and domains, enabling **decentralized ownership with centralized governance**. + +### The Problem dbt Mesh Solves + +- A single monolithic dbt project becomes unwieldy at scale (1000+ models, 10+ teams). +- Teams need to own their data independently but depend on models from other teams. +- No cross-project visibility or contracts between teams. + +### Key Concepts + +| Concept | Description | +|---|---| +| **Multi-project collaboration** | Different teams maintain separate dbt repos/projects | +| **Cross-project `ref`** | Use `ref('model_name')` across projects via `dependencies.yml` | +| **Model contracts** | Enforced column names, types, and constraints on public models | +| **Access control** | `public` / `protected` / `private` model access modifiers | +| **Versioning** | Semantic versioning for model contracts | +| **Discovery API** | Query metadata across all projects | + +### Setting Up dbt Mesh + +**Producer (upstream) project:** + +```yaml +# models/_models.yml +models: + - name: dim_customers + access: public # can be used by other projects + config: + contract: + enforced: true + columns: + - name: customer_id + data_type: int + constraints: [not_null, unique] + - name: customer_name + data_type: varchar(256) + - name: email + data_type: varchar(256) +``` + +**Consumer (downstream) project:** + +```yaml +# dependencies.yml +packages: + - name: upstream_core + version: 1.0.0 + # For dbt Cloud: + # (handled via Project Dependencies UI) + # For dbt Core (with dbt-loom or similar): + git: "https://github.com/team-a/dbt-core-project.git" +``` + +```sql +-- models/marts/fct_orders.sql +SELECT * +FROM {{ ref('upstream_core', 'dim_customers') }} -- cross-project ref +``` + +### Model Contracts + +Contracts enforce a "schema on write" for downstream consumers: + +```yaml +models: + - name: dim_customers + config: + contract: + enforced: true # dbt will fail if the model's SQL doesn't match the declared columns + columns: + - name: customer_id + data_type: int + constraints: + - type: not_null + - type: primary_key + - name: email + data_type: varchar(256) +``` + +### Benefits + +- **Team autonomy**: Each team manages their own dbt project, CI/CD, and deployments. +- **Governance**: Model contracts prevent breaking changes across teams. +- **Scalability**: Reduced DAG complexity per project, faster CI, independent deploy cycles. +- **Reusability**: Shared domain models (e.g., `dim_customers`, `dim_dates`) are versioned and consumed by many projects. + +### Tools for Multi-Project Without dbt Cloud + +| Tool | Description | +|---|---| +| **dbt-loom** | Open-source CLI tool for cross-project `ref()` resolution in dbt Core | +| **dbt-meshify** | CLI by dbt Labs to assist splitting monolithic projects | +| **Custom scripts** | `git submodule` or multi-repo CI strategies | + +--- + +## Appendix A: Essential dbt Commands + +```bash +dbt init # Create a new dbt project +dbt deps # Install packages from packages.yml +dbt debug # Verify warehouse connection +dbt seed # Load CSV files (seeds) +dbt run # Execute all models +dbt run --select +model_name # Run a model + its upstream dependencies +dbt run --select model_name+ # Run a model + its downstream dependents +dbt run --exclude tag:stale # Run everything except models tagged 'stale' +dbt test # Run all tests +dbt test --select model_name # Run tests only for a specific model +dbt build # seed + run + test (in one command, DAG-ordered) +dbt snapshot # Execute snapshots +dbt source freshness # Check source table freshness +dbt docs generate # Build documentation +dbt docs serve # Serve documentation locally +dbt ls # List all resources (models, tests, etc.) +dbt compile # Compile SQL without executing +dbt parse # Validate project without running anything +``` + +## Appendix B: YAML Schema File Pattern + +Organize YAML files alongside models. Naming convention: + +| File | Contains | +|---|---| +| `_sources.yml` | Source declarations | +| `_models.yml` | Model descriptions, column docs, tests | +| `_metrics.yml` | Metric definitions | +| `_exposures.yml` | Exposure declarations | +| `_macros.yml` | Macro documentation | + +## Appendix C: Key Resources + +- [Official dbt Documentation](https://docs.getdbt.com/) +- [dbt GitHub (dbt-core)](https://github.com/dbt-labs/dbt-core) +- [dbt_utils Package](https://github.com/dbt-labs/dbt-utils) +- [dbt_expectations Package](https://github.com/calogica/dbt-expectations) +- [dbt Discourse Community](https://discourse.getdbt.com/) +- [dbt Best Practices Guide](https://docs.getdbt.com/best-practices) +- [dbt Mesh Docs](https://docs.getdbt.com/docs/mesh) + +--- + +*Document produced for data-engineering methodology skill reference. June 2026.* diff --git a/data-engineering/references/etl-pipeline-design.md b/data-engineering/references/etl-pipeline-design.md new file mode 100644 index 0000000..a5c0d95 --- /dev/null +++ b/data-engineering/references/etl-pipeline-design.md @@ -0,0 +1,93 @@ +# ETL/ELT Pipeline Design + +## ETL vs ELT + +| Approach | Transform location | When to use | Tools | +|----------|-------------------|-------------|-------| +| ETL (Extract, Transform, Load) | Staging server before load | Strict schema enforcement, legacy systems | Custom scripts, Spark, Python | +| ELT (Extract, Load, Transform) | Target database after load | Cloud data warehouses, modern stacks | dbt, BigQuery, Snowflake | + +Modern data engineering overwhelmingly favors ELT. The data warehouse is the transformation engine — load raw data first, then transform with SQL. + +## Pipeline Architecture Patterns + +| Pattern | Latency | Complexity | Best for | +|---------|---------|------------|----------| +| Batch (scheduled) | Hours/days | Low | Reporting, BI, historical analysis | +| Micro-batch (frequent) | Minutes | Medium | Near-real-time dashboards, ML features | +| Streaming (continuous) | Seconds | High | Real-time alerts, fraud detection, monitoring | +| Lambda (batch + streaming) | Mixed | High | Systems needing both real-time and historical | +| Delta (unified batch/stream) | Mixed | Medium | Lakehouse architectures (Delta Lake, Iceberg) | + +## Extraction Strategies + +| Strategy | Mechanism | Freshness | Load on source | +|----------|-----------|-----------|----------------| +| Full refresh | `SELECT * FROM source` | Per schedule | High — reads everything | +| Incremental (watermark) | `WHERE updated_at > last_max` | Minutes | Low — only new/changed rows | +| CDC (change data capture) | Binlog/WAL replication | Real-time | Minimal — reads transaction log | +| Snapshot diff | Compare periodic snapshots | Hours | Medium — stores full snapshots | +| API polling | Paginated API calls | Configurable | Varies — respects rate limits | + +## Transformation Layers (dbt-style) + +``` +Raw (source) → Staging → Intermediate → Marts (facts/dimensions) +``` + +| Layer | Purpose | Materialization | Idempotent | +|-------|---------|-----------------|------------| +| **Staging** | Clean, type, rename, deduplicate source data | View or ephemeral | Yes — always full-refresh safe | +| **Intermediate** | Business logic, joins, aggregations, pivots | Ephemeral or table | Yes — recomputable from staging | +| **Facts** | Measurable business events | Table or incremental | Yes — idempotent merge/upsert | +| **Dimensions** | Descriptive business entities | Table or slowly-changing | Yes — SCD Type 2 tracked | + +## Incremental Load Patterns + +### Watermark Pattern +```sql +-- Pseudocode pattern +SELECT * FROM source_table +WHERE updated_at > ( + SELECT MAX(updated_at) FROM target_table +); +``` + +### Merge/Upsert Pattern +```sql +-- PostgreSQL +INSERT INTO target (id, value, updated_at) +SELECT id, value, updated_at FROM source +ON CONFLICT (id) DO UPDATE +SET value = EXCLUDED.value, + updated_at = EXCLUDED.updated_at; +``` + +### Snapshot Pattern (Full Replace) +```sql +-- For small dimensions: truncate and reload +TRUNCATE TABLE dim_small; +INSERT INTO dim_small SELECT * FROM source; +``` + +## Validation Gates + +Every pipeline stage should validate: + +| Gate | What it catches | Implementation | +|------|-----------------|----------------| +| Schema validation | Column count, type mismatch | Compare source schema to expected | +| Null check | Required field missing | `HAVING COUNT(*) = SUM(CASE WHEN col IS NULL THEN 1 ELSE 0 END)` | +| Row count | Missing data, truncation | Compare source row count to target row count | +| Uniqueness | Duplicate records | `COUNT(*) vs COUNT(DISTINCT pk)` | +| Freshness | Stale data pipeline | Timestamp threshold check | +| Distribution | Data quality drift | Min/max/avg comparison to historical | + +## Error Handling + +| Error type | Strategy | Example | +|------------|----------|---------| +| Transient (network, timeout) | Retry with exponential backoff | 3 retries, 30s/60s/120s intervals | +| Data quality (null key, type error) | Reject to dead letter queue, alert | Log bad rows, continue pipeline | +| Schema drift (new column) | Alert, optionally adapt | Detect, log, notify, proceed with null | +| Catastrophic (source down) | Halt pipeline, alert, wait for manual recovery | Preserve pipeline state for resume | diff --git a/data-engineering/references/graph-databases.md b/data-engineering/references/graph-databases.md new file mode 100644 index 0000000..9fa8edd --- /dev/null +++ b/data-engineering/references/graph-databases.md @@ -0,0 +1,1148 @@ +# Graph Databases for Data Engineering: Neo4j & Cypher Reference + +> A practical reference for data engineers working with Neo4j, the Cypher query language, +> and graph data modeling patterns in production pipelines. + +--- + +## Table of Contents + +1. [Graph Data Modeling Principles](#1-graph-data-modeling-principles) +2. [Neo4j Deployment Models](#2-neo4j-deployment-models) +3. [Cypher Query Language Fundamentals](#3-cypher-query-language-fundamentals) +4. [Graph Data Modeling Patterns by Domain](#4-graph-data-modeling-patterns-by-domain) +5. [Cypher Aggregation & Graph Traversal](#5-cypher-aggregation--graph-traversal) +6. [Importing Data into Neo4j](#6-importing-data-into-neo4j) +7. [Indexing & Performance Optimization](#7-indexing--performance-optimization) +8. [Graph Algorithms Library (GDS)](#8-graph-algorithms-library-gds) +9. [Integrating Neo4j into Data Pipelines](#9-integrating-neo4j-into-data-pipelines) +10. [Graph vs. Relational: When to Use Which](#10-graph-vs-relational-when-to-use-which) + +--- + +## 1. Graph Data Modeling Principles + +### 1.1 The Property Graph Model + +Neo4j uses the **labeled property graph** model. A graph is composed of: + +| Element | Description | Example | +|---------|-------------|---------| +| **Node** | A discrete entity/object | `(:Person {name: "Alice"})` | +| **Relationship** | A directed connection between two nodes | `(Alice)-[:KNOWS]->(Bob)` | +| **Label** | A node category/type (a node can have many) | `:Person`, `:Company`, `:Customer` | +| **Property** | A key-value pair on a node or relationship | `{name: "Alice", age: 30}` | +| **Relationship Type** | The semantic category of a relationship | `:KNOWS`, `:WORKS_FOR`, `:PURCHASED` | + +**Key distinction from relational**: Relationships are first-class citizens, not foreign-key joins computed at query time. In Neo4j, a relationship is a physically stored pointer — traversal is O(1) per hop regardless of graph size. + +### 1.2 Core Modeling Principles + +**1. Model the Domain, Not the Schema** +- In a relational DB you design tables first, then JOIN them. +- In a graph you ask "what are the real-world entities and how do they connect?" +- A node label groups entities by role; a relationship type captures the verb. + +**2. Favor Relationships Over Join Tables** +- A join table in SQL (e.g., `user_roles`) becomes a relationship in Neo4j. +- If the relationship itself has data (e.g., `since` date on an employment relationship), put properties on the relationship. + +```cypher +-- Relational: JOIN table with data columns +-- user_id, role_id, assigned_date + +-- Graph: relationship carries the data +(:User)-[:HAS_ROLE {assigned_date: "2024-01-15"}]->(:Role) +``` + +**3. Nodes for Nouns, Relationships for Verbs** +- `:Invoice`, `:Product`, `:Customer` → nodes +- `:PURCHASED`, `:SHIPPED_TO`, `:CONTAINS` → relationships +- If you find yourself creating `:Transaction` as a node connecting two other nodes, first ask whether you need properties on the connection itself. + +**4. Avoid "Meta-Relationships"** (relationships that should be nodes) +- If a relationship has enough data to be an entity itself (especially if it connects more than two nodes), promote it to a node. + +``` +BAD: (User)-[:TRANSACTION {amount, date}]->(Product) +BETTER: (User)-[:MADE]->(Transaction {amount, date})-[:FOR]->(Product) +``` + +**5. Use Labels as Index Categories** +- Every query starts with a label match: `MATCH (p:Person)`. +- Labels separate entity types. A node can have multiple labels: `(:Person:Customer:Premium)`. + +### 1.3 Common Anti-Patterns + +| Anti-Pattern | Why It's Wrong | Fix | +|---|---|---| +| One giant "Thing" label for everything | Every query scans everything | Use specific labels `:Person`, `:Invoice` | +| Properties on relationship that belong on the target node | Bloated traversals | Put the property on the destination node | +| Creating a node for a simple scalar value | Unnecessary overhead | Keep it as a property | +| Over-labeling (10+ labels on one node) | Index overhead, confusion | Consolidate; use at most 3-4 per node | +| Chaining relationships where a direct one suffices | Slower queries | Shortest path wins; add direct relationships for frequent patterns | + +--- + +## 2. Neo4j Deployment Models + +### 2.1 Comparison Matrix + +| Feature | Self-Hosted (Community) | Self-Hosted (Enterprise) | AuraDB Free | AuraDB Professional | AuraDB Enterprise | +|---------|------------------------|-------------------------|-------------|--------------------|--------------------| +| Cost | Free | License fee | Free (limited) | Consumption-based | Consumption-based | +| Scaling | Single instance | Clustering (primary-replica) | Auto-scaled | Auto-scaled | Auto-scaled | +| HA/DR | None | Full clustering, backups | Built-in | Built-in | Multi-region | +| Cypher | Full | Full with fabric | Full | Full | Full | +| GDS Library | Manual install | Included | Via plugin | Via plugin | Via plugin | +| APOC | Manual install | Included | Via plugin | Via plugin | Via plugin | +| Backup | Manual/`neo4j-admin` | Online, incremental | Automated | Automated | Automated | +| SLA | None | Optional | 99.9% | 99.95% | 99.995% | +| Max DB size | Limited by hardware | Limited by cluster | 200K nodes | Pay-as-you-grow | Pay-as-you-grow | + +### 2.2 When to Choose Each + +**Self-Hosted (Community)** +- Development, prototyping, small internal tools +- Air-gapped environments +- Cost-sensitive projects with under ~50M nodes +- You already manage your own infrastructure + +**Self-Hosted (Enterprise)** +- Regulatory requirements (data residency, SOC2 on your own infra) +- Need full clustering (up to 200+ core servers + read replicas) +- Custom security policies (LDAP, Kerberos, custom plugins) +- You have a DBRE team + +**AuraDB** +- "I just want a graph database, not a server to manage" +- Serverless scaling without capacity planning +- Graph apps in production that need HA out of the box +- CI/CD and dev/staging/prod environments you spin up/down +- Small team without DBA headcount + +### 2.3 Deployment Infrastructure Quickstart + +**Docker (local dev):** +```bash +docker run \ + --name neo4j \ + -p 7474:7474 -p 7687:7687 \ + -e NEO4J_AUTH=neo4j/strongpassword \ + -e NEO4J_PLUGINS='["apoc","graph-data-science"]' \ + neo4j:enterprise +``` + +**Kubernetes (production):** +- Use the Neo4j Helm chart: `helm repo add neo4j https://helm.neo4j.com` +- Supports Core/Read-replica topology +- PersistentVolumeClaims for data durability + +**AuraDB (managed):** +- Sign up at https://neo4j.com/cloud/aura-free/ +- Download `.env` with credentials +- Connect via `neo4j+s://.databases.neo4j.io` + +--- + +## 3. Cypher Query Language Fundamentals + +Cypher is a declarative, pattern-matching query language inspired by ASCII-art syntax for graph patterns. + +### 3.1 Core Clauses + +#### `MATCH` — Find Patterns in the Graph + +```cypher +-- Simple node match +MATCH (p:Person) +RETURN p.name + +-- Pattern match (relationship) +MATCH (a:Person)-[:KNOWS]->(b:Person) +RETURN a.name, b.name + +-- Property filter +MATCH (p:Person {name: "Alice"}) +RETURN p.email + +-- WHERE clause (equivalent — use for complex predicates) +MATCH (p:Person) +WHERE p.name STARTS WITH "A" AND p.age > 25 +RETURN p + +-- Variable-length traversal +MATCH (a:Person)-[:KNOWS]->{1..3}(b:Person) +RETURN DISTINCT b.name +``` + +#### `CREATE` — Add Nodes and Relationships + +```cypher +-- Create a node +CREATE (p:Person {name: "Charlie", age: 35}) + +-- Create a relationship between existing nodes +MATCH (a:Person {name: "Alice"}) +MATCH (b:Person {name: "Bob"}) +CREATE (a)-[:KNOWS {since: 2020}]->(b) + +-- Create both (avoid for large imports to prevent OOM) +CREATE (a:Person {name: "Dave"})-[:WORKS_FOR]->(:Company {name: "Acme"}) +``` + +#### `MERGE` — Find or Create (Upsert) + +```cypher +-- Find a node by ID, create if missing +MERGE (p:Person {id: "alice-001"}) +ON CREATE SET p.name = "Alice", p.createdAt = datetime() +ON MATCH SET p.lastSeen = datetime() + +-- MERGE a relationship (only creates if not exists) +MATCH (a:Person {id: $id1}) +MATCH (b:Person {id: $id2}) +MERGE (a)-[:KNOWS]->(b) +``` + +**MERGE pitfall**: `MERGE (a)-[:R]->(b)` without MATCHing a and b first will create duplicate nodes. Always MATCH both endpoints first. + +#### `RETURN` — Shape Query Output + +```cypher +-- Return specific properties (for LLM/API consumption) +MATCH (p:Person) +RETURN p.name AS name, p.email AS email + +-- Map projection (concise) +MATCH (p:Person) +RETURN p { .name, .email, .age } + +-- Return entire node (for visualization tools) +MATCH path = (a:Person)-[:KNOWS*1..3]->(:Person) +RETURN path +``` + +#### `WHERE` — Filter Results + +```cypher +-- Comparisons +WHERE p.age >= 18 AND p.age <= 65 + +-- String patterns +WHERE p.name STARTS WITH "A" +WHERE p.name ENDS WITH "son" +WHERE p.name CONTAINS "li" + +-- List membership +WHERE p.name IN ["Alice", "Bob", "Charlie"] + +-- Existence +WHERE exists { (p)-[:KNOWS]->() } + +-- Negation +WHERE NOT (p)-[:KNOWS]->() +``` + +### 3.2 Essential Patterns (Cypher 25+) + +Cypher 25 introduced cleaner syntax. Avoid deprecated patterns. + +| Old (Deprecated) | New (Preferred) | +|---|---| +| `shortestPath((a)-[*]-(b))` | `SHORTEST 1 (a)-[*]-(b)` | +| `()-[*1..5]-()` | `()-[]{1,5}-()` | +| `WITH collect(x)` | `COLLECT { MATCH ... RETURN ... }` | +| `WITH count(*)` | `COUNT { MATCH ... }` | +| `RETURN exists((n)-[:R]->())` | `RETURN exists { (n)-[:R]->() }` | + +### 3.3 Parameterization + +**Always use `$parameters` — never string-interpolate.** + +```python +# BAD: string interpolation (SQL injection risk) +query = f"MATCH (p:Person {{name: '{user_input}'}}) RETURN p" + +# GOOD: parameterized +records, _, _ = driver.execute_query( + "MATCH (p:Person {name: $name}) RETURN p.email", + name=user_input +) +``` + +```cypher +// Cypher-side (parameters passed by driver) +MATCH (p:Person {name: $name}) +RETURN p +``` + +### 3.4 Subqueries and Chaining + +```cypher +-- COUNT subquery (Cypher 25) +MATCH (p:Person) +RETURN p.name, COUNT { (p)-[:KNOWS]->() } AS friend_count + +-- COLLECT subquery +MATCH (p:Person) +RETURN p.name, COLLECT { + MATCH (p)-[:KNOWS]->(friend) + RETURN friend.name +} AS friends + +-- WITH for pipeline chaining +MATCH (p:Person)-[:PURCHASED]->(item:Product) +WITH p, count(item) AS purchase_count +WHERE purchase_count > 5 +RETURN p.name, purchase_count +``` + +--- + +## 4. Graph Data Modeling Patterns by Domain + +### 4.1 Knowledge Graph + +**Pattern**: Entities connected by typed, often hierarchical relationships. + +```cypher +-- Schema: documents, concepts, entities with semantic relationships +(:Document {id, title, published_date}) + -[:CONTAINS]->(:Chunk {id, text, embedding}) + -[:MENTIONS]->(:Entity {id, name, type}) + +(:Entity)-[:RELATED_TO {weight, relationship_type}]->(:Entity) +(:Entity)-[:SUBCLASS_OF]->(:Entity) -- taxonomy hierarchy +``` + +**Common queries**: +```cypher +-- Multi-hop: find concepts reachable from a document +MATCH (d:Document {id: $doc_id})-[:CONTAINS]->(:Chunk)-[:MENTIONS]->(e:Entity) +RETURN DISTINCT e.name, e.type + +-- Graph traversal for GraphRAG: entities connected to a seed through 2 hops +MATCH (e:Entity {name: $seed})-[]->{1,2}(related:Entity) +RETURN related.name, related.type +``` + +**When it wins vs. relational**: Multi-hop queries (`book → author → institution → location`) that would require 4+ JOINs in SQL are a single variable-length pattern match in Cypher. + +### 4.2 Recommendation Engine + +**Pattern**: Users, items, and interactions as relationships carrying weight/timestamp. + +```cypher +-- Schema +(:User {id, preferences, embeddding}) + -[:RATED {score, timestamp}]->(:Item {id, category, tags}) + -[:BELONGS_TO]->(:Category {name}) + -[:SIMILAR_TO {score}]->(:Category) + +(:User)-[:FRIENDS_WITH]->(:User) +(:User)-[:VIEWED]->(:Item) +(:Item)-[:CO_OCCURS {count}]->(:Item) -- "bought together" +``` + +**Common queries**: +```cypher +-- Collaborative filtering: "users like you also liked" +MATCH (me:User {id: $user_id})-[:RATED]->(item:Item) +WHERE item.rating >= 4 +MATCH (other:User)-[:RATED]->(item) +WHERE other.id <> $user_id +MATCH (other)-[:RATED]->(rec:Item) +WHERE NOT exists { (me)-[:RATED]->(rec) } +RETURN rec.id, avg(other.rating) AS predicted_rating +ORDER BY predicted_rating DESC +LIMIT 20 + +-- Content-based: "similar to items you liked" +MATCH (me:User {id: $user_id})-[:RATED {score: 5}]->(liked:Item) +MATCH (liked)-[:BELONGS_TO]->(cat:Category) +MATCH (rec:Item)-[:BELONGS_TO]->(cat) +WHERE NOT exists { (me)-[:RATED]->(rec) } +RETURN rec.id, count(*) AS matches +ORDER BY matches DESC +LIMIT 10 +``` + +**When it wins vs. relational**: The `other-[:RATED]->item` join pattern (users-to-items-to-users) avoids a three-table self-join. Variable-length traversal replaces recursive CTEs for path-based similarity. + +### 4.3 Network/Mesh (Infrastructure & Topology) + +**Pattern**: Physical or logical nodes connected by directed/undirected links, often with layered abstraction. + +```cypher +-- Schema: cloud infrastructure +(:Server {id, hostname, ip, region, provider}) + -[:HOSTS]->(:Container {id, image, status}) + -[:RUNS]->(:Service {name, version, port}) + +(:Server)-[:CONNECTS_TO {bandwidth, latency_ms}]->(:Server) +(:Service)-[:DEPENDS_ON]->(:Service) +(:Service)-[:EXPOSES]->(:Endpoint {path, method}) +(:Subnet {cidr})-[r:CONTAINS]->(:Server) +``` + +**Common queries**: +```cypher +-- Blast radius: all services reachable from a failing server +MATCH (s:Server {id: $server_id})-[:HOSTS]->(:Container)-[:RUNS]->(svc:Service) +RETURN svc.name + +-- Dependency chain: find all transitive dependencies +MATCH (svc:Service {name: $svc_name})-[:DEPENDS_ON]->{1..10}(dependency:Service) +RETURN DISTINCT dependency.name, length(path) AS depth + +-- Shortest network path between two servers +MATCH SHORTEST 1 (a:Server {ip: $ip1})-[:CONNECTS_TO*]-(b:Server {ip: $ip2}) +RETURN [x IN nodes(path) | x.hostname] AS route +``` + +**When it wins vs. relational**: Blast-radius analysis and transitive dependency resolution are O(1)-per-hop tree traversals in a graph vs. recursive CTEs (which hit recursive query limits and degrade with depth). + +### 4.4 Access Control (RBAC / ReBAC) + +**Pattern**: Users, roles, permissions, and resources as nodes; grants and assignments as relationships. + +```cypher +-- Schema: Relationship-Based Access Control (ReBAC) +(:User {id, email}) + -[:HAS_ROLE]->(:Role {name, level}) + -[:GRANTS]->(:Permission {action, resource_type}) + +(:User)-[:MEMBER_OF]->(:Group {name}) +(:Group)-[:HAS_ROLE]->(:Role) + +(:Permission)-[:ON]->(:Resource {id, type, owner_id}) + +-- Direct access via ownership +(:User)-[:OWNS]->(:Resource) + +-- Organization hierarchy +(:OrgUnit)-[:CONTAINS]->(:OrgUnit) +(:User)-[:BELONGS_TO]->(:OrgUnit) +``` + +**Common queries**: +```cypher +-- Is user authorized to perform action on resource? +MATCH (u:User {id: $user_id}) +MATCH (r:Resource {id: $resource_id}) +CALL { + WITH u + // Direct role assignment + MATCH (u)-[:HAS_ROLE]->(role:Role)-[:GRANTS]->(perm:Permission) + WHERE perm.action = $action + RETURN perm + UNION + // Group membership + MATCH (u)-[:MEMBER_OF]->(:Group)-[:HAS_ROLE]->(role:Role)-[:GRANTS]->(perm:Permission) + WHERE perm.action = $action + RETURN perm + UNION + // Ownership + MATCH (u)-[:OWNS]->(r) + RETURN null AS perm +} +RETURN count(*) > 0 AS is_authorized + +-- Compute effective permissions for a user +MATCH (u:User {id: $user_id}) +OPTIONAL MATCH path = (u)-[:MEMBER_OF|HAS_ROLE|GRANTS*]->(p:Permission) +RETURN p.action, p.resource_type, min(length(path)) AS shortest_path +``` + +**When it wins vs. relational**: ReBAC requires modeling nested group membership and inheritance chains. In SQL this is a many-many join across 5+ tables with recursive CTEs. In Cypher it's a variable-length traversal with union. + +--- + +## 5. Cypher Aggregation & Graph Traversal + +### 5.1 Aggregation Functions + +| Function | Purpose | Example | +|----------|---------|---------| +| `count()` | Count rows or distinct values | `RETURN count(*)` | +| `collect()` | Aggregate into a list | `RETURN p.name, collect(friend.name)` | +| `avg()` | Average of numeric values | `RETURN avg(r.rating)` | +| `sum()` | Sum of values | `RETURN sum(o.total)` | +| `min()` / `max()` | Min/max | `RETURN max(p.salary)` | +| `stDev()` / `stDevP()` | Sample/population stddev | `RETURN stDev(p.age)` | + +**GROUP BY is implicit**: any non-aggregated column in `RETURN` is a grouping key. + +```cypher +MATCH (o:Order)-[:CONTAINS]->(p:Product) +RETURN p.category, count(o) AS order_count, avg(o.total) AS avg_order_value +ORDER BY order_count DESC +``` + +### 5.2 Graph Traversal Patterns + +**Variable-length path traversal:** +```cypher +-- Depth 1 to 3 +MATCH (a:Person)-[:KNOWS]->{1,3}(b:Person) +RETURN a.name, collect(DISTINCT b.name) AS network + +-- Exactly 3 hops +MATCH (a:Person)-[:KNOWS]->{3}(b:Person) +``` + +**Shortest/fastest paths (Cypher 25):** +```cypher +-- Single shortest path +MATCH SHORTEST 1 (a:Airport {code: "LAX"})-[:ROUTE*]-(b:Airport {code: "JFK"}) +RETURN [x IN nodes(path) | x.code] AS route + +-- All shortest paths +MATCH ALL SHORTEST (a)-[:KNOWS*]-(b) +RETURN count(path) AS path_count + +-- Cost-based (using relationship property as weight) +MATCH SHORTEST 1 (a:City {name: "NYC"})-[:ROAD*]-(b:City {name: "SF"}) +WHERE reduce(cost = 0, r IN relationships(path) | cost + r.distance) < 5000 +RETURN path, reduce(cost = 0, r IN relationships(path) | cost + r.distance) AS total_distance +``` + +**Quantified path patterns (Cypher 25):** +```cypher +-- Named quantified path: each hop can be any of several relationship types +MATCH (a:Person) ((:Person)-[:KNOWS|:FRIENDS_WITH]->(:Person)){1,3} (b:Person) +RETURN a.name, b.name +``` + +### 5.3 Path Projections and Analysis + +```cypher +-- Extract node names from a path +MATCH p = (a:Person)-[:KNOWS*1..3]->(b:Person) +RETURN [n IN nodes(p) | n.name] AS name_chain, + length(p) AS depth + +-- Sum relationship properties along a path +MATCH p = (a:User)-[:TRANSFERRED*]-(b:User) +RETURN reduce(total = 0, r IN relationships(p) | total + r.amount) AS total_transferred + +-- Find paths where a condition holds at each step +MATCH p = (a:Car {status: "active"})-[:BELONGS_TO*]-(org:Org) +WHERE all(n IN nodes(p) WHERE n.active = true) +RETURN p +``` + +--- + +## 6. Importing Data into Neo4j + +### 6.1 `LOAD CSV` — Online, Incremental + +Best for small to medium datasets (up to ~10M rows). Runs as a Cypher query. + +```cypher +// Simple import +LOAD CSV WITH HEADERS FROM 'file:///users.csv' AS row +CREATE (:User { + id: row.id, + name: row.name, + email: row.email, + created_at: datetime(row.created_at) +}) + +// With MERGE and relationships +LOAD CSV WITH HEADERS FROM 'https://s3.amazonaws.com/bucket/orders.csv' AS row +MATCH (u:User {id: row.user_id}) +MATCH (p:Product {id: row.product_id}) +MERGE (u)-[:PURCHASED {amount: toFloat(row.amount), date: date(row.date)}]->(p) + +// Periodic commit (for large files, though Cypher 25 handles streaming better) +:auto USING PERIODIC COMMIT 5000 +LOAD CSV WITH HEADERS FROM 'file:///large.csv' AS row +CREATE (:Event {id: row.id}) +``` + +**Performance tips for `LOAD CSV`:** +- Always use `WITH HEADERS` for readability +- Create indexes/lookup constraints on `id` fields before loading relationships +- Pre-`MATCH` / `MERGE` by indexed property, not label scan +- Limit to 10M rows per `LOAD CSV` call for practical performance +- For larger datasets, batch with `UNWIND` and `IN TRANSACTIONS` + +```cypher +// Batched import via UNWIND (Cypher 25) +UNWIND $batch_of_rows AS row +CALL (row) { + MERGE (u:User {id: row.user_id}) + SET u.name = row.name, u.email = row.email +} IN TRANSACTIONS OF 5000 ROWS +``` + +### 6.2 APOC Load (`apoc.load.*`) + +The APOC library provides more powerful import capabilities. + +```cypher +// Load from JSON API +CALL apoc.load.json("https://api.example.com/users") +YIELD value +MERGE (u:User {id: value.id}) +SET u.name = value.name, u.email = value.email + +// Load from CSV with more control +CALL apoc.load.csv("data.csv", {header: true, sep: "|"}) +YIELD map AS row +CREATE (:Record {id: row.id, value: row.val}) + +// Load from Parquet / ORC (via apoc.nlp or custom plugins) +CALL apoc.load.parquet("s3://bucket/data.parquet") +YIELD row +MERGE (p:Product {sku: row.sku}) +SET p.price = row.price + +// Conditional import +CALL apoc.periodic.iterate( + "LOAD CSV WITH HEADERS FROM 'file:///data.csv' AS row RETURN row", + "MATCH (c:Customer {id: row.id}) + CREATE (c)-[:PURCHASED]->(:Order {id: row.order_id, total: toFloat(row.total)})", + {batchSize: 1000, parallel: true} +) +``` + +### 6.3 `neo4j-admin database import` — Bulk Offline Import + +Best for initial bulk loads (hundreds of millions to billions of nodes). Requires the database to be offline. + +```bash +# Stop Neo4j first, then: +neo4j-admin database import full \ + --nodes=import/users_header.csv,import/users.csv \ + --nodes=import/products_header.csv,import/products.csv \ + --relationships=import/purchases_header.csv,import/purchases.csv \ + --delimiter="," \ + --verbose + +# With a single header file per entity type +# users_header.csv: id:ID, name, email:STRING, age:INT, :LABEL +# purchases_header.csv: :START_ID, :END_ID, amount:FLOAT, date:DATE, :TYPE +``` + +**Performance:** +- 1-2 billion nodes per hour on reasonable hardware +- Creates the database from scratch (no merge logic) +- Best for initial data loads, then use CDC/incremental for updates + +### 6.4 Import Strategy Decision + +| Data Volume | Approach | Latency | Complexity | +|-------------|----------|---------|------------| +| < 100K rows | `LOAD CSV` | Minutes | Low | +| 100K – 10M | `apoc.periodic.iterate` | Minutes | Medium | +| 10M – 100M | `LOAD CSV` + `IN TRANSACTIONS` | Hours | Medium | +| 100M – 1B+ | `neo4j-admin database import` | Minutes (offline) | High | +| Streaming / real-time | CDC + `MERGE` | Sub-second | High | +| Incremental updates | `apoc.periodic.iterate` / CDC | Variable | Medium | + +--- + +## 7. Indexing & Performance Optimization + +### 7.1 Index Types + +| Index Type | Syntax | Use Case | +|---|---|---| +| **BTREE** (default) | `CREATE INDEX FOR (p:Person) ON (p.name)` | Equality, range, prefix queries | +| **RANGE** (Cypher 25) | `CREATE RANGE INDEX FOR (p:Person) ON (p.name)` | Same as BTREE; preferred syntax | +| **TEXT** | `CREATE TEXT INDEX FOR (p:Person) ON (p.name)` | Full-text `CONTAINS`, `STARTS WITH` | +| **POINT** | `CREATE POINT INDEX FOR (l:Location) ON (l.coords)` | Spatial queries (`distance()`, `point.withinBBox()`) | +| **VECTOR** | `CREATE VECTOR INDEX FOR (c:Chunk) ON (c.embedding)` | ANN similarity search for embeddings | +| **FULLTEXT** | `CREATE FULLTEXT INDEX names FOR (p:Person) ON EACH [p.name]` | Language-aware full-text search | + +### 7.2 Constraints (Which Also Create Indexes) + +```cypher +-- Unique constraint (creates a backing index) +CREATE CONSTRAINT FOR (p:Person) REQUIRE p.id IS UNIQUE + +-- Node key constraint (composite uniqueness) +CREATE CONSTRAINT FOR (p:Person) REQUIRE (p.first_name, p.last_name) IS NODE KEY + +-- Existence constraint +CREATE CONSTRAINT FOR (p:Person) REQUIRE p.email IS NOT NULL +``` + +**Rule of thumb**: Every property you filter on in `WHERE` should have an index. Every property used for `MERGE` should have a uniqueness constraint. + +### 7.3 Query Performance Rules + +1. **Use labels always**: `MATCH (n)` scans everything → always write `MATCH (n:Label)`. +2. **Index lookup before traversal**: Put selective filters first. + ```cypher + -- Fast: narrows to one user first, then traverses + MATCH (u:User {id: $id})-[:PURCHASED]->(o:Order) + RETURN o + + -- Slow: might scan all orders first + MATCH (o:Order)<-[:PURCHASED]-(u:User {id: $id}) + RETURN o + ``` +3. **Use `PROFILE` and `EXPLAIN`**: + ```cypher + PROFILE MATCH (u:User {id: $id})-[:PURCHASED]->(o:Order) RETURN o + ``` + Look for `NodeByLabelScan` (bad) vs `NodeUniqueIndexSeek` (good). +4. **Always `LIMIT` unbounded traversals** in user-facing queries. +5. **Avoid `RETURN n`** for large result sets — project specific properties. +6. **Use `WHERE n.property = $param`** over `{property: $param}` when the filter is a precondition. + +### 7.4 Caching Strategy + +Neo4j uses a page cache (mmap-based). Everything that fits in cache runs at memory speed. + +```ini +# neo4j.conf +# Set page cache to 50-70% of available RAM for graph workloads +server.memory.pagecache.size=8G +# Heap for query execution and transactions +server.memory.heap.max_size=4G +# Off-heap for GDS algorithms +server.memory.off_heap.max_size=2G +``` + +**Cache hit ratio monitoring:** +```cypher +CALL dbms.listConfig() YIELD name, value +WHERE name STARTS WITH "server.memory" +RETURN name, value +``` + +--- + +## 8. Graph Algorithms Library (GDS) + +The Neo4j Graph Data Science library provides in-database parallel graph algorithms. Algorithms operate on an **in-memory graph projection**, not the stored graph directly. + +### 8.1 Workflow + +``` +Stored Graph → Project → In-Memory Graph → Run Algorithm → Stream/Write Results +``` + +```cypher +-- 1. Project a graph into memory +CALL gds.graph.project( + 'myGraph', + ['Person', 'Company'], + ['KNOWS', 'WORKS_FOR'] +) + +-- 2. Run an algorithm +CALL gds.pageRank.stream('myGraph') +YIELD nodeId, score +RETURN gds.util.asNode(nodeId).name AS name, score +ORDER BY score DESC +LIMIT 10 + +-- 3. Write results back to stored graph +CALL gds.pageRank.write('myGraph', {writeProperty: 'pagerank'}) + +-- 4. Drop the in-memory graph when done +CALL gds.graph.drop('myGraph') +``` + +### 8.2 Algorithm Categories + +#### Pathfinding + +| Algorithm | Use Case | Syntax Hint | +|---|---|---| +| **Shortest Path (Dijkstra)** | Weighted shortest route | `gds.shortestPath.dijkstra.stream()` | +| **A\*** | Geospatial routing (with coordinates) | `gds.shortestPath.astar.stream()` | +| **All Pairs / Single Source** | Distance matrix computation | `gds.allPairsShortestPath.stream()` | +| **Yen's K-Shortest Paths** | Top-N alternative routes | `gds.shortestPath.yens.stream()` | + +```cypher +-- Weighted shortest path +MATCH (a:Airport {code: "LAX"}), (b:Airport {code: "JFK"}) +CALL gds.shortestPath.dijkstra.stream('flightGraph', { + sourceNode: a, + targetNode: b, + relationshipWeightProperty: 'distance' +}) +YIELD nodeIds, totalCost +RETURN [id IN nodeIds | gds.util.asNode(id).code] AS route, totalCost +``` + +#### Centrality (Node Importance) + +| Algorithm | What It Measures | Use Case | +|---|---|---| +| **PageRank** | Inbound link importance | Influence ranking, recommendation | +| **Betweenness Centrality** | Node bridge/connector importance | Identifying chokepoints, fraud rings | +| **Closeness Centrality** | Average distance to all other nodes | Information propagation speed | +| **Degree Centrality** | Number of connections | Hub identification | +| **Eigenvector Centrality** | Influence of connected nodes | Authority ranking | +| **ArticleRank** | PageRank variant for co-citation | Academic citation analysis | + +```cypher +-- PageRank for influencer detection +CALL gds.pageRank.stream('socialGraph', { + maxIterations: 20, + dampingFactor: 0.85 +}) +YIELD nodeId, score +RETURN gds.util.asNode(nodeId).name AS influencer, score +ORDER BY score DESC +LIMIT 50 + +-- Betweenness for bridge detection (identify fraud mules) +CALL gds.betweenness.stream('transactionGraph') +YIELD nodeId, score +RETURN gds.util.asNode(nodeId).id AS account_id, score +ORDER BY score DESC +``` + +#### Community Detection + +| Algorithm | Type | Use Case | +|---|---|---| +| **Louvain** | Hierarchical clustering | General community detection, org structure | +| **Label Propagation** | Fast, near-linear | Large-scale community assignment | +| **Weakly Connected Components** | Connectivity check | Isolated subgraph detection | +| **Strongly Connected Components** | Directed connectivity | Dependency cycles | +| **Triangle Count / Clustering Coefficient** | Local connectivity density | Fraud rings, highly clustered groups | +| **K-1 Coloring** | Graph coloring | Resource allocation, scheduling | +| **Modularity Optimization** | Quality measure for communities | Evaluating cluster quality | + +```cypher +-- Louvain community detection +CALL gds.louvain.stream('interactionGraph') +YIELD nodeId, communityId, intermediateCommunityIds +RETURN gds.util.asNode(nodeId).name AS name, communityId +ORDER BY communityId + +-- Label Propagation (for billion-node graphs) +CALL gds.labelPropagation.stream('hugeGraph', {maxIterations: 10}) +YIELD nodeId, communityId +RETURN communityId, count(*) AS member_count +ORDER BY member_count DESC + +-- Triangle count (fraud detection: dense subgraphs) +CALL gds.triangleCount.stream('transactionGraph') +YIELD nodeId, triangleCount +WHERE triangleCount > 10 +RETURN gds.util.asNode(nodeId).id AS account, triangleCount +ORDER BY triangleCount DESC +``` + +#### Node Embedding (Graph ML) + +| Algorithm | Description | +|---|---| +| **FastRP** | Fast random-projection embeddings | +| **Node2Vec** | Random-walk based embeddings | +| **GraphSAGE** | GNN-based inductive embeddings | +| **HashGNN** | Scalable GNN embeddings | + +### 8.3 GDS Production Tips + +- **Mutate mode** (`{mutateProperty: '...'}`) — stores results in the in-memory graph without writing to the stored graph. Useful for chaining: run Louvain, use communities as features for Node2Vec. +- **Write mode** (`{writeProperty: '...'}`) — persists to the stored graph for dashboard queries. +- **Tiered projections**: Create progressively smaller projections for iterative algorithm chaining. +- **Memory estimation**: Always call `gds..estimate()` before running on large graphs to avoid OOM. + +--- + +## 9. Integrating Neo4j into Data Pipelines + +### 9.1 Change Data Capture (CDC) + +Neo4j CDC (GA since 2024) streams transaction log changes to Kafka or directly to consumers. + +```bash +# Enable CDC on the database +ALTER DATABASE neo4j SET cdc ENABLED; +``` + +```python +# Python CDC consumer (via Neo4j Kafka Connector or direct capture) +from neo4j import GraphDatabase + +def watch_changes(driver): + # Poll the CDC stream + with driver.session() as session: + result = session.run(""" + CALL cdc.current() + YIELD eventId, operation, metadata, change + RETURN eventId, operation, metadata, change + ORDER BY eventId + LIMIT 100 + """) + for record in result: + handle_change(record) +``` + +**Neo4j Connector for Apache Kafka:** +``` +Source: Neo4j → CDC → Kafka topic → Sink (downstream systems) +``` + +CDC captures every `CREATE`, `UPDATE`, `DELETE` on nodes and relationships with before/after snapshots. + +### 9.2 Querying Neo4j from Applications + +**Python (neo4j driver):** +```python +from neo4j import GraphDatabase + +class Neo4jConnection: + def __init__(self, uri, user, password): + self.driver = GraphDatabase.driver(uri, auth=(user, password)) + + def close(self): + self.driver.close() + + def find_person_network(self, name): + with self.driver.session(database="neo4j") as session: + result = session.run(""" + MATCH (p:Person {name: $name})-[:KNOWS]->{1,3}(contacts) + RETURN contacts.name AS name, + labels(contacts) AS labels + LIMIT 100 + """, name=name) + return [record.data() for record in result] + +# Singleton pattern — one driver per process +conn = Neo4jConnection("neo4j+s://myinstance.databases.neo4j.io", "neo4j", os.getenv("PASSWORD")) +network = conn.find_person_network("Alice") +conn.close() +``` + +**HTTP Query API (driverless — useful for serverless/lambda):** +```bash +curl -X POST https://.databases.neo4j.io/db/neo4j/query/v2 \ + -u neo4j:$PASSWORD \ + -H "Content-Type: application/json" \ + -d '{ + "statement": "MATCH (p:Person {name: $name}) RETURN p.email", + "parameters": {"name": "Alice"} + }' +``` + +**Airflow integration (custom hook or operator):** +```python +from airflow.providers.common.sql.hooks import SqlHook + +# Use a custom Neo4j hook or the generic DB API hook +# Alternatively, use the PythonOperator with the neo4j driver directly +def pull_graph_data(**context): + driver = GraphDatabase.driver(...) + with driver.session() as session: + result = session.run("MATCH ... RETURN ...") + return [r.data() for r in result] +``` + +### 9.3 Pipeline Architecture Patterns + +**Batch ETL (Daily/Weekly):** +``` +Source DB → (CSV/Parquet) → S3/GCS → LOAD CSV / apoc.load → Neo4j +``` + +**Streaming / Micro-batch:** +``` +Source DB → Debezium → Kafka → Neo4j Connector (CDC sink) → Neo4j +``` + +**Dual-write / Transactional:** +``` +App → (Write to Postgres + Neo4j in same transaction) → Both databases consistent +``` + +**Graph as enrichment layer:** +``` +Data Lake → Spark (featurization using GDS) → ML model training + ↓ + Neo4j (graph features joined back) +``` + +### 9.4 Neo4j + Spark Integration + +The Neo4j Spark Connector supports reading/writing via DataFrames: + +```python +# Read from Neo4j into Spark +df = spark.read \ + .format("org.neo4j.spark.DataSource") \ + .option("url", "neo4j+s://...") \ + .option("query", "MATCH (p:Person)-[:KNOWS]->(f:Person) RETURN p.name, collect(f.name) AS friends") \ + .load() + +# Write from Spark to Neo4j +df.write \ + .format("org.neo4j.spark.DataSource") \ + .option("url", "neo4j+s://...") \ + .option("labels", ":Person") \ + .mode("Overwrite") \ + .save() +``` + +### 9.5 Neo4j + GraphQL + +Neo4j GraphQL Library auto-generates a GraphQL API from the graph model: + +```javascript +const { Neo4jGraphQL } = require("@neo4j/graphql"); +const { Neo4jDriver } = require("neo4j-driver"); + +const typeDefs = ` + type Person { + name: String! + knows: [Person!]! @relationship(type: "KNOWS", direction: OUT) + } +`; + +const neoSchema = new Neo4jGraphQL({ typeDefs, driver }); +const schema = await neoSchema.getSchema(); +// Expose as Apollo Server, Express, etc. +``` + +--- + +## 10. Graph vs. Relational: When to Use Which + +### 10.1 Decision Matrix + +| Criteria | Choose Graph (Neo4j) | Choose Relational (Postgres, etc.) | +|---|---|---| +| **Connection depth** | Deep traversals (3+ hops) frequent | Shallow joins (1-2 tables) | +| **Relationship cardinality** | Many-to-many, recursive, hierarchical | One-to-many, simple FK lookups | +| **Schema evolution** | Frequent, ad-hoc, per-instance | Stable, predefined migrations | +| **Query pattern** | "Who/what is connected to X through Y?" | "What are the attributes of X?" | +| **Write volume** | Moderate (OLTP) or batch (analytics) | High-velocity OLTP | +| **Data volume** | Hundreds of millions of relationships | Trillions of rows (columnar) | +| **Team expertise** | Data scientists, ML engineers | DBAs, backend engineers | +| **Reporting** | Graph-based analytics (GDS) | SQL BI, OLAP cubes | + +### 10.2 When SQL JOINs Become Painful + +**Query**: "Find all products purchased by people who bought the same product as Alice and live in the same city as Bob" + +```sql +-- SQL (6 JOINs, deeply nested) +SELECT DISTINCT p2.name +FROM users alice +JOIN orders o1 ON alice.id = o1.user_id +JOIN order_items oi1 ON o1.id = oi1.order_id +JOIN products p1 ON oi1.product_id = p1.id +JOIN order_items oi2 ON p1.id = oi2.product_id +JOIN orders o2 ON oi2.order_id = o2.id +JOIN users u2 ON o2.user_id = u2.id +JOIN users bob ON bob.name = 'Bob' +WHERE alice.name = 'Alice' + AND u2.city = bob.city + AND u2.id != alice.id; +``` + +```cypher +-- Cypher (natural pattern match) +MATCH (alice:User {name: "Alice"})-[:PURCHASED]->(:Product)<-[:PURCHASED]-(other:User), + (bob:User {name: "Bob"}) +WHERE other.city = bob.city AND other <> alice +MATCH (other)-[:PURCHASED]->(rec:Product) +WHERE NOT (alice)-[:PURCHASED]->(rec) +RETURN DISTINCT rec.name +``` + +### 10.3 Hybrid Approaches + +Many production systems use both: +- **Postgres** for transactional data (orders, users, inventory) +- **Neo4j** for recommendations, fraud detection, and relationship analytics +- **Elasticsearch** for full-text search +- Sync via CDC (Debezium → Kafka → Neo4j connector) + +```python +# Dual database pattern +def get_recommendations(user_id): + # 1. Get user profile from Postgres (OLTP) + user = pg_client.query("SELECT * FROM users WHERE id = %s", user_id) + + # 2. Get recommendations from Neo4j (graph traversal) + with neo4j_driver.session() as session: + result = session.run(""" + MATCH (me:User {id: $uid})-[:PURCHASED]->(:Product) + <-[:PURCHASED]-(other:User) + MATCH (other)-[:PURCHASED]->(rec:Product) + WHERE NOT (me)-[:PURCHASED]->(rec) + RETURN rec.id, count(*) AS score + ORDER BY score DESC LIMIT 10 + """, uid=user_id) + return [record["rec.id"] for record in result] +``` + +### 10.4 Cost & Operational Comparison + +| Factor | Relational (RDS Postgres) | Graph (Neo4j Aura) | +|--------|--------------------------|-------------------| +| **Query time** (3-hop join) | 500ms – 5s (depending on indexes) | 5ms – 50ms | +| **Query time** (10-hop recursive) | Minutes or timeout | 100ms – 500ms | +| **Schema migration** | ALTER TABLE (locking) | Add label/relationship at runtime | +| **Backup size** | Larger (normalized with indexes) | More compact (pointer-based) | +| **Learning curve** | Widely known | Specialized (Cypher, GDS) | +| **Tool ecosystem** | Mature (every BI tool) | Growing (Bloom, Neodash, GraphQL) | + +### 10.5 Rule of Thumb + +> **Use a graph database when the relationships between your entities are as important as, or more important than, the entities themselves.** + +If your primary query pattern is "find me X by its attributes" with occasional FK lookups → use relational. + +If your primary query pattern is "find me everything connected to X through N degrees of separation" → use a graph. + +If you need both → use both (polyglot persistence). + +--- + +## Appendix A: Quick Reference — Cypher by Analogy to SQL + +| SQL | Cypher | +|-----|--------| +| `SELECT col FROM table` | `RETURN n.prop` | +| `FROM table AS t` | `MATCH (t:Label)` | +| `WHERE t.col = val` | `WHERE t.prop = val` or `MATCH (t {prop: val})` | +| `JOIN t1 ON t1.id = t2.fk` | `(a)-[:REL]->(b)` | +| `LEFT JOIN` | `OPTIONAL MATCH` | +| `GROUP BY col` | Implicit in `RETURN` with aggregation | +| `ORDER BY col LIMIT n` | `ORDER BY col LIMIT n` | +| `INSERT INTO` | `CREATE` or `MERGE` | +| `UPDATE` | `SET n.prop = val` | +| `DELETE` | `DETACH DELETE n` | +| `UNION` | `UNION` | +| `WITH (CTE)` | `WITH` (pipeline) | +| Recursive CTE | Variable-length `[]->{1..n}` | +| `ROW_NUMBER() OVER (PARTITION BY ...)` | Reduce to pattern match + collect | + +## Appendix B: Essential CLI Tools + +```bash +# neo4j-admin — backup, restore, import +neo4j-admin database dump neo4j --to-backup=/backups/ +neo4j-admin database load neo4j --from-backup=/backups/ + +# cypher-shell — direct Cypher execution +echo "MATCH (n) RETURN count(n)" | cypher-shell -u neo4j -p password + +# neo4j-cli — unified agent-friendly CLI +neo4j-cli aura create myinstance --region us-east-1 --type professional +neo4j-cli cypher "MATCH (n) RETURN count(n)" +neo4j-cli schema describe + +# Install neo4j-cli +curl -sSfL https://neo4j.sh/install.sh | bash +``` + +--- + +*Generated: 2025-06-05 | Based on Neo4j 5.x / Cypher 25 / GDS 2.x* diff --git a/data-engineering/references/sql-analytical-patterns.md b/data-engineering/references/sql-analytical-patterns.md new file mode 100644 index 0000000..2cacf5b --- /dev/null +++ b/data-engineering/references/sql-analytical-patterns.md @@ -0,0 +1,1266 @@ +# Data Engineering SQL & Relational Database Reference + +**Purpose:** A thorough reference for data engineers covering analytical SQL patterns, +ETL/ELT patterns, query performance, data modeling, testing, and engine comparisons. +This is methodology-level guidance — not a tutorial, but a field manual. + +--- + +## Table of Contents + +1. [Analytical SQL Patterns](#1-analytical-sql-patterns) +2. [ETL/ELT SQL Patterns](#2-etlelt-sql-patterns) +3. [Query Performance Patterns](#3-query-performance-patterns) +4. [Data Modeling for Analytics](#4-data-modeling-for-analytics) +5. [SQL Testing & Validation Patterns](#5-sql-testing--validation-patterns) +6. [Analytical SQL Engine Comparison](#6-analytical-sql-engine-comparison) + +--- + +## 1. Analytical SQL Patterns + +### 1.1 Window Functions + +Window functions perform calculations across a set of rows related to the current +row, without collapsing rows into a single output (unlike GROUP BY). + +**Syntax anatomy:** +```sql +() OVER ( + [PARTITION BY col1, col2, ...] + [ORDER BY col1 [ASC|DESC], ...] + [frame_spec] +) +``` + +**Frame specifications (critical for correctness):** +| Clause | Behavior | +|---|---| +| `ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW` | Physical — counts actual rows regardless of value ties | +| `RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW` | Logical — includes peers (rows with same ORDER BY value) | +| `ROWS BETWEEN n PRECEDING AND n FOLLOWING` | Sliding physical window of 2n+1 rows | +| `RANGE BETWEEN INTERVAL '7' DAY PRECEDING AND CURRENT ROW` | Time-based frame (Date/Time ORDER BY) | +| `ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING` | Entire partition (like SUM with no frame) | + +**Window function families:** + +| Family | Functions | Use Case | +|---|---|---| +| **Ranking** | `ROW_NUMBER()`, `RANK()`, `DENSE_RANK()`, `NTILE(n)` | Dedup, pagination, top-N-per-group | +| **Value** | `LAG(col, n)`, `LEAD(col, n)`, `FIRST_VALUE()`, `LAST_VALUE()`, `NTH_VALUE()` | Time-series shifts, YoY comparison, filling gaps | +| **Aggregate** | `SUM()`, `AVG()`, `COUNT()`, `MIN()`, `MAX()` over window | Running totals, moving averages, cumulative stats | +| **Distribution** | `PERCENT_RANK()`, `CUME_DIST()`, `PERCENTILE_CONT()`, `PERCENTILE_DISC()` | Statistical distributions, median calculation | + +**Running total (cumulative sum):** +```sql +SELECT + order_date, + amount, + SUM(amount) OVER (ORDER BY order_date + ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS running_total +FROM orders; +``` + +**Moving average (7-day):** +```sql +SELECT + date, + revenue, + AVG(revenue) OVER (ORDER BY date + ROWS BETWEEN 6 PRECEDING AND CURRENT ROW) AS ma_7d +FROM daily_revenue; +``` + +**First value in partition (fill-forward):** +```sql +SELECT + user_id, + login_date, + FIRST_VALUE(login_date) OVER (PARTITION BY user_id + ORDER BY login_date + ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) AS first_login +FROM user_logins; +``` + +**Deduplication with ROW_NUMBER:** +```sql +WITH ranked AS ( + SELECT *, + ROW_NUMBER() OVER (PARTITION BY id ORDER BY updated_at DESC) AS rn + FROM raw_table +) +SELECT * FROM ranked WHERE rn = 1; +``` + +--- + +### 1.2 Common Table Expressions (CTEs) + +CTEs improve query readability, enable recursion, and allow stepwise logic. + +**Non-recursive CTE:** +```sql +WITH monthly_sales AS ( + SELECT + DATE_TRUNC('month', order_date) AS month, + SUM(amount) AS total + FROM orders + WHERE order_date >= '2024-01-01' + GROUP BY 1 +), +ranked_months AS ( + SELECT *, + RANK() OVER (ORDER BY total DESC) AS rank + FROM monthly_sales +) +SELECT * FROM ranked_months WHERE rank <= 5; +``` + +**Recursive CTE (hierarchy traversal — org chart, bill of materials):** +```sql +WITH RECURSIVE org_tree AS ( + -- Anchor: top-level + SELECT id, name, manager_id, 1 AS level + FROM employees + WHERE manager_id IS NULL + + UNION ALL + + -- Recursive step + SELECT e.id, e.name, e.manager_id, t.level + 1 + FROM employees e + JOIN org_tree t ON e.manager_id = t.id +) +SELECT * FROM org_tree; +``` + +**CTE vs subquery guidance:** +- Use CTEs for readability when the same subquery is referenced multiple times. +- CTEs are **optimization fences** in some engines (PostgreSQL materializes them by + default; BigQuery inlines them). Test performance with real data. +- In Snowflake and DuckDB, CTEs are usually inlined unless forced with materialization hints. + +--- + +### 1.3 Pivot / Unpivot + +**Pivot (rows to columns):** + +Most engines provide a `PIVOT` or `CROSSTAB` function. The fallback is conditional aggregation. + +*Explicit PIVOT (Snowflake, BigQuery, SQL Server):* +```sql +SELECT * +FROM sales +PIVOT ( + SUM(amount) + FOR category IN ('Electronics', 'Clothing', 'Food') +) AS p; +``` + +*Conditional aggregation fallback (works everywhere):* +```sql +SELECT + region, + SUM(CASE WHEN category = 'Electronics' THEN amount ELSE 0 END) AS electronics, + SUM(CASE WHEN category = 'Clothing' THEN amount ELSE 0 END) AS clothing, + SUM(CASE WHEN category = 'Food' THEN amount ELSE 0 END) AS food +FROM sales +GROUP BY region; +``` + +**Unpivot (columns to rows):** + +*Explicit UNPIVOT (Snowflake, BigQuery, SQL Server):* +```sql +SELECT region, category, amount +FROM regional_sales +UNPIVOT ( + amount FOR category IN (electronics, clothing, food) +); +``` + +*CROSS JOIN LATERAL / UNION ALL fallback:* +```sql +SELECT region, 'electronics' AS category, electronics AS amount FROM regional_sales +UNION ALL +SELECT region, 'clothing' AS category, clothing AS amount FROM regional_sales +UNION ALL +SELECT region, 'food' AS category, food AS amount FROM regional_sales; +``` + +--- + +### 1.4 Rolling Aggregates + +Rolling aggregates extend window functions for time-series analytics. + +**Year-over-year comparison:** +```sql +SELECT + month, + revenue, + LAG(revenue, 12) OVER (ORDER BY month) AS revenue_12m_ago, + (revenue - LAG(revenue, 12) OVER (ORDER BY month)) + / NULLIF(LAG(revenue, 12) OVER (ORDER BY month), 0) * 100 AS yoy_pct +FROM monthly_revenue; +``` + +**Rolling 30-day sum (period-to-date-style):** +```sql +SELECT + date, + amount, + SUM(amount) OVER (ORDER BY date + RANGE BETWEEN INTERVAL '29' DAY PRECEDING AND CURRENT ROW) AS rolling_30d +FROM daily_data; +``` + +**Sessionized aggregates (reset per partition):** +```sql +SELECT + user_id, + event_time, + SUM(value) OVER (PARTITION BY user_id + ORDER BY event_time + ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS session_running_total +FROM user_events; +``` + +--- + +### 1.5 Date/Time Bucketing + +Bucketing dates into intervals is essential for rollups and time-series. + +**DATE_TRUNC (standard in PostgreSQL, DuckDB, Snowflake, BigQuery):** +```sql +-- Bucket to hour, day, week, month, quarter, year +SELECT + DATE_TRUNC('month', event_timestamp) AS bucket, + COUNT(*) AS events +FROM events +GROUP BY 1 +ORDER BY 1; +``` + +**Custom bucket sizes (DuckDB: `date_bin`):** +```sql +SELECT + date_bin(INTERVAL '15 minutes', event_timestamp, TIMESTAMP '2024-01-01') AS bucket_15min, + COUNT(*) AS events +FROM events +GROUP BY 1; +``` + +**ISO week and year extraction:** +```sql +SELECT + EXTRACT(YEAR FROM order_date) AS yr, + EXTRACT(WEEK FROM order_date) AS wk, + SUM(amount) AS total +FROM orders +GROUP BY yr, wk; +``` + +**Fiscal calendar bucketing (when standard months don't fit):** +```sql +SELECT + CASE + WHEN EXTRACT(MONTH FROM order_date) >= 2 THEN EXTRACT(YEAR FROM order_date) + ELSE EXTRACT(YEAR FROM order_date) - 1 + END AS fiscal_year, + SUM(amount) AS total +FROM orders +GROUP BY fiscal_year; +``` + +**Period-over-period difference using DATE_TRUNC and LAG:** +```sql +WITH weekly AS ( + SELECT + DATE_TRUNC('week', order_date) AS week, + SUM(amount) AS revenue + FROM orders + GROUP BY 1 +) +SELECT + week, + revenue, + LAG(revenue) OVER (ORDER BY week) AS prev_week_rev, + revenue - LAG(revenue) OVER (ORDER BY week) AS wow_change +FROM weekly; +``` + +--- + +## 2. ETL/ELT SQL Patterns + +### 2.1 Incremental Loading + +**Watermark / High-Water Mark pattern:** + +Use a monotonically increasing column (timestamp, auto-increment ID) to track what +has already been loaded. + +```sql +-- Extract: pull rows newer than the last watermark +INSERT INTO target_table (id, col1, col2, loaded_at) +SELECT id, col1, col2, CURRENT_TIMESTAMP +FROM source_table +WHERE updated_at > (SELECT MAX(loaded_at) FROM target_table); +``` + +**Last-modified pattern with checksum for changed detection:** +```sql +WITH source AS ( + SELECT id, MD5(col1 || col2) AS row_hash, updated_at + FROM source_table + WHERE updated_at > (SELECT MAX(watermark_ts) FROM load_watermarks WHERE table_name = 'target') +) +SELECT s.* +FROM source s +LEFT JOIN target_table t ON s.id = t.id +WHERE t.id IS NULL OR s.row_hash != t.row_hash; +``` + +**Best practices:** +- Store watermarks in a control table (`table_name`, `watermark_ts`, `row_count`, `run_id`). +- Use `BEGIN`/`COMMIT` to make extract-and-update-watermark atomic. +- Prefer timestamp columns that are indexed in the source. +- For append-only sources (event logs), use an auto-increment ID as the watermark. + +--- + +### 2.2 Merge / Upsert (MERGE / INSERT ON CONFLICT) + +**PostgreSQL (`INSERT ... ON CONFLICT DO UPDATE`):** +```sql +INSERT INTO target (id, col1, col2, updated_at) +VALUES (1, 'val1', 'val2', NOW()) +ON CONFLICT (id) DO UPDATE SET + col1 = EXCLUDED.col1, + col2 = EXCLUDED.col2, + updated_at = EXCLUDED.updated_at; +``` + +**Standard SQL MERGE (Snowflake, BigQuery, SQL Server, DuckDB):** +```sql +MERGE INTO target AS t +USING source AS s + ON t.id = s.id +WHEN MATCHED AND ( + t.col1 != s.col1 OR t.col2 != s.col2 OR (t.col1 IS NULL AND s.col1 IS NOT NULL) +) THEN UPDATE SET + col1 = s.col1, + col2 = s.col2, + updated_at = CURRENT_TIMESTAMP +WHEN NOT MATCHED THEN + INSERT (id, col1, col2, created_at, updated_at) + VALUES (s.id, s.col1, s.col2, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP); +``` + +**BigQuery MERGE (with DML):** +```sql +MERGE INTO `project.dataset.target` AS t +USING `project.dataset.source` AS s +ON t.id = s.id +WHEN MATCHED THEN + UPDATE SET col1 = s.col1, col2 = s.col2 +WHEN NOT MATCHED THEN + INSERT (id, col1, col2) VALUES (id, col1, col2); +``` + +**DuckDB MERGE (note: single UPDATE/DELETE per WHEN MATCHED):** +```sql +MERGE INTO target AS t +USING source AS s +ON t.id = s.id +WHEN MATCHED AND s._is_deleted THEN DELETE +WHEN MATCHED THEN UPDATE SET col1 = s.col1, col2 = s.col2 +WHEN NOT MATCHED THEN INSERT (id, col1, col2) VALUES (s.id, s.col1, s.col2); +``` + +**Engine-specific notes:** +| Engine | Upsert Method | Notes | +|---|---|---| +| PostgreSQL | `INSERT ... ON CONFLICT DO UPDATE` | Also supports `DO NOTHING`; requires unique index | +| Snowflake | `MERGE` | Also supports `INSERT OVERWRITE` for tables | +| BigQuery | `MERGE` | Charges for all bytes processed, even if no rows change | +| DuckDB | `INSERT OR REPLACE` or `MERGE` | DuckDB v1.3+: `MERGE` with single action per clause | +| Redshift | `MERGE` (via `UPDATE`/`INSERT` or `MERGE` since RA3) | Older versions: separate UPDATE then INSERT | +| ClickHouse | `ReplacingMergeTree` engine or `ALTER TABLE DELETE` | ClickHouse is append-optimized; upserts are not idiomatic | + +--- + +### 2.3 Change Data Capture (CDC) Patterns + +**1. Debezium-style (log-based CDC):** +- Source database captures changes via transaction log (PostgreSQL WAL, MySQL binlog). +- Events streamed to Kafka -> consumed and written to staging tables. +- Target SQL: merge staged changes into the final table. + +```sql +-- Staging table holds INSERT, UPDATE, DELETE events +WITH latest_changes AS ( + SELECT DISTINCT ON (id) id, col1, col2, op, change_ts + FROM cdc_staging + ORDER BY id, change_ts DESC +) +MERGE INTO target t +USING latest_changes s ON t.id = s.id +WHEN MATCHED AND s.op = 'DELETE' THEN DELETE +WHEN MATCHED AND s.op IN ('INSERT', 'UPDATE') THEN UPDATE SET col1 = s.col1, col2 = s.col2 +WHEN NOT MATCHED AND s.op IN ('INSERT', 'UPDATE') THEN INSERT (id, col1, col2) + VALUES (s.id, s.col1, s.col2); +``` + +**2. Audit-column CDC (watermark + last-modified):** +- Source table has `updated_at` and optionally a version column. +- Periodic poll queries `WHERE updated_at > last_watermark`. +- Works for sources that cannot stream logs. + +**3. Trigger-based CDC (SQL Server, PostgreSQL):** +- Database triggers write changes to a change-tracking table. +- Downstream reads the change table and clears processed rows. + +```sql +-- PostgreSQL trigger-captured changes +CREATE TABLE _audit_accounts ( + audit_id BIGSERIAL PRIMARY KEY, + op TEXT, -- 'INSERT', 'UPDATE', 'DELETE' + old_row JSONB, + new_row JSONB, + changed_at TIMESTAMPTZ DEFAULT NOW() +); +``` + +**4. Snapshot-diff CDC:** +- Periodically snapshot the entire source table. +- Compare the new snapshot with the previous snapshot to find changes. +- Works for small reference tables; wasteful for large fact tables. + +--- + +### 2.4 Full Refresh vs Incremental Decision Matrix + +| Scenario | Strategy | +|---|---| +| Small dimension tables (< 10K rows) | Full refresh (simpler, idempotent) | +| Large fact tables (millions of rows) | Incremental with watermark | +| Append-only event streams | Incremental by ID or timestamp | +| Slow-changing reference data | Full refresh on schedule | +| Source has no reliable watermark column | Full refresh or snapshot-diff CDC | +| Source supports CDC (logical replication) | Stream-based CDC (lowest latency) | + +--- + +## 3. Query Performance Patterns + +### 3.1 Execution Plan Analysis + +**Reading EXPLAIN output:** + +Every plan is a tree of *nodes*. Each node has cost estimates and actuals (with ANALYZE). + +``` +Seq Scan on orders (cost=0.00..1234.56 rows=56789 width=32) + Filter: (amount > 100) +``` + +| Component | Meaning | +|---|---| +| `cost=0.00..1234.56` | Startup cost .. total cost (arbitrary units) | +| `rows=56789` | Estimated rows produced by this node | +| `width=32` | Average row width in bytes | +| `actual time=12.3..45.6` | (With EXPLAIN ANALYZE) actual timing in ms | + +**Node types you'll see (PostgreSQL):** + +| Node | Meaning | Usually okay? | +|---|---|---| +| `Seq Scan` | Full table scan | Yes for small tables, bad for large filtered queries | +| `Index Scan` | Single index lookup | Good for point queries | +| `Index Only Scan` | All needed data in index | Excellent (avoids heap fetch) | +| `Bitmap Heap Scan` + `Bitmap Index Scan` | Reads index, builds bitmap, then fetches pages | Good for medium-selectivity queries | +| `Nested Loop` | For each outer row, probe inner index | Good with small outer set | +| `Hash Join` | Build hash table on one side, probe with other | Good for medium-large joins | +| `Merge Join` | Sort both sides, merge | Good for pre-sorted data | +| `Sort` / `Incremental Sort` | Ordering operation | Expensive; avoid if possible | +| `Aggregate` (Hash/GroupAgg) | GROUP BY or aggregation | HashAgg is faster; GroupAgg requires sorted input | + +**Red flags in execution plans:** +- Sequential scans on large tables (>1M rows) with selective filters (<1% of rows) +- Nested Loop joins where the outer input is large (tens of thousands+) +- Sort operations on unindexed columns driving GROUP BY or ORDER BY +- `rows` estimates far off from `actual rows` (sign of stale statistics) +- Spilling to disk (temp files) for sort/hash operations + +**EXPLAIN ANALYZE checklist:** +```sql +-- 1. Check estimated vs actual row counts (accuracy) +-- 2. Check actual time (where is the most time spent?) +-- 3. Check for sequential scans on large tables +-- 4. Check for sorts that could use indexes +-- 5. Check for loops in Nested Loop (high loop count = bad) +EXPLAIN (ANALYZE, BUFFERS, TIMING) SELECT ... +``` + +--- + +### 3.2 Index Strategies for Analytical Queries + +**Type comparison:** + +| Index Type | Best For | Avoid When | +|---|---|---| +| **B-Tree** | Equality + range queries, primary keys, foreign keys | High-cardinality columns with wide values (text blobs) | +| **BRIN** (Block Range Index) | Large, append-only, naturally ordered tables (time-series, logs) | Randomly distributed data, high-update tables | +| **Hash Index** | Exact-equality lookups only | Anything with range/order | +| **GIN** (Generalized Inverted Index) | Array columns, full-text search, JSONB | Simple = lookups on scalar columns | +| **GiST** | Geometric/geospatial data, range overlap, full-text | General-purpose analytical queries | +| **Z-ordering** (Delta/BigQuery) | Multi-dimensional range queries on several columns | Single-column queries (use simple sort instead) | + +**Analytical index patterns:** + +*Covering index (index-only scans):* +```sql +-- Avoid heap fetches by including all needed columns +CREATE INDEX idx_sales_date_amount ON sales (sale_date) INCLUDE (amount, product_id); +``` + +*Partial index (filtered):* +```sql +-- Only index active records +CREATE INDEX idx_orders_active ON orders (order_date) WHERE status = 'active'; +``` + +*Composite B-Tree for analytical filter patterns:* +```sql +-- Order columns by: equality -> range -> group/order +CREATE INDEX idx_sales_region_date ON sales (region, sale_date); +-- Supports: WHERE region = 'US' AND sale_date BETWEEN '2024-01-01' AND '2024-06-30' +``` + +*BRIN for time-series (low maintenance, tiny index):* +```sql +-- 10x smaller than B-Tree on ordered timestamps +CREATE INDEX idx_events_ts_brin ON events USING brin(created_at) + WITH (pages_per_range = 32); +``` + +**Indexing anti-patterns for analytics:** +- Don't index every column — write throughput suffers. +- Don't index low-cardinality columns alone (e.g., `gender`) — full scan is faster. +- Don't use B-Tree on timestamp columns in append-only tables — use BRIN. +- Don't forget `VACUUM`/`ANALYZE` after bulk loads — stale stats cause bad plans. + +--- + +### 3.3 Partitioning + +**When to partition:** +- Table > 100 GB or > 100M rows +- Queries always filter by a partition key (e.g., `order_date`) +- Old data can be dropped by dropping partitions (time-series retention) +- Maintenance operations (VACUUM, index rebuild) can target individual partitions + +**Partition strategies:** + +| Strategy | Key | Use Case | +|---|---|---| +| **Range** | Date, timestamp | Time-series data, event logs | +| **List** | Region, status, category | Discrete value partitions | +| **Hash** | ID, customer_id | Even data distribution, parallelism | + +**PostgreSQL range partitioning:** +```sql +CREATE TABLE orders ( + id BIGSERIAL, + order_date DATE NOT NULL, + amount NUMERIC +) PARTITION BY RANGE (order_date); + +CREATE TABLE orders_2024_q1 PARTITION OF orders + FOR VALUES FROM ('2024-01-01') TO ('2024-04-01'); +CREATE TABLE orders_2024_q2 PARTITION OF orders + FOR VALUES FROM ('2024-04-01') TO ('2024-07-01'); +``` + +**BigQuery partitioning (table creation):** +```sql +CREATE TABLE `project.dataset.orders` +PARTITION BY DATE(order_timestamp) +CLUSTER BY region, product_id +OPTIONS(require_partition_filter=true); +``` + +**Snowflake clustering (automatic):** +```sql +ALTER TABLE orders CLUSTER BY (order_date, region); +``` + +**Partition pruning verification:** +```sql +-- PostgreSQL: check for "Append" node showing only relevant partitions +EXPLAIN SELECT * FROM orders WHERE order_date = '2024-02-15'; +``` + +**Key rules:** +- Aim for 100-500 partitions (too few = no benefit; too many = metadata overhead). +- Always filter queries by the partition key. +- Use partition pruning verification after implementation. +- Consider declarative partitioning over manual table inheritance. + +--- + +### 3.4 Clustering (within-partition ordering) + +Clustering physically co-locates rows with similar cluster-key values. This reduces +the amount of data scanned by filter/aggregation queries. + +| Engine | Feature | Notes | +|---|---|---| +| BigQuery | `CLUSTER BY` | Automatic re-clustering; no maintenance | +| Snowflake | `CLUSTER BY` | Automatic, but reclustering costs credits | +| Redshift | `SORTKEY` compound/interleaved | Manual; re-sort with `VACUUM SORT ONLY` | +| DuckDB | `ORDER BY` within `CREATE TABLE AS` | Manual; use WITH clause or ordering | +| PostgreSQL | CLUSTER command | One-time reorder; not maintained automatically | + +**Strategy:** +```sql +-- BigQuery +CREATE TABLE `project.dataset.orders` +PARTITION BY DATE(order_date) +CLUSTER BY customer_id, region; +``` + +```sql +-- Redshift +CREATE TABLE orders ( + id BIGINT, + order_date DATE, + customer_id BIGINT, + region VARCHAR(50) +) SORTKEY (customer_id, order_date); +``` + +**Cluster key ordering rules:** +- High-cardinality filter columns first. +- Equality filter columns before range filter columns. +- Columns frequently used in GROUP BY or ORDER BY. +- Avoid columns that are monotonically increasing (like timestamps) as the + *first* cluster key if the table is also partitioned by time — it adds no extra benefit. + +--- + +### 3.5 Materialized Views + +Materialized views pre-compute and store query results. They trade storage for +query speed. + +**PostgreSQL materialized view:** +```sql +CREATE MATERIALIZED VIEW mv_monthly_sales AS +SELECT + DATE_TRUNC('month', order_date) AS month, + region, + SUM(amount) AS total_sales, + COUNT(*) AS order_count +FROM orders +GROUP BY 1, 2; + +-- Refresh (blocking — table locked during refresh) +REFRESH MATERIALIZED VIEW mv_monthly_sales; + +-- Concurrent refresh (non-blocking, requires unique index) +CREATE UNIQUE INDEX idx_mv_monthly_sales_key ON mv_monthly_sales (month, region); +REFRESH MATERIALIZED VIEW CONCURRENTLY mv_monthly_sales; +``` + +**BigQuery materialized views (auto-refreshed):** +```sql +CREATE MATERIALIZED VIEW `project.dataset.monthly_sales` +AS +SELECT + DATE_TRUNC(order_date, MONTH) AS month, + region, + SUM(amount) AS total_sales +FROM `project.dataset.orders` +GROUP BY 1, 2; +``` + +**Snowflake materialized views (auto-maintained, credits incurred):** +```sql +CREATE MATERIALIZED VIEW mv_monthly_sales AS +SELECT + DATE_TRUNC('month', order_date) AS month, + region, + SUM(amount) AS total_sales +FROM orders +GROUP BY 1, 2; +``` + +**When to use materialized views:** +- Slow-running aggregations that are queried frequently. +- Dashboard/report queries with known filter patterns. +- Pre-joined dimension+fact denormalizations. +- Data that changes infrequently (or you can tolerate stale data). + +**When NOT to use materialized views:** +- Highly volatile data (refresh cost exceeds query savings). +- Ad-hoc query workloads with unpredictable filter patterns. +- Tables under 50M rows (incremental query is often fast enough). +- When the view depends on tables with complex streaming updates. + +--- + +### 3.6 Sorting within Analytical Engines + +| Engine | Default Physical Sort | Notes | +|---|---|---| +| PostgreSQL | Heap-organized (CTID = physical order of insertion) | CLUSTER reorders once | +| DuckDB | Row-group columnar layout | `ORDER BY` in `COPY` or `CREATE TABLE AS` optimizes scan | +| ClickHouse | ORDER BY columns specified in table engine | Primary key determines sort | +| BigQuery | Capacitor columnar format, no physical sort guarantee | `CLUSTER BY` controls block layout | +| Snowflake | Micro-partition metadata tracks column min/max | Automatic via clustering | +| Redshift | SORTKEY determines block order | Compound vs interleaved | + +--- + +## 4. Data Modeling for Analytics + +### 4.1 Star Schema + +**Structure:** One central *fact table* surrounded by *dimension tables*. + +``` + +--------------+ + | Date (Dim) | + | date_key |<-------+ + +--------------+ | + | ++----------------+ +------------------+ +| Product (Dim) | | Sales (Fact) | +| product_key |<-----------------| product_key (FK) | +| product_name | | customer_key (FK)| +| category | | date_key (FK) | ++----------------+ | store_key (FK) | + | quantity | ++----------------+ | unit_price | +| Store (Dim) | | discount | +| store_key |<-----------------+------------------+ +| store_name | +| region | +------------------+ ++----------------+ | Customer (Dim) | + | customer_key (FK)| + +------------------+ +``` + +**Fact table design rules:** +- Grain: explicitly define what one row represents (e.g., one row per product per store per day). +- Foreign keys: reference dimension surrogate keys, not natural keys. +- Measures: additive (quantity, amount), semi-additive (balance), non-additive (ratio). +- Avoid storing NULLs in numeric measure columns — use 0 if meaningful. + +**Dimension table design rules:** +- Surrogate key (auto-increment or UUID) as primary key. +- Natural key stored as a separate attribute (business key). +- Split hierarchical attributes into role-playing dimensions where appropriate. +- Include descriptive text, codes, and categorization columns. + +--- + +### 4.2 Snowflake Schema + +**Structure:** Dimensions are normalized into multiple related tables. + +``` ++----------------+ +------------------+ +------------------+ +| Category | | Subcategory | | Product | +| category_id |<---| category_id (FK) |<---| subcategory_id | +| category_name | | subcategory_id | | product_key | ++----------------+ | subcategory_name | | product_name | + +------------------+ +------------------+ +``` + +**Star vs Snowflake decision:** + +| Factor | Star | Snowflake | +|---|---|---| +| Query simplicity | Simple (fewer joins) | Complex (more joins) | +| Storage | Redundant (denormalized) space | Normalized (less space) | +| ETL complexity | Simple (single table) | Complex (multiple related tables) | +| BI tool performance | Fast (fewer joins) | Slower (more joins) | +| Maintenance | Update all rows in denormalized table | Update one row in normalized table | +| Dimensional hierarchy | Flattened into one table | Separate tables per level | + +**Rule of thumb:** Start with star schema. Only normalize to snowflake when: +- Dimension has more than 5 hierarchical levels. +- Dimension rows are shared across multiple fact tables. +- Storage cost savings from normalization are significant. +- The ETL/maintenance overhead of snowflake is acceptable. + +--- + +### 4.3 Dimensional Modeling (Kimball) + +Kimball's four-step dimensional design process: + +1. **Select the business process** (e.g., sales, inventory, customer orders). +2. **Declare the grain** (e.g., one row per product per store per day). +3. **Identify the dimensions** (who, what, where, when, why). +4. **Identify the facts** (measures: how many, how much). + +**Conformed dimensions:** Dimensions that are shared across multiple fact tables +with the same keys, attributes, and meanings. This enables cross-process analysis +(e.g., compare sales to inventory by product). + +**Degenerate dimensions:** Dimension attributes stored in the fact table because +they have no separate dimension table (e.g., order number for a line-item fact). + +**Junk dimensions:** A single dimension table combining multiple low-cardinality +flags and indicators (e.g., `is_new_customer`, `is_express_shipping`, `is_promo`) +into one table to keep the fact table lean. + +**Fact table types:** + +| Type | Description | Example | +|---|---|---| +| **Transactional** | One row per event | Line-item sales, web clicks | +| **Periodic Snapshot** | One row per period | Daily account balance, monthly inventory | +| **Accumulating Snapshot** | One row per process lifecycle | Order fulfillment (order -> ship -> deliver) | + +--- + +### 4.4 Slowly Changing Dimensions (SCD) + +**SCD Type 0 — Retain original:** +- Dimension attributes never change once written. +- Use for immutable reference data (date of birth, timestamp). + +**SCD Type 1 — Overwrite:** +- No history; current value overwrites the old value. +```sql +UPDATE customer_dim +SET email = 'new@email.com' +WHERE customer_id = 123; +``` + +**SCD Type 2 — Add new row (most common for analytics):** +- Each change creates a new row with effective dates. +```sql +UPDATE customer_dim +SET end_date = CURRENT_DATE - 1 +WHERE customer_id = 123 AND end_date IS NULL; -- expire old + +INSERT INTO customer_dim (customer_id, name, email, start_date, end_date) +VALUES (123, 'John', 'new@email.com', CURRENT_DATE, NULL); -- add new +``` + +*Additional columns for Type 2:* +- `start_date`, `end_date` — effective date range +- `is_current` — boolean flag for active row +- `version_number` — incrementing version + +**SCD Type 3 — Add new column:** +- Track limited history by adding a "previous value" column. +```sql +ALTER TABLE customer_dim ADD COLUMN previous_email VARCHAR(255); +UPDATE customer_dim +SET previous_email = email, email = 'new@email.com' +WHERE customer_id = 123; +``` + +**SCD Type 4 — Mini-dimension:** +- Rapidly changing attributes are split into a separate dimension table. +- The main dimension stores the current value; the mini-dimension tracks changes. +- Useful when attributes change faster than the dimension can accommodate Type 2. + +**SCD Type 6 (Hybrid 1+2+3):** +- Combines Type 1 (current value), Type 2 (history via rows), and Type 3 (previous value column). +- Useful for "as-is" and "as-was" reporting in the same table. + +**Decision table:** + +| SCD Type | Use When | +|---|---| +| 0 | Attribute never changes (birth date, original SKU) | +| 1 | History not needed, audit not required (email, phone) | +| 2 | Full history required (address, department) | +| 3 | Quick access to previous value only (territory assignment) | +| 4 | Attributes change very frequently (credit score, loyalty tier) | +| 6 | Need both current and historical in same query (compliance) | + +--- + +### 4.5 Fact Table Design — Advanced + +**Additive vs Semi-Additive vs Non-Additive:** + +| Measure Type | Add Across All Dims | Add Across Time | Example | +|---|---|---|---| +| Additive | Yes | Yes | Sales amount, quantity | +| Semi-additive | Yes | No | Account balance, inventory level | +| Non-additive | No | No | Ratio, percentage, unit price | + +*Semi-additive handling:* Use `SUM()` across other dimensions, but `AVG()` or +`LAST_VALUE()` across time. + +**Null handling in facts:** +- Numeric facts: use 0 for additive nulls (quantity, amount). Use NULL for + non-applicable values (e.g., discount on non-promotional sale). +- Foreign keys: avoid NULLs — use a "Unknown" dimension row (key = -1). + +**Factless fact tables:** +- A fact table with only foreign keys and no measures. +- Records an event or relationship (e.g., product-to-campaign assignment, student attendance). + +**Transaction header + line-item fact modeling:** +- Grain = line item. +- Header-level attributes (order date, customer, store) are degenerate dimensions. +- Headers with multiple grains may split into separate fact tables. + +--- + +## 5. SQL Testing & Validation Patterns + +### 5.1 Data Quality Testing with SQL + +**Category: Uniqueness / Primary Key** +```sql +-- EXPECT: 0 rows (all IDs are unique) +SELECT id, COUNT(*) +FROM target_table +GROUP BY id +HAVING COUNT(*) > 1; +``` + +**Category: Not Null** +```sql +-- EXPECT: 0 rows (no nulls in required columns) +SELECT COUNT(*) AS null_count +FROM target_table +WHERE required_column IS NULL; +``` + +**Category: Referential Integrity** +```sql +-- EXPECT: 0 rows (all foreign keys exist in parent) +SELECT DISTINCT ft.fk_column +FROM fact_table ft +LEFT JOIN dim_table dt ON ft.fk_column = dt.pk +WHERE dt.pk IS NULL; +``` + +**Category: Accepted Values (enum/dimension)** +```sql +-- EXPECT: 0 rows (all values in allowed set) +SELECT DISTINCT status +FROM target_table +WHERE status NOT IN ('active', 'inactive', 'pending', 'cancelled'); +``` + +**Category: Freshness (data recency)** +```sql +-- EXPECT: max date within acceptable lag +SELECT MAX(loaded_at) AS last_load +FROM target_table; +-- Alert if last_load < CURRENT_TIMESTAMP - INTERVAL '24 hours' +``` + +**Category: Row Count Consistency** +```sql +-- EXPECT: row counts match (within tolerance) +SELECT 'source' AS source, COUNT(*) AS cnt FROM source_table +UNION ALL +SELECT 'target', COUNT(*) FROM target_table; +``` + +**Category: Distribution / Outlier Detection** +```sql +-- EXPECT: no rows outside 3 standard deviations +WITH stats AS ( + SELECT + AVG(amount) AS avg, + STDDEV(amount) AS std + FROM orders +) +SELECT * +FROM orders, stats +WHERE ABS(orders.amount - stats.avg) > 3 * stats.std; +``` + +**Category: Duplicate Detection (multi-column)** +```sql +-- EXPECT: 0 rows +SELECT natural_key_1, natural_key_2, COUNT(*) +FROM target_table +GROUP BY natural_key_1, natural_key_2 +HAVING COUNT(*) > 1; +``` + +--- + +### 5.2 dbt Test Patterns + +dbt provides four built-in generic tests: + +```yaml +# schema.yml +version: 2 +models: + - name: orders + columns: + - name: order_id + tests: + - unique + - not_null + - name: status + tests: + - accepted_values: + values: ['placed', 'shipped', 'completed', 'cancelled'] + - name: customer_id + tests: + - not_null + - relationships: + to: ref('customers') + field: customer_id +``` + +**Custom singular tests (dbt):** +```sql +-- tests/custom/positive_revenue.sql +-- EXPECT: 0 rows returned +SELECT order_id, revenue +FROM {{ ref('orders') }} +WHERE revenue < 0; +``` + +**Custom generic tests (dbt):** +```sql +-- tests/generic/test_is_positive.sql +{% test is_positive(model, column_name) %} +SELECT * +FROM {{ model }} +WHERE {{ column_name }} < 0 +{% endtest %} +``` + +--- + +### 5.3 Testing Pipeline Patterns + +**Unit testing (transformation logic):** + +```sql +-- Given: a known input +WITH test_data AS ( + SELECT 'US' AS country, 100 AS amount, DATE '2024-01-15' AS order_date + UNION ALL + SELECT 'UK', 200, DATE '2024-02-20' +) +-- When: apply transformation +, transformed AS ( + SELECT + country, + amount, + CASE WHEN country = 'US' THEN amount * 1.0 ELSE amount * 1.2 END AS amount_usd + FROM test_data +) +-- Then: assert expected output +SELECT * +FROM transformed +WHERE (country = 'US' AND amount_usd != 100) + OR (country = 'UK' AND amount_usd != 240); +``` + +**Regression testing (compare output across versions):** + +- Store known-good output as a reference table or CSV. +- Run the new version of the query. +- EXPECT: row-perfect match (or within delta for floating-point). + +**Schema drift detection:** +```sql +-- Compare column schemas between source and target +SELECT column_name, data_type +FROM information_schema.columns +WHERE table_name = 'source' +EXCEPT +SELECT column_name, data_type +FROM information_schema.columns +WHERE table_name = 'target'; +``` + +**Reconciliation (cross-system):** + +```sql +SELECT + COALESCE(a.order_id, b.order_id) AS order_id, + a.total AS source_total, + b.total AS target_total, + COALESCE(a.total, 0) - COALESCE(b.total, 0) AS diff +FROM source_system.orders a +FULL OUTER JOIN target_system.orders b + ON a.order_id = b.order_id +WHERE a.total IS DISTINCT FROM b.total; +``` + +--- + +## 6. Analytical SQL Engine Comparison + +### 6.1 Engine Overview + +| Feature | PostgreSQL | DuckDB | ClickHouse | BigQuery | Snowflake | Redshift | +|---|---|---|---|---|---|---| +| **Architecture** | Row-store, monolithic | Columnar, embedded | Columnar, MPP | Serverless, columnar | Virtual warehouses, columnar | Columnar, MPP | +| **Deployment** | Self-hosted / managed | Embedded / MotherDuck cloud | Self-hosted / ClickHouse Cloud | GCP only | AWS / Azure / GCP | AWS only | +| **SQL dialect** | SQL:2011 | SQL:2011 + extensions | Custom SQL (MySQL-like) | GoogleSQL | SnowflakeSQL | PostgreSQL-like | +| **ACID** | Full | Full | Per-table | Row-level (recent) | Snapshot isolation | Serial isolation | +| **Concurrency model** | Connection-based | Single-user (per process) | High-concurrency reads | Massive concurrency | Virtual warehouse scale | WLM queues | + +### 6.2 Performance Characteristics + +| Metric | PostgreSQL | DuckDB | ClickHouse | BigQuery | Snowflake | Redshift | +|---|---|---|---|---|---|---| +| **Scan speed (single node)** | ~50 MB/s | ~500 MB/s | ~2-5 GB/s | ~GB/s (distributed) | ~MB/s per node | ~GB/s per slice | +| **Aggregation throughput** | Moderate | Very high | Extremely high | Very high | High | High | +| **JOIN performance** | Excellent (indexed) | Good (hash join) | Good (need careful schema) | Excellent | Excellent | Good | +| **Sub-second queries** | Yes (small data) | Yes (in-memory) | Yes (columnar) | Yes (with cached) | Yes (with cached) | Yes (with SORTKEY) | +| **Full table scan** | Slow | Fast | Very fast | Fast | Moderate | Fast | +| **Concurrent queries** | Good (configurable) | Limited (single process) | Excellent | Excellent | Good (per warehouse) | Good (per WLM queue) | + +### 6.3 When Each Engine Makes Sense + +**PostgreSQL — The transactional foundation:** +- Source of record for OLTP systems. +- Small-to-medium analytical workloads (< 50 GB). +- When you need full ACID and complex joins. +- Data engineering: staging area, metadata store, Airflow backend. +- NOT for: multi-TB datasets, high-cardinality aggregations on billions of rows. + +**DuckDB — The embedded analyst:** +- Local data exploration on Parquet/CSV files. +- Single-machine analytical workloads (up to ~100 GB comfortably in memory). +- Data engineering: dbt development, local testing, transform-in-place. +- Embedded analytics (in-process OLAP). +- NOT for: multi-user production APIs, concurrent write workloads. + +**ClickHouse — The real-time powerhouse:** +- Real-time dashboards and observability. +- High-ingestion-rate event data (logs, metrics, clickstreams). +- Sub-second aggregations on billions of rows. +- Data engineering: time-series analytics, real-time monitoring, product analytics. +- NOT for: point-lookup queries, frequent small updates/deletes, complex joins. + +**BigQuery — The serverless warehouse:** +- When you don't want to manage infrastructure. +- Petabyte-scale analytics with auto-scaling. +- Integration with Google Cloud ecosystem (Dataflow, Looker, Vertex AI). +- Data engineering: ELT-heavy workflows, ad-hoc analysis at scale. +- NOT for: transactional workloads, predictable monthly spend (cost can be spiky). + +**Snowflake — The enterprise data cloud:** +- When multi-cloud or multi-region is required. +- Data sharing across organizations (Snowflake Marketplace). +- Separation of compute and storage with automatic scaling. +- Data engineering: production data warehouses, data sharing, BI backends. +- NOT for: real-time streaming (ingest latency is seconds), budget-constrained workloads. + +**Redshift — The AWS-native warehouse:** +- Heavily invested in the AWS ecosystem. +- Predictable performance for well-defined workloads. +- Integration with S3, Glue, Spectrum, QuickSight. +- Data engineering: large-scale batch processing on AWS, BI workloads. +- NOT for: ad-hoc multi-user queries without careful WLM tuning, multi-cloud. + +### 6.4 Key Feature Differences + +| Feature | PostgreSQL | DuckDB | ClickHouse | BigQuery | Snowflake | Redshift | +|---|---|---|---|---|---|---| +| **Materialized views** | Manual REFRESH | Not built-in (use dbt) | Materialized views | Auto-refresh | Auto-refresh, cost credits | Late-binding views | +| **MERGE support** | `INSERT ON CONFLICT` | `MERGE` (v1.3+) | `ALTER TABLE .. DELETE` + INSERT | `MERGE` | `MERGE` | `MERGE` (RA3+) | +| **External tables** | FDW (postgres_fdw) | `read_parquet`, `read_csv` | `CREATE TABLE .. ENGINE=Kafka/MySQL` | External tables | External tables | Spectrum | +| **Window functions** | Full support | Full support | Full support | Full support | Full support | Full support | +| **Recursive CTEs** | Yes | Yes | No (non-recursive only) | Yes | Yes | Yes | +| **PIVOT** | `crosstab()` extension | `PIVOT` | No (use `GROUP BY` + arrays) | `PIVOT` | `PIVOT` | No (use CASE) | +| **Semi-structured** | JSONB | JSON, Struct, Array | JSON, Array, Tuple, Nested | REPEATED, RECORD | VARIANT | SUPER (JSON-like) | +| **Time travel** | pg_rewind (limited) | Not built-in | Not built-in (use snapshot) | Query any point in 7 days | `AT (TIMESTAMP)` up to 90 days | `RESTORE TABLE` | +| **Cost model** | Licensing + hardware | Free / MotherDuck consumption | Open source / Cloud credits | Pay per byte scanned | Pay per compute credit | Pay per node-hour | + +### 6.5 Pricing and Cost Considerations + +| Engine | Cost Character | Best Cost Profile | Worst Cost Profile | +|---|---|---|---| +| PostgreSQL | Fixed (HW/license) | Predictable, moderate volume | Very large datasets (no auto-scale) | +| DuckDB | Free / MotherDuck usage | Sub-TB workloads, OLAP queries | Multi-user concurrent access | +| ClickHouse | HW/cloud credits | High-volume, high-throughput real-time | Small workloads (overhead of cluster) | +| BigQuery | Per-byte scanned | Ad-hoc, infrequent large queries | Repeated full scans of large tables | +| Snowflake | Per-credit (compute) | Variable workloads with auto-suspend | Always-on large warehouse | +| Redshift | Per-node-hour (fixed) | Steady-state batch workloads | Idle clusters (pay for what you allocate) | + +### 6.6 Engine Selection Matrix + +| Workload Profile | Recommended Engine | Runner-Up | +|---|---|---| +| Small team, local analysis | DuckDB | PostgreSQL | +| Cloud-native analytics, GCP shop | BigQuery | Snowflake | +| Enterprise data warehouse, multi-cloud | Snowflake | BigQuery | +| AWS ecosystem, steady workloads | Redshift | Snowflake | +| Real-time observability, logs | ClickHouse | BigQuery (streaming) | +| Embedded analytics (SaaS product) | DuckDB | ClickHouse | +| Transactional + reporting (single system) | PostgreSQL | -- | +| Petabyte-scale ad-hoc | BigQuery | Snowflake | +| Budget-constrained, large batch | ClickHouse (self-hosted) | DuckDB (MotherDuck) | + +--- + +## Appendix: Quick Reference SQL Snippets + +**Common analytical queries:** + +```sql +-- Top-N per group +SELECT * FROM ( + SELECT *, ROW_NUMBER() OVER (PARTITION BY category ORDER BY revenue DESC) AS rn + FROM sales +) WHERE rn <= 10; + +-- Running total +SELECT date, SUM(amount) OVER (ORDER BY date ROWS UNBOUNDED PRECEDING) AS running_total +FROM daily; + +-- Month-over-month change +WITH monthly AS ( + SELECT DATE_TRUNC('month', date) AS month, SUM(val) AS val + FROM data GROUP BY 1 +) +SELECT month, val, + LAG(val) OVER (ORDER BY month) AS prev, + (val - LAG(val) OVER (ORDER BY month)) / NULLIF(LAG(val) OVER (ORDER BY month), 0) * 100 AS mom_pct +FROM monthly; + +-- Rolling 7-day average +SELECT date, AVG(amount) OVER (ORDER BY date ROWS BETWEEN 6 PRECEDING AND CURRENT ROW) AS ma_7 +FROM daily_revenue; + +-- Deduplication (keep latest) +WITH deduped AS ( + SELECT *, ROW_NUMBER() OVER (PARTITION BY business_key ORDER BY updated_at DESC) AS rn + FROM raw +) +SELECT * FROM deduped WHERE rn = 1; + +-- Fill forward (last non-null value) +SELECT + date, + amount, + LAST_VALUE(amount IGNORE NULLS) OVER (ORDER BY date ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS filled +FROM sparse_data; +``` + +--- + +*End of reference document.* diff --git a/data-engineering/references/time-series-databases.md b/data-engineering/references/time-series-databases.md new file mode 100644 index 0000000..c6c9641 --- /dev/null +++ b/data-engineering/references/time-series-databases.md @@ -0,0 +1,986 @@ +# Time-Series Database Patterns for Data Engineering + +> A thorough reference on time-series databases with a focus on InfluxDB, written for the data engineering methodology skill. + +--- + +## Table of Contents + +1. [When to Choose a Time-Series DB Over a Relational DB](#1-when-to-choose-a-time-series-db-over-a-relational-db) +2. [InfluxDB Data Model](#2-influxdb-data-model) +3. [Schema Design for Time-Series](#3-schema-design-for-time-series) +4. [Flux Query Language Fundamentals](#4-flux-query-language-fundamentals) +5. [InfluxQL vs Flux vs SQL](#5-influxql-vs-flux-vs-sql) +6. [Downsampling and Continuous Queries / Tasks](#6-downsampling-and-continuous-queries--tasks) +7. [Data Lifecycle Management](#7-data-lifecycle-management) +8. [Ingest Patterns](#8-ingest-patterns) +9. [Integration with Data Engineering Pipelines](#9-integration-with-data-engineering-pipelines) +10. [Comparison: InfluxDB vs TimescaleDB vs Prometheus vs QuestDB](#10-comparison-influxdb-vs-timescaledb-vs-prometheus-vs-questdb) + +--- + +## 1. When to Choose a Time-Series DB Over a Relational DB + +### Time-Series Data Characteristics + +Not all timestamped data is time-series data. True time-series data has these properties: + +- **Append-heavy**: New data points arrive continuously; updates/upserts are rare. +- **Time-ordered**: Write order closely follows the timestamp order (recent data is hot). +- **Time-centric queries**: Analysts almost always filter, aggregate, and slice by time ranges. +- **Downsampling pattern**: Old data is routinely summarized into lower-resolution rollups. +- **Immutable by nature**: Historical records are almost never modified. + +### Decision Matrix + +| Factor | Choose Relational (PostgreSQL/MySQL) | Choose Time-Series DB (InfluxDB/TimescaleDB) | +|---|---|---| +| Write pattern | Mixed read-write, UPDATE-heavy | Append-only streaming writes | +| Data volume | Millions of rows | Billions to trillions of data points | +| Query pattern | OLTP: single-row lookups, JOINs, transactions | Time-bucket aggregates, range scans | +| Retention | Keep everything indefinitely, DELETE rare | Auto-expire raw data after N days | +| Schema | Frequently evolving, normalized | Stable, denormalized per measurement | +| Consistency | ACID strongly required | Eventual or tunable consistency acceptable | +| Cardinality | Low (user IDs, order IDs) | Can range from low to very high (device IDs, container IDs) | + +### When to Use a General-Purpose TSDB (like InfluxDB) + +- **Metrics infrastructure**: Server/container CPU, memory, disk, network (DevOps/SRE). +- **IoT/IIoT sensor data**: Temperature, pressure, vibration readings at high frequency. +- **Application telemetry**: Request latencies, error rates, user counts. +- **Industrial/historian workloads**: Replacing PI System, OSIsoft, or other historians. +- **Financial tick data**: Stock trades, order book snapshots (though QuestDB may be better here). + +### When to Stick with a Relational DB + Time-Series Extension + +- You already have a PostgreSQL ecosystem and want to avoid another infrastructure stack. +- Your time-series data has complex relational joins (e.g., sensor metadata normalized across 5 tables). +- You need full SQL with window functions, CTEs, and transactional guarantees. +- *Solution: TimescaleDB hypertables on PostgreSQL.* + +### When to Use an Analytical Columnar DB (ClickHouse) Instead + +- You run large ad-hoc analytical queries on time-series data (OLAP-style). +- You need sub-second aggregation over billions of rows across many dimensions. +- Your query patterns are more "GROUP BY time, dimension" than "latest value per series." + +--- + +## 2. InfluxDB Data Model + +InfluxDB v1 and v2 share a four-component data model. InfluxDB 3 (the current recommended version) retains compatibility with this model but adds SQL-on-Parquet support. + +### The Four Components + +``` +Measurement -> Logical table name (e.g., "cpu", "sensor_temp") +Tag set -> Indexed metadata key=value pairs (e.g., host=server01, region=us-east) +Field set -> Actual data values (e.g., temperature=98.6, cpu_usage=0.85) +Timestamp -> Nanosecond-precision Unix timestamp +``` + +### Line Protocol Format + +This is the canonical way to write data into InfluxDB (all versions): + +``` +[,=[,=...]] =[,=...] [] +``` + +**Concrete example:** + +``` +sensor_temp,host=server01,region=us-east temperature=98.6,humidity=0.45 1717610400000000000 +``` + +Or in InfluxDB 3 SQL terms, the CREATE TABLE equivalent would be: + +```sql +-- InfluxDB 3 uses SQL for schema management +CREATE TABLE sensor_temp ( + time TIMESTAMP, + host STRING, -- tag + region STRING, -- tag + temperature DOUBLE, -- field + humidity DOUBLE -- field +); +``` + +### How InfluxDB Stores Data + +- **Tags are indexed** — they form the *series key*. Every unique combination of measurement + tag set defines a **time series**. +- **Fields are not indexed** — querying by field values requires a full scan. +- **Timestamp** is the primary sort key within each series. +- In InfluxDB 3 (IOx engine), data is stored in **Parquet files** on object storage, with an in-memory catalog for indexing. +- In InfluxDB 1.x/2.x (TSM engine), data is stored in **TSM (Time-Structured Merge Tree)** files with an in-memory index. + +### Series Cardinality + +``` +series cardinality = number of unique (measurement, tag set) combinations +``` + +**Example:** If you have measurement `cpu` with tags `host` (1000 values) and `region` (5 values), you have up to 5,000 series. + +**High cardinality is the #1 performance killer in InfluxDB v1/v2 (TSM engine).** InfluxDB 3 (IOx) largely solves this by using a columnar storage engine, but high cardinality still affects memory for the catalog. + +**Values that cause high cardinality and should NEVER be tags:** +- Request IDs +- Session IDs +- User IDs (if unique per user and high volume) +- Timestamps as strings +- Email addresses +- Any value with millions of unique values + +--- + +## 3. Schema Design for Time-Series + +### Measurement Design + +**Rule of thumb:** One measurement per logical data source type. + +``` +GOOD: measurement="cpu" fields={usage_user, usage_system, usage_idle} +GOOD: measurement="memory" fields={used_bytes, free_bytes, total_bytes} +BAD: measurement="metrics" fields={cpu_user, cpu_system, mem_used, mem_free, disk_read, disk_write} +``` + +### Tag vs Field Decision Guide + +| Put in TAGS if... | Put in FIELDS if... | +|---|---| +| Low to moderate cardinality (< 100K unique values) | The actual measured numeric value | +| You filter or GROUP BY this attribute | High cardinality (request IDs, UUIDs) | +| It's static/reusable metadata (host, region, data_center) | It's the payload/metric value itself | +| You need fast indexed lookups | It changes on every data point | + +### Tag Cardinality Management + +**For InfluxDB v1/v2 (TSM):** +- Keep total series cardinality under 10 million per node (hard limit ~10-20M). +- Keep per-measurement cardinality under 1 million for good performance. +- Use TSI (Time Series Index) for higher cardinality in v1.7+ but expect memory pressure. + +**For InfluxDB 3 (IOx/columnar):** +- Catalog memory scales with number of unique tag values, not combinations. +- Can handle 100M+ unique series; watch catalog memory (~2-4 GB per 100M series). + +**Anti-patterns to avoid:** +- Putting timestamps, dates, or high-entropy strings as tags. +- Using tags for values that change on every write. +- Over-tagging (10+ tags per measurement when 3-4 would suffice). +- Tag values that grow unboundedly (e.g., container IDs in Kubernetes). + +### Retention Policies (v1) vs Buckets (v2) + +**InfluxDB v1 — Retention Policies (RPs):** +```sql +-- Create a retention policy: keep data for 30 days, 1 replica +CREATE RETENTION POLICY "thirty_days" ON "mydb" DURATION 30d REPLICATION 1 DEFAULT; +``` + +**InfluxDB v2 — Buckets:** +```bash +# A bucket combines a database + retention policy from v1 +influx bucket create --name "sensor_data_30d" --retention 30d +``` + +**InfluxDB 3 — Retention at Database Level:** +```bash +# InfluxDB 3 Core: retention set at database level +# Core OSS enforces a 72-hour default; Cloud Dedicated and Enterprise allow custom retention +influxdb3 create database iot_sensors_prod --retention-period 90d +``` + +**Best practices:** +- Use separate buckets for different retention durations. +- For long-term storage, downsample raw data into a separate measurement with longer retention. +- In InfluxDB 3, the retention period is set per database and defines how long data is kept before automatic deletion. + +--- + +## 4. Flux Query Language Fundamentals + +> **Note:** Flux was introduced with InfluxDB v2.x. InfluxDB 3 now recommends **SQL** as the primary query language. Flux is still supported in InfluxDB 2.x and for backward compatibility, but new development on InfluxDB 3 should favor SQL. This section is retained for teams maintaining v2.x workloads. + +### Basic Structure + +Flux is a **functional, piped-data language**. Every query is a chain of transformations with data flowing left-to-right through pipes (`|>`). + +``` +data_source + |> transformation_1() + |> transformation_2() + |> transformation_3() +``` + +### Core Functions + +```flux +// 1. Define the data source and time range +from(bucket: "sensor_data") + |> range(start: -1h) // last hour of data + |> filter(fn: (r) => r._measurement == "cpu") + |> filter(fn: (r) => r._field == "usage_user") + |> filter(fn: (r) => r.host == "server01") + |> yield(name: "cpu_usage") +``` + +### Common Flux Patterns + +**Aggregation with windowing (downsampling):** +```flux +from(bucket: "sensor_data") + |> range(start: -7d) + |> filter(fn: (r) => r._measurement == "sensor_temp") + |> aggregateWindow(every: 1h, fn: mean) + |> yield(name: "hourly_mean") +``` + +**Multiple aggregations in one query:** +```flux +from(bucket: "sensor_data") + |> range(start: -24h) + |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_user") + |> aggregateWindow(every: 15m, fn: mean) + |> duplicate(column: "_stop", as: "_time") + |> drop(columns: ["_start", "_stop"]) + |> set(key: "_field", value: "usage_user_mean") + |> to(bucket: "downsampled_cpu") +``` + +**Pivoting to wide format (useful for Grafana):** +```flux +from(bucket: "sensor_data") + |> range(start: -1h) + |> filter(fn: (r) => r._measurement == "cpu") + |> pivot(rowKey: ["_time"], columnKey: ["_field"], valueColumn: "_value") + |> yield(name: "wide") +``` + +**Joining data streams:** +```flux +cpu = from(bucket: "sensor_data") + |> range(start: -1h) + |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_user") + +mem = from(bucket: "sensor_data") + |> range(start: -1h) + |> filter(fn: (r) => r._measurement == "mem" and r._field == "used_percent") + +join(tables: {cpu: cpu, mem: mem}, on: ["_time", "host"]) + |> yield(name: "joined") +``` + +### Flux Task (recurring script) + +```flux +// Run every hour +option task = { + name: "downsample_cpu_hourly", + every: 1h, + offset: 5m +} + +from(bucket: "raw_sensor_data") + |> range(start: -2h) + |> filter(fn: (r) => r._measurement == "cpu") + |> aggregateWindow(every: 1h, fn: mean) + |> to(bucket: "downsampled_cpu") +``` + +--- + +## 5. InfluxQL vs Flux vs SQL + +### Overview + +| Feature | InfluxQL | Flux | SQL (InfluxDB 3) | +|---|---|---|---| +| Era | InfluxDB 1.x | InfluxDB 2.x | InfluxDB 3 (current) | +| Style | SQL-like | Functional / piped | Standard SQL | +| Complexity | Low | Medium-High | Low | +| Learning curve | Easy (if you know SQL) | Steep | Easy (if you know SQL) | +| Multi-bucket queries | No | Yes | Yes | +| Joins | Limited (subqueries only) | Native | Full SQL JOINs | +| Scripting | No | Yes (variables, conditionals, functions) | Via SQL functions | +| Window functions | Limited | Native | Yes | +| Performance | Good | Medium (interpreted) | Best (compiled) | +| Status in InfluxDB 3 | Read-only compatibility | Supported for compatibility | **Recommended** | + +### When to Use Which + +``` +Use SQL (InfluxDB 3): Default for ALL new projects on InfluxDB 3. +Use Flux (v2.x only): Existing v2.x deployments, complex transformation pipelines. +Use InfluxQL (v1.x): Existing v1.x deployments, minimal migration path. +``` + +### InfluxQL vs Flux: Equivalent Queries + +**InfluxQL:** +```sql +SELECT mean("usage_user") +FROM "cpu" +WHERE time > now() - 1h +AND "host" = 'server01' +GROUP BY time(15m) +``` + +**Flux:** +```flux +from(bucket: "mydb/autogen") + |> range(start: -1h) + |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_user" and r.host == "server01") + |> aggregateWindow(every: 15m, fn: mean) +``` + +**SQL (InfluxDB 3):** +```sql +SELECT DATE_BIN(INTERVAL '15 minutes', time) AS bucket, + AVG(usage_user) AS avg_usage +FROM cpu +WHERE time > now() - INTERVAL '1 hour' + AND host = 'server01' +GROUP BY bucket +ORDER BY bucket; +``` + +### Key Migration Notes + +- InfluxQL `GROUP BY time(interval)` → Flux `aggregateWindow(every: interval, fn: ...)` +- InfluxQL `INTO` (downsample+write) → Flux `... |> to(bucket: "...")` +- Continuous Queries (InfluxQL) → Tasks (Flux) +- InfluxDB 3: Rewrite InfluxQL CQs as SQL scheduled queries or use external orchestrators. + +--- + +## 6. Downsampling and Continuous Queries / Tasks + +### The Downsampling Pattern + +Downsampling is the most critical data engineering pattern for time-series: + +``` +RAW DATA (1-second resolution, keep 30 days) + | + v [downsample every hour] +HOURLY ROLLUPS (1-minute aggregates, keep 1 year) + | + v [downsample every day] +DAILY ROLLUPS (1-hour aggregates, keep 5 years) +``` + +Each tier provides exponentially smaller storage footprint while preserving analytical value. + +### InfluxDB 1.x: Continuous Queries (CQs) + +```sql +CREATE CONTINUOUS QUERY "cq_cpu_hourly" ON "mydb" +BEGIN + SELECT mean("usage_user") AS "mean_usage" + INTO "mydb"."downsampled"."cpu_hourly" + FROM "cpu" + GROUP BY time(1h), "host" +END; +``` + +CQs run automatically at the end of each time window. They are simple but limited: only one aggregation function per CQ, no chaining. + +### InfluxDB 2.x: Tasks (Flux-based) + +```flux +// downsample_cpu_hourly +option task = { + name: "downsample_cpu_hourly", + every: 1h, + offset: 10m +} + +from(bucket: "raw_data") + |> range(start: -task.every) + |> filter(fn: (r) => r._measurement == "cpu") + |> aggregateWindow(every: 1h, fn: mean) + |> set(key: "_measurement", value: "cpu_hourly") + |> to(bucket: "downsampled_data") +``` + +**Chained (hierarchical) downsampling with tasks:** +``` +Task 1: raw_1s -> hourly (runs every hour) +Task 2: hourly -> daily (runs every day, queries the hourly bucket) +Task 3: daily -> monthly (runs monthly, queries the daily bucket) +``` + +### InfluxDB 3: Scheduled Queries / External Orchestration + +InfluxDB 3 Core does not have built-in continuous aggregates. Recommended approaches: + +1. **External scheduler** (Airflow, cron, Prefect): + ```sql + -- Run hourly via Airflow + INSERT INTO cpu_hourly + SELECT DATE_BIN(INTERVAL '1 hour', time) AS bucket, + host, + AVG(usage_user) AS avg_usage, + MIN(usage_user) AS min_usage, + MAX(usage_user) AS max_usage, + COUNT(*) AS sample_count + FROM cpu + WHERE time > now() - INTERVAL '2 hours' + GROUP BY bucket, host; + ``` + +2. **InfluxDB 3 processing engine plugins** for real-time transformations on write. + +3. **Write-time processing** via Telegraf aggregator plugins: + ```toml + [[processors.aggregate]] + period = "60s" + + [[processors.aggregate.config]] + measurement = "cpu" + columns = ["usage_user", "usage_system"] + functions = ["mean", "max", "min"] + ``` + +### Downsampling Best Practices + +- Store raw data at full resolution for the shortest practical window. +- Always include `COUNT(*)` in aggregates to track sample density. +- Use hierarchical aggregation (aggregate hourly data into daily, not raw into daily). +- Align your downsampling schedule with your retention policies. +- Consider **two-phase downsampling**: real-time via Telegraf aggregators for the first 1-5 minutes, then batch tasks for correction/backfill. + +--- + +## 7. Data Lifecycle Management + +### Storage Engine Architecture + +**InfluxDB 1.x / 2.x (TSM Engine):** +- Data organized into **shards** by time range (typically 7 days). +- Each shard is a set of TSM files + Write-Ahead Log (WAL). +- **Compaction** merges smaller TSM files into larger ones, removing deleted/overwritten data. +- **Shard management**: Default 7-day shard duration; configurable. +- Memory index (in-memory) maps series keys to TSM file locations. + +**InfluxDB 3 (IOx Engine):** +- Data stored as **Parquet files** in object storage (S3, local FS). +- Catalog (SQLite or PostgreSQL) tracks table/column metadata. +- **Compaction** merges Parquet files for read efficiency. +- No shards per se — data is partitioned by time but managed at the file level. + +### Retention Lifecycle Strategy + +``` +Example: 3-tier retention for IoT sensor data + +Tier 1: Raw (1-second resolution) -> 30 days -> bucket "raw_30d" +Tier 2: Hourly aggregates -> 1 year -> bucket "hourly_1y" +Tier 3: Daily aggregates -> 5 years -> bucket "daily_5y" +``` + +**InfluxDB v2/v3 implementation:** +1. Create three buckets with different retention periods. +2. Run downsampling tasks from raw -> hourly -> daily. +3. InfluxDB automatically deletes data older than each bucket's retention period. + +### Compaction + +**TSM compaction stages:** +- **Level 1** (snapshot): WAL -> TSM file (when WAL reaches threshold). +- **Level 2** (merge): 2-4 small TSM files -> 1 larger TSM file. +- **Level 3** (full): Multiple TSM files -> 1 optimized TSM file (deduplicates, removes tombstones). +- Compaction runs automatically; tune `cache-snapshot-write-cold-duration` and `compact-full-write-cold-duration` for write-heavy workloads. + +**IOx/Parquet compaction:** +- Merges small Parquet files (< 100 MB) into larger ones (~100-500 MB). +- Runs automatically but can be triggered manually via API. +- Compaction also applies retention deletion. + +### Shard Management (InfluxDB v1/v2) + +``` +Command: Effect: +ALTER RETENTION Change shard duration (default 7d) + POLICY ... DURATION +DROP SHARD Force-delete a specific shard and all its data +SHOW SHARDS List all shards with durations, sizes, status +influx_inspect Low-level TSM inspection and recovery tools +``` + +**Choosing shard duration:** +- Short shard duration (1-7d): More granular retention, easier to drop old data, more overhead. +- Long shard duration (1-4w): Less overhead, faster range queries, slower retention enforcement. +- Rule: shard duration should be ≤ 1/2 of your retention period for efficient expiry. + +### Cold / Tiered Storage + +- **InfluxDB Cloud Serverless**: Automatically tiers data to object storage. +- **InfluxDB Cloud Dedicated**: Configurable cold storage with Parquet. +- **TimescaleDB**: Native tiering to S3 via `tiering` policies. +- **InfluxDB OSS**: No built-in tiering; manage via external scripts or data migration. + +--- + +## 8. Ingest Patterns + +### Line Protocol + +The core ingestion format for all InfluxDB versions. + +**Format:** +``` +measurement,tag1=val1,tag2=val2 field1=val1,field2=val2 timestamp +``` + +**Data types:** +- Tags: strings only (no quoting needed if no special chars). +- Fields: floats (default), integers (trailing `i`), strings (`"quoted"`), booleans (`t`/`f`/`true`/`false`). +- Timestamp: nanosecond epoch (default); configurable precision (s, ms, us, ns). + +**Examples:** +```ini +# Float fields (default) +weather,location=us-midwest temperature=82.0 1465839830100400200 + +# Integer field (trailing i) +weather,location=us-midwest wind_speed=15i 1465839830100400200 + +# String field (double-quoted) +weather,location=us-midwest conditions="partly cloudy" 1465839830100400200 + +# Boolean field +weather,location=us-midwest is_raining=t 1465839830100400200 + +# Multiple fields +weather,location=us-midwest temperature=82.0,humidity=71.2 1465839830100400200 +``` + +**Write via HTTP API:** +```bash +curl -X POST \ + "http://localhost:8086/write?db=mydb&precision=s" \ + --data-raw "weather,location=us-midwest temperature=82.0 1465839830" +``` + +### Batch vs Streaming Writes + +| Factor | Batch | Streaming | +|---|---|---| +| Frequency | Every N seconds or N points | Every point as it arrives | +| Overhead | Low (HTTP overhead amortized) | High (per-request overhead) | +| Throughput | High (10K-100K points/s per node) | Low (1K-10K points/s per node) | +| Latency | Seconds to minutes | Sub-second | +| Use case | Backfill, batch ETL | Real-time monitoring, Telegraf | + +**Best practice:** Always batch writes — send 1,000-10,000 points per HTTP request. Never send single points. + +### Telegraf Agent + +Telegraf is InfluxData's plugin-driven collection agent. + +**Architecture:** +``` +Input Plugins -> Aggregator/Processor Plugins -> Output Plugins + | | | + (CPU, disk, (aggregate, (InfluxDB, + MQTT, Kafka, transform, Prometheus, + Prometheus, enrich, file, Kafka, + syslog, SNMP, filter) CloudWatch) + Docker, k8s...) +``` + +**Example Telegraf config (`telegraf.conf`):** +```toml +# Global settings +[agent] + interval = "10s" + flush_interval = "10s" + metric_batch_size = 5000 + +# Input: CPU metrics +[[inputs.cpu]] + percpu = true + totalcpu = true + +# Input: MQTT subscriber +[[inputs.mqtt_consumer]] + servers = ["tcp://broker.local:1883"] + topics = ["sensors/#"] + data_format = "json" + json_time_key = "timestamp" + json_time_format = "unix_ms" + tag_keys = ["device_id", "location"] + +# Processor: apply transformation +[[processors.enum]] + [[processors.enum.mapping]] + tag = "status" + value_mappings = {online = 1, offline = 0} + +# Aggregator: downsample in real-time +[[processors.aggregate]] + period = "60s" + [[processors.aggregate.config]] + measurement = "cpu" + columns = ["usage_idle", "usage_user"] + functions = ["mean", "min", "max"] + +# Output: InfluxDB v2 +[[outputs.influxdb_v2]] + urls = ["http://localhost:8086"] + token = "${INFLUX_TOKEN}" + organization = "myorg" + bucket = "sensor_data" + +# Output: backup to file (for audit trail) +[[outputs.file]] + files = ["/var/log/telegraf_audit.log"] + data_format = "json" +``` + +**Telegraf best practices:** +- Use `metric_batch_size` (5,000-10,000) and `flush_interval` (5-10s) for efficient batching. +- Tag inputs with consistent metadata (data center, region, environment). +- Use processor and aggregator plugins rather than sending raw data and downsampling later. +- Set `fieldpass`/`fielddrop` on inputs to avoid collecting unused metrics. +- File-based logging output for auditability and data recovery. + +### Ingest Performance Tuning + +| Parameter | InfluxDB v1/v2 (TSM) | InfluxDB 3 (IOx) | +|---|---|---| +| Write batch size | 5,000-10,000 points | 10,000-50,000 points | +| HTTP workers | 8-16 | 16-64 | +| WAL flush interval | 1-10s (lower = safer) | N/A (no WAL) | +| Max points per second (single node) | 500K-1M | 3M-10M | +| Bottleneck | CPU (index updates) | Network I/O (Parquet writes) | + +### Kafka Integration + +**Telegraf as Kafka consumer:** +```toml +[[inputs.kafka_consumer]] + brokers = ["kafka:9092"] + topics = ["sensor_events"] + group_id = "telegraf_ingest" + data_format = "json" + consumer_fetch_min = "100KB" + consumer_fetch_default = "1MB" + max_undelivered_messages = 10000 +``` + +**Telegraf as Kafka producer:** +```toml +[[outputs.kafka]] + brokers = ["kafka:9092"] + topic = "aggregated_metrics" + data_format = "json" + compression_codec = 2 # snappy + required_acks = -1 # all +``` + +--- + +## 9. Integration with Data Engineering Pipelines + +### Apache Airflow DAG Example + +```python +from airflow import DAG +from airflow.providers.http.operators.http import HttpOperator +from airflow.operators.python import PythonOperator +from datetime import datetime, timedelta + +default_args = { + 'owner': 'data_engineering', + 'retries': 2, + 'retry_delay': timedelta(minutes=5), +} + +dag = DAG( + 'influxdb_downsample_hourly', + schedule='0 * * * *', # Every hour + start_date=datetime(2025, 1, 1), + catchup=False, +) + +def generate_line_protocol(**context): + """Generate downsampled data from raw and write hourly rollup.""" + import requests + import json + + # Query raw hourly aggregates via SQL (InfluxDB 3) + raw_data = requests.get( + f"{INFLUX_HOST}/api/v3/query", + params={"db": "sensor_raw"}, + headers={"Authorization": f"Bearer {INFLUX_TOKEN}"}, + data={ + "query": """ + SELECT DATE_BIN(INTERVAL '1 hour', time) AS bucket, + host, region, AVG(temperature) AS avg_temp, + MIN(temperature) AS min_temp, MAX(temperature) AS max_temp, + COUNT(*) AS sample_count + FROM sensor_readings + WHERE time >= now() - INTERVAL '2 hours' + AND time < now() - INTERVAL '1 hour' + GROUP BY bucket, host, region + """ + } + ) + + # Convert to line protocol and write to hourly bucket + points = [] + for row in raw_data.json(): + lp = ( + f"sensor_hourly,host={row['host']},region={row['region']} " + f"avg_temp={row['avg_temp']},min_temp={row['min_temp']}," + f"max_temp={row['max_temp']},sample_count={row['sample_count']}i " + f"{row['bucket']}" + ) + points.append(lp) + + # Batch write + requests.post( + f"{INFLUX_HOST}/api/v2/write", + params={"bucket": "sensor_hourly", "precision": "ms"}, + headers={"Authorization": f"Bearer {INFLUX_TOKEN}"}, + data="\n".join(points), + ) +``` + +### Apache Spark / PySpark Integration + +```python +from pyspark.sql import SparkSession + +spark = SparkSession.builder.appName("influxdb_etl").getOrCreate() + +# Read from InfluxDB v3 using JDBC (PostgreSQL-compatible driver) +df = spark.read \ + .format("jdbc") \ + .option("url", f"jdbc:postgresql://{INFLUX_HOST}:5432/mydb") \ + .option("query", """ + SELECT time, host, region, temperature + FROM sensor_readings + WHERE time >= now() - INTERVAL '24 hours' + """) \ + .option("user", INFLUX_USER) \ + .option("password", INFLUX_PASS) \ + .option("driver", "org.postgresql.Driver") \ + .load() + +# Transform +hourly_agg = df.groupBy( + F.window("time", "1 hour").alias("bucket"), + "host", "region" +).agg( + F.avg("temperature").alias("avg_temp"), + F.min("temperature").alias("min_temp"), + F.max("temperature").alias("max_temp"), + F.count("*").alias("sample_count") +) + +# Write back to InfluxDB +hourly_agg.write \ + .format("jdbc") \ + .option("url", f"jdbc:postgresql://{INFLUX_HOST}:5432/downsampled_db") \ + .option("dbtable", "sensor_hourly") \ + .option("user", INFLUX_USER) \ + .option("password", INFLUX_PASS) \ + .mode("append") \ + .save() +``` + +### Grafana Integration + +Grafana is the most common visualization layer for InfluxDB: + +``` +Grafana Data Source: + Type: InfluxDB + URL: http://influxdb:8086 + Database: mydb (v1) / Organization: myorg, Bucket: sensor_data (v2/v3) + Min time interval: 10s (matches collection interval) + Version: InfluxQL (v1) / Flux (v2) / SQL (v3) +``` + +### ETL Pipeline Patterns + +``` +Pattern 1: Telegraf -> InfluxDB -> Grafana + (simplest, real-time monitoring) + +Pattern 2: Sensors -> MQTT/Kafka -> Telegraf -> InfluxDB -> Grafana + (Kafka for buffering, backpressure handling) + +Pattern 3: Sensors -> Kafka -> Flink/Spark -> InfluxDB -> Airflow (downsample) -> InfluxDB + (heavy stream processing + scheduled downsampling) + +Pattern 4: Application -> InfluxDB -> Airflow/Spark -> Parquet -> S3/Data Lake + (time-series data lake architecture) +``` + +--- + +## 10. Comparison: InfluxDB vs TimescaleDB vs Prometheus vs QuestDB + +### At a Glance + +| Feature | InfluxDB 3 | TimescaleDB | Prometheus | QuestDB | +|---|---|---|---|---| +| **Type** | Purpose-built TSDB | PostgreSQL extension | Monitoring + TSDB | Purpose-built TSDB | +| **Engine** | Columnar (Parquet/IOx) | Hybrid row-columnar (Hypercore) | Custom TSDB engine | Columnar (custom) | +| **Query Language** | SQL + Flux + InfluxQL | Full SQL + PG extensions | PromQL | SQL | +| **Ingest Protocol** | Line Protocol, SQL, InfluxDB v2 API | PostgreSQL INSERT, COPY | Push via remote write | Line Protocol, InfluxDB, PostgreSQL wire | +| **Storage** | Parquet files on object storage | PostgreSQL on local/cloud disk | Local TSDB blocks | Memory-mapped files + disk | +| **Compression** | Good (Parquet) | Excellent (up to 95%) | Good (snappy) | Good | +| **High Availability** | Enterprise / Cloud (multi-node) | Streaming replication (PG built-in) | Sidecar (Thanos/Cortex) | Enterprise (pending) | +| **Clustering** | Enterprise only | PG-based | Built-in with Thanos | Enterprise | +| **Retention** | Per-database config | Per-hypertable via policies | Configurable (local) | Partition-based | +| **Continuous Aggregation** | External only (Core) | Built-in (continuous aggregates) | Recording rules | Materialized views | +| **Downsampling** | Tasks / external orchestration | Continuous aggregates | Recording rules + federation | SAMPLE BY + scheduled queries | +| **Real-time performance** | Excellent (3M+ points/s per node) | Very good (1M+ points/s) | Good (1M samples/s) | Excellent (5M+ points/s) | +| **SQL compatibility** | High (PostgreSQL-like) | **Complete (PostgreSQL)** | None (PromQL only) | High (custom SQL) | +| **Data lake export** | Native Parquet output | Via PG tools | Remote write / Thanos | Native Parquet output | +| **Best for...** | General TSDB, IoT, app metrics, edge | Teams already on PostgreSQL, need full SQL | Kubernetes monitoring, site reliability | **Lowest-latency**, financial tick data, HFT | + +### Detailed Comparison + +#### InfluxDB 3 +- **Strengths:** Mature ecosystem (Telegraf, 300+ plugins), multiple deployment options (edge to cloud), native line protocol (de facto standard), good for heterogeneous data sources. +- **Weaknesses:** Flux deprecation path creates migration friction; Core OSS has 72h retention limit; no built-in continuous aggregates in Core; clustering only in Enterprise. +- **Best fit:** General-purpose time-series, DevOps/SRE monitoring, IoT/IIoT, replacing legacy historians. + +#### TimescaleDB (Tiger Data) +- **Strengths:** Full PostgreSQL compatibility (all SQL, all PG extensions, all tools), continuous aggregates with incremental refresh, hierarchical aggregation, excellent compression (up to 95%), hypertables with automatic partitioning, mature replication/PITR/HA from PostgreSQL. +- **Weaknesses:** Slightly lower raw ingest throughput than InfluxDB 3 or QuestDB; requires PostgreSQL knowledge; not as lightweight for edge deployments; parent company rebranded to Tiger Data (some confusion). +- **Best fit:** Data teams already on PostgreSQL; workloads needing complex JOINs, transactions, or full SQL analytics; long-term historical storage. + +#### Prometheus +- **Strengths:** Cloud-native standard for Kubernetes monitoring, simple operational model (single binary), PromQL is excellent for alerting and service-level metrics, pull-based model works well for dynamic infra. +- **Weaknesses:** Not a general-purpose TSDB (no SQL, no complex aggregations), limited retention (default 15d), single-node, no native HA (requires Thanos/Cortex), poor for IoT or high-cardinality label sets. +- **Best fit:** Kubernetes and container monitoring, service-level dashboards, alerting (PagerDuty/AlertManager), infra metrics. + +#### QuestDB +- **Strengths:** **Highest raw ingest throughput** (claimed 5M+ points/s on single node), lowest query latency for time-bucket aggregations, designed for capital markets/finance, native InfluxDB line protocol and PostgreSQL wire protocol support, SQL-compatible, non-blocking ingestion (immutable append), parallelized and vectorized query execution. +- **Weaknesses:** Smaller ecosystem (fewer integrations, fewer client libraries), newer project (less mature), clustering is Enterprise-only, less tooling for alerting/monitoring out of the box. +- **Best fit:** Financial tick data, high-frequency trading, real-time dashboards demanding microsecond query latency, capital markets infrastructure. + +### Decision Flowchart + +``` +Q: What infrastructure are you already running? + | + +-- PostgreSQL everywhere? + | |--> TimescaleDB (stay in PG ecosystem) + | + +-- Kubernetes / containers? + | |--> Prometheus for infra monitoring + | +--> InfluxDB for app metrics and IoT + | + +-- Need lowest possible latency (< 1ms queries)? + | |--> QuestDB (financial, HFT) + | + +-- Heterogeneous environment, edge devices, IoT? + |--> InfluxDB (Telegraf ecosystem, multiple deployment options) + +Q: What query language do you need? + |--> Full SQL with JOINs, CTEs, window functions? -> TimescaleDB or InfluxDB 3 + |--> Time-series specific: PromQL? -> Prometheus + |--> Time-series specific: Flux? -> InfluxDB 2.x + |--> DevOps dashboards: Grafana + search? -> Any (Grafana supports all) + +Q: How much data are you ingesting? + |--> < 100K points/s -> Any + |--> 100K-1M points/s -> InfluxDB or TimescaleDB + |--> > 1M points/s -> QuestDB or InfluxDB 3 + |--> > 5M points/s -> QuestDB (benchmark leader) +``` + +### Version Guidance (InfluxDB Specific) + +| Deployment | Best For | Query Language | +|---|---|---| +| InfluxDB 3 Cloud Serverless | Rapid prototyping, variable workloads | **SQL** (recommended) | +| InfluxDB 3 Cloud Dedicated | Predictable production workloads | **SQL** (recommended) | +| InfluxDB 3 Enterprise | Self-managed HA production | **SQL** (recommended) | +| InfluxDB 3 Core | Edge, dev, prototypes | **SQL** (recommended) | +| InfluxDB OSS v2 | Existing v2.x deployments | Flux (migrate to SQL when moving to v3) | +| InfluxDB OSS v1 | Legacy, no migration budget yet | InfluxQL (migrate to SQL when possible) | + +--- + +## Appendix: Quick Reference + +### Line Protocol Cheatsheet + +``` +# Measurement name +weather +# Tags (comma-separated after measurement, space before fields) +weather,location=us-midwest,station=A +# Fields (comma-separated, space before timestamp) +weather temperature=82.0,humidity=71.2 +# Timestamp (nanoseconds since epoch; space after fields) +weather temperature=82.0 1465839830100400200 + +# Data type suffixes: +# Integer: value=42i +# Float: value=3.14 (default) +# String: value="hello world" +# Boolean: value=t (t/true/True/TRUE) +# value=f (f/false/False/FALSE) +# Timestamp: value=1465839830100400200 (nanosecond) + +# Escaping: +# Commas in tag values: tag=hello\,world +# Spaces in tag values: tag=hello\ world +# Equals in tag values: tag=hello\=world +# Double-quotes in string fields: field="say \"hello\"" +# Backslashes: tag=path\\to\\dir +``` + +### Key InfluxDB v2/v3 CLI Commands + +```bash +# InfluxDB v2 +influx bucket create --name my_bucket --retention 30d +influx task create --file downsample.flux +influx query --file my_query.flux + +# InfluxDB 3 Core +influxdb3 create database my_db --retention-period 90d +influxdb3 write --db my_db --file data.lp +influxdb3 query --db my_db "SELECT * FROM cpu WHERE time > now() - INTERVAL '1 hour'" +``` + +### Key TimescaleDB SQL Commands + +```sql +-- Create hypertable +SELECT create_hypertable('sensor_readings', 'time'); + +-- Add compression +ALTER TABLE sensor_readings SET ( + timescaledb.compress, + timescaledb.compress_segmentby = 'sensor_id' +); +SELECT add_compression_policy('sensor_readings', INTERVAL '7 days'); + +-- Continuous aggregate +CREATE MATERIALIZED VIEW sensor_hourly +WITH (timescaledb.continuous) AS +SELECT time_bucket('1 hour', time) AS bucket, + sensor_id, + AVG(value) AS avg_value +FROM sensor_readings +GROUP BY bucket, sensor_id; +``` + +--- + +*Generated: 2025-06-05 | InfluxDB 3 (IOx/columnar engine) is the current recommended version. Flux is supported for v2.x compatibility; new projects should prefer SQL.* diff --git a/data-engineering/references/vector-db-operations.md b/data-engineering/references/vector-db-operations.md new file mode 100644 index 0000000..032107e --- /dev/null +++ b/data-engineering/references/vector-db-operations.md @@ -0,0 +1,30 @@ +# Vector Database Operations + +## Collection Lifecycle + +| Phase | Activities | +|-------|-----------| +| Design | Schema definition, dimension selection, distance metric (cosine, euclidean, dot), index type selection | +| Create | Collection provisioning, index creation, partition configuration, alias setup | +| Ingest | Batch loading, streaming ingestion, data validation, consistency verification | +| Maintain | Index rebuilding, compaction, collection health monitoring, performance tuning | +| Migrate | Dimension changes, index type changes, cluster migration, data reindexing | +| Decommission | Data archival, collection backup, alias reassignment, collection drop | + +## Index Type Selection + +| Index Type | Best for | Tradeoffs | +|-----------|----------|-----------| +| IVF_FLAT | Balanced accuracy/speed | Higher memory, good recall | +| HNSW | High-recall, large datasets | Higher memory, slower build | +| IVF_SQ8 | Memory-efficient | Lower recall than IVF_FLAT | +| FLAT | Exact search, small datasets | O(n) search, exact recall | + +## Migration Patterns + +| Scenario | Approach | +|----------|----------| +| Dimension change | Create new collection with target dimension, dual-write during migration, batch reindex old data, swap alias | +| Index type change | Online index rebuild if supported, otherwise parallel collection with dual-write | +| Cluster migration | Backup → restore on target, validate row counts and sample queries, cut over via alias | +| Embedding model change | Full reindex: read source → generate new embeddings → write to new collection → verify → swap | diff --git a/frontend-engineering/README.md b/frontend-engineering/README.md new file mode 100644 index 0000000..653e1b1 --- /dev/null +++ b/frontend-engineering/README.md @@ -0,0 +1,26 @@ +# Frontend Engineering + +Frontend engineering methodology — component architecture, state management, API integration, responsive layout, client-side performance, and frontend testing patterns. Framework agnostic, focused on web frontend implementation. + +## Why Install This Skill + +Your agent applies proven component architecture, state management, and performance patterns instead of reinventing frontend structure each time. + +## What You Get + +| Directory | Purpose | +|-----------|---------| +| `SKILL.md` | Core methodology, trigger conditions, reference index | +| `references/` | Deep-dive reference files loaded on demand | + +## Triggers + +Building UI components, choosing state management approaches, integrating APIs, optimizing Core Web Vitals, or setting up responsive layouts. + +## Requirements + +Platform-agnostic. Framework-agnostic patterns applicable to React, Vue, Svelte, or vanilla JS. + +## Quick Start + +Load SKILL.md for the methodology overview and reference table, then load specific references as needed for the task at hand. diff --git a/frontend-engineering/SKILL.md b/frontend-engineering/SKILL.md new file mode 100644 index 0000000..bc6d285 --- /dev/null +++ b/frontend-engineering/SKILL.md @@ -0,0 +1,49 @@ +--- +name: frontend-engineering +description: Frontend engineering methodology — component architecture, state management, + API integration, responsive layout, client-side performance, and frontend testing + patterns. Framework agnostic, focused on web frontend implementation. +license: MIT +metadata: + tags: frontend, web, ui, components, state-management, performance, javascript, + typescript, responsive, testing + source_repo: https://github.com/magnus919/hermes-profiles +--- + +# Frontend Engineering Methodology + +Frontend engineering is the craft of building the user-facing layer of applications — components, state management, API integration, responsive layout, and client-side performance. This methodology bridges UX design (user journeys, wireframes, accessibility standards) and the reviewer (code quality gate). + +## The Frontend Engineer's Domain + +| You own | You don't own | +|---------|--------------| +| Component implementation — UI component composition, props/state interfaces, rendering patterns, lifecycle | User journeys, wireframes, accessibility standards, interaction design — that's the ux-designer | +| State management — client-side state architecture, data fetching patterns, caching, optimistic updates | API contract design — that's the technical-architect | +| API integration — frontend-to-backend data flow, auth flows (OAuth, JWT), real-time updates | Test strategy and automation — that's the QA-engineer | +| Responsive design implementation — layout systems, breakpoints, cross-device testing | Visual identity and brand guidelines — that's the brand-designer | +| Client-side performance — bundle optimization, lazy loading, Core Web Vitals, render optimization | Editorial content and copy — that's the writer | +| Frontend testing — component tests, integration tests, visual regression, accessibility tests | Code review and quality gates — that's the reviewer | +| Build tooling — bundler config, TypeScript config, linting, formatting, dev environment | CI/CD pipeline infrastructure — that's the platform-engineer | + +## Reference Files + +| Reference | When to load | +|-----------|-------------| +| `references/component-architecture.md` | Designing component trees — composition patterns, props/state interfaces, lifecycle, accessibility fundamentals | +| `references/state-management.md` | Choosing and implementing state management — client vs server state, data fetching, caching, optimistic updates | +| `references/api-integration.md` | Connecting frontend to backend — API client design, auth token flow, error handling in the UI, real-time subscriptions | +| `references/responsive-layout-testing.md` | Implementing responsive designs (layout system selection — Grid vs Flexbox vs Container Queries, breakpoint strategies, cross-device testing methodology) and testing frontend code (component testing with Testing Library, integration testing with Playwright/Cypress, visual regression, accessibility testing with axe-core and Lighthouse CI, test data management) | +| `references/performance.md` | Optimizing client-side performance — Core Web Vitals, bundle analysis, code splitting, render optimization | + +## Core Principles + +**Components are the unit of composition, not pages** — Design and build components as reusable, composable units. Pages are assembled from components, not built as monoliths. A well-designed component can be reused in contexts its creator never imagined. + +**Co-locate state with the components that need it** — Not every piece of state belongs in a global store. Local state stays local. Server state is fetched and cached. Only truly shared application state belongs in a global context. + +**Design for every state, not just the happy path** — Every data-dependent component has at least four states: loading, empty, error, and success. Designing for all four is not a nicety — it creates a resilient user experience. + +**Accessibility is not a feature, it's a requirement** — Keyboard navigation, screen reader support, color contrast, and focus management are not enhancements. They are part of the implementation contract. + +**Performance is a UX concern** — Every millisecond of load time, every layout shift, every janky interaction erodes user trust. Performance budgeting, bundle analysis, and render optimization are part of frontend engineering, not an afterthought. diff --git a/frontend-engineering/references/api-integration.md b/frontend-engineering/references/api-integration.md new file mode 100644 index 0000000..dc6b5ef --- /dev/null +++ b/frontend-engineering/references/api-integration.md @@ -0,0 +1,35 @@ +# API Integration + +## API Client Design + +| Layer | Responsibility | +|-------|---------------| +| Client instance | Base URL, default headers, timeout, interceptors | +| Request interceptor | Auth token injection, request logging, request ID | +| Response interceptor | Token refresh on 401, error normalization, response logging | +| Service module | Typed API methods per domain, request/response transformation | +| Hook layer | Data fetching hooks with loading/error/empty states | + +## Auth Token Flow + +``` +Login → Store token → Attach to requests → Detect expiry → Refresh → Retry + ↓ + Re-login on refresh failure +``` + +| Storage location | Pros | Cons | +|-----------------|------|------| +| HTTP-only cookie | Secure against XSS | CSRF vulnerability, harder for SPA to read | +| Memory (variable) | Most secure, not persisted | Lost on page refresh | +| localStorage | Survives refresh, simple | Accessible by any JS on the page | +| Session storage | Survives refresh in same tab | Cleared on tab close | + +## Real-Time Updates + +| Protocol | When to use | Connection management | +|----------|-------------|---------------------| +| WebSocket | Bidirectional, low-latency | Reconnect with backoff, heartbeat, fallback to polling | +| SSE (Server-Sent Events) | Server-to-client only, simpler than WebSocket | Automatic reconnection, event ID for resume | +| Short polling | Simple, no server push support | Fixed interval, wasteful when idle | +| Long polling | When WebSocket/SSE unavailable | Persistent connection, complex timeout handling | diff --git a/frontend-engineering/references/component-architecture.md b/frontend-engineering/references/component-architecture.md new file mode 100644 index 0000000..8442085 --- /dev/null +++ b/frontend-engineering/references/component-architecture.md @@ -0,0 +1,31 @@ +# Component Architecture + +## Composition Patterns + +| Pattern | When to use | Example | +|---------|-------------|---------| +| Atomic design | Design systems with clear hierarchy | `Button → FormField → AddressForm → CheckoutPage` | +| Compound components | Related components that share implicit state | `Select.Trigger`, `Select.Options`, `Select.Option` | +| Render props | Maximum flexibility in component behavior | Data provider that delegates rendering to consumer | +| Controlled vs uncontrolled | Form inputs, external state management | Controlled: state lives in parent. Uncontrolled: state lives in component | +| Higher-order components | Cross-cutting concerns (auth, logging) | `withAuth(Component)`, `withAnalytics(Component)` | + +## Props and State Interface Design + +| Aspect | Guideline | +|--------|-----------| +| Props should be minimal | Pass only what the component needs. Avoid prop drilling with context. | +| Defaults for optional props | Every optional prop has a sensible default. | +| Boolean props are named as questions | `isLoading`, `hasError`, `isDisabled`, `canSubmit` | +| Callback props describe the event | `onClick`, `onSubmit`, `onChange`, `onClose` | +| Avoid overloaded props | A prop should do one thing. `variant="primary|secondary|danger"`, not `mode="view|edit|admin"` | + +## Accessibility Fundamentals + +Every component must support: + +- **Keyboard navigation** — All interactive elements reachable and operable via keyboard (Tab, Enter, Escape, Arrow keys) +- **Focus management** — Visible focus indicators, logical tab order, focus trapping in modals +- **Screen reader support** — ARIA labels, roles, live regions, landmarks +- **Color contrast** — Text meets WCAG AA (4.5:1 for normal text, 3:1 for large text) +- **Reduced motion** — Respect `prefers-reduced-motion` for animations and transitions diff --git a/frontend-engineering/references/performance.md b/frontend-engineering/references/performance.md new file mode 100644 index 0000000..e2c0618 --- /dev/null +++ b/frontend-engineering/references/performance.md @@ -0,0 +1,30 @@ +# Client-Side Performance + +## Core Web Vitals Targets + +| Metric | Good | Needs improvement | Poor | +|--------|------|-------------------|------| +| LCP (Largest Contentful Paint) | ≤ 2.5s | 2.5s - 4.0s | > 4.0s | +| FID (First Input Delay) / INP | ≤ 100ms | 100ms - 300ms | > 300ms | +| CLS (Cumulative Layout Shift) | ≤ 0.1 | 0.1 - 0.25 | > 0.25 | + +## Bundle Optimization + +| Technique | Impact | Effort | +|-----------|--------|--------| +| Code splitting by route | High | Low (built-in with most frameworks) | +| Dynamic imports for heavy components | Medium | Low | +| Tree shaking unused exports | Medium | Low (enabled by default in bundlers) | +| Import cost awareness | Medium | Medium (lint rules, CI checks) | +| Image optimization (format, sizing, lazy loading) | High | Medium | +| Dependency audit (remove unused, find lighter alternatives) | Medium | High | + +## Render Optimization + +| Pattern | When to use | Mechanism | +|---------|-------------|-----------| +| Memoization | Pure components that re-render often | `React.memo`, `useMemo`, `useCallback` | +| Virtualization | Long lists (1000+ items) | `react-window`, `react-virtuoso` | +| Debouncing | High-frequency events (search input, scroll) | Debounce by 300-500ms | +| Throttling | Rate-limited updates (resize, scroll position) | Throttle by 100-200ms | +| Progressive hydration | Heavy interactive content below the fold | Lazy hydrate on visibility | diff --git a/frontend-engineering/references/responsive-layout-testing.md b/frontend-engineering/references/responsive-layout-testing.md new file mode 100644 index 0000000..c37bb61 --- /dev/null +++ b/frontend-engineering/references/responsive-layout-testing.md @@ -0,0 +1,1062 @@ +# Frontend Engineering Methodology Reference + +> Comprehensive reference covering responsive layout systems, breakpoint strategy, +> cross-device testing, and frontend testing patterns (component, integration, +> visual regression, accessibility, and test data management). +> +> Compiled: June 2026 + +--- + +## Table of Contents + +1. [Responsive Layout Systems](#1-responsive-layout-systems) +2. [Breakpoint Strategy](#2-breakpoint-strategy) +3. [Cross-Device Testing Methodology](#3-cross-device-testing-methodology) +4. [Frontend Testing Overview](#4-frontend-testing-overview) +5. [Component Testing](#5-component-testing) +6. [Integration Testing](#6-integration-testing) +7. [Visual Regression Testing](#7-visual-regression-testing) +8. [Accessibility Testing](#8-accessibility-testing) +9. [Test Data Management](#9-test-data-management) + +--- + +## 1. Responsive Layout Systems + +### Core Philosophy + +> **"CSS Grid is for layout; Flexbox is for alignment."** +> +> Use the right tool for the dimensionality of the problem. + +Modern CSS provides three layout primitives, each suited to different concerns. +The modern approach combines all three rather than choosing one. + +### 1.1 CSS Grid — Two-Dimensional Layout + +**Best for:** Page-level structure, complex multi-axis layouts, precise spatial +control, and layouts where rows AND columns matter simultaneously. + +| Use Case | Example | +|---|---| +| Full-page templates | Header, sidebar, main, footer regions | +| Card grids | Gallery, dashboard, e-commerce listings | +| Overlapping elements | Hero sections, magazine layouts | +| Gap-native layouts | `gap` property avoids margin hacks | +| Fraction-based sizing | `fr` units, `minmax()`, `auto-fill`/`auto-fit` | + +**Key patterns:** + +```css +/* Responsive grid without media queries */ +.grid { + display: grid; + grid-template-columns: repeat(auto-fill, minmax(16rem, 1fr)); + gap: 1rem; +} + +/* Named grid areas for page layout */ +.page { + display: grid; + grid-template-areas: + "header header" + "sidebar main" + "footer footer"; + grid-template-columns: 1fr 3fr; +} +``` + +**When to choose Grid over Flexbox:** +- You need control over both rows and columns simultaneously +- You want explicit placement (grid-column / grid-row) +- You have a predefined layout structure (layout-first design) +- You need overlapping elements without hacks +- You're building page-level templates + +### 1.2 Flexbox — One-Dimensional Alignment + +**Best for:** Component-level layouts, linear sequences, dynamic content flow, +centering, and distributing items along a single axis. + +| Use Case | Example | +|---|---| +| Navigation bars | Horizontal link lists, toolbars | +| Card internal layout | Row of label + value, button groups | +| Centering | Vertically/horizontally centering content | +| Dynamic wrapping | Tags, chips, badge lists | +| Flexible spacing | `justify-content: space-between` on footers | +| Reordering | `order` property for responsive reflow | + +**Key patterns:** + +```css +/* Centering */ +.container { + display: flex; + align-items: center; + justify-content: center; +} + +/* Responsive wrapping */ +.tag-list { + display: flex; + flex-wrap: wrap; + gap: 0.5rem; +} + +/* Holy grail of spacing */ +.toolbar { + display: flex; + justify-content: space-between; + align-items: center; +} +``` + +**When to choose Flexbox over Grid:** +- Content flows in one direction (row OR column, not both) +- Item sizes are unknown and should determine layout (content-first design) +- You need simple centering or alignment +- You're building reusable UI components (buttons, navs, toolbars) +- Items need to wrap to the next line naturally + +### 1.3 Container Queries — Component-Level Responsiveness + +**Best for:** Reusable components that must adapt to their parent container's +size rather than the viewport. The paradigm shift from viewport-centric to +container-centric design. + +```css +/* Establish containment context */ +.card-grid { + container-type: inline-size; + container-name: cards; +} + +/* Query against the container */ +@container cards (width >= 30rem) { + .card { + display: grid; + grid-template: "media body" auto / 2fr 3fr; + } + .card__media { + block-size: 100%; + object-fit: cover; + } +} + +@container cards (width >= 60rem) { + .card { + grid-template: "media body aside" auto / 1fr 2fr 1fr; + } +} +``` + +**Container units:** +| Unit | Meaning | +|---|---| +| `cqw` | 1% of container width | +| `cqh` | 1% of container height | +| `cqi` | 1% of container inline size | +| `cqb` | 1% of container block size | +| `cqmin` | Smaller of `cqi` and `cqb` | +| `cqmax` | Larger of `cqi` and `cqb` | + +**Style queries** (CSS 2025): Conditionally style based on a container's custom +properties or state. + +```css +@container style(--density: compact) { + .card { padding: 0.75rem; gap: 0.5rem; } +} + +@container style(--theme: surface) { + .card { background: #fff; color: #111; } +} +``` + +**When to use container queries:** +- The same component appears in multiple contexts (sidebar vs. main content) +- You want truly reusable design-system components +- Component breakpoints differ from page-level breakpoints +- You need to scale typography relative to the component, not the viewport + +**Progressive enhancement pattern:** + +```css +/* Base — works everywhere */ +.card { display: block; } + +/* Enhancement — only if supported */ +@supports (container-type: inline-size) { + .card-wrapper { container-type: inline-size; } + @container (width >= 25rem) { + .card { display: grid; grid-template-columns: 1fr 2fr; } + } +} +``` + +### 1.4 Decision Matrix: Grid vs. Flexbox vs. Container Queries + +| Criterion | CSS Grid | Flexbox | Container Queries | +|---|---|---|---| +| **Dimensionality** | 2D (rows + cols) | 1D (row OR col) | N/A (context only) | +| **Primary use** | Page layout | Component alignment | Component adaptation | +| **Content vs. layout driven** | Layout-first | Content-first | Container-driven | +| **Gap support** | Native `gap` | Native `gap` | Via host layout | +| **Overlap support** | Native (grid placement) | Not designed for | Not applicable | +| **Reordering** | Via placement | Via `order` | N/A | +| **Responsive technique** | `auto-fill`/`minmax()` + MQ | `flex-wrap` + MQ | `@container` queries | +| **Browser support** | Universal | Universal | ~90%+ (2026) | +| **Reusable components** | Possible, but rigid | Good | Best fit | + +### 1.5 Modern Fluid Layout Toolkit (2025+) + +Beyond the three primitives, modern CSS offers fluid sizing tools that reduce +or eliminate media queries: + +```css +/* Fluid typography */ +h1 { font-size: clamp(1.5rem, 2.5vw + 1rem, 3rem); } + +/* Fluid grid columns */ +.grid { grid-template-columns: repeat(auto-fill, minmax(clamp(12rem, 30%, 24rem), 1fr)); } + +/* Intrinsic sizing with aspect-ratio */ +.card { aspect-ratio: 16 / 9; } + +/* Logical properties for RTL support */ +.card { margin-inline: 1rem; padding-block: 2rem; } +``` + +--- + +## 2. Breakpoint Strategy + +### 2.1 Device-Agnostic vs. Content-Driven + +**The industry consensus in 2025-2026 is: content-driven breakpoints, not +device-based presets.** + +| Approach | Description | Verdict | +|---|---|---| +| **Device-agnostic** | Breakpoints at key widths where content breaks (e.g., 480px, 768px, 1024px) | Legacy best practice — better than fixed device targeting, but still viewport-centric | +| **Content-driven** | Breakpoints determined by the content itself — resize until it looks wrong, then add a breakpoint | Modern best practice | +| **Container-driven** | Breakpoints live on components via `@container`, not the viewport | Cutting edge (2025+) | + +### 2.2 The Problem With Fixed Breakpoints + +Traditional breakpoint strategy used device classes: + +```css +/* Avoid: device-specific */ +@media (max-width: 575px) { /* phones */ } +@media (min-width: 576px) { /* tablets */ } +@media (min-width: 992px) { /* laptops */ } +@media (min-width: 1200px) { /* desktops */ } +``` + +This fails because: +- New devices appear constantly (foldables, ultra-wides, 2-in-1s) +- The same component might need different breakpoints in different contexts +- It couples layout logic to arbitrary screen widths that may not match actual content needs + +### 2.3 Content-Driven Breakpoint Strategy + +**Methodology:** + +1. **Design in the browser** — resize gradually; stop every time the layout breaks +2. **Add a breakpoint at each breaking point**, naming it after what breaks, not the pixel value +3. **Use `rem` not `px`** for breakpoints — respects user font-size preferences +4. **Prefer fluid techniques first** (clamp, minmax, auto-fill) before adding media queries + +```css +/* Good: content-driven breakpoints in rem */ +/* Breakpoints at 30rem, 48rem, 64rem */ + +/* Prefer: fluid techniques before media queries */ +.card-grid { + display: grid; + grid-template-columns: repeat(auto-fill, minmax(18rem, 1fr)); +} + +/* Only add media queries when fluid isn't enough */ +@media (width >= 64rem) { + .card-grid { grid-template-columns: repeat(3, 1fr); } +} +``` + +### 2.4 Recommended Breakpoint Ranges + +Not fixed values, but ranges where content typically breaks: + +| Range (approx) | Common name | Behaviour | +|---|---|---| +| < 30rem (~480px) | Single-column | Stack everything | +| 30-48rem (~480-768px) | Narrow | 2-column grids possible | +| 48-64rem (~768-1024px) | Medium | 3-column layouts, sidebars | +| 64-90rem (~1024-1440px) | Wide | Full layouts, multi-column | +| > 90rem (~1440px) | Extra-wide | Max-width constraints, whitespace | + +> **Key insight:** These are guidelines, not dogma. Let YOUR content determine +> the exact values. A data table might need 50rem; a long-form article might +> need only 35rem. + +### 2.5 Modern Media Query Syntax + +CSS Media Queries Level 4+ introduced range syntax (widely supported in 2025): + +```css +/* Old syntax */ +@media (min-width: 768px) and (max-width: 1024px) { } + +/* New range syntax — cleaner */ +@media (768px <= width <= 1024px) { } +@media (width >= 48rem) { } +@media (width < 30rem) { } +``` + +### 2.6 Preference Queries (Accessibility-Aware) + +Beyond size, modern responsive design queries user preferences: + +```css +/* Respect reduced motion */ +@media (prefers-reduced-motion: reduce) { + * { animation-duration: 0.01ms !important; } +} + +/* Respect reduced transparency */ +@media (prefers-reduced-transparency: reduce) { + .glass { background: solid; } +} + +/* Respect dark mode */ +@media (prefers-color-scheme: dark) { + :root { --bg: #111; --text: #eee; } +} + +/* Respect increased contrast */ +@media (prefers-contrast: more) { + .card { border: 2px solid; } +} +``` + +--- + +## 3. Cross-Device Testing Methodology + +### 3.1 Test Pyramid for Responsive Design + +``` + /\ + / \ Manual device testing + / M \ (real hardware for critical paths) + / a u \ + / n a \ Device-emulation E2E + / u l \ (Playwright emulation matrix) + / a t e \ + / l e s t \ +/_______________\ Automated layout-safe checks + (container queries, fluid validation) +``` + +### 3.2 Device Emulation Matrix (Playwright/Cypress) + +Define your test matrix based on actual user analytics. Common strategy: + +```javascript +// Playwright config — device emulation +const devices = [ + { name: 'iPhone 15', width: 390, height: 844, deviceScaleFactor: 3 }, + { name: 'Pixel 8', width: 412, height: 915, deviceScaleFactor: 2.625 }, + { name: 'iPad Air', width: 820, height: 1180, deviceScaleFactor: 2 }, + { name: 'Desktop 1440', width: 1440, height: 900, deviceScaleFactor: 1 }, + { name: 'Desktop 1920', width: 1920, height: 1080, deviceScaleFactor: 1 }, +]; +``` + +### 3.3 Responsive Testing Checklist + +| Check | Method | Tooling | +|---|---|---| +| Content doesn't overflow | Automated CSS assertion | Playwright `toHaveCSS` | +| Touch targets >= 44px | Automated size check | Playwright bounding box | +| No horizontal scrollbar | Visual regression | Percy / Chromatic | +| Font size >= 16px (iOS zoom) | Emulation check | Safari/iOS device | +| Tap targets don't overlap | Layout check | axe-core / manual | +| All interactive elements work on touch | E2E test | Playwright touch emulation | +| Viewport meta tag present | Lint check | Lighthouse | + +### 3.4 Real Device Testing Strategy + +**Automated emulation covers ~80% of responsive bugs.** Real devices are needed for: + +1. **Touch interactions** — hover states, drag, swipe, force touch +2. **Hardware-specific** — notch, dynamic island, camera cutouts, safe areas +3. **Performance** — real CPU/memory constraints, network throttling +4. **Rendering differences** — Safari vs. Chrome font rendering, sub-pixel differences + +**Practical approach:** +- **CI/CD:** Emulation matrix (Playwright + devices) +- **Pull request review:** Visual regression (Percy/Chromatic) +- **Pre-release:** Real device cloud (BrowserStack / Sauce Labs / AWS Device Farm) +- **Critical paths:** Physical devices owned by the team + +### 3.5 Environment Simulation + +```javascript +// Playwright — responsive + environment simulation +test('homepage on slow 3G', async ({ page }) => { + await page.emulate({ viewport: { width: 390, height: 844 } }); + await page.context().addInitScript(() => { + // Simulate reduced motion + window.matchMedia = (query) => ({ + matches: query.includes('reduce-motion'), + media: query, + addListener: () => {}, + removeListener: () => {}, + }); + }); + await page.goto('/', { waitUntil: 'networkidle' }); + // assertions... +}); +``` + +--- + +## 4. Frontend Testing Overview + +### 4.1 The Testing Trophy (Modern Frontend) + +Replace the traditional "testing pyramid" with Kent C. Dodds' **Testing Trophy**, +which better reflects frontend priorities: + +``` + /\ + / \ Static analysis (TypeScript, ESLint) + / \ Unit/Component tests (Vitest + Testing Library) + / \ Integration tests (Playwright / Cypress) +/________\ E2E tests (critical user journeys only) +``` + +**Static analysis** catches type errors and lint issues at compile time. +**Component tests** verify isolated UI behaviour. +**Integration tests** (the bulk of your test suite) verify features work together. +**E2E tests** cover the most critical user journeys end-to-end. + +### 4.2 Testing Matrix Summary + +| Layer | Tool | Scope | Speed | Flakiness | CI cost | +|---|---|---|---|---|---| +| Static | TypeScript, ESLint | Types, lint | Instant | Never | Free | +| Component | Vitest + Testing Library | Individual components | Fast (ms) | Low | Cheap | +| Integration | Playwright / Cypress | Features, page interactions | Medium (s) | Low-Med | Moderate | +| Visual regression | Percy / Chromatic | Pixel-level UI | Medium (s) | Medium | Higher | +| Accessibility | axe-core + Lighthouse | WCAG violations | Fast (ms-s) | Low | Cheap | +| E2E critical | Playwright | Full user journeys | Slow (min) | Medium | Highest | + +--- + +## 5. Component Testing + +### 5.1 Vitest + React Testing Library + +**Standard setup (2025-2026):** + +```javascript +// vitest.config.js +import { defineConfig } from 'vitest/config'; +import react from '@vitejs/plugin-react'; + +export default defineConfig({ + plugins: [react()], + test: { + globals: true, + environment: 'jsdom', + setupFiles: './src/test/setup.js', + css: true, // process CSS imports + }, +}); +``` + +```javascript +// src/test/setup.js +import '@testing-library/jest-dom'; +import { cleanup } from '@testing-library/react'; +import { afterEach } from 'vitest'; + +afterEach(() => { cleanup(); }); +``` + +### 5.2 Core Query Priority + +**Test as users experience the UI:** + +``` +1. getByRole — Preferred for almost everything +2. getByLabelText — Form fields +3. getByPlaceholderText — Input hints +4. getByText — Non-interactive text +5. getByDisplayValue — Form values +6. getByAltText — Images +7. getByTitle — Tooltips +8. getByTestId — Last resort (data-testid) +``` + +### 5.3 Component Test Patterns + +**Render + Interaction + Assert:** + +```javascript +import { render, screen } from '@testing-library/react'; +import userEvent from '@testing-library/user-event'; +import { Counter } from './Counter'; + +it('increments count when button clicked', async () => { + const user = userEvent.setup(); + render(); + + await user.click(screen.getByRole('button', { name: /increment/i })); + + expect(screen.getByText('Count: 1')).toBeInTheDocument(); +}); +``` + +**Test behaviour, not implementation:** + +```javascript +// Bad: testing internal state +expect(counter.state.count).toBe(1); + +// Good: testing what the user sees +expect(screen.getByText('Count: 1')).toBeInTheDocument(); +``` + +**Mock external dependencies:** + +```javascript +import axios from 'axios'; +vi.mock('axios'); + +it('displays posts after fetch', async () => { + const posts = [{ id: 1, title: 'Hello' }]; + axios.get.mockResolvedValue({ data: posts }); + + render(); + + await waitFor(() => { + expect(screen.getByText('Hello')).toBeInTheDocument(); + }); +}); +``` + +**Custom hook testing:** + +```javascript +import { renderHook, waitFor } from '@testing-library/react'; + +it('returns data from fetch', async () => { + const { result } = renderHook(() => useFetch('/api/data')); + + await waitFor(() => expect(result.current.loading).toBe(false)); + + expect(result.current.data).toEqual({ id: 1 }); +}); +``` + +### 5.4 Best Practices (Component Testing) + +| Practice | Rationale | +|---|---| +| Test user-visible outcomes, not internals | Refactoring doesn't break tests | +| One assertion per test (when practical) | Clear failure messages | +| Use `userEvent`, not `fireEvent` | Realistic interaction simulation | +| Mock API calls, not modules | Tests stay fast and focused | +| Prefer `screen.` methods over destructured render | Keeps tests maintainable | +| Always clean up DOM between tests | Prevents test pollution | +| Use descriptive test names | `it('disables button while submitting')` | +| Write the test for the component's contract, not its internals | Tests validate behaviour | + +--- + +## 6. Integration Testing + +### 6.1 Playwright vs. Cypress (2025-2026) + +| Dimension | Playwright | Cypress | +|---|---|---| +| **Browser support** | Chromium, Firefox, WebKit | Chromium, Firefox (limited), WebKit (beta) | +| **Language** | JS/TS, Python, Java, .NET | JS/TS only | +| **Architecture** | Browser protocol (CDP) — runs outside browser | In-browser — runs inside the browser | +| **Multi-tab/window** | Native support | Limited | +| **Network mocking** | Route interception | cy.intercept | +| **Parallel execution** | Native, sharding | Dashboard required | +| **Cross-origin iframes** | Full support | Limited | +| **Mobile emulation** | Built-in device descriptors | cy.viewport only | +| **API testing** | Same context as browser | cy.request | +| **Community** | Larger momentum (91% satisfaction in State of JS 2025) | Mature, but satisfaction declined (72%) | +| **CI integration** | Zero config | Dashboard or plugin | + +**Bottom line (2026):** Playwright has become the default choice for new +projects due to broader browser support, multi-tab handling, and stronger +momentum. Cypress remains viable for teams already invested in its ecosystem. + +### 6.2 Playwright Integration Test Patterns + +**Page Object Model (POM) — recommended:** + +```typescript +// pages/LoginPage.ts +export class LoginPage { + constructor(private page: Page) {} + + async goto() { await this.page.goto('/login'); } + async login(email: string, password: string) { + await this.page.fill('[data-testid="email"]', email); + await this.page.fill('[data-testid="password"]', password); + await this.page.click('[data-testid="submit"]'); + } + async getErrorMessage() { + return this.page.textContent('[data-testid="error"]'); + } +} +``` + +```typescript +// tests/login.spec.ts +import { test, expect } from '@playwright/test'; +import { LoginPage } from '../pages/LoginPage'; + +test('shows error on invalid credentials', async ({ page }) => { + const loginPage = new LoginPage(page); + await loginPage.goto(); + await loginPage.login('bad@email.com', 'wrong'); + + await expect(loginPage.getErrorMessage()).toContain('Invalid credentials'); +}); +``` + +**Responsive integration test with device emulation:** + +```typescript +test('mobile navigation is usable', async ({ page }) => { + // Emulate mobile viewport + await page.setViewportSize({ width: 390, height: 844 }); + + await page.goto('/'); + await page.click('[data-testid="hamburger"]'); + await expect(page.locator('[data-testid="nav-menu"]')).toBeVisible(); + + // Touch targets meet minimum size (44x44 CSS pixels) + const links = page.locator('nav a'); + const count = await links.count(); + for (let i = 0; i < count; i++) { + const box = await links.nth(i).boundingBox(); + expect(box?.width).toBeGreaterThanOrEqual(44); + expect(box?.height).toBeGreaterThanOrEqual(44); + } +}); +``` + +**API mocking in integration tests:** + +```typescript +test('displays empty state when no results', async ({ page }) => { + // Intercept API call and return empty results + await page.route('**/api/search**', async route => { + await route.fulfill({ json: { results: [] } }); + }); + + await page.goto('/search'); + await page.fill('[name="q"]', 'nonexistent'); + await page.press('[name="q"]', 'Enter'); + + await expect(page.getByText('No results found')).toBeVisible(); +}); +``` + +### 6.3 Test Organization + +``` +tests/ + e2e/ + login.spec.ts + checkout.spec.ts + integration/ + api/ + search.spec.ts + user.spec.ts + features/ + filters.spec.ts + pagination.spec.ts + visual/ + homepage.spec.ts + product-card.spec.ts + accessibility/ + homepage.a11y.spec.ts + form.a11y.spec.ts +``` + +--- + +## 7. Visual Regression Testing + +### 7.1 Approaches + +| Approach | Tool | Pros | Cons | +|---|---|---|---| +| **Pixel-by-pixel screenshot diff** | Percy, Applitools, Chromatic | Catches every visual change | Baseline management, flakiness from animated content | +| **DOM snapshot** | Jest/Vitest snapshots | Fast, no browser needed | Fragile, doesn't catch CSS-only changes | +| **CSS-in-JS snapshot** | Storybook + Chromatic | Component-level, integrated with design system | Requires Storybook setup | +| **Layout diff** | Playwright screenshot | Full-page, device-emulated | Slower, per-environment diffs | +| **AI-assisted** | Percy AI, Applitools Eyes | Smart change detection, reduced false positives | Cost, vendor lock-in | + +### 7.2 Percy (BrowserStack) + +```javascript +// Cypress + Percy +cy.visit('/'); +cy.percySnapshot('Homepage'); + +// Playwright + Percy +import percySnapshot from '@percy/playwright'; +test('homepage visual', async ({ page }) => { + await page.goto('/'); + await percySnapshot(page, 'Homepage'); +}); +``` + +### 7.3 Chromatic (Storybook) + +```javascript +// .github/workflows/chromatic.yml +name: Chromatic +on: push +jobs: + chromatic: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + with: + fetch-depth: 0 + - run: npm ci + - uses: chromaui/action@v11 + with: + projectToken: ${{ secrets.CHROMATIC_PROJECT_TOKEN }} +``` + +### 7.4 Playwright Native Visual Testing + +```typescript +import { test, expect } from '@playwright/test'; + +test('homepage matches snapshot', async ({ page }) => { + await page.goto('/'); + await expect(page).toHaveScreenshot('homepage.png', { + maxDiffPixels: 100, + fullPage: true, + }); +}); +``` + +### 7.5 Visual Testing Best Practices + +| Practice | Detail | +|---|---| +| Mock dynamic content | Replace real data with fixtures for stable screenshots | +| Freeze animations | Use `page.addStyleTag()` to disable CSS animations | +| Isolate component states | Test loading, empty, error, and edge case states | +| Set consistent viewport | Always snapshot from known viewport sizes | +| Use CI diff thresholds | Allow configurable pixel tolerance to reduce flakiness | +| Review diffs in PRs | Block merging on unapproved visual changes | +| Rebase baselines intentionally | Not on every commit — only when changes are expected | + +--- + +## 8. Accessibility Testing + +### 8.1 Automation Coverage + +Automated accessibility testing catches roughly **30-40%** of WCAG violations. +This covers the "low-hanging fruit" — common, detectable issues. Manual testing +is still required for the remaining 60-70%. + +**What automation catches well:** +- Missing alt text on images +- Missing form labels +- Insufficient colour contrast +- Duplicate IDs +- Missing ARIA attributes +- Invalid ARIA usage +- Missing lang attributes +- Empty links / buttons + +**What requires manual testing:** +- Logical reading order +- Keyboard navigation flow +- Screen reader announcements +- Focus management +- Meaningful alt text +- Colour-only information transmission + +### 8.2 axe-core Integration + +**In Vitest/unit tests (jest-axe):** + +```javascript +import { render } from '@testing-library/react'; +import { axe, toHaveNoViolations } from 'jest-axe'; +expect.extend(toHaveNoViolations); + +it('has no accessibility violations', async () => { + const { container } = render(); + const results = await axe(container); + expect(results).toHaveNoViolations(); +}); +``` + +**In Playwright E2E tests:** + +```typescript +import AxeBuilder from '@axe-core/playwright'; + +test('homepage has no a11y violations', async ({ page }) => { + await page.goto('/'); + const results = await new AxeBuilder({ page }).analyze(); + expect(results.violations).toEqual([]); +}); + +// Specific WCAG levels +test('meets WCAG AA requirements', async ({ page }) => { + await page.goto('/'); + const results = await new AxeBuilder({ page }) + .withTags(['wcag2a', 'wcag2aa', 'wcag21a', 'wcag21aa']) + .analyze(); + expect(results.violations).toEqual([]); +}); + +// Targeted scan — specific element +test('navigation menu is accessible', async ({ page }) => { + await page.goto('/'); + await page.getByRole('button', { name: 'Menu' }).click(); + const results = await new AxeBuilder({ page }) + .include('#nav-flyout') + .analyze(); + expect(results.violations).toEqual([]); +}); +``` + +### 8.3 Lighthouse CI + +```javascript +// lighthouserc.json +{ + "ci": { + "collect": { + "numberOfRuns": 3, + "staticDistDir": "./build", + "settings": { "onlyCategories": ["accessibility"] } + }, + "assert": { + "assertions": { + "categories:accessibility": ["error", { "minScore": 0.9 }] + } + }, + "upload": { + "target": "temporary-public-storage" + } + } +} +``` + +### 8.4 CI/CD Integration + +```yaml +# .github/workflows/accessibility.yml +name: Accessibility +on: [pull_request] +jobs: + a11y: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-node@v4 + with: + node-version: 20 + cache: npm + - run: npm ci + - run: npm run build + - run: npx playwright install + - name: Run a11y tests + run: npx playwright test --project=a11y +``` + +### 8.5 Accessibility Testing Maturity Model + +| Level | What you do | Coverage | +|---|---|---| +| 1. None | No accessibility testing | 0% | +| 2. Manual only | Occasional Lighthouse audits | 30-40% (sporadic) | +| 3. Automated in CI | axe-core on every PR in Playwright tests | 30-40% (consistent) | +| 4. Automated + enforcement | Lighthouse CI score gates | 30-40% + regression detection | +| 5. Integrated into component tests | jest-axe on every component | 30-40% at component level | +| 6. Full pipeline | Axe + Lighthouse + manual audits + screen reader | 30-40% automated + 60-70% manual | + +--- + +## 9. Test Data Management + +### 9.1 Data Generation Strategies + +| Strategy | Description | When to use | +|---|---|---| +| **Fixtures** | Static, pre-defined data in JSON/YAML files | Test data that doesn't change often | +| **Factories** | Programmatic data generation with overrides | Many tests with slight data variations | +| **Faker** | Random realistic data (names, emails, addresses) | Stress testing, large datasets | +| **Seed data** | Known, reproducible database state | E2E tests needing a consistent baseline | +| **API mocks** | Intercepted network responses | Integration/component tests without a backend | + +### 9.2 Fixture Pattern + +```json +// src/test/fixtures/user.json +{ + "id": 1, + "name": "Alice Johnson", + "email": "alice@example.com", + "role": "admin" +} +``` + +```javascript +// Using fixture in a test +import userFixture from './fixtures/user.json'; + +it('renders user profile', () => { + render(); + expect(screen.getByText('Alice Johnson')).toBeInTheDocument(); +}); +``` + +### 9.3 Factory Pattern + +```javascript +// src/test/factories/user.js +import { faker } from '@faker-js/faker'; + +export function buildUser(overrides = {}) { + return { + id: faker.number.int({ min: 1, max: 10000 }), + name: faker.person.fullName(), + email: faker.internet.email(), + role: faker.helpers.arrayElement(['user', 'admin', 'moderator']), + avatar: faker.image.avatar(), + createdAt: faker.date.past().toISOString(), + ...overrides, + }; +} + +// Usage +const admin = buildUser({ role: 'admin', name: 'Admin User' }); +const users = Array.from({ length: 20 }, () => buildUser()); +``` + +### 9.4 Request Mocking Patterns + +```javascript +// MSW (Mock Service Worker) — recommended approach +import { http, HttpResponse } from 'msw'; +import { setupServer } from 'msw/node'; + +const server = setupServer( + http.get('/api/users/:id', ({ params }) => { + return HttpResponse.json({ + id: params.id, + name: 'Mocked User', + }); + }), +); + +beforeAll(() => server.listen()); +afterEach(() => server.resetHandlers()); +afterAll(() => server.close()); + +it('fetches and displays user', async () => { + render(); + await waitFor(() => { + expect(screen.getByText('Mocked User')).toBeInTheDocument(); + }); +}); +``` + +### 9.5 Test Data Isolation + +| Concern | Solution | +|---|---| +| **State leakage between tests** | `afterEach` cleanup, MSW handler reset | +| **Database state for E2E** | Database seeding before test suite, teardown after | +| **Shared state in factories** | Return new objects, not singletons | +| **Crypto-dependent IDs** | Deterministic faker seed: `faker.seed(123)` | +| **Dates / timers** | Fake timers: `vi.useFakeTimers()` | + +### 9.6 Test Data Setup Patterns + +```javascript +// Pattern 1: Setup in describe block +describe('UserList', () => { + const users = Array.from({ length: 5 }, (_, i) => + buildUser({ id: i + 1 }) + ); + + it('renders all users', () => { + render(); + expect(screen.getAllByRole('listitem')).toHaveLength(5); + }); +}); + +// Pattern 2: Custom render +function renderWithProviders(ui, { users = [], ...options } = {}) { + const wrapper = ({ children }) => ( + + {children} + + ); + return render(ui, { wrapper, ...options }); +} + +// Pattern 3: Test factory function +function setupUserList(overrides = {}) { + const users = overrides.users ?? [buildUser()]; + const onSelect = vi.fn(); + const utils = render(); + return { ...utils, users, onSelect }; +} +``` + +### 9.7 Frontend Test Data Best Practices + +| Practice | Detail | +|---|---| +| Use realistic data | Catch real rendering issues, not just schema validation | +| Separate data from assertions | Factories encourage readable, intention-revealing tests | +| Prefer MSW over vi.mock | MSW works at the network level, doesn't need module mocking | +| Seed deterministically | `faker.seed(123)` for reproducible failures | +| Mock at the right level | API mocking > module mocking > global mocking | +| Clean up between tests | Prevent state leakage that causes flaky tests | +| Keep fixtures small | Only include fields the test actually exercises | + +--- + +## References & Further Reading + +- **CSS Grid vs Flexbox**: blog.logrocket.com/css-flexbox-vs-css-grid +- **Container Queries Guide**: caisy.io/blog/css-container-queries +- **Beyond Media Queries (2025)**: medium.com/@orami98/beyond-media-queries +- **Modern Breakpoint Strategy**: penpot.app/blog/how-to-use-css-and-media-query-breakpoints +- **Playwright Accessibility Testing**: playwright.dev/docs/accessibility-testing +- **Vitest + Testing Library Setup**: freecodecamp.org/news/how-to-test-react-applications-with-vitest +- **Accessibility in CI/CD**: testparty.ai/blog/accessibility-testing-cicd +- **Visual Regression Guide**: desplega.ai/blog/deep-dive-7-visual-regression-testing-ui-bugs +- **Responsive Web Design Basics**: web.dev/articles/responsive-web-design-basics +- **Ten Modern Layouts in One Line of CSS**: web.dev/articles/one-line-layouts diff --git a/frontend-engineering/references/state-management.md b/frontend-engineering/references/state-management.md new file mode 100644 index 0000000..c2cd572 --- /dev/null +++ b/frontend-engineering/references/state-management.md @@ -0,0 +1,30 @@ +# State Management + +## State Classification + +| State type | Where it lives | How to manage | Example | +|-----------|----------------|---------------|---------| +| Local UI state | Component | `useState`, `useReducer` | Form input values, toggle open/closed | +| Shared UI state | Context or store | `useContext`, Zustand, Redux | Theme preference, sidebar collapsed state | +| Server state | Cache layer | React Query, SWR, Apollo | User profile, product list, search results | +| URL state | Browser URL | `useRouter`, search params | Current page, sort order, active filters | +| Form state | Form library | React Hook Form, Formik | Form values, validation errors, submission status | + +## Data Fetching Patterns + +| Pattern | When to use | Loading | Error | Empty | +|---------|-------------|---------|-------|-------| +| Fetch on render | Data needed immediately on page load | Skeleton/spinner | Error toast or inline error | Empty state message | +| Fetch on interaction | Data needed after user action | Button loading state | Inline error next to trigger | Handle in response | +| Prefetch | Next likely interaction | Background fetch, no loading UI | Silent failure, retry on explicit action | Handle on navigation | +| Infinite scroll | Paginated lists | Loading indicator at bottom | Inline error with retry | "No more results" | +| Optimistic update | Actions with predictable success | Instant UI update, rollback on error | Revert optimistic update, show error toast | N/A | + +## Caching Strategy + +| Aspect | Approach | +|--------|----------| +| Cache duration | Configurable per query type (stale-while-revalidate) | +| Invalidation | On mutation success, optimistic update, or manual refetch | +| Deduplication | Identical in-flight queries share a single request | +| Garbage collection | Unused cache entries evicted after configurable TTL | diff --git a/go-to-market/README.md b/go-to-market/README.md new file mode 100644 index 0000000..52b64da --- /dev/null +++ b/go-to-market/README.md @@ -0,0 +1,26 @@ +# Go To Market + +CMO methodology — positioning and messaging frameworks (April Dunford's positioning, message hierarchy), customer acquisition strategy (paid, organic, PLG, SLG), brand architecture (brand house vs house of brands), growth modeling (CAC/LTV by channel, cohort analysis), market entry strategy (beachhead, land-and-expand), competitive response (pricing wars, feature races, brand defense). + +## Why Install This Skill + +Your agent reasons about positioning, acquisition channels, and growth economics with real frameworks and numbers instead of generic marketing advice. + +## What You Get + +| Directory | Purpose | +|-----------|---------| +| `SKILL.md` | Core methodology, trigger conditions, reference index | +| `references/` | Deep-dive reference files loaded on demand | + +## Triggers + +Developing positioning/messaging, planning acquisition strategy, modeling CAC/LTV, designing brand architecture, or evaluating market entry approaches. + +## Requirements + +No technical requirements. Frameworks from April Dunford, PLG/SLG playbooks, and growth modeling. + +## Quick Start + +Load SKILL.md for the methodology overview and reference table, then load specific references as needed for the task at hand. diff --git a/go-to-market/SKILL.md b/go-to-market/SKILL.md new file mode 100644 index 0000000..cf853fe --- /dev/null +++ b/go-to-market/SKILL.md @@ -0,0 +1,57 @@ +--- +name: go-to-market +description: CMO methodology — positioning and messaging frameworks (April Dunford's + positioning, message hierarchy), customer acquisition strategy (paid, organic, PLG, + SLG), brand architecture (brand house vs house of brands), growth modeling (CAC/LTV + by channel, cohort analysis), market entry strategy (beachhead, land-and-expand), + competitive response (pricing wars, feature races, brand defense). +license: MIT +metadata: + tags: go-to-market, cmo, marketing, positioning, messaging, acquisition, brand-architecture, + growth-modeling, competitive-response, plg, slg + source_repo: https://github.com/magnus919/hermes-profiles +--- + +# Go-to-Market — CMO Methodology + +CMO-level methodology for go-to-market strategy, positioning, acquisition, brand, and growth modeling. This skill provides the frameworks and reference material for a chief marketing officer profile. + +## When to Load + +| Trigger | What's Needed | +|---------|---------------| +| Define positioning and messaging | `references/positioning-messaging.md` — Dunford framework, message hierarchy | +| Build acquisition channel strategy | `references/acquisition-strategy.md` — channel mix, PLG/SLG, funnel metrics | +| Design brand architecture | `references/acquisition-strategy.md` — brand systems, visual identity, brand health | +| Model growth economics | `references/growth-modeling.md` — CAC/LTV, cohort analysis, market entry | +| Develop competitive response | `references/acquisition-strategy.md` — pricing wars, feature races, brand defense | +| Plan market entry | `references/growth-modeling.md` — beachhead, land-and-expand, channel economics | + +## Loading Order + +```text +skill_view('go-to-market') +# Then domain-specific references: +skill_view('go-to-market', file_path='references/positioning-messaging.md') +skill_view('go-to-market', file_path='references/acquisition-strategy.md') +skill_view('go-to-market', file_path='references/growth-modeling.md') +``` + +## Reference Files + +| Reference | Purpose | +|-----------|---------| +| `references/positioning-messaging.md` | April Dunford positioning, message hierarchy (elevator pitch → value prop → narrative), positioning diagnostic | +| `references/acquisition-strategy.md` | Channel taxonomy, PLG vs SLG playbooks, sales funnel ratios, competitive response playbook, brand architecture | +| `references/growth-modeling.md` | CAC/LTV deep dive, cohort analysis practical guide, NRR, market entry strategy (beachhead, land-and-expand) | + +## Output Contract + +The profile using this skill produces artifact pyramids. The response to any caller is the absolute path to `00-index.md`. See `artifact-pyramids` skill for the specification. + +## Related Skills + +- `artifact-pyramids` — output contract +- `product-strategy` — CPO methodology (product vision, PMF, market sizing) +- `brand-designer` — visual brand identity design +- `seo-content-optimization` — organic search and content strategy diff --git a/go-to-market/references/acquisition-strategy.md b/go-to-market/references/acquisition-strategy.md new file mode 100644 index 0000000..1a25384 --- /dev/null +++ b/go-to-market/references/acquisition-strategy.md @@ -0,0 +1,156 @@ +# Acquisition Strategy + +## Channel Strategy + +### Building a Channel Mix + +No single channel scales forever. Build a diversified portfolio: + +``` +Stage 1 (Validate PMF): + └─ One channel you know well (e.g., content marketing for a dev tool) + └─ Optimize ruthlessly before adding channels + +Stage 2 (Scale): + └─ Add 1-2 complementary channels (e.g., paid search + partnerships) + └─ Ensure each channel has a clear attribution model + +Stage 3 (Optimize): + └─ 3-5 channels in the mix, each with different CAC trajectories + └─ 70/20/10 split: core channels / test channels / moonshots +``` + +### Channel Economics + +| Channel | Typical CAC range | Payback period | Scalability | +|---------|------------------|----------------|-------------| +| Organic search | $0-$100 | Immediate | Medium (takes months to build) | +| Paid search | $50-$500+ | 3-6 months | High (scales with budget) | +| Social ads | $20-$300 | 1-3 months | High (but auction pressure) | +| Content marketing | $100-$500 | 6-12 months | Medium (compounding) | +| Sales outbound | $5,000-$50,000+ | 6-18 months | High (headcount dependent) | +| Partnerships | $100-$5,000 | 3-12 months | Medium (relationship dependent) | +| Community | $0-$50 | 3-6 months | Low (organic, can't force) | +| Events | $500-$5,000 | 6-12 months | Low (one-off) | + +## Product-Led Growth (PLG) + +### The PLG Flywheel + +``` +User discovers → Tries self-serve → Hits aha moment → Invites team → Org adopts + ↑ | + └──────────────────────────┘ + (network effects / virality) +``` + +### PLG Metrics + +| Metric | Definition | Target | +|--------|------------|--------| +| **Activation rate** | % who hit the "aha moment" in first session | >30% | +| **Time to value** | Time from signup to first meaningful outcome | <10 minutes | +| **PQL (Product-Qualified Lead)** | User who hit a usage milestone indicating they're ready to buy | Varies | +| **Virality coefficient (K)** | Average number of new users each existing user invites | >1.0 for viral growth | +| **Self-serve conversion** | % of active users who convert to paid | 2-10% | +| **Expansion revenue** | Revenue from upgrades and add-ons | >120% NRR for PLG companies | + +### PLG Playbook + +1. **Remove friction from signup** — No credit card, no demo call, no sales contact +2. **Design for the aha moment** — The first experience should showcase core value within 60 seconds +3. **Build collaboration into the product** — Shared workspaces, team invites, public profiles +4. **Instrument for PQLs** — Score users on usage intensity, feature adoption, and team size +5. **Handoff to sales at the right moment** — PQL triggers a gentle CTA to talk to sales (not the other way around) + +## Sales-Led Growth (SLG) + +### The Sales Funnel + +``` +Top of Funnel (TOFU) + └─ Inbound leads (marketing-generated) + └─ Outbound leads (SDR-generated) + │ + Marketing-Qualified Leads (MQLs) + │ + Sales-Qualified Leads (SQLs) + │ + Opportunities (pipeline) + │ + ┌──────┴──────┐ + │ Demo/Proposal │ + └──────┬──────┘ + │ + Negotiation/Closing + │ + Customer Onboarded +``` + +### Key SLG Ratios + +| Metric | Definition | Healthy range | +|--------|------------|---------------| +| **MQL → SQL** | % of MQLs that qualify | 10-30% | +| **SQL → Opportunity** | % of SQLs that enter pipeline | 20-40% | +| **Win rate** | % of opportunities closed-won | 20-40% | +| **Sales cycle** | Days from lead to closed-won | 30-90 days (SMB) / 90-180 days (Enterprise) | +| **Quota attainment** | % of reps hitting quota | 60-70% | + +## Competitive Response Playbook + +### Pricing Wars + +| Situation | Don't | Do | +|-----------|-------|----| +| Competitor drops price | Match immediately in public | Analyze: are they buying market share or forced by cost? Decide whether to compete or segment away. | +| Competitor launches free tier | Panic and make your paid tier free | Evaluate if their free tier is viable. If it's a loss leader, let them burn cash. | +| Customer asks for price match | Cave without context | Frame your value in ROI terms. Offer a packaging concession instead of a rate cut. | + +### Feature Races + +| Situation | Don't | Do | +|-----------|-------|----| +| Competitor ships your differentiator | Treat it as existential | Evaluate how well they executed. Often, they ship a 60% version. Your 100% version remains differentiated. | +| Competitor ships a highly visible feature you lack | Build a copy in a sprint | Assess whether it matters to your best customers. If not, let them have the headline; you have the substance. | + +### Brand Defense + +| Threat | Strategy | +|--------|----------| +| Competitor claims they're "the [your category]" | Reinforce what makes the category meaningful. If the category is well-defined, they can't easily co-opt it. | +| Negative comparative marketing | Don't respond in kind — it elevates them. Use the "fan principle": ignore attackers, engage respectful comparators. | +| Competitor outspends on brand | Outspend on proof. Customer stories, benchmarks, and analyst recognition are harder to counter than ad spend. | + +## Brand Architecture + +### Building a Brand System + +A brand system ensures consistency across every touchpoint: + +| Element | Purpose | Format | +|---------|---------|--------| +| **Brand strategy** | Why the brand exists | Positioning document, brand soul | +| **Visual identity** | How the brand looks | Logo, colors, typography, imagery, spacing | +| **Verbal identity** | How the brand sounds | Voice, tone, vocabulary, messaging principles | +| **Applications** | Where the brand appears | Website, product, email, social, decks, swag | +| **Guidelines** | How to apply consistently | Brand book with do's and don'ts | + +### Visual Identity Components + +1. **Logo** — Primary lockup, icon only, horizontal, vertical, favicon +2. **Color palette** — Primary, secondary, neutral, accent, functional (success/error/warning) +3. **Typography** — Headline, body, monospace (code), sizing scale +4. **Imagery** — Photography style, illustration system, iconography +5. **Spacing** — Grid system, component spacing, padding conventions + +### Measuring Brand Health + +| Metric | What it measures | Source | +|--------|-----------------|--------| +| **Awareness** | % of target audience who know your brand | Surveys, brand lift studies | +| **Consideration** | % who would consider buying from you | Surveys | +| **Preference** | % who prefer you over alternatives | Surveys, choice modeling | +| **Net Promoter Score** | Willingness to recommend | Transactional surveys | +| **Share of voice** | % of category conversation | Social listening, media monitoring | +| **Brand equity** | Willingness to pay more for your brand over generic | Pricing studies | diff --git a/go-to-market/references/growth-modeling.md b/go-to-market/references/growth-modeling.md new file mode 100644 index 0000000..1835792 --- /dev/null +++ b/go-to-market/references/growth-modeling.md @@ -0,0 +1,132 @@ +# Brand Architecture + +## Brand Systems Frameworks + +### The Brand Hierarchy + +| Level | Description | Example (Apple) | +|-------|-------------|-----------------| +| **Corporate brand** | The parent company | Apple Inc. | +| **Product brand** | Individual product lines | iPhone, Mac, iPad | +| **Sub-brand** | Variants within product lines | iPhone Pro, MacBook Air | +| **Endorsed brand** | Standalone brands backed by corporate | Beats by Dr. Dre (Apple) | +| **Descriptor** | Clarifies what the brand does | Google "Cloud," Amazon "Web Services" | + +### Brand Relationship Spectrum + +A continuum from fully integrated to fully independent: + +``` +Branded House ─ Sub-brands ─ Endorsed Brands ─ House of Brands + (Google) (Apple) (Marriott) (P&G) +``` + +**Decision framework:** The more distinct the customer, channel, and price point, the further right you go. The more integration, cross-sell, and efficiency you need, the further left. + +## Brand Voice & Tone + +### Voice Dimensions + +| Dimension | Spectrum | Example | +|-----------|----------|---------| +| Formal ↔ Casual | Language complexity | Legal docs ↔ Slack messages | +| Serious ↔ Playful | Emotional tone | Medical device ↔ Gaming | +| Respectful ↔ Irreverent | Power distance | Bank ↔ Startup | +| Enthusiastic ↔ Matter-of-fact | Energy level | Social media ↔ Documentation | +| Concrete ↔ Abstract | Specificity | "Save 2 hours/week" ↔ "Empower your team" | + +### Tone Shifts by Context + +| Context | Tone shift | Example | +|---------|-----------|---------| +| **Error message** | Empathetic, solution-oriented | "Something went wrong — we've been notified. Retry?" | +| **Marketing** | Benefit-driven, active | "Ship faster with automated deployments." | +| **Legal/Compliance** | Precise, careful | "This statement does not constitute a guarantee of performance." | +| **Support** | Personal, ownership | "I'll personally ensure this gets resolved for you." | +| **Internal** | Direct, transparent | "Here's what we know and what we're doing about it." | + +## Growth Modeling + +### CAC / LTV Deep Dive + +#### Calculating CAC + +``` +CAC = (Sales expenses + Marketing expenses + Sales tools + S&M overhead) + └─────────────────────────────────────────────────────────────────┘ + ÷ + New customers acquired in period +``` + +**Blended vs Paid vs Organic:** +- **Blended CAC** — Total S&M spend ÷ all new customers (hides channel mix) +- **Paid CAC** — Paid channel spend ÷ paid-acquired customers (for efficiency analysis) +- **Organic CAC** — Organic channel spend ÷ organically acquired customers (usually near $0) + +#### LTV Calculation Methods + +**Simple LTV (for subscription):** +``` +LTV = ARPU × Gross Margin × (1 / Monthly Churn) +``` + +**Cohort LTV (more accurate):** +Track actual revenue from each cohort over 6-12 months. Plot the curve and extrapolate: + +``` +LTV_t = Σ(revenue per user in month t) × survival rate to month t +``` + +**Net Revenue Retention (NRR) LTV:** +``` +LTV with expansion = ARPU × Gross Margin / (1 - (Net Retention / 100)) +``` +Net retention >100% means LTV is infinite (in theory) — customers expand faster than churn. + +### Cohort Analysis — Practical Guide + +#### Setting Up Cohorts + +| Cohort type | When to use | Example | +|-------------|-------------|---------| +| **Time-based** | You want to track improvements over time | Users who signed up in January vs February | +| **Behavior-based** | You want to compare path-dependent outcomes | Trial-started vs trial-converted | +| **Segment-based** | You want to compare persona outcomes | Enterprise vs SMB users | +| **Channel-based** | You want to compare acquisition quality | Organic vs paid vs referral | + +#### The Cohort Retention Table + +``` +Cohort | Month 1 | Month 2 | Month 3 | Month 4 | Month 5 | Month 6 +Jan | 100% | 42% | 31% | 28% | 26% | 25% +Feb | 100% | 48% | 35% | 30% | 27% | +Mar | 100% | 51% | 38% | 32% | | +Apr | 100% | 48% | 34% | | | +May | 100% | 45% | | | | +Jun | 100% | | | | | +``` + +**Reading the table:** The vertical axis shows cohort quality (improving or declining over time). The horizontal axis shows stickiness (how quickly users fall off, and where they plateau). + +### Market Entry Strategy + +#### Beachhead Selection Matrix + +| Criterion | Weight | Segment A | Segment B | Segment C | +|-----------|--------|-----------|-----------|-----------| +| Urgent need (1-5) | 25% | 5 | 3 | 4 | +| Accessible (1-5) | 20% | 3 | 5 | 2 | +| Low competition (1-5) | 20% | 2 | 4 | 5 | +| Budget available (1-5) | 15% | 5 | 3 | 3 | +| Reference value (1-5) | 10% | 4 | 3 | 2 | +| Scale potential (1-5) | 10% | 5 | 3 | 1 | +| **Weighted score** | 100% | **4.10** | **3.60** | **3.15** | + +#### Land-and-Expand Playbook + +| Phase | Duration | Tactics | Success signal | +|-------|----------|---------|----------------| +| **Land** | 0-3 months | Solve a narrowly-scoped pain; white-glove onboarding; overinvest in success | First renewal, first public reference | +| **Prove** | 3-6 months | Track usage metrics; build case study; gather sponsor quotes | CSAT >90%, NPS >50 | +| **Expand** | 6-12 months | Introduce adjacent use cases; invite more teams; API access | Seat count 3x, feature adoption >3 modules | +| **Entrench** | 12-24 months | Custom integrations; exec relationships; steering committee | Multi-year renewal, product feedback loops | diff --git a/go-to-market/references/positioning-messaging.md b/go-to-market/references/positioning-messaging.md new file mode 100644 index 0000000..254a592 --- /dev/null +++ b/go-to-market/references/positioning-messaging.md @@ -0,0 +1,145 @@ +# Positioning & Messaging + +## April Dunford's Positioning Framework + +Positioning is not your tagline — it defines the context in which your product wins. Dunford's framework has 10 steps, distilled here into three essential layers: + +### Layer 1: Market Context + +Define the competitive alternative — what customers use *instead* of your product: + +| If customers are using... | Your positioning reference | +|---------------------------|---------------------------| +| A direct competitor | Differentiate on capabilities unmet by them | +| A manual process (spreadsheets, email) | Position as the first real solution to a known pain | +| An adjacent category solution | Reposition the need — "this is different from X" | +| Nothing / unaware of the problem | Educate on the problem first, then position the solution | + +### Layer 2: Unique Value + +Your unique value is the intersection of: + +``` +Unique Value = What you're best at × What customers care most about × What competitors can't easily copy +``` + +Map this as a Venn diagram. The center is your positioning sweet spot. + +### Layer 3: Market Definition + +Name the category or create a new one: +- **Existing category** — "We're the best [category] for [subsegment]" +- **New category** — "We're defining [new category], and here's why it matters" + +## Message Hierarchy + +Build a tiered messaging system that works at every touchpoint: + +``` +Level 1: Elevator Pitch (3 seconds) + └─ One line: "We help [segment] [outcome] by [capability]." + +Level 2: Value Proposition (30 seconds) + └─ Three sentences: + 1. The problem we solve (and why current approaches fall short) + 2. Our solution (and why it's different/better) + 3. The outcome (measurable results for our customers) + +Level 3: Supporting Narrative (3 minutes) + └─ Proof points: + - Customer stories / case studies + - Data and benchmarks + - Analyst recognition + - Technical depth (how it works) + +Level 4: Objection Handling + └─ Preempt the top 3-5 objections with reframed narratives +``` + +### The Message Testing Checklist + +| Check | Question | +|-------|----------| +| **Clarity** | Can a non-expert understand this in one read? | +| **Specificity** | Are we claiming something concrete, not generic ("best," "leading")? | +| **Differentiation** | Does a competitor's name slot in without sounding wrong? | +| **Evidence** | Can we back every claim with a data point or customer story? | +| **Memorability** | Would someone repeat this to a colleague? | + +## Acquisition Strategy + +### Channel Taxonomy + +| Channel Type | Examples | Best for | CAC trajectory | +|-------------|----------|----------|----------------| +| **Paid** | SEM, social ads, display, sponsorships | Scalable, predictable | Usually increases over time (auction pressure) | +| **Organic** | SEO, content marketing, social organic, W-o-M | High LTV:CAC | Decreases over time (compounding) | +| **Product-led** | Freemium, free trial, self-serve | Low-touch, high-volume | Low per user, high acquisition cost for power users | +| **Sales-led** | Outbound, field sales, channel partners | High-ACV, complex deals | High per deal, efficient at scale | +| **Community-led** | Forums, events, user groups, advocacy | Niche, high-trust markets | Very low, but slow to build | + +### PLG vs SLG Decision Matrix + +| Factor | PLG (Product-Led Growth) | SLG (Sales-Led Growth) | +|--------|--------------------------|------------------------| +| **ACV** | <$10K | >$25K | +| **Buyer** | End user (bottom-up) | Executive (top-down) | +| **Evaluation** | Self-serve trial | Demo, POC, procurement | +| **Sales cycle** | Days to weeks | Months to quarters | +| **Sales team** | Inbound product-qualified leads (PQLs) | Outbound, SDRs, AEs, SEs | + +**Hybrid model:** PLG for awareness and bottom-up adoption → SLG for expansion and enterprise close. + +## Brand Architecture + +### Brand House vs House of Brands + +| Structure | Description | Example | +|-----------|-------------|---------| +| **Branded House** | One master brand, sub-brands are descriptors | Google (Google Drive, Google Maps, Google Cloud) | +| **House of Brands** | Independent brands, each with its own identity | Procter & Gamble (Tide, Pampers, Gillette) | +| **Hybrid** | Endorsed brands — independent but carry a parent mark | Marriott (JW Marriott, Courtyard, Ritz-Carlton) | +| **Sub-Brand** | Master brand + modifier | Apple iPhone, Apple Watch | + +### Brand Architecture Decision Criteria + +| Decide Branded House When | Decide House of Brands When | +|---------------------------|----------------------------| +| One promise across all products | Products serve different customer needs | +| Customers trust the parent brand | Each brand needs its own positioning | +| Cross-selling is a core strategy | Brands compete in different price tiers | +| Business model is integrated | Business model is holding-company | +| Marketing efficiency matters | Brand equity of individual brands is valuable | + +## Growth Modeling + +### CAC / LTV Framework + +``` +CAC = Total sales & marketing spend / Number of new customers acquired +LTV = ARPU × Gross margin × (1 / Churn rate) +LTV:CAC ratio target: >3:1 for healthy growth +``` + +### Cohort Analysis + +Track groups of users who signed up in the same period and compare their behavior over time. + +| Metric | What it reveals | Action | +|--------|-----------------|--------| +| **Per-cohort retention** | Is each new cohort sticking or churning faster? | If declining, recent channels or onboarding are broken | +| **Revenue per cohort** | Are later cohorts monetizing as well? | If declining, pricing or packaging needs adjustment | +| **Feature adoption by cohort** | Are new cohorts using features differently? | May indicate shifting persona or segment | +| **Time to value** | How long until each cohort hits the "aha moment"? | Shorten onboarding for slow cohorts | + +### Growth Loops + +A growth loop is a self-reinforcing system where output feeds back into input: + +| Loop type | How it works | Example | +|-----------|-------------|---------| +| **Virality** | Each user brings more users | Slack (shared channels) | +| **Content** | Content → SEO → users → more content | HubSpot | +| **Sales-assisted** | Deal → case study → more deals | Salesforce | +| **Product-driven** | Feature → engagement → referral → more users | Dropbox (shared folders) | +| **Network effects** | More users → more value → more users | Uber, Airbnb | diff --git a/legal-strategy/README.md b/legal-strategy/README.md new file mode 100644 index 0000000..e2feac1 --- /dev/null +++ b/legal-strategy/README.md @@ -0,0 +1,26 @@ +# Legal Strategy + +CLO/General Counsel methodology — regulatory landscape analysis (GDPR, CCPA, AI Act, sector-specific), IP strategy (patent, trademark, trade secret, open source licensing), contract risk assessment (indemnification, liability caps, force majeure), data privacy frameworks (privacy-by-design, DPIAs, data mapping), corporate governance (board responsibilities, fiduciary duties, shareholder rights), employment law (classification, IP assignment, non-competes). + +## Why Install This Skill + +Your agent applies structured legal analysis — GDPR articles, liability cap tiers, IP decision frameworks — instead of hand-waving about compliance. + +## What You Get + +| Directory | Purpose | +|-----------|---------| +| `SKILL.md` | Core methodology, trigger conditions, reference index | +| `references/` | Deep-dive reference files loaded on demand | + +## Triggers + +Analyzing regulatory requirements, assessing contract risk, designing data privacy programs, evaluating IP strategy, or reviewing corporate governance. + +## Requirements + +No technical requirements. Covers GDPR, CCPA, EU AI Act, HIPAA, and general corporate/employment law frameworks. + +## Quick Start + +Load SKILL.md for the methodology overview and reference table, then load specific references as needed for the task at hand. diff --git a/legal-strategy/SKILL.md b/legal-strategy/SKILL.md new file mode 100644 index 0000000..b03e7b9 --- /dev/null +++ b/legal-strategy/SKILL.md @@ -0,0 +1,59 @@ +--- +name: legal-strategy +description: CLO/General Counsel methodology — regulatory landscape analysis (GDPR, + CCPA, AI Act, sector-specific), IP strategy (patent, trademark, trade secret, open + source licensing), contract risk assessment (indemnification, liability caps, force + majeure), data privacy frameworks (privacy-by-design, DPIAs, data mapping), corporate + governance (board responsibilities, fiduciary duties, shareholder rights), employment + law (classification, IP assignment, non-competes). +license: MIT +metadata: + tags: legal-strategy, clo, general-counsel, regulatory, ip-strategy, contract-risk, + data-privacy, corporate-governance, employment-law + source_repo: https://github.com/magnus919/hermes-profiles +--- + +# Legal Strategy — CLO/General Counsel Methodology + +CLO-level methodology for legal strategy, regulatory compliance, IP management, contract risk, data privacy, corporate governance, and employment law. This skill provides the frameworks and reference material for a chief legal officer or general counsel profile. + +## When to Load + +| Trigger | What's Needed | +|---------|---------------| +| Regulatory compliance analysis | `references/regulatory-analysis.md` — GDPR, CCPA, AI Act, HIPAA, cross-border transfers | +| IP strategy development | `references/ip-strategy.md` — patents, trademarks, trade secrets, open source licensing | +| Contract risk assessment | `references/contract-risk.md` — indemnification, liability caps, force majeure, DPA | +| Data privacy framework design | `references/data-privacy.md` — privacy-by-design, DPIAs, data mapping, breach response | +| Corporate governance counsel | `references/regulatory-analysis.md` — board duties, fiduciary obligations | +| Employment law consideration | `references/ip-strategy.md` — IP assignment, classification, non-competes | + +## Loading Order + +```text +skill_view('legal-strategy') +# Then domain-specific references: +skill_view('legal-strategy', file_path='references/regulatory-analysis.md') +skill_view('legal-strategy', file_path='references/ip-strategy.md') +skill_view('legal-strategy', file_path='references/contract-risk.md') +skill_view('legal-strategy', file_path='references/data-privacy.md') +``` + +## Reference Files + +| Reference | Purpose | +|-----------|---------| +| `references/regulatory-analysis.md` | GDPR, CCPA, AI Act risk categories, sector-specific (HIPAA, SOX, GLBA, PCI DSS), children's privacy, cross-border data transfers | +| `references/ip-strategy.md` | Patent types and process, trademark clearance and registration, trade secret management, open source licensing (permissive/copyleft), IP portfolio management | +| `references/contract-risk.md` | Indemnification clauses, liability cap tiers, force majeure (post-COVID), limitation of liability, data protection addenda, contract lifecycle management, risk scoring | +| `references/data-privacy.md` | Privacy-by-design (7 principles), DPIA process, data mapping/RoPA, breach response checklist, vendor privacy assessment | + +## Output Contract + +The profile using this skill produces artifact pyramids. The response to any caller is the absolute path to `00-index.md`. See `artifact-pyramids` skill for the specification. + +## Related Skills + +- `artifact-pyramids` — output contract +- `security-audit-methodology` — technical security posture (complementary to privacy/regulatory) +- `opensource-contributions` — open source contribution compliance and CLAs diff --git a/legal-strategy/references/contract-risk.md b/legal-strategy/references/contract-risk.md new file mode 100644 index 0000000..bff05df --- /dev/null +++ b/legal-strategy/references/contract-risk.md @@ -0,0 +1,103 @@ +# Contract Risk Assessment + +## Key Contract Provisions + +### Indemnification + +The indemnification clause determines who bears the cost of a third-party claim arising from the contract. + +| Type | What it covers | Who bears risk | +|------|----------------|----------------| +| **Mutual indemnification** | Each party indemnifies for claims arising from their own breach/negligence | Balanced | +| **One-way indemnification** | Only one party indemnifies the other | Unbalanced — typically favors the provider | +| **IP indemnification** | Claims that the product infringes third-party IP | Provider bears infringement risk | +| **Data breach indemnification** | Claims from a security incident involving customer data | Party responsible for the breach | + +**Negotiation tips:** +- Push for mutual indemnification whenever possible +- IP indemnification should be capped at the same amount as general liability +- Exclude claims arising from customer's modifications, misuse, or combination with third-party products +- Ensure coverage for settlement amounts, not just judgment + +### Liability Caps + +| Term | Typical range | Strategy | +|------|---------------|----------| +| **Liability cap** | 1x to 12x monthly or annual fees | Customer: push for multiple of fees. Provider: push for single digit multiple | +| **Uncapped exceptions** | IP infringement, breach of confidentiality, death/injury, fraud | Never cap these | +| **Mutual vs one-sided** | Some caps apply only to one party | Push for mutual caps with same exceptions | + +### The Liability Cap Tiers + +| Tier | Multiplier | Context | +|------|------------|---------| +| Enterprise deal, strong negotiating position | 12x annual fees | Customer-side enterprise procurement | +| Mid-market | 6x annual fees | Balanced negotiation | +| SMB / standard terms | 1x-3x annual fees | Provider's standard terms | +| Uncapped | N/A | Only for IP, confidentiality, fraud | + +### Force Majeure + +| Element | Description | Key issue | +|---------|-------------|-----------| +| **Events** | Acts of god, war, terrorism, pandemic, government action, labor strike | Is pandemic explicitly included? (Post-COVID: yes) | +| **Effect** | Performance becomes impossible, illegal, or impracticable | "Impossible" is narrower than "impracticable" | +| **Duration** | How long until termination rights arise | 30-90 days is typical | +| **Obligation** | Notice requirement, mitigation duty | Must notify within X days of triggering event | +| **Exclusions** | What is NOT force majeure | Economic hardship, market changes, supplier failure (unless force majeure) | + +**Post-COVID update:** Force majeure clauses are now reviewed carefully. Ensure pandemics are explicitly included (many pre-2020 contracts excluded them). Some contracts now have a separate "business continuity" or "pandemic" clause. + +### Limitation of Liability + +The limitation of liability clause excludes certain types of damages: + +| Excluded damages | Typical scope | Negotiation strategy | +|-----------------|---------------|---------------------| +| **Consequential damages** | Indirect, incidental, special, punitive | Never exclude damages for breach of confidentiality, IP infringement, or data breach | +| **Lost profits** | Revenue the customer expected to generate | Exclude for everyone — too speculative | +| **Lost data** | Cost of recreating or recovering data | Provider should exclude; customer should carve out if provider destroys customer data | + +### Data Protection / Privacy Addendum + +Must include: + +1. **Data Processing Agreement (DPA)** — Required under GDPR Art. 28 +2. **Data Security Schedule** — Specific security measures, encryption standards, breach notification timeline +3. **Data Transfer Mechanism** — SCCs or adequacy decision for cross-border transfers +4. **Data Retention and Deletion** — Timeframes for deletion after contract termination +5. **Sub-processor List** — Approved sub-processors, change notification, objection rights +6. **Audit Rights** — Customer's right to audit provider's security practices + +## Contract Lifecycle + +### Pre-Signature Checklist + +- [ ] Scope of work clearly defined (SOW or exhibit) +- [ ] Pricing and payment terms unambiguous +- [ ] Term, renewal, and termination provisions clear +- [ ] Liability cap (mutual, with appropriate exceptions) +- [ ] Indemnification scope matches risk allocation +- [ ] DPA/data protection addendum attached (if processing personal data) +- [ ] Governing law and dispute resolution specified +- [ ] Confidentiality obligations mutual +- [ ] IP ownership: each party retains its pre-existing IP; deliverables IP assigned to customer +- [ ] SLA (if applicable) with service credits + +### Post-Signature Management + +| Phase | Action | Owner | +|-------|--------|-------| +| **Day 1** | File signed contract in repository, assign to contract manager, set renewal reminder | Legal ops | +| **Ongoing** | Track obligations (reporting, compliance, renewal deadlines) | Contract manager | +| **At renewal** | Review performance against SLA, negotiate improvements, update terms | Legal + Business | +| **At termination** | Confirm data deletion, final payment, close-out obligations | Legal ops | + +## Contract Risk Scoring + +| Risk level | Characteristics | Review requirement | +|------------|----------------|--------------------| +| **Low** | Standard terms on provider's paper, minimal data processing, low value | Fast-track, no redlines | +| **Medium** | Custom terms, moderate data processing, some redlines expected | Counsel review, key terms only | +| **High** | Customer's paper, aggressive redlines, high-value, significant data processing | Full counsel review, exec approval | +| **Critical** | Uncapped liability, one-way indemnification, unusual data handling | External counsel, board approval | diff --git a/legal-strategy/references/data-privacy.md b/legal-strategy/references/data-privacy.md new file mode 100644 index 0000000..c4e1501 --- /dev/null +++ b/legal-strategy/references/data-privacy.md @@ -0,0 +1,211 @@ +# Data Privacy Framework + +## Privacy-by-Design (PbD) + +The seven foundational principles, as articulated by Dr. Ann Cavoukian and codified in GDPR Article 25: + +### 1. Proactive Not Reactive; Preventative Not Remedial + +| Traditional approach | PbD approach | +|---------------------|--------------| +| Respond to breaches after they happen | Anticipate and prevent privacy-invasive events | +| Privacy is a compliance checkbox | Privacy is a design requirement | +| Privacy reviewed at launch | Privacy considered from the first spec | + +### 2. Privacy as the Default Setting + +Data is automatically protected without the user having to take action: + +- **Data minimization**: Collect the minimum data needed +- **Purpose limitation**: Process data only for specified purposes +- **Limited accessibility**: Only authorized personnel can access +- **Limited retention**: Automatically delete data when no longer needed +- **Privacy-preserving defaults**: Opt-in, not opt-out + +### 3. Privacy Embedded into Design + +Privacy is not bolted on after the fact — it's an integral part of the system: + +- Architecture diagrams include data flows and privacy controls +- User stories include privacy acceptance criteria +- Code reviews check for privacy compliance +- Testing includes privacy scenarios + +### 4. Full Functionality — Positive-Sum, Not Zero-Sum + +Reject the false tradeoff between privacy and functionality. Design for both: + +- **Privacy + Security** — Encryption is both a privacy and security measure +- **Privacy + Analytics** — Differential privacy allows aggregate analytics without individual identification +- **Privacy + Personalization** — On-device processing enables personalization without data collection + +### 5. End-to-End Security + +Privacy depends on security. Full lifecycle protection: + +| Stage | Security measure | +|-------|------------------| +| **In transit** | TLS 1.3, mutual TLS, VPN | +| **At rest** | Encryption (AES-256), key rotation, HSM | +| **In use** | Confidential computing, differential privacy | +| **Processing** | Access controls, audit logging, anomaly detection | +| **Deletion** | Cryptographic erasure, secure wipe verification | + +### 6. Visibility and Transparency + +All stakeholders operate with the knowledge that the system is privacy-assuring: + +- **Privacy notice** — Clear, specific, accessible +- **Data flow map** — Where data goes, who touches it, how it's protected +- **Processing records** — Article 30 RoPA (Record of Processing Activities) +- **Incident reporting** — Breach notification process +- **Audit trail** — Log of who accessed what and when + +### 7. Respect for User Privacy + +User-centric design — keep the individual's interests paramount: + +- **Granular consent** — Separate consents for separate purposes +- **Easy exercise of rights** — Access, rectification, erasure, portability +- **Usable privacy controls** — Settings are easy to find and understand +- **User education** — Clear explanations of data practices + +## Data Protection Impact Assessment (DPIA) + +### When a DPIA is Required (GDPR Art. 35) + +A DPIA is mandatory when processing is likely to result in high risk to individuals' rights and freedoms, specifically: + +1. Systematic and extensive profiling with significant effects +2. Large-scale processing of special categories of data (health, biometric, genetic) +3. Systematic monitoring of a publicly accessible area on a large scale (CCTV) +4. Other high-risk processing as identified by the supervisory authority + +### DPIA Process + +``` +Step 1: Identify need ──────────────────────────────┐ + │ │ + v │ +Step 2: Describe processing │ + │ (nature, scope, context, purposes) │ + v │ +Step 3: Assess necessity & proportionality │ + │ (is this the least privacy-invasive way?) │ + v │ +Step 4: Identify and assess risks ───────────────────┤ + │ (likelihood × severity for each risk) │ + v │ +Step 5: Identify mitigations ────────────────────────┤ + │ (reduce risk to acceptable level) │ + v │ +Step 6: Document, sign off, integrate │ + │ (record decision, obtain approval, implement) │ + v │ +Step 7: Review │ + (periodic review — update if processing changes) │ +``` + +### DPIA Risk Matrix + +| Likelihood \ Severity | Low | Medium | High | +|-----------------------|-----|--------|------| +| **High** | Medium risk | High risk | Critical | +| **Medium** | Low risk | Medium risk | High risk | +| **Low** | Low risk | Low risk | Medium risk | + +**Response by risk level:** +- **Critical** — Processing cannot proceed as described. Redesign or abandon. +- **High** — Implement additional mitigations. Consult DPO/supervisory authority if mitigations cannot reduce to acceptable level. +- **Medium** — Mitigate; standard controls are sufficient. +- **Low** — Document and proceed. + +## Data Mapping (Record of Processing Activities — RoPA) + +### Required Under GDPR Article 30 + +| Field | Description | +|-------|-------------| +| **Controller/Processor** | Name and contact details of each | +| **Purposes of processing** | Why this data is processed | +| **Categories of data subjects** | Whose data (employees, customers, website visitors) | +| **Categories of personal data** | What data (name, email, health data, location) | +| **Categories of recipients** | Who receives it (processors, third parties, authorities) | +| **Transfers to third countries** | Any cross-border data flows and safeguard mechanism | +| **Retention periods** | How long data is kept | +| **Technical/organizational measures** | Security measures applied | + +### Data Mapping Methodology + +1. **Inventory data collection points** — Every system, form, API, integration, and manual process +2. **Map data flows** — From collection through processing, storage, sharing, to deletion +3. **Identify data elements** — What specific personal data is involved +4. **Classify by sensitivity** — Regular vs special category vs sensitive +5. **Assess legal basis** — What lawful basis applies to each processing purpose +6. **Document retention** — How long each data element is retained +7. **Review periodically** — Annual or on significant process change + +## Breach Response + +### The 72-Hour Notification Clock (GDPR Art. 33) + +``` +Breach detected + │ + v +T+0 hours: Contain and assess + ├── Isolate affected systems + ├── Preserve evidence / logs + ├── Determine scope (what data, who is affected) + └── Assign breach response lead + │ + v +T+24 hours: Notify internal stakeholders + ├── Legal counsel + ├── Security team + ├── DPO + ├── Comms / PR + └── Executive team + │ + v +T+48 hours: File (if required) – draft notification + ├── Likelihood of risk to individuals? + ├── Categories and approximate number of data subjects + ├── Categories and approximate number of records + ├── Likely consequences of the breach + └── Measures taken or proposed to address the breach + │ + v +T+72 hours: Submit notification to supervisory authority + ├── Notify affected individuals (if high risk) + └── Begin post-mortem +``` + +**Not required to notify if:** +- Data was encrypted (and key was not compromised) +- Breach is unlikely to result in risk to rights and freedoms +- Affected data was pseudonymized and cannot be re-associated + +### Breach Notification Contents (GDPR Art. 33(3)) + +1. Description of the nature of the breach +2. Categories and approximate number of data subjects concerned +3. Categories and approximate number of personal data records concerned +4. Name and contact details of the DPO or other contact point +5. Likely consequences of the breach +6. Measures taken or proposed to address the breach + +## Vendor Privacy Assessment + +### Pre-Engagement Checklist + +- [ ] Does vendor process personal data on our behalf? +- [ ] Is a DPA/Data Processing Agreement in place? +- [ ] Does vendor have ISO 27001, SOC 2 Type II certification? +- [ ] Is vendor's sub-processor list current and acceptable? +- [ ] What is vendor's breach notification timeline? (GDPR requires 72h — the same requirement flows down) +- [ ] Where is data physically stored? (Jurisdiction matters for transfers) +- [ ] Does vendor have a published data retention and deletion policy? +- [ ] Is vendor's liability cap sufficient for the data risk? +- [ ] What audit rights do we have? +- [ ] Does the contract survive a change-of-control at the vendor? diff --git a/legal-strategy/references/ip-strategy.md b/legal-strategy/references/ip-strategy.md new file mode 100644 index 0000000..6580370 --- /dev/null +++ b/legal-strategy/references/ip-strategy.md @@ -0,0 +1,132 @@ +# IP Strategy — Patents, Trademarks, Trade Secrets, and Open Source Licensing + +## Patent Strategy + +### Types of Patents + +| Type | Duration | Subject matter | Example | +|------|----------|----------------|---------| +| **Utility patent** | 20 years from filing | Processes, machines, articles of manufacture, compositions of matter | Software algorithm, hardware device | +| **Design patent** | 15 years (US) | Ornamental design of a functional item | Icon design, product shape | +| **Provisional patent** | 12 months (placeholder) | Establishes priority date, doesn't become a patent automatically | "Patent pending" status | + +### Patent Filing Strategy + +| Strategy | When to use | Example | +|----------|-------------|---------| +| **Defensive filing** | Core technology you want to protect | Foundational algorithm | +| **Offensive filing** | Blocking competitors in key areas | Patent thicket around a technology domain | +| **Landscape filling** | Identify gaps in competitor portfolios and file there | Adjacent use cases competitors haven't claimed | +| **Standard-essential** | Technology required by an industry standard | 5G, Wi-Fi, video codecs | + +### The Patent Process + +``` +Disclosure → Prior art search → Drafting → Filing → Examination → Grant/Maintenance + ↓ + Patent pools / licensing / enforcement +``` + +**Key decision:** Trade secret vs patent. Patents require public disclosure; trade secrets can last indefinitely. If the technology can be reverse-engineered with moderate effort, patent. If the secret can be kept (Coca-Cola formula, Google PageRank algorithm), consider trade secret. + +## Trademark Strategy + +### Trademark Types + +| Type | Examples | Protection | +|------|----------|------------| +| **Word mark** | "Google," "Apple" | The word itself, in any stylization | +| **Design mark** | Nike swoosh, Apple logo | The visual design | +| **Sound mark** | Intel jingle, MGM lion roar | Sonic brand identity | +| **Trade dress** | Coca-Cola bottle shape, Tiffany blue | Product appearance or packaging | + +### Trademark Clearance + +Before adopting a mark, conduct: + +1. **Screening search** — Internal database, general web search +2. **Full availability search** — USPTO / EUIPO trademark database +3. **Common law search** — Business registries, domain names, social media handles +4. **International search** — Madrid Protocol if filing in multiple jurisdictions + +**Risk levels:** +| Finding | Risk | Action | +|---------|------|--------| +| No conflicting marks | Low | Proceed | +| Conflicting mark in different class/geography | Medium | File with careful monitoring | +| Direct conflict with active mark in same class | High | Abandon or acquire | + +## Trade Secret Management + +### Legal Requirements (US — DTSA / State Law) + +A trade secret must: +1. Have **independent economic value** from not being generally known +2. Be subject to **reasonable measures** to maintain secrecy + +### Reasonable Secrecy Measures + +| Measure | Implementation | +|---------|----------------| +| **Access controls** | Need-to-know basis, role-based permissions, physical locks | +| **NDAs** | Employee, contractor, and partner non-disclosure agreements | +| **Exit procedures** | Return of materials, reminder of ongoing obligations, access revocation | +| **Labeling** | Clearly mark "CONFIDENTIAL — Trade Secret" on documents | +| **Training** | Annual training on trade secret handling | +| **Segmentation** | Compartmentalize — no one person knows the whole secret | +| **Audit trails** | Log access to trade secret repositories | + +### Litigation Risks + +- **Inevitable disclosure doctrine** — Former employee's new role inevitably requires disclosing trade secrets (injunction available in some jurisdictions) +- **Reverse engineering** — Legal unless prohibited by contract; cannot protect against it with trade secret law alone + +## Open Source Licensing + +### License Categories + +| Category | Examples | Requirements | Commercial implications | +|----------|----------|-------------|------------------------| +| **Permissive** | MIT, Apache 2.0, BSD | Attribution only | Can use in proprietary products | +| **Weak copyleft** | LGPL, MPL, EPL | Modifications to the library itself must be open-sourced | Can link from proprietary code | +| **Strong copyleft** | GPL 2.0/3.0, AGPL | Derivative works must be open-sourced under same license | Usually incompatible with proprietary products | +| **Network copyleft** | AGPL | Software accessed over a network must be distributed with source | Affects SaaS companies | + +### Strategic Decisions + +| Decision | Consideration | +|----------|--------------| +| **Why open source?** | Community adoption, talent attraction, commoditize complement, standards setting | +| **License choice** | Permissive for maximum adoption; copyleft to prevent proprietary forks | +| **CLA (Contributor License Agreement)** | Required for corporate projects to relicense later | +| **Dual licensing** | Open source (GPL) + commercial license for proprietary users (Qt, MySQL model) | +| **Trademark policy** | Prevent confusion: who can use the project name/logo | + +### Open Source Compliance + +| Process | Description | +|---------|-------------| +| **SBOM generation** | Software Bill of Materials — list every dependency and its license | +| **License scanning** | Automated tools (FOSSA, Black Duck, Snyk) to detect license obligations | +| **Policy creation** | Approved licenses list, obligation matrix, approval workflow for exceptions | +| **Distribution compliance** | Include license notices, provide source code on request (GPL) | +| **Audit readiness** | Maintain a compliance artifact: notices file, source code archive, obligation log | + +## IP Portfolio Management + +### The IP Lifecycle + +``` +Creation → Protection → Maintenance → Monetization + ↓ + Licensing / Sale / Enforcement +``` + +### IP Budget Allocation + +| % of Budget | Category | Activity | +|-------------|----------|----------| +| 40-50% | **Defensive core** | Patent filing for core technology, trademark registration | +| 20-30% | **Offensive/IP landscape** | Competitive blocking, freedom-to-operate analysis | +| 15-20% | **Maintenance** | Renewal fees, trademark renewal, portfolio pruning | +| 10-15% | **Enforcement** | Cease-and-desist, licensing negotiations, litigation | diff --git a/legal-strategy/references/regulatory-analysis.md b/legal-strategy/references/regulatory-analysis.md new file mode 100644 index 0000000..8d2d6ab --- /dev/null +++ b/legal-strategy/references/regulatory-analysis.md @@ -0,0 +1,122 @@ +# Regulatory Analysis — GDPR, CCPA, AI Act + +## General Data Protection Regulation (GDPR) + +### Scope & Applicability + +| Aspect | Detail | +|--------|--------| +| **Enforcement** | May 25, 2018 | +| **Territorial scope** | EU establishment; OR targeting/monitoring data subjects in the EU (Article 3) | +| **Material scope** | Personal data processed wholly or partly by automated means (Article 2) | +| **Penalties** | Up to €20M or 4% of global annual turnover, whichever is higher | + +### Key Principles (Article 5) + +1. **Lawfulness, fairness, transparency** — Must have a lawful basis; be clear about how data is used +2. **Purpose limitation** — Collect for specified, explicit, legitimate purposes; don't repurpose +3. **Data minimization** — Collect only what's necessary for the stated purpose +4. **Accuracy** — Keep data accurate and up to date; rectify inaccuracies +5. **Storage limitation** — Keep only as long as necessary for the purpose +6. **Integrity and confidentiality** — Appropriate security measures +7. **Accountability** — Must be able to demonstrate compliance with all principles + +### Lawful Bases for Processing (Article 6) + +| Basis | Description | Best for | +|-------|-------------|----------| +| **Consent** | Freely given, specific, informed, unambiguous | Marketing, non-essential cookies | +| **Contract** | Necessary to fulfill a contract with the data subject | Order processing, account management | +| **Legal obligation** | Required by law | Tax reporting, fraud prevention | +| **Vital interests** | Necessary to protect someone's life | Emergency medical data | +| **Public interest** | Official authority or public task | Government services | +| **Legitimate interests** | Balanced against data subject's rights | Analytics, security (not public authorities) | + +### Data Subject Rights + +| Right | Description | Response timeline | +|-------|-------------|------------------| +| **Right to be informed** | Privacy notice at collection point | At time of collection | +| **Right of access** | Copy of personal data and processing info | 1 month (extendable to 2) | +| **Right to rectification** | Correct inaccurate data | 1 month | +| **Right to erasure** ("Right to be forgotten") | Delete personal data | 1 month (with exceptions) | +| **Right to restrict processing** | Limit how data is used | 1 month | +| **Right to data portability** | Receive data in machine-readable format | 1 month | +| **Right to object** | Object to processing (including profiling/marketing) | Without undue delay | +| **Rights related to automated decision-making** | Not be subject to solely automated decisions with legal effects | N/A | + +## California Consumer Privacy Act (CCPA / CPRA) + +### Scope + +Applies to for-profit businesses that: +- Gross annual revenue >$25M; OR +- Buy, sell, or share personal data of 100,000+ California residents/year; OR +- Derive 50%+ of revenue from selling/sharing personal data + +### Consumer Rights + +- **Right to know** — Categories and specific pieces of personal data collected, sources, purpose, third parties +- **Right to delete** — Subject to exceptions (complete transaction, security, legal compliance) +- **Right to opt out** — Of sale or sharing of personal data (includes cross-context behavioral advertising) +- **Right to correct** — Inaccurate personal data +- **Right to limit** — Use of sensitive personal data +- **Right to non-discrimination** — No retaliation for exercising rights + +## EU AI Act + +### Risk Categories + +| Category | Examples | Requirements | +|----------|----------|--------------| +| **Unacceptable risk** (banned) | Social scoring, real-time biometric surveillance, manipulative AI | Prohibited entirely | +| **High risk** | Critical infrastructure, education, employment, law enforcement, migration, justice | Conformity assessment, risk management, human oversight, transparency, accuracy, cybersecurity | +| **Limited risk** | Chatbots, AI-generated content | Transparency obligations (disclose AI interaction) | +| **Minimal risk** | AI-enabled video games, spam filters | No additional obligations (voluntary codes of conduct) | + +### High-Risk System Requirements + +1. **Risk management system** — Continuous, iterative process throughout lifecycle +2. **Data governance** — Training, validation, and testing data must be relevant, representative, and free from biases +3. **Technical documentation** — Design specifications, development methodology, accuracy/robustness benchmarks +4. **Record-keeping** — Automatic logging of events during operation +5. **Transparency** — Clear disclosure to users +6. **Human oversight** — Appropriate for the risk level +7. **Accuracy, robustness, cybersecurity** — Appropriate levels of each + +### Penalties + +| Violation | Penalty | +|-----------|---------| +| Unacceptable risk practices | €35M or 7% of global annual turnover | +| Non-compliance with high-risk obligations | €15M or 3% of global annual turnover | +| Providing incorrect information | €7.5M or 1% of global annual turnover | + +## Sector-Specific Regulations + +### Healthcare (US — HIPAA) + +- Protected Health Information (PHI) — 18 identifiers +- Privacy Rule — Use/disclosure limits +- Security Rule — Administrative, physical, technical safeguards +- Breach Notification Rule — 60-day notification requirement + +### Financial Services (US — SOX, GLBA) + +- **SOX** — Internal controls over financial reporting, records retention (7 years), CEO/CFO certification +- **GLBA** — Financial Privacy Rule, Safeguards Rule, Pretexting Protection +- **PCI DSS** — Payment card data security (not law, but contractual requirement) + +### Children's Privacy + +- **COPPA (US)** — Under 13: verifiable parental consent, privacy policy, data minimization +- **GDPR Art. 8** — Under 16 (varies by member state down to 13): parental consent + +## Cross-Border Data Transfers + +| Mechanism | GDPR | Notes | +|-----------|------|-------| +| Adequacy decision | EU deems country's protections adequate | UK, Japan, South Korea, others | +| Standard Contractual Clauses (SCCs) | EU Commission-approved contracts | Most common mechanism | +| Binding Corporate Rules (BCRs) | Multi-national group policies | Complex to implement | +| Derogations | Specific situations (consent, contract necessity) | Limited use — not a long-term solution | diff --git a/ml-engineering/README.md b/ml-engineering/README.md new file mode 100644 index 0000000..35b3168 --- /dev/null +++ b/ml-engineering/README.md @@ -0,0 +1,26 @@ +# Ml Engineering + +Machine learning engineering methodology — model training, fine-tuning (LoRA/QLoRA), evaluation, quantization, deployment, and MLOps pipeline design. Grounded in practical engineering patterns for production ML systems. + +## Why Install This Skill + +Your agent makes informed decisions about fine-tuning approaches, quantization trade-offs, GPU selection, and serving architecture with real VRAM budgets and benchmarks. + +## What You Get + +| Directory | Purpose | +|-----------|---------| +| `SKILL.md` | Core methodology, trigger conditions, reference index | +| `references/` | Deep-dive reference files loaded on demand | + +## Triggers + +Setting up fine-tuning runs, quantizing models, selecting training infrastructure, deploying inference servers, or evaluating model quality. + +## Requirements + +Assumes familiarity with PyTorch/HuggingFace ecosystem. References cover vLLM, llama.cpp, TGI, DeepSpeed, and accelerate. + +## Quick Start + +Load SKILL.md for the methodology overview and reference table, then load specific references as needed for the task at hand. diff --git a/ml-engineering/SKILL.md b/ml-engineering/SKILL.md new file mode 100644 index 0000000..8664f1d --- /dev/null +++ b/ml-engineering/SKILL.md @@ -0,0 +1,47 @@ +--- +name: ml-engineering +description: Machine learning engineering methodology — model training, fine-tuning + (LoRA/QLoRA), evaluation, quantization, deployment, and MLOps pipeline design. Grounded + in practical engineering patterns for production ML systems. +license: MIT +metadata: + tags: ml, machine-learning, fine-tuning, training, evaluation, quantization, mlops, + inference, vllm, gguf + source_repo: https://github.com/magnus919/hermes-profiles +--- + +# ML Engineering Methodology + +Machine learning engineering is the bridge between model research and production systems. This methodology covers the engineering disciplines needed to train, evaluate, deploy, and maintain ML models reliably. + +## The ML Engineer's Domain + +| You own | You don't own | +|---------|--------------| +| Model training — LoRA/QLoRA fine-tuning, full fine-tuning, distributed training | Statistical modeling and experimental design — that's the data scientist | +| Model evaluation — benchmark suites, custom eval sets, regression testing | Causal inference and hypothesis testing — that's the data scientist | +| Quantization — GGUF, GPTQ, AWQ, bitsandbytes | Training data collection and labeling — that's the data/ML ops team | +| Inference serving — vLLM, llama.cpp, TGI, Triton | Business metrics and KPI definition — that's the product manager | +| Evaluation harness — lm-eval-harness, custom pipelines | Data pipeline architecture — that's the data engineer | +| Model deployment — containerization, versioning, A/B testing | Infrastructure provisioning — that's the platform engineer | + +## Reference Files + +| Reference | When to load | +|-----------|-------------| +| `references/fine-tuning.md` | Setting up a LoRA/QLoRA/ full fine-tuning run — data prep, hyperparameters, validation strategy | +| `references/evaluation.md` | Evaluating a model — benchmark selection, custom eval sets, regression tracking, comparison methodology | +| `references/quantization-inference.md` | Quantizing a model and serving it — GGUF/GPTQ/AWQ/bitsandbytes comparison, calibration data strategies, KV cache quantization, vLLM/llama.cpp/TGI/Triton architecture, production considerations | +| `references/training-infrastructure.md` | Selecting and provisioning training infrastructure — GPU selection, VRAM budgeting, multi-GPU strategies (DDP/FSDP/DeepSpeed), cloud vs on-prem, storage, monitoring | + +## Core Principles + +**Measure before you optimize** — Never quantize, prune, or distill a model without first measuring its baseline performance. Optimization without measurement is guessing. + +**Reproducibility is non-negotiable** — Every training run needs a reproducible config: seed, data version, hyperparameters, and evaluation methodology. If you can't reproduce it, you can't ship it. + +**Baseline first** — Before running an expensive fine-tuning run, establish a baseline with the base model. If the base model is already good enough, the fine-tuning budget is better spent elsewhere. + +**Test at the boundary** — Model evaluation is most informative at the edges of the capability distribution, not at the center. Hard examples reveal more than easy ones. + +**The evaluation set is a liability** — Every example in your eval set is a potential test-set leak. Use held-out sets, rotate examples, and periodically audit for contamination. diff --git a/ml-engineering/references/evaluation.md b/ml-engineering/references/evaluation.md new file mode 100644 index 0000000..2c87663 --- /dev/null +++ b/ml-engineering/references/evaluation.md @@ -0,0 +1,31 @@ +# Model Evaluation + +## Benchmark Selection + +| What you want to measure | Recommended benchmarks | +|--------------------------|----------------------| +| General reasoning | MMLU-Pro, GPQA, ARC-Challenge | +| Code generation | HumanEval+, SWE-Bench, BigCodeBench | +| Instruction following | MT-Bench, AlpacaEval, Arena-Hard | +| Tool calling | BFCL, ToolBench, jdhodges tool-call eval | +| Safety | TruthfulQA, BBQ, ToxiGen | +| Math | GSM8K, MATH, AIME | + +## Custom Eval Design + +When off-the-shelf benchmarks don't capture your domain: + +1. **Collect 50-200 representative examples** from your actual use cases +2. **Define a scoring rubric** — what constitutes correct, partially correct, and incorrect +3. **Include adversarial examples** — edge cases, ambiguous inputs, known failure modes +4. **Run baseline (base model) first** — establish the ceiling before fine-tuning +5. **Track per-example** — aggregate scores hide regressions in specific capabilities + +## Regression Tracking + +| Before change | After change | Interpretation | +|---|---|---| +| Score A | Score A ± noise | No detectable effect | +| Score A | Score A - delta | Regression — investigate | +| Score A | Score A + delta | Improvement — verify on held-out set | +| Score A, B regressed | Score A + delta, B regressed | Tradeoff — intentional? | diff --git a/ml-engineering/references/fine-tuning.md b/ml-engineering/references/fine-tuning.md new file mode 100644 index 0000000..ed12fbb --- /dev/null +++ b/ml-engineering/references/fine-tuning.md @@ -0,0 +1,32 @@ +# Fine-Tuning + +## Approach Selection + +| Approach | When to use | VRAM | Data needed | +|----------|-------------|------|-------------| +| LoRA | Adapting to a new domain/task, limited VRAM | 8-16 GB for 7B | 100-10K examples | +| QLoRA | Very limited VRAM, large base model | 6-12 GB for 7B | 100-10K examples | +| Full fine-tune | Maximum capability shift, sufficient VRAM | 40-80 GB for 7B | 10K+ examples | +| DoRA | Weight-decomposed adaptation, better than LoRA | +5% over LoRA | Same as LoRA | + +## Training Config Checklist + +- [ ] Seed set for reproducibility +- [ ] Learning rate schedule selected (cosine, linear, constant) +- [ ] Warmup steps configured (typically 5-10% of total) +- [ ] Gradient accumulation steps set +- [ ] Mixed precision (bf16/fp16) enabled if hardware supports +- [ ] Evaluation during training (every N steps) +- [ ] Checkpoint saving with best-model tracking (eval loss) +- [ ] WandB or local logging configured +- [ ] Data shuffled before each epoch +- [ ] Train/validation split verified (no cross-contamination) + +## Validation Strategy + +| Data regime | Validation approach | +|-------------|-------------------| +| < 500 examples | K-fold cross-validation (k=5) | +| 500-10K | 80/10/10 train/val/test split | +| 10K+ | 90/5/5 split with stratified sampling | +| Imbalanced classes | Stratified split by class distribution | diff --git a/ml-engineering/references/quantization-inference.md b/ml-engineering/references/quantization-inference.md new file mode 100644 index 0000000..3c74d10 --- /dev/null +++ b/ml-engineering/references/quantization-inference.md @@ -0,0 +1,912 @@ +# ML Model Quantization & Inference Serving — Reference Manual + +> **Purpose:** Engineering methodology reference for production ML serving. Covers quantization approaches, calibration strategies, quality assessment, mixed precision, KV cache optimization, and inference serving architectures. +> +> **Last updated:** 2025-06-05 + +--- + +## Table of Contents + +1. [Quantization Fundamentals](#1-quantization-fundamentals) +2. [Quantization Approaches Compared](#2-quantization-approaches-compared) + - 2.1 GGUF + - 2.2 GPTQ + - 2.3 AWQ + - 2.4 bitsandbytes NF4 + - 2.5 EXL2 + - 2.6 HQQ + - 2.7 Comparison Table +3. [Calibration Data Strategies](#3-calibration-data-strategies) +4. [Quality Impact Assessment](#4-quality-impact-assessment) +5. [Mixed Precision Patterns](#5-mixed-precision-patterns) +6. [KV Cache Quantization](#6-kv-cache-quantization) +7. [Quantization Workflow](#7-quantization-workflow) +8. [Inference Serving Architecture](#8-inference-serving-architecture) + - 8.1 vLLM + - 8.2 llama.cpp Server + - 8.3 Text Generation Inference (TGI) + - 8.4 Triton Inference Server + - 8.5 Serving Framework Comparison Table +9. [Production Considerations](#9-production-considerations) +10. [References & Further Reading](#10-references--further-reading) + +--- + +## 1. Quantization Fundamentals + +**Quantization** reduces the numerical precision of model weights (and optionally activations / KV cache) from full-precision (FP32, BF16) to lower-bit representations (INT8, INT4, FP8, NF4). This shrinks memory footprint, reduces memory bandwidth pressure, and accelerates inference — especially on bandwidth-bound decode steps. + +### Key concepts + +| Concept | Description | +|---|---| +| **Weight quantization** | Map each weight tensor's values from a high-precision range into a low-bit grid. Reduces model size by 2-4x at common bit widths. | +| **Activation quantization** | Quantize intermediate activations at runtime. Harder than weight quantization due to dynamic range variation. Common in INT8 pipelines (e.g., TensorRT). | +| **KV cache quantization** | Quantize the key-value cache during autoregressive generation. Critical for long-context serving where KV cache dominates GPU memory. | +| **Symmetric vs. asymmetric** | Symmetric: zero point = 0, range is [-max, max]. Asymmetric: zero point can shift. Asymmetric generally preserves more info for non-normalized distributions. | +| **Per-tensor vs. per-channel (per-group)** | Finer granularity (per-channel or per-group) captures outlier distributions better at the cost of storing more scale/zero-point metadata. | +| **Post-training quantization (PTQ)** | Quantize after training. Dominant paradigm for LLMs given cost of training. | +| **Quantization-aware training (QAT)** | Simulate quantization during training (e.g., FakeQuant ops). More accurate but expensive. Used in some production pipelines (e.g., NVIDIA TensorRT). | + +### Why quantize LLMs? + +- **Memory:** A 70B model at FP16 requires ~140 GB VRAM. At INT4, ~35 GB — fitting on a single H100/A100-80GB. +- **Throughput:** Lower-precision weights reduce memory bandwidth consumption, which is the primary bottleneck for autoregressive token generation (memory-bound, not compute-bound). +- **Cost:** Enables deployment on cheaper/consumer hardware (RTX 4090, Apple Silicon unified memory, CPU-only). + +--- + +## 2. Quantization Approaches Compared + +### 2.1 GGUF + +**Type:** File format + quantization scheme +**Ecosystem:** llama.cpp, Ollama, LM Studio +**Bit widths:** Q2_K through Q8_0, plus Q4_K_M, Q5_K_M, Q6_K, etc. + +GGUF (GPT-Generated Unified Format) is the successor to GGML. It packages a model's weights, tokenizer, and metadata into a single file. The quantization variants use a **k-quant** scheme that assigns different bit widths to different layers based on their importance: + +- **Q4_K_M** — recommended sweet spot. Mixture of 4-bit and 6-bit quantization across layers. ~4.5 bits/weight effective. +- **Q5_K_M** — higher quality, ~5.5 bits/weight effective. +- **Q8_0** — near-lossless 8-bit, ~8.5 GB for a 7B model. +- **Q2_K** — aggressive 2-3 bit mix, significant quality loss. + +**Key features:** +- Supports CPU inference natively (no GPU required for modest models). +- Can offload layers to GPU via `--n-gpu-layers`. +- Single-file distribution simplifies deployment. +- Supports a wide range of architectures (Llama, Mistral, Falcon, Gemma, etc.). +- **Imatrix** (importance matrix) quantization: weights are quantized with per-layer importance scores, improving quality at a given bit rate. + +**Strengths:** Universal format, best CPU/edge support, large pre-quantized Hub ecosystem (TheBloke, etc.), excellent for local/offline use. + +**Weaknesses:** Not natively supported by HuggingFace `transformers` or vLLM (though vLLM added GGUF support in 2025). GPU performance trails AWQ/GPTQ on NVIDIA hardware. + +--- + +### 2.2 GPTQ + +**Type:** Post-training quantization (weight only) +**Ecosystem:** AutoGPTQ, HuggingFace optimum, vLLM, TGI +**Bit widths:** 2-8 bits (most common: 4-bit), supports group size 32/64/128 + +GPTQ (GPT Post-Training Quantization) uses approximate second-order optimization (Hessian-based) to find weight quantizations that minimize output error. It was the first widely adopted 4-bit LLM quantization method. + +**How it works:** +1. Sample a calibration dataset (typically 128 sequences from the training distribution). +2. Compute the approximate Hessian (Fisher information) for each weight column. +3. Quantize weights column-by-column, using the Hessian to prioritize preserving important weights. +4. Update remaining unquantized weights to compensate for quantization error (optimal brain quantization / OBC family). + +**Key parameters:** +- **Group size** (g128, g64, g32): Smaller groups = higher accuracy but more scale storage overhead. g128 is common. g64 preferred for quality. +- **Desc_act / act_order** (activation order): When True, reorders columns by activation magnitude. Increases accuracy significantly but reduces speed in older implementations. vLLM's Marlin kernel makes desc_act fast. + +**Strengths:** Strong quality at 4-bit, mature ecosystem, widely supported in serving frameworks. + +**Weaknesses:** Calibration dataset required (cannot quantize a model from scratch without data). Quantization is slower than AWQ or NF4 due to Hessian computation. + +--- + +### 2.3 AWQ + +**Type:** Post-training quantization (weight only) +**Ecosystem:** AutoAWQ, vLLM, TGI, TensorRT-LLM +**Bit widths:** 4-bit (most common), also 2-bit, 3-bit variants + +AWQ (Activation-Aware Weight Quantization) observes that a small fraction (~1%) of weight channels are "salient" — they handle large activations and are disproportionately important. AWQ protects these channels by scaling them up before quantization, then scaling the output down. + +**How it works:** +1. Run a few calibration samples to collect activation statistics. +2. Identify salient channels (those with large activation magnitudes). +3. Apply per-channel scaling factors to redistribute quantization error from salient → non-salient channels. +4. Quantize with simple round-to-nearest. + +**Key advantages:** +- **No group size dependency:** AWQ INT4 often matches GPTQ g128 quality without requiring groups, simplifying kernel implementation. +- **Very fast quantization:** Minutes instead of hours. No Hessian computation needed. +- **Excellent GPU kernel support:** Marlin kernel (for GPTQ-compatible AWQ) and AWQ-specific kernels in vLLM achieve near-peak hardware utilization. +- **Good hardware compatibility:** Works well on NVIDIA, AMD, and Apple Silicon via MLX. + +**Strengths:** Best quality-to-speed tradeoff at 4-bit, fastest quantize time, strong production support in vLLM. + +**Weaknesses:** Primarily designed for 4-bit (less flexible than GGUF's range of bit widths). Requires activation statistics → needs calibration data. + +--- + +### 2.4 bitsandbytes NF4 + +**Type:** Post-training quantization (weight only) +**Ecosystem:** HuggingFace `bitsandbytes`, `transformers`, PEFT/LoRA +**Bit widths:** 4-bit (NF4), 8-bit (INT8) + +Bitsandbytes (BnB) is a library from Tim Dettmers that implements efficient GPU quantization kernels. Its 4-bit variant uses **NormalFloat (NF4)** — a non-uniform quantization grid that assumes normally distributed weights. + +**Key concepts:** +- **NF4:** A 4-bit data type with 16 levels, non-uniformly spaced to match the cumulative distribution function (CDF) of a normal distribution. This gives higher resolution near zero where most weight values cluster. +- **Double quantization:** Quantizes the quantization constants (scale/offset) themselves to save additional memory. Reduces the 4-bit overhead from ~0.5 bits/weight to ~0.127 bits/weight. +- **QLoRA:** Fine-tune quantized models with LoRA adapters. The base model stays in NF4; only the LoRA parameters are updated in FP16. + +**Key features:** +- Native `transformers` integration via `BitsAndBytesConfig` — load any model in 4-bit with a single config object. +- Best for fine-tuning (QLoRA) and rapid prototyping. +- No calibration data needed — quantization is "on the fly" at load time. + +**Strengths:** Simplest API, no calibration required, excellent for fine-tuning, HuggingFace-native. + +**Weaknesses:** Slower inference than AWQ/GPTQ (dequantization at every forward pass). Kernels are less optimized for serving throughput. Not suitable for high-throughput production serving on its own. + +--- + +### 2.5 EXL2 + +**Type:** Post-training quantization +**Ecosystem:** ExLlamaV2 +**Bit widths:** Mixed 2-8 bits per layer + +EXL2 is the quantization format for ExLlamaV2, a high-throughput inference engine. It supports mixed-precision within a single model — different layers can use different bit widths. + +**Key features:** +- Fine-grained per-layer bit allocation for optimal quality/size tradeoffs. +- Very fast GPU inference — ExLlamaV2 kernels are among the fastest for single-batch inference. +- Less widely supported than GGUF/GPTQ/AWQ but excellent for local GPU use. + +**Strengths:** Fastest local GPU inference for many models, flexible per-layer bit allocation. + +**Weaknesses:** Smaller ecosystem, primarily desktop/local use. Not widely supported in production serving frameworks. + +--- + +### 2.6 HQQ + +**Type:** Post-training quantization +**Ecosystem:** HuggingFace, independent +**Bit widths:** 1-8 bits + +HQQ (Half-Quadratic Quantization) uses a half-quadratic splitting approach to compute optimal quantization. It's notable for being extremely fast to quantize (no calibration data, no Hessian) and supporting very low bit widths (2-bit, 3-bit). + +**Strengths:** Fastest PTQ (no data needed), supports 1-3 bits, good quality at lower bits. + +**Weaknesses:** Needs `torch.compile` for reasonable inference speed; otherwise dequantization overhead is high. Less mature ecosystem. + +--- + +### 2.7 Quantization Method Comparison Table + +| Property | GGUF (Q4_K_M) | GPTQ (g128) | AWQ | BnB NF4 | EXL2 | HQQ | +|---|---|---|---|---|---|---| +| **Effective bits/weight** | ~4.5 | ~4.125 | ~4.0 | ~4.127 | Variable | 1-8 | +| **File format** | Single .gguf | HF safetensors | HF safetensors | HF safetensors | Custom | HF safetensors | +| **Calibration data req.** | No* | Yes (128 seq) | Yes (128 seq) | No | Yes | No | +| **Quantize speed** | Fast** | Slow (hrs) | Fast (min) | Instant (load) | Moderate | Very fast | +| **GPU inference speed** | Moderate | Fast (Marlin) | Very fast (Marlin) | Slow | Very fast | Moderate | +| **CPU inference** | Native | No | No | No | No | No | +| **Apple Silicon** | Native (MLX) | Via MLX | Via MLX | No | No | Possible | +| **Serving support** | vLLM, llama.cpp | vLLM, TGI | vLLM, TGI, TRT-LLM | Limited | ExLlamaV2 | Limited | +| **Fine-tuning support** | No | No | No | Yes (QLoRA) | No | Possible | +| **Ecosystem maturity** | Very high | High | High | Very high | Moderate | Low | +| **Typical PPL increase (7B)** | +0.15 | +0.10 | +0.10 | +0.20 | +0.10 | +0.12 | + +\* GGUF quantization typically does not use calibration data; imatrix quantization does. +\** Quantization is an explicit step via `llama-quantize` or similar tools (not on-the-fly). + +--- + +## 3. Calibration Data Strategies + +Some quantization methods (GPTQ, AWQ, EXL2, imatrix) require a **calibration dataset** — a small set of representative text samples used to compute activation statistics or Hessian information. + +### Recommended calibration datasets + +| Dataset | Typical Size | Use Case | +|---|---|---| +| **Wikitext-2** | 128 seq × 2048 tokens | Standard benchmark, general text | +| **C4 (Colossal Clean Crawled Corpus)** | 128-256 seq | General web text, diverse | +| **Pile** | 128-256 seq | General, diverse (books, code, academic) | +| **Custom task-specific** | 128-512 seq | Domain adaptation (medical, legal, code) | +| **Random from training data** | 128 seq | Best if available (closest to training distribution) | + +### Best practices + +1. **Size:** 128-256 sequences of 2048 tokens is typically sufficient. More calibration data has diminishing returns and can even harm quality (overfitting the calibration set). +2. **Diversity:** Calibration data should broadly match the model's training distribution. A model trained on code + text benefits from a calibration mix of both. +3. **Sequences vs. random tokens:** Always use natural text sequences, not random tokens. Random tokens produce meaningless activation statistics. +4. **Avoid duplication:** Deduplicate calibration data. Repeated samples can skew Hessian estimates. +5. **Prompt-like structure:** For instruction-tuned models, including representative prompts in calibration data can improve downstream quality. +6. **Multiple calibration runs:** Some advanced pipelines run calibration on multiple small datasets and average the quantized parameters. + +### When calibration data matters most + +- **GPTQ** — critically important. Poor calibration data leads to significantly higher perplexity. +- **AWQ** — important but more robust than GPTQ. The scaling factor approach is less sensitive to calibration data quality. +- **GGUF imatrix** — uses importance matrices computed from calibration data. Worth the effort for best-quality GGUF quants. +- **NF4 / HQQ** — no calibration data needed. + +--- + +## 4. Quality Impact Assessment + +### Quantization degradation patterns + +| Bit Width | Quality Impact | +|---|---| +| **FP16 / BF16** | Baseline (lossless reference) | +| **INT8 (Q8_0, BnB INT8)** | Near-lossless. Negligible PPL increase (<0.01). Output-level differences often undetectable. | +| **6-bit (Q6_K)** | Very minor PPL increase (~0.02). Safe for production. | +| **5-bit (Q5_K_M, Q5_0)** | Small PPL increase (~0.05-0.15). Generally safe. | +| **4-bit (Q4_K_M, GPTQ, AWQ, NF4)** | Moderate PPL increase (~0.10-0.35). Noticeable on complex reasoning tasks. AWS/GPTQ typically best, NF4 worst at same bit width. | +| **3-bit (Q3_K_S, HQQ int3)** | Significant degradation. PPL +0.5-1.5. Tasks requiring multi-step reasoning (CoT) degrade notably. | +| **2-bit (Q2_K)** | Heavy degradation. Only usable for very tolerant tasks. | + +### Task-level sensitivity + +Not all tasks degrade equally: + +| Task Type | Sensitivity | Notes | +|---|---|---| +| **Perplexity / next-token prediction** | Low | Relatively robust to quantization. | +| **Single-token classification (MMLU)** | Low-Moderate | 4-bit typically loses 1-2% accuracy. | +| **Multi-step reasoning (CoT, MATH)** | High | 4-bit can lose 3-5%+ on math reasoning. 3-bit often fails entirely. | +| **Code generation** | Moderate | Functional correctness degrades at aggressive quantization. | +| **Creative writing** | Low | Quality differences are subtle at 4-bit; 3-bit may produce incoherence. | +| **Instruction following** | Moderate | Longer, multi-step instructions become harder at lower precision. | +| **Few-shot learning** | Moderate | Degrades faster than zero-shot performance. | + +### Empirical data (Llama 3 8B, from LessWrong benchmarks) + +| Method | MMLU (0-shot) | WMDP | The Pile PPL | +|---|---|---|---| +| BF16 (baseline) | 63.87% | 54.99% | 8.283 | +| BnB INT8 | 63.05% | 54.96% | 8.305 | +| HQQ INT8 | 63.87% | 54.66% | 8.298 | +| AWQ INT4 | 61.84% | 54.55% | 8.483 | +| HQQ INT4 | 62.29% | 54.23% | 8.482 | +| GPTQ INT4 | 61.58% | 53.30% | 8.575 | +| BnB NF4 | 61.44% | 54.42% | 8.499 | +| BnB INT4 | 60.80% | 52.73% | 8.633 | +| HQQ INT3 | 62.26% | 51.23% | 8.872 | + +> **Key takeaway:** AWQ = GPTQ > HQQ > NF4 > BnB INT4 at 4-bit. At 8-bit, all methods are essentially lossless. Differential sensitivity across tasks means eval should always be task-specific. + +### Recommended evaluation framework + +1. **Perplexity** — quick sanity check. Compute on withheld validation split (100k tokens minimum). +2. **Task-specific accuracy** — MMLU, HumanEval, GSM8K, or domain-specific benchmarks. +3. **A/B comparison** — Run paired generations from FP16 and quantized model. Human eval or LLM-as-judge for quality differences. +4. **Downstream metric** — For RAG systems, measure retrieval precision. For chatbots, measure response acceptability. + +--- + +## 5. Mixed Precision Patterns + +Mixed precision assigns different numerical precisions to different parts of the model or computation graph. This is distinct from per-layer variable-width quantization. + +### Common mixed precision patterns + +#### 5.1 Weight quantization + high-precision compute + +``` +Weights: INT4 / NF4 +Activations: FP16 / BF16 +Gradients (training): FP32 +``` + +- Most common pattern for LLM inference. +- Weights are dequantized on-the-fly to FP16 for computation. +- Offered by AWQ, GPTQ, BnB, GGUF. +- **Tradeoff:** Dequantization overhead. AWQ/GPTQ minimize this via fused kernels (Marlin). + +#### 5.2 Low-precision compute (FP8 matmul) + +``` +Weights: FP8 +Activations: FP8 +Accumulation: FP16/FP32 +``` + +- NVIDIA H100/H200 supports native FP8 tensor cores (2x throughput vs. FP16). +- TensorRT-LLM and vLLM (FP8 support) use this for high-throughput serving. +- Activation ranges are calibrated (per-tensor or per-channel) at export time. +- Quantization-aware scaling (QTS) ensures accuracy. + +#### 5.3 INT8 compute with INT4 weights (W4A8) + +``` +Weights: INT4 +Activations: INT8 +Compute: INT8 tensor cores +``` + +- Emerging pattern for maximum throughput on hardware with INT8 tensor cores (all NVIDIA GPUs since Volta). +- Requires activation quantization at inference time — more complex. +- Used by TensorRT-LLM and some custom serving stacks. + +#### 5.4 Per-layer variable precision + +``` +Layer 1: Q4_K +Layer 5: Q6_K +Layer 14: Q5_K +... +``` + +- GGUF k-quant and EXL2 use this pattern. +- Sensitive layers (e.g., embedding, lm_head, early/late transformer layers) get higher precision. +- Reduces average bit width without sacrificing critical layers. + +#### 5.5 FP16 weights + INT8 KV cache + +``` +Weights: FP16 +KV Cache: INT8/FP8 +Compute: FP16/BF16 +``` + +- KV cache is the memory bottleneck for long contexts. +- Quantizing only the KV cache (not weights) saves 50-75% of KV cache memory. +- Supported by vLLM and TensorRT-LLM. + +### Precision selection decision tree + +``` +Is model size > GPU VRAM? +├── YES → Can we tolerate quality loss? +│ ├── YES → INT4 weight quantization (AWQ/GPTQ) + optional KV cache quant +│ └── NO → FP8 weight quant (if H100) or BF16 with tensor parallelism +└── NO → Is latency critical? + ├── YES → INT4 weights + FP16 activations (Marlin kernel) + └── NO → BF16 baseline is fine +``` + +--- + +## 6. KV Cache Quantization + +### The KV cache problem + +During autoregressive generation, each transformer layer computes Key (K) and Value (V) tensors that are cached for all previous tokens. For a batch of size `b`, `n_layers` layers, `n_heads` attention heads, sequence length `s`, and dimension `d_per_head`: + +``` +KV cache size = 2 × b × n_layers × n_heads × s × d_per_head × precision_bytes +``` + +At FP16, a 32K-token sequence with Llama 3 70B (80 layers, 8 KV heads, d=128) requires ~320 GB for the KV cache alone. Quantizing to INT8 halves this; to INT4 quarters it. + +### Attention architecture impact + +| Architecture | KV Cache per token (FP16) | Notes | +|---|---|---| +| **MHA** (Multi-Head Attention) | 2 × n_layers × n_heads × d | Largest cache. Every layer has full key/value for all heads. | +| **MQA** (Multi-Query Attention) | 2 × n_layers × 1 × d | One KV head shared across all query heads. 8-32x smaller than MHA. | +| **GQA** (Grouped-Query Attention) | 2 × n_layers × n_kv_heads × d | Middle ground. Llama 2/3 uses 8 KV heads for 32+ query heads. | +| **MLA** (Multi-Head Latent Attention) | 2 × n_layers × d_latent | DeepSeek's approach. Compresses KV into a low-rank latent space. ~2-4x smaller than GQA. | + +### KV cache quantization methods + +| Method | Bit Width | Strategy | +|---|---|---| +| **KVTuner** | INT4/INT8 | Sensitivity-aware per-layer mixed-precision. Key layers get 8-bit, others 4-bit. | +| **KVQuant** | INT4/FP8 | Per-channel + per-token quantization with non-uniform grids. Targets 10M+ context. | +| **FP8 KV cache** (H100 native) | FP8 | Uses H100 FP8 tensor cores. Minimal quality loss. | +| **INT8 KV cache** (vLLM) | INT8 | Per-tensor symmetric quantization. Standard vLLM feature. | +| **INT4 KV cache** (experimental) | INT4 | Per-channel asymmetric. Quality loss noticeable at very long contexts. | + +### Production guidelines + +1. **Start with GQA/MLA architecture** — architecture-level KV cache reduction is more impactful than quantization. +2. **FP8 KV cache on H100** — essentially lossless, 2x memory reduction. Enable in vLLM with `--kv-cache-dtype fp8`. +3. **INT8 KV cache** — good tradeoff for A100/H100. Minimal quality impact for contexts under 32K tokens. +4. **INT4 KV cache** — quality impact grows with sequence length. Evaluate carefully for long-context applications. +5. **Layer-wise KV quantization** — tools like KVTuner offer better quality at same average bit width by allocating higher precision to critical layers. + +--- + +## 7. Quantization Workflow + +The canonical workflow for applying and validating quantization in production: + +### Step 1: Establish Baseline + +- Load the model in FP16/BF16. +- Run evaluation benchmark (MMLU, perplexity, domain-specific tasks). +- Record latency and throughput at relevant batch sizes. +- Record VRAM usage. +- This is the reference against which all quantized variants are compared. + +### Step 2: Select Quantization Method + +Use the comparison table (Section 2.7) to choose based on: +- **Target hardware** (CPU? GPU? Apple Silicon? Cloud instance type?) +- **Deployment framework** (vLLM? llama.cpp? TGI? TensorRT-LLM?) +- **Quality constraints** (must match FP16 within X%?) +- **Compute budget** (time available for quantization) + +### Step 3: Quantize + +```bash +# GGUF (via llama.cpp) +python3 convert.py --outtype f16 --model ./model --outpath model-f16.gguf +./llama-quantize model-f16.gguf model-q4km.gguf Q4_K_M + +# GPTQ (via AutoGPTQ) +python3 -m auto_gptq --model ./model --quantize --bits 4 --group-size 128 --dataset c4 + +# AWQ (via AutoAWQ) +python3 -m awq.quantize --model_path ./model --quant_path ./awq-model --calib-data wikitext + +# BnB NF4 (via transformers — on-the-fly) +# Simply load with BitsAndBytesConfig +``` + +### Step 4: Evaluate Quality + +- **Primary metric:** Same evaluation benchmark as baseline (Step 1). +- **Secondary metric:** Perplexity on a held-out validation set (e.g., The Pile test split). +- **Tertiary metric:** A/B test with LLM-as-judge for generative tasks. +- **Threshold:** Define acceptable degradation (e.g., < 0.5% MMLU drop, < 0.3 PPL increase). + +### Step 5: Benchmark Performance + +- Measure tokens/second at batch size 1 (latency-sensitive). +- Measure throughput at max batch size (throughput-sensitive). +- Record peak VRAM usage. +- Compare to baseline and to alternative quantization methods. + +### Step 6: Select the Winner + +| Criteria | Decision | +|---|---| +| Quality within threshold, best throughput | Choose that method | +| Quality outside threshold | Try higher-precision variant (Q4_K_M → Q5_K_M; g128 → g64) | +| All methods fail threshold | Consider FP16 with tensor parallelism, or switch to a different architecture | +| Throughput insufficient | Consider FP8 (if H100) or lower-precision quant with faster kernel | + +### Step 7: Production Deployment + +- Store quantized model in model registry. +- Configure inference server with appropriate settings. +- Monitor quality metrics continuously (drift detection). +- Set up A/B test vs. previous version. + +--- + +## 8. Inference Serving Architecture + +### 8.1 vLLM + +**Developed by:** UC Berkeley (Kwatra, Stoica) +**Language:** Python/C++/CUDA +**GitHub:** github.com/vllm-project/vllm +**License:** Apache 2.0 + +vLLM is the most widely adopted open-source LLM serving framework, known for its combination of throughput and flexibility. + +#### Core innovations + +**PagedAttention** +- Inspired by OS virtual memory paging. +- KV cache is divided into fixed-size **blocks** (typically 16 or 32 tokens each). +- Blocks are stored in a non-contiguous page table, eliminating fragmentation. +- Enables near-zero memory waste vs. the 60-80% waste in traditional pre-allocated KV cache. +- Allows memory sharing across sequences for techniques like beam search and parallel sampling. + +**Continuous Batching** +- Also called **in-flight batching** or **iteration-level batching**. +- Traditional servers wait for all sequences in a batch to finish before starting a new batch. +- Continuous batching adds/removes sequences from the batch **after every iteration** (every decoding step). +- Dramatically improves GPU utilization, especially when sequences have variable lengths. +- vLLM achieves up to **24x higher throughput** than HuggingFace Transformers on the same hardware. + +**Chunked Prefill** +- Splits long prefill (prompt processing) into smaller chunks that can interleave with decode steps. +- Prevents long prompts from blocking decode-only sequences. +- Reduces time-to-first-token (TTFT) variability. + +**Speculative Decoding** +- Uses a small draft model to propose multiple tokens, verified by the target model in one forward pass. +- 1.5-2.5x latency improvement on latency-sensitive workloads. + +#### Key features + +| Feature | Status | Notes | +|---|---|---| +| AWQ quantization | ✅ Native | Marlin kernel support | +| GPTQ quantization | ✅ Native | Marlin kernel support | +| GGUF quantization | ✅ Added 2025 | Via llama.cpp backend | +| FP8 (H100) | ✅ Native | Requires H100 | +| KV cache INT8/FP8 | ✅ Native | `--kv-cache-dtype` flag | +| Tensor parallelism | ✅ | Across GPU nodes | +| Pipeline parallelism | ✅ | Limited | +| Prefix caching | ✅ | Automatic KV cache reuse | +| OpenAI-compatible API | ✅ | Drop-in replacement | +| Multi-LoRA serving | ✅ | Efficient LoRA adapter switching | +| Guided decoding | ✅ | JSON schema, grammar | +| Disaggregated prefill/decode | ✅ | 2025 feature | + +#### Typical deployment + +```bash +# Start vLLM server +python3 -m vllm.entrypoints.openai.api_server \ + --model /path/to/model \ + --quantization awq \ + --dtype auto \ + --max-model-len 8192 \ + --gpu-memory-utilization 0.90 \ + --tensor-parallel-size 2 \ + --enable-prefix-caching +``` + +#### When to choose vLLM + +- **High-throughput production serving** (chatbots, API endpoints). +- **Multi-model or multi-LoRA setups**. +- **OpenAI-compatible API needed**. +- **Heterogeneous GPU setups** (supports various NVIDIA GPUs, AMD ROCm). +- **Need for speculative decoding or prefix caching**. + +--- + +### 8.2 llama.cpp Server + +**Developed by:** Georgi Gerganov & community +**Language:** C/C++ +**GitHub:** github.com/ggml-org/llama.cpp +**License:** MIT + +llama.cpp is a C/C++ inference engine focused on local/edge deployment with minimal dependencies. The `llama-server` component provides an HTTP API. + +#### Architecture + +- **No external dependencies** — pure C/C++ implementation with BLAS-optimized matrix operations. +- **ggml backend** — custom tensor library supporting CPU, CUDA, Metal, Vulkan, SYCL, and more. +- **Pure CPU inference** — unique among major serving frameworks. Can run 7B models at 10-20 tok/s on modern CPUs with AVX2. +- **GPU offloading** — `--n-gpu-layers N` offloads N transformer layers to GPU. The rest runs on CPU. +- **Quantization-native** — designed from the ground up for GGUF quantized models. + +#### Key features + +| Feature | Status | Notes | +|---|---|---| +| GGUF quantization | ✅ Native | Full k-quant suite | +| AWQ/GPTQ | ❌ Not native | Via conversions | +| CPU inference | ✅ Best-in-class | AVX2, AVX-512, NEON | +| GPU offloading | ✅ | CUDA, Metal, Vulkan | +| Batch inference | ✅ | Server mode with continuous batching | +| KV cache reuse | ✅ | Automatic | +| OpenAI-compatible API | ✅ | Built into `llama-server` | +| Grammar sampling | ✅ | GBNF grammar engine | +| Embedding endpoint | ✅ | Via `/v1/embeddings` | +| Vision (multimodal) | ✅ | Llava, etc. | +| Structured output | ✅ | JSON schema mode | + +#### Typical deployment + +```bash +./llama-server \ + --model /path/to/model.gguf \ + --host 0.0.0.0 \ + --port 8080 \ + --n-gpu-layers 32 \ + --threads $(nproc) \ + --ctx-size 8192 \ + --rope-scaling yarn \ + --cache-type-k q8_0 \ + --cache-type-v q8_0 +``` + +#### When to choose llama.cpp + +- **CPU-only or hybrid CPU/GPU deployments**. +- **Apple Silicon** (Metal backend is excellent). +- **Local/edge inference** (privacy-sensitive, offline). +- **Single-user or low-concurrency serving**. +- **Experimentation** (fastest iteration for trying different quant levels). +- **No dependency on Python or CUDA toolkit**. + +--- + +### 8.3 Text Generation Inference (TGI) + +**Developed by:** Hugging Face +**Language:** Rust/Python +**GitHub:** github.com/huggingface/text-generation-inference +**License:** Apache 2.0 + +TGI is Hugging Face's production-grade inference server, used to power HuggingChat and the Hugging Face Inference API. + +#### Architecture + +- **Rust core** for HTTP routing and request management (high concurrency, low overhead). +- **Python/CUDA backend** for model execution. +- **Flash Attention 2** integration for efficient attention computation. +- **PagedAttention** added in v2.x (also called "Paged Attention in TGI"). +- **Continuous batching** similar to vLLM. +- **Safetensors** and `transformers` integration — loads models directly from Hugging Face Hub. + +#### Key features + +| Feature | Status | Notes | +|---|---|---| +| AWQ quantization | ✅ | Via optimum | +| GPTQ quantization | ✅ | Via optimum | +| FP8 quantization | ✅ | H100 support | +| Bitsandbytes | ✅ | Via transformers | +| Tensor parallelism | ✅ | | +| Flash Attention 2 | ✅ | Default | +| PagedAttention (v2.x) | ✅ | Added after vLLM | +| Watermarking | ✅ | SynthID-Text | +| Message API | ✅ | Native chat templates | +| Streaming | ✅ | Server-Sent Events | +| Speculative decoding | ✅ | | + +#### Typical deployment + +```bash +docker run --gpus all \ + -p 8080:80 \ + -v /path/to/models:/data \ + ghcr.io/huggingface/text-generation-inference:latest \ + --model-id /data/model \ + --max-total-tokens 8192 \ + --quantize awq \ + --num-shard 2 +``` + +#### When to choose TGI + +- **Deep HuggingFace ecosystem integration** (Hub, optimum, tokenizers). +- **Production serving with AWS Inferentia** (TGI has native Inferentia2 support). +- **Message-based chat APIs** (native chat template handling). +- **When watermarking or model-level guardrails are needed**. + +--- + +### 8.4 Triton Inference Server + +**Developed by:** NVIDIA +**Language:** C++/CUDA (backend), Python (frontend) +**GitHub:** github.com/triton-inference-server/server +**License:** BSD-3-Clause + +Triton is NVIDIA's production inference server. It is model-framework-agnostic and designed for enterprise-grade deployments. + +#### Architecture + +- **Multi-framework backend:** Supports TensorRT, TensorRT-LLM, PyTorch, ONNX Runtime, vLLM, Python, and custom backends. +- **Concurrent model serving:** Multiple models (and multiple versions of the same model) served from a single instance. +- **Ensemble scheduler:** Chain multiple models together without custom code (e.g., embedding → re-rank → LLM). +- **Dynamic batching:** Client-side and server-side batching with configurable timeouts. +- **GPU/CPU/accelerator support:** Concurrent serving across heterogeneous hardware. +- **Prometheus metrics:** Native monitoring endpoint. + +#### TensorRT-LLM backend + +The TensorRT-LLM backend is the primary LLM serving path within Triton: + +1. **Model optimization phase:** Convert model to TensorRT engine (FP16, INT8, INT4, FP8). +2. **Graph optimizations:** Kernel fusion, layer fusion, attention optimization. +3. **In-flight batching:** Equivalent to continuous batching. +4. **PagedAttention:** Adopted from vLLM's approach. +5. **Multi-node tensor parallelism:** Up to hundreds of GPUs. + +#### Key features + +| Feature | Status | Notes | +|---|---|---| +| Multi-framework | ✅ | Not just LLMs | +| TensorRT-LLM backend | ✅ | Highest throughput on H100 | +| INT4/FP8/INT8 quantization | ✅ | Through TensorRT | +| PagedAttention | ✅ | Via TensorRT-LLM | +| In-flight batching | ✅ | | +| Dynamic batching | ✅ | Server-side | +| Ensemble inference | ✅ | Pipeline multiple models | +| Concurrent model versions | ✅ | A/B test, gradual rollout | +| Model repository | ✅ | Pull models at startup | +| Prometheus monitoring | ✅ | | +| Custom metrics | ✅ | | +| Decoupled API | ✅ | Streaming responses | +| Request prioritization | ✅ | QoS support | + +#### Typical deployment + +```yaml +# Model repository structure +model_repository/ + ensemble_model/ + 1/ + model.py (ensemble definition) + tensorrt_llm/ + 1/ + config.pbtxt + model.engine + embedding_model/ + 1/ + config.pbtxt + model.plan +``` + +```bash +docker run --gpus all --shm-size=4g \ + -p 8000:8000 -p 8001:8001 -p 8002:8002 \ + -v /path/to/model_repo:/models \ + nvcr.io/nvidia/tritonserver:24.12-trtllm-python-py3 \ + tritonserver --model-repository=/models +``` + +#### When to choose Triton + +- **Enterprise production serving** (SLOs, multi-model, heterogeneous hardware). +- **Multi-model pipelines** (embed → re-rank → generate). +- **Multi-framework environments** (mixing TensorRT, PyTorch, ONNX). +- **High-performance LLM serving on H100/H200 clusters** (TensorRT-LLM path). +- **Need for request prioritization, A/B testing, multi-version serving**. +- **Kubernate-native deployments** (Triton has first-class K8s support). + +--- + +### 8.5 Serving Framework Comparison + +| Property | vLLM | llama.cpp Server | TGI | Triton + TRT-LLM | +|---|---|---|---|---| +| **Language** | Python/C++/CUDA | C/C++ | Rust/Python/CUDA | C++/CUDA | +| **Primary hardware** | NVIDIA GPU (+ AMD, Intel) | CPU, Apple, any GPU | NVIDIA GPU | NVIDIA GPU | +| **Best quantization** | AWQ, GPTQ, FP8 | GGUF (all k-quants) | AWQ, GPTQ, FP8 | INT4/FP8 via TRT | +| **Throughput (7B)** | Very high | Moderate | High | Highest (on H100) | +| **Latency (single request)** | Low | Low | Low | Very low | +| **CPU-only support** | No | Yes (best) | No | No | +| **Apple Silicon** | No | Yes (Metal) | No | No | +| **Multi-model serving** | Limited | No (one model) | No (one model) | Yes (full) | +| **Ensemble pipelines** | No | No | No | Yes | +| **OpenAI API compat** | ✅ Native | ✅ Built-in | ✅ Native | Requires NIM | +| **Ecosystem** | OSS community | OSS community | HuggingFace | NVIDIA | +| **License** | Apache 2.0 | MIT | Apache 2.0 | BSD-3 | + +--- + +## 9. Production Considerations + +### Model registry & versioning + +- Store quantized models alongside their FP16 originals in a model registry (e.g., MLflow, HuggingFace Hub, S3). +- Tag each quantized model with: base model version, quantization method, bit width, calibration dataset, validation metrics. +- Never overwrite a quantized model — always create a new version. + +### A/B testing in production + +- Serve both FP16 and quantized variants simultaneously. +- Route a fraction of traffic to each variant. +- Compare quality (user feedback, downstream metrics), latency (p50, p95, p99), and throughput. +- Gradual rollout: 5% → 25% → 50% → 100%. + +### Monitoring + +| Metric | What to Watch | Alert Threshold | +|---|---|---| +| **p50/p99 TTFT** | Time to first token | +30% from baseline | +| **p50/p99 TPOT** | Time per output token | +20% from baseline | +| **Throughput** | Tokens/second | <80% of expected | +| **GPU memory utilization** | VRAM usage | >95% persistent | +| **KV cache utilization** | vs. allocated | >90% (good) | +| **Error rate** | 4xx/5xx responses | >1% | +| **Perplexity (eval)** | Quality drift | +0.5 from baseline | +| **Generation quality** | LLM-as-judge or human eval | Periodic | + +### Hardware selection guide + +| Deployment | Recommended Hardware | Recommended Setup | +|---|---|---| +| **Single user, local** | RTX 4090 (24 GB) | 7-13B, Q4_K_M GGUF, llama.cpp | +| **Low concurrency API** | A100-40GB or RTX 6000 | 7-13B, AWQ, vLLM | +| **Mid-scale production** | A100-80GB (x2-4) | 70B, AWQ/GPTQ, vLLM, TP=2-4 | +| **High-scale production** | H100-80GB (x8+) | 70B-405B, FP8/INT4, TRT-LLM, TP=8 | +| **Edge / CPU-only** | Modern x86 with AVX-512 | 7B, Q4_K_M, llama.cpp | +| **Apple Silicon** | M2 Ultra / M4 Ultra | 7-13B, GGUF, llama.cpp Metal | +| **Cost-sensitive** | L4 (24 GB) | 7-13B, AWQ, vLLM | + +### Memory budget calculation + +For a model with `P` parameters, quantized to `B` bits/weight: + +``` +Model weights: P × B / 8 bytes +KV cache: 2 × n_layers × n_kv_heads × head_dim × max_seq_len × 2 (FP16) bytes +Activations: ~20% of model weights (rough estimate) +Overhead: CUDA context, framework, ~1-2 GB +``` + +Example — Llama 3 70B, AWQ INT4, seq_len 8192, batch_size 1: +``` +Weights: 70B × 0.5 = ~35 GB +KV cache: 2 × 80 × 8 × 128 × 8192 × 2 = ~2.7 GB +Activations + overhead: ~8 GB +Total: ~46 GB → fits on a single A100-80GB or H100 +``` + +### Cold start / warm-up + +- Quantized models may produce garbage tokens for the first few inference steps (cold-start artifacts). +- Always run a warm-up prompt (e.g., "Hello") before production traffic. +- For serverless deployments, keep a warm standby or use model repository pre-loading. + +### Throughput vs. latency tradeoffs + +| Configuration | TTFT | TPOT | Throughput | Use Case | +|---|---|---|---|---| +| Batch size 1 | Lowest | Moderate | Lowest | Real-time chat | +| Max batch, parallel | Higher | Higher | Highest | Offline batch | +| Chunked prefill | Moderate | Moderate | High | Mixed workloads | +| Speculative decoding | Low | Low | Moderate | Latency-sensitive | + +### Security considerations + +- **GGUF models are executable files** — only load from trusted sources. A malicious GGUF can execute arbitrary code. +- **Safetensors** (used by AWQ/GPTQ) are safer but not invulnerable. +- Validate model provenance: + - Check SHA256 hashes against published values. + - Only load from trusted registries (HuggingFace verified orgs, internal registry). +- Harden the inference server: + - Run as non-root user. + - Use network isolation (no external access for the server). + - Rate-limit API endpoints. + +--- + +## 10. References & Further Reading + +### Foundational papers + +- **GPTQ:** Frantar et al., "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers" (2023) — [arXiv:2210.17323](https://arxiv.org/abs/2210.17323) +- **AWQ:** Lin et al., "AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration" (2024) — [arXiv:2306.00978](https://arxiv.org/abs/2306.00978) +- **Bitsandbytes / NF4 / QLoRA:** Dettmers et al., "QLoRA: Efficient Finetuning of Quantized Language Models" (2023) — [arXiv:2305.14314](https://arxiv.org/abs/2305.14314) +- **PagedAttention:** Kwon et al., "Efficient Memory Management for Large Language Model Serving with PagedAttention" (2023) — [arXiv:2309.06180](https://arxiv.org/abs/2309.06180) +- **vLLM performance analysis:** "Anatomy of a High-Throughput LLM Inference System" (2025) — [vLLM Blog](https://vllm.ai/blog/2025-09-05-anatomy-of-vllm) +- **KV cache quantization (KVTuner):** Liu et al., "KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization" (2025) — [OpenReview](https://openreview.net/forum?id=zDwipF6h06) +- **MLA:** "TransMLA: Multi-head Latent Attention Is All You Need" (2025) — [arXiv:2502.07864](https://arxiv.org/abs/2502.07864) + +### Guides & benchmarks + +- "Which Quantization Method is Right for You (GPTQ vs. GGUF vs. AWQ)" — [Maarten Grootendorst](https://newsletter.maartengrootendorst.com/p/which-quantization-method-is-right) +- "Comparing Quantized Performance in Llama Models" (2024) — [LessWrong](https://www.lesswrong.com/posts/qmPXQbyYA66DuJbht/comparing-quantized-performance-in-llama-models) +- "The Complete Guide to LLM Quantization with vLLM" (2026) — [Jarvis Labs](https://jarvislabs.ai/blog/vllm-quantization-complete-guide-benchmarks) +- "An Empirical Study of Qwen3 Quantization" (2025) — [arXiv:2505.02214](https://arxiv.org/abs/2505.02214) +- "LLM Inference at scale with TGI" (2024) — [HuggingFace Blog](https://huggingface.co/blog/martinigoyanes/llm-inference-at-scale-with-tgi) +- "Continuous Batching: The Single Biggest GPU Utilization Unlock" (2026) — [Tian Pan](https://tianpan.co/blog/2026-04-09-continuous-batching-llm-inference) + +### Tools & repositories + +- **vLLM** — [github.com/vllm-project/vllm](https://github.com/vllm-project/vllm) +- **llama.cpp** — [github.com/ggml-org/llama.cpp](https://github.com/ggml-org/llama.cpp) +- **TGI** — [github.com/huggingface/text-generation-inference](https://github.com/huggingface/text-generation-inference) +- **TensorRT-LLM** — [github.com/NVIDIA/TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM) +- **Triton Inference Server** — [github.com/triton-inference-server/server](https://github.com/triton-inference-server/server) +- **AutoGPTQ** — [github.com/PanQiWei/AutoGPTQ](https://github.com/PanQiWei/AutoGPTQ) +- **AutoAWQ** — [github.com/casper-hansen/AutoAWQ](https://github.com/casper-hansen/AutoAWQ) +- **bitsandbytes** — [github.com/bitsandbytes-foundation/bitsandbytes](https://github.com/bitsandbytes-foundation/bitsandbytes) +- **ExLlamaV2** — [github.com/turboderp/exllamav2](https://github.com/turboderp/exllamav2) +- **Awesome LLM Quantization** — [github.com/pprp/awesome-llm-quantization](https://github.com/pprp/awesome-llm-quantization) + +--- + +> **End of reference document.** This is a living document — update benchmarks and framework versions as the ecosystem evolves. diff --git a/ml-engineering/references/training-infrastructure.md b/ml-engineering/references/training-infrastructure.md new file mode 100644 index 0000000..0ef07c9 --- /dev/null +++ b/ml-engineering/references/training-infrastructure.md @@ -0,0 +1,102 @@ +# Training Infrastructure + +## GPU Selection + +| GPU | VRAM | Best For | Notes | +|-----|------|----------|-------| +| RTX 4090 | 24 GB | LoRA/QLoRA 7B–13B, inference | Consumer, no NVLink | +| RTX 5090 | 32 GB | LoRA 13B–30B, QLoRA 70B | Consumer, no NVLink | +| A6000 | 48 GB | Full fine-tune 7B, LoRA 30B–70B | Prosumer, NVLink pair | +| A100 80GB | 80 GB | Full fine-tune 13B–30B, multi-GPU | Datacenter, NVLink | +| H100 80GB | 80 GB | Full fine-tune 30B–70B, RLHF | Datacenter, NVLink, FP8 | +| H200 141GB | 141 GB | Full fine-tune 70B+, long context | Datacenter, NVLink | + +## VRAM Budgeting + +Rule of thumb for training memory (mixed precision, AdamW): + +``` +VRAM ≈ params × (2 + 2 + 4 + 4) bytes [weights + grads + optimizer states] + + activation memory (batch-dependent) +``` + +| Technique | VRAM multiplier | Example: 7B model | +|-----------|----------------|-------------------| +| Full fine-tune (FP16 + Adam) | ~16× params | ~112 GB | +| LoRA (rank 64) | ~2.5× params | ~18 GB | +| QLoRA (4-bit + LoRA) | ~0.8× params | ~6 GB | +| Inference only (FP16) | ~2× params | ~14 GB | +| Inference only (4-bit) | ~0.5× params | ~4 GB | + +## Multi-GPU Training + +| Strategy | When | Framework | +|----------|------|-----------| +| DataParallel (DP) | Single node, quick experiments | PyTorch native | +| DistributedDataParallel (DDP) | Single node, production training | `torchrun --nproc_per_node=N` | +| FSDP / DeepSpeed ZeRO | Model doesn't fit one GPU | `accelerate`, DeepSpeed config | +| Pipeline parallelism | Very large models (>70B) | DeepSpeed, Megatron-LM | +| Tensor parallelism | Latency-critical inference | vLLM, TensorRT-LLM | + +### DDP Launch Pattern + +```bash +torchrun --nproc_per_node=4 --master_port=29500 train.py \ + --model_name meta-llama/Llama-3-8B \ + --per_device_train_batch_size 4 \ + --gradient_accumulation_steps 8 \ + --bf16 true +``` + +Effective batch size = `per_device × nproc × grad_accum` = 4 × 4 × 8 = 128. + +### FSDP Config (accelerate) + +```yaml +# accelerate_config.yaml +compute_environment: LOCAL_MACHINE +distributed_type: FSDP +fsdp_config: + fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP + fsdp_sharding_strategy: FULL_SHARD # ZeRO-3 equivalent + fsdp_state_dict_type: SHARDED_STATE_DICT + fsdp_cpu_ram_efficient_loading: true +mixed_precision: bf16 +num_processes: 4 +``` + +## Cloud vs On-Prem Decision + +| Factor | Cloud (Lambda, RunPod, Vast) | On-Prem | +|--------|------------------------------|---------| +| Utilization < 30% | ✅ Pay per hour | ❌ Idle hardware | +| Utilization > 60% | ❌ Expensive at scale | ✅ Amortizes in ~8 months | +| Data sensitivity | ❌ Data leaves premises | ✅ Stays local | +| Burst capacity | ✅ Scale to 8×H100 on demand | ❌ Fixed ceiling | +| Ops burden | ❌ Zero (managed) | ✅ You maintain cooling, power, drivers | +| Experiment velocity | ✅ Spin up, tear down | ⚠️ Queue contention on shared cluster | + +### Cloud Cost Reference (spot/on-demand, 2025) + +| GPU | On-demand $/hr | Spot $/hr | +|-----|----------------|-----------| +| A100 80GB | $1.80–$2.50 | $0.90–$1.40 | +| H100 80GB | $3.50–$5.00 | $2.00–$3.00 | +| RTX 4090 | $0.40–$0.70 | $0.25–$0.45 | + +## Storage and Data Pipeline + +- Training data on NVMe or tmpfs — network storage stalls GPUs +- Checkpoints to object storage (S3/GCS) or NAS — never only local disk +- Use `safetensors` format — faster load, no pickle security risk +- Pre-tokenize datasets for large corpora — tokenization at load time wastes GPU-hours + +## Monitoring Training + +| Metric | Healthy Range | Red Flag | +|--------|--------------|----------| +| GPU utilization | > 90% | < 70% = data pipeline bottleneck | +| GPU memory | Stable after warmup | Growing = leak (check grad accumulation) | +| Loss curve | Smooth decrease | Spikes = LR too high; plateau = converged or stuck | +| Grad norm | Stable or decreasing | Exploding = reduce LR or add clipping | +| Throughput (samples/sec) | Consistent | Degrading = thermal throttle or I/O | diff --git a/operational-design/README.md b/operational-design/README.md new file mode 100644 index 0000000..4ca243e --- /dev/null +++ b/operational-design/README.md @@ -0,0 +1,26 @@ +# Operational Design + +COO methodology for process design, organizational scaling, operational metrics, compliance and audit, vendor management, and team topology. Covers value stream mapping, BPMN, bottleneck analysis, scaling from 10 to 100 to 1000 people, KPI design, balanced scorecard, SOC 2, ISO 27001, GDPR readiness, RFP processes, SLA design, vendor scorecards, team topologies, Conway's Law, and Dunbar's Number. + +## Why Install This Skill + +Your agent applies COO-level frameworks — value stream mapping, scaling stages with concrete triggers, SOC 2 readiness, RFP evaluation — instead of generic ops advice. + +## What You Get + +| Directory | Purpose | +|-----------|---------| +| `SKILL.md` | Core methodology, trigger conditions, reference index | +| `references/` | Deep-dive reference files loaded on demand | + +## Triggers + +Designing processes, planning organizational scaling, defining operational metrics, preparing for compliance audits, or running vendor selection. + +## Requirements + +No technical requirements. Covers VSM, BPMN, balanced scorecard, SOC 2, ISO 27001, and team topologies. + +## Quick Start + +Load SKILL.md for the methodology overview and reference table, then load specific references as needed for the task at hand. diff --git a/operational-design/SKILL.md b/operational-design/SKILL.md new file mode 100644 index 0000000..a83e70e --- /dev/null +++ b/operational-design/SKILL.md @@ -0,0 +1,81 @@ +--- +name: operational-design +description: COO methodology for process design, organizational scaling, operational + metrics, compliance and audit, vendor management, and team topology. Covers value + stream mapping, BPMN, bottleneck analysis, scaling from 10 to 100 to 1000 people, + KPI design, balanced scorecard, SOC 2, ISO 27001, GDPR readiness, RFP processes, + SLA design, vendor scorecards, team topologies, Conway's Law, and Dunbar's Number. +license: MIT +metadata: + tags: coo, operations, process-design, scaling, compliance, vendor-management, team-topologies + source_repo: https://github.com/magnus919/hermes-profiles +--- + +# Operational Design + +COO methodology for designing and scaling operations, managing compliance, selecting and managing vendors, and measuring operational health. These frameworks help a COO build the systems and processes that enable the organization to execute reliably at scale. + +## Domain Model + +| Domain | Covers | Artifact | +|--------|--------|----------| +| **Process Design** | Value stream mapping, BPMN, bottleneck analysis, workflow optimization | Process maps, VSM current/future state | +| **Scaling Frameworks** | 10-to-100-to-1000 transitions, organizational design, delegation patterns | Scaling plan, org design | +| **Operational Metrics** | KPI design, balanced scorecard, leading vs lagging indicators | Operations dashboard | +| **Compliance & Audit** | SOC 2, ISO 27001, GDPR readiness, audit preparation | Compliance roadmap, control matrix | +| **Vendor Management** | RFP process, SLA design, vendor scorecards, relationship tiers | Vendor management framework | +| **Organizational Patterns** | Team topologies, Conway's Law, Dunbar's number, span of control | Team design, communication model | + +## When to Load + +Load this skill when the task involves: + +- Mapping and optimizing a business process (value stream, BPMN) +- Planning organizational scaling through growth phases +- Designing operational KPIs and a balanced scorecard +- Preparing for SOC 2, ISO 27001, or GDPR compliance +- Running an RFP or vendor selection process +- Designing SLAs and vendor scorecards +- Restructuring teams using team topologies +- Analyzing bottlenecks and throughput constraints +- Designing delegation and span-of-control models + +## Loading Order + +``` +skill_view('operational-design') # This — methodology index +skill_view('executive-methodology') # Shared decision frameworks +skill_view('artifact-pyramids') # Output contract +skill_view('operational-design', file_path='references/process-design.md') +skill_view('operational-design', file_path='references/scaling-frameworks.md') +skill_view('operational-design', file_path='references/operational-metrics.md') +skill_view('operational-design', file_path='references/compliance.md') +skill_view('operational-design', file_path='references/vendor-management.md') +``` + +## Reference Files + +| Reference | Load When | File | +|-----------|-----------|------| +| Process Design | You need to map, analyze, or optimize a business process | `references/process-design.md` | +| Scaling Frameworks | You're planning organizational growth or restructuring | `references/scaling-frameworks.md` | +| Operational Metrics | You're designing KPIs, dashboards, or a balanced scorecard | `references/operational-metrics.md` | +| Compliance & Audit | You're preparing for SOC 2, ISO 27001, or GDPR compliance | `references/compliance.md` | +| Vendor Management | You're running an RFP, designing SLAs, or evaluating vendors | `references/vendor-management.md` | + +## Design Principles + +1. **Process before automation.** Automating a bad process makes bad output faster. Map and optimize the workflow before selecting tools. +2. **Scale is a discontinuous function.** An organization that works at 10 people will break at 50, 200, and 1,000 — each requires a different operating model. Design for the next phase, not the current one. +3. **Lead with leading indicators.** Lagging indicators tell you what already happened. Leading indicators tell you what will happen. A good operations dashboard has a balanced mix of both. +4. **Compliance is a system, not a project.** SOC 2 certification is not a one-time effort. Compliance requires embedded controls, continuous monitoring, and periodic testing. +5. **Vendors are partners, not passengers.** The cheapest vendor is usually the most expensive in total cost. Invest in vendor relationships proportional to business criticality. +6. **Structure follows strategy.** Team topology should be driven by the communication patterns the work requires (Conway's Law), not by reporting lines that are convenient for management. + +## Related Skills + +- `executive-methodology` — shared decision frameworks and governance +- `strategy-frameworks` — CEO-side strategic direction and competitive analysis +- `technology-radar` — CTO-side technology evaluation and architecture governance +- `financial-modeling` — CFO-side unit economics and SaaS metrics +- `artifact-pyramids` — output contract specification diff --git a/operational-design/references/compliance.md b/operational-design/references/compliance.md new file mode 100644 index 0000000..235990f --- /dev/null +++ b/operational-design/references/compliance.md @@ -0,0 +1,176 @@ +# Compliance and Audit Frameworks + +Compliance is a system, not a project. It requires embedded controls, continuous monitoring, and periodic testing. The frameworks below cover the most common compliance regimes for SaaS and technology companies. + +## SOC 2 + +SOC 2 (System and Organization Controls 2) is the most common compliance framework for SaaS companies. It reports on controls related to security, availability, processing integrity, confidentiality, and privacy. + +### Trust Services Criteria (TSC) + +| Category | Criteria | What It Covers | +|----------|----------|---------------| +| **Security** | CC1-CC9 | The system is protected against unauthorized access. The foundational criteria that everyone must meet. | +| **Availability** | A1 | The system is available for operation and use as committed or agreed. | +| **Processing Integrity** | PI1 | System processing is complete, valid, accurate, timely, and authorized. | +| **Confidentiality** | C1 | Information designated as confidential is protected. | +| **Privacy** | P1-P4 | Personal information is collected, used, retained, disclosed, and disposed in conformity with commitments. | + +### SOC 2 Report Types + +| Type | Description | When to Get It | +|------|-------------|----------------| +| **Type I** | Controls are designed properly at a point in time | First audit, getting started | +| **Type II** | Controls operated effectively over a period (typically 6-12 months) | Customer demands, vendor due diligence | + +### Key Controls by Domain + +**Security (CC1-CC9):** +- Access control policy and enforcement +- Logical access reviews (quarterly) +- Change management process +- Incident response plan and testing +- Vendor risk management +- Encryption at rest and in transit +- Physical security (office/data center) +- Monitoring and alerting +- Background checks +- Security awareness training + +**Availability (A1):** +- Uptime SLAs and monitoring +- Business continuity / disaster recovery plan +- Backup and restore testing +- Capacity planning + +**Processing Integrity (PI1):** +- Input validation controls +- Error handling and corrective action +- Batch job monitoring + +**Confidentiality (C1):** +- Data classification policy +- Access restrictions on confidential data +- Data retention and disposal + +### SOC 2 Readiness Checklist + +| Phase | Activities | Timeline | +|-------|-----------|----------| +| **Scoping** | Define system boundaries, identify in-scope services and data | 2-4 weeks | +| **Risk Assessment** | Identify risks to TSC criteria, document control objectives | 2-4 weeks | +| **Control Design** | Document existing controls, design new controls for gaps | 4-8 weeks | +| **Control Implementation** | Build and deploy controls, update policies | 4-12 weeks | +| **Evidence Collection** | Run controls, collect audit evidence | 3-6 months (Type II) | +| **Audit** | External auditor reviews controls and evidence | 4-8 weeks | + +--- + +## ISO 27001 + +An international standard for Information Security Management Systems (ISMS). Broader than SOC 2 — requires a management system, not just controls. + +### The ISO 27001 Approach + +| Component | Description | +|-----------|-------------| +| **ISMS** | A systematic approach to managing sensitive information | +| **Annex A Controls** | 93 controls across 4 domains (organizational, people, physical, technological) | +| **PDCA Cycle** | Plan-Do-Check-Act continuous improvement | +| **Risk Assessment** | Risk-based approach — controls are selected based on risk, not checklist | +| **Statement of Applicability** | Which Annex A controls apply and why | + +### ISO 27001 vs SOC 2 + +| Dimension | SOC 2 | ISO 27001 | +|-----------|-------|-----------| +| Focus | Controls effectiveness | Management system | +| Flexibility | Fixed TSC criteria | Risk-based, choose your controls | +| Certification | Auditor's opinion letter | Certificate issued | +| Recognition | US-focused | International | +| Renewal | Annual | Three-year certification + surveillance audits | + +### Key Requirements + +1. **ISMS Scope** — What's included and excluded, with justification +2. **Information Security Policy** — Top-level policy, reviewed annually +3. **Risk Assessment** — Systematic risk assessment methodology +4. **Risk Treatment Plan** — How risks will be addressed +5. **Internal Audit** — Regular internal audits of the ISMS +6. **Management Review** — Top management reviews ISMS performance +7. **Continuous Improvement** — Non-conformities are tracked and resolved + +--- + +## GDPR Readiness + +The General Data Protection Regulation governs how personal data of EU residents is handled, regardless of where the company is based. + +### Key Principles + +| Principle | Requirement | +|-----------|-------------| +| **Lawfulness, fairness, transparency** | Legal basis for processing, clear privacy notices | +| **Purpose limitation** | Data collected for specified, explicit purposes only | +| **Data minimization** | Collect only what's necessary | +| **Accuracy** | Keep data accurate and up to date | +| **Storage limitation** | Delete data when no longer needed | +| **Integrity and confidentiality** | Appropriate security measures | +| **Accountability** | Demonstrate compliance (documentation, DPO, records) | + +### Data Subject Rights + +| Right | Description | Response Time | +|-------|-------------|---------------| +| **Right to be informed** | Privacy notice at collection point | At collection | +| **Right of access** | Individuals can request their data | 30 days | +| **Right to rectification** | Correct inaccurate data | 30 days | +| **Right to erasure** ("Right to be forgotten") | Delete personal data | 30 days | +| **Right to restrict processing** | Limit how data is used | 30 days | +| **Right to data portability** | Export data in machine-readable format | 30 days | +| **Right to object** | Opt out of processing (including marketing) | At any time | +| **Rights related to automated decision-making** | Explanation of algo-based decisions | Upon request | + +### GDPR Compliance Roadmap + +| Phase | Activities | Timeline | +|-------|-----------|----------| +| **Discovery** | Data mapping, identify all personal data processing | 4-8 weeks | +| **Gap Analysis** | Current state vs GDPR requirements | 2-4 weeks | +| **Remediation** | Update policies, implement controls, update contracts | 8-16 weeks | +| **Implementation** | DPO appointment, privacy notices, consent management | 4-8 weeks | +| **Ongoing** | DSAR handling process, breach notification procedure, annual review | Continuous | + +### GDPR Breach Notification + +``` +Under GDPR, a breach must be reported to the supervisory authority +within 72 hours of becoming aware of it. + +If the breach is likely to result in high risk to individuals, +they must also be informed without undue delay. +``` + +--- + +## Common Compliance Pitfalls + +| Pitfall | Symptom | Fix | +|---------|---------|-----| +| **Compliance once, not continuous** | Controls atrophy between audits | Build continuous monitoring, automate evidence collection | +| **Documentation but not implementation** | Policies exist but aren't followed | Test controls, not just documentation | +| **Scope creep** | Trying to cover everything at once | Start narrow, expand scope over time | +| **No executive ownership** | Compliance is delegated to IT without business support | Assign executive sponsor, report compliance at board level | +| **Vendor blind spot** | Vendors have access to your data but no compliance validation | Vendor risk management program, contract reviews | +| **Over-relying on automation** | Tools replace thinking | Automation supports controls, doesn't replace judgment | + +### Evidence Collection Strategy + +| Evidence Type | Examples | Collection Method | +|---------------|----------|-------------------| +| **System logs** | Access logs, change logs, audit trails | Automated log aggregation | +| **Configuration** | IAM policies, encryption settings | Infrastructure-as-code, CI/CD | +| **Process artifacts** | Signed forms, approved change requests | Document management system | +| **Training records** | Completed security training | LMS reports | +| **Review evidence** | Access review sign-offs | Quarterly automated workflow | +| **Testing results** | Penetration test reports, DR test results | Scheduled external testing | diff --git a/operational-design/references/operational-metrics.md b/operational-design/references/operational-metrics.md new file mode 100644 index 0000000..6136e4d --- /dev/null +++ b/operational-design/references/operational-metrics.md @@ -0,0 +1,137 @@ +# Operational Metrics + +Metrics are how you know whether the operation is performing as designed. The right metrics create alignment. The wrong metrics drive the wrong behavior. + +## KPI Design Framework + +### Good KPIs vs Bad KPIs + +| Good KPI | Bad KPI | +|----------|---------| +| Specific and measurable | Vague ("improve quality") | +| Actionable — can be influenced | Purely informative ("stock price") | +| Owner assigned | No one responsible | +| Tied to a target or threshold | No context for good/bad | +| Leading or lagging with clear relationship | Random measures without connection | +| Few in number (5-7 per department) | Too many to focus on | + +### The KPI Hierarchy + +**Level 1: Company North Star (1-2 metrics)** +The single metric that best captures whether the company is succeeding. Everything else supports this. + +**Level 2: Departmental KPIs (3-5 per department)** +What each department must achieve to support the North Star. + +**Level 3: Team/Individual Metrics (3-5 per team)** +What teams and individuals can directly influence. + +**Level 4: Process Metrics (as needed)** +Real-time operational measures that feed into Level 3. + +### KPI Template + +``` +Name: [Metric name] +Formula: [How it's calculated] +Frequency: [Daily, weekly, monthly] +Owner: [Who is accountable] +Target: [Good, acceptable, critical thresholds] +Data source: [Where the data comes from] +Lag/Lead: [Leading or lagging indicator] +``` + +--- + +## Leading vs Lagging Indicators + +| Type | Definition | Examples | When to Use | +|------|-----------|----------|-------------| +| **Lagging** | Measures outcomes after they happen | Revenue, profit, churn, NPS | Strategic review, board reporting | +| **Leading** | Predicts future outcomes | Pipeline creation, demo requests, activation rate | Operational management, early warning | + +### Leading Indicators by Function + +| Function | Lagging Indicator | Leading Indicator | +|----------|------------------|-------------------| +| **Sales** | Revenue closed | Pipeline created, demo-to-close ratio, win rate | +| **Marketing** | Leads generated | Website traffic, content engagement, conversion rate | +| **Product** | Feature adoption | Time to value, activation rate, session frequency | +| **Engineering** | System uptime | Deployment quality, alert volume, WIP limits | +| **Support** | Customer satisfaction | First response time, resolution time, ticket volume trend | +| **HR** | Attrition rate | Engagement survey score, promotion readiness, time-to-hire | + +### The Lead-Lag Chain + +Build a causal model connecting leading indicators to lagging outcomes: + +``` +More demos (lead) → Higher pipeline (lead) → More closed deals (lag) → Revenue growth (lag) +Faster time to value (lead) → Higher activation (lead) → Lower churn (lag) → Higher LTV (lag) +Faster deployment frequency (lead) → Shorter lead time (lead) → Lower change failure rate (lag) → Higher uptime (lag) +``` + +--- + +## Balanced Scorecard + +A strategic planning and management system that goes beyond financial metrics to include customer, process, and learning perspectives. + +### The Four Perspectives + +| Perspective | Question | Typical Metrics | +|-------------|----------|-----------------| +| **Financial** | How do we look to shareholders? | Revenue growth, profitability, ROIC, cash flow | +| **Customer** | How do customers see us? | NPS, retention, satisfaction, time to value | +| **Internal Process** | What must we excel at? | Quality, cycle time, cost, throughput | +| **Learning & Growth** | Can we continue to improve? | Employee engagement, skill development, innovation pipeline | + +### Building a Balanced Scorecard + +1. **Define the strategy.** What is the organization's strategic objective for the next 12-24 months? +2. **Identify 3-5 objectives per perspective.** What must happen in each perspective to achieve the strategy? +3. **Define 1-2 measures per objective.** How will you know the objective is being achieved? +4. **Set targets.** What does good look like? What's the stretch target? +5. **Identify initiatives.** What projects or programs drive movement on these measures? + +### Scorecard Example (SaaS Company) + +| Perspective | Objective | Measure | Target | Initiative | +|-------------|-----------|---------|--------|------------| +| Financial | Grow recurring revenue | ARR growth rate | 40% YoY | Sales team expansion | +| Financial | Improve efficiency | Rule of 40 | ≥ 40% | Cost optimization program | +| Customer | Improve retention | Net Revenue Retention | ≥ 120% | Customer success automation | +| Customer | Shorten time to value | Days to first activation | < 7 days | Onboarding redesign | +| Internal | Improve delivery speed | Lead time for changes | < 1 day | CI/CD pipeline investment | +| Internal | Ensure quality | Change failure rate | < 5% | Automated testing increase | +| Learning | Develop leadership | Internal promotion rate | 40%+ | Leadership development program | +| Learning | Improve engagement | eNPS score | > 50 | Engagement action planning | + +--- + +## Designing Operational Dashboards + +### Dashboard Layers + +| Layer | Audience | Refresh | Content | +|-------|----------|---------|---------| +| **Strategic** | Executives | Monthly | North Star, high-level KPIs, trend lines | +| **Tactical** | Department heads | Weekly | Departmental KPIs, variance vs plan, top issues | +| **Operational** | Team leads, ICs | Daily | Process metrics, queues, real-time status | + +### Dashboard Design Principles + +1. **One page, one purpose.** Don't try to serve everyone with one dashboard. Create separate views for different audiences. +2. **Show the target.** A metric without a target is just a number. Show the actual vs target and the direction. +3. **Trend over time.** A single number is meaningless without context. Show at least the last 12 periods. +4. **Highlight exceptions.** The dashboard should surface what needs attention, not just report what's normal. +5. **Limit to 7-10 metrics per view.** More than that and the dashboard becomes noise. +6. **Label everything.** Metric name, unit, frequency, owner. If someone can't understand the dashboard without asking, it's not done. + +### Common Dashboard Anti-Patterns + +- **Vanity metrics.** "Total registered users" sounds impressive but tells you nothing about health. Use active users, cohort retention, and conversion rates instead. +- **Data puking.** 50 charts on one page. More data is not more insight. Edit ruthlessly. +- **No drill-down.** The dashboard shows revenue is down. Can you click to see by product line? By region? By segment? Layer in drill-down capability. +- **Stale data.** A dashboard that's updated monthly for operational use is misleading. Match refresh frequency to decision frequency. +- **Automated but unowned.** Every metric needs a human owner. If a number goes red, someone should know who to call. diff --git a/operational-design/references/process-design.md b/operational-design/references/process-design.md new file mode 100644 index 0000000..9f7c228 --- /dev/null +++ b/operational-design/references/process-design.md @@ -0,0 +1,148 @@ +# Process Design + +Frameworks for understanding, documenting, and improving business processes. Process design is the foundation of operational excellence — you cannot improve what you haven't mapped. + +## Value Stream Mapping (VSM) + +A lean-management technique for analyzing the flow of materials and information required to deliver a product or service to a customer. + +### Standard VSM Elements + +| Symbol | Name | Meaning | +|--------|------|---------| +| □ | Process box | A process step (department, system, person) | +| ▼ | Inventory | Work-in-progress between steps | +| → | Push arrow | Material moves without pull signal | +| ☰ | Information flow | Communication (manual or electronic) | +| ⚡ | Kaizen burst | Improvement opportunity identified | +| ⏰ | Timeline | Value-added vs non-value-added time | + +### Building a Current-State VSM + +1. **Define the product/service.** What's the specific product, order, or request you're mapping? +2. **Walk the process.** Physically follow the work from start to finish. Don't map from memory. +3. **Map process steps.** Each major step is a process box. Include wait states and handoffs. +4. **Collect data for each step:** + - Cycle time (time to complete the step) + - Changeover time (time to switch between types) + - Uptime / reliability + - First-pass yield (% of work done right first time) + - Number of operators +5. **Map information flow.** How does each step know what to do? Email? System? Verbal? +6. **Add the timeline.** Calculate value-added time vs total lead time. + +### Value-Added vs Non-Value-Added + +| Category | Definition | Examples | +|----------|-----------|----------| +| **Value-Added (VA)** | Changes the product/service in a way the customer cares about and pays for | Manufacturing, code development, customer consultation | +| **Business Non-Value-Added (BNVA)** | Required by regulation or business policy but not valued by customer | Compliance checks, reporting, approvals | +| **Non-Value-Added (NVA)** | Pure waste. Customer would not pay for this. | Rework, waiting, handoffs, unnecessary steps | + +### The Efficiency Metric + +``` +Process Cycle Efficiency = Total Value-Added Time / Total Lead Time +``` + +| Efficiency | Classification | +|------------|---------------| +| > 25% | Excellent. Lean process. | +| 10-25% | Good. Room for improvement. | +| 5-10% | Typical for most organizations. Significant waste. | +| < 5% | High waste environment. Major opportunity. | + +--- + +## BPMN (Business Process Model and Notation) + +A standardized notation for process modeling. Use BPMN when you need formal, unambiguous process documentation. + +### Core Elements + +| Element | Notation | Meaning | +|---------|----------|---------| +| Event | Circle | Something that happens (start, end, timer, message) | +| Activity | Rounded rectangle | Work performed (task, subprocess) | +| Gateway | Diamond | Decision point (XOR, AND, OR) | +| Sequence Flow | Solid arrow | Order of activities | +| Message Flow | Dashed arrow | Communication between participants | +| Pool | Large rectangle | A participant in the process | +| Lane | Nested section within a pool | Role or department within a participant | + +### Gateway Types + +| Gateway | Logic | Visual | Use When | +|---------|-------|--------|----------| +| **XOR (Exclusive)** | Exactly one path | Standard diamond | Yes/No decisions, routing | +| **AND (Parallel)** | All paths execute | Diamond with + | Tasks can happen simultaneously | +| **OR (Inclusive)** | One or more paths | Diamond with O | Multiple conditions may be true | + +--- + +## Bottleneck Analysis + +In any process, the slowest step determines the throughput of the entire system. Bottleneck analysis identifies that step. + +### Theory of Constraints (Goldratt) + +1. **Identify** the constraint (the bottleneck). The step with the smallest capacity or longest cycle time. +2. **Exploit** the constraint. Maximize the bottleneck's throughput. Don't let it wait. +3. **Subordinate** everything else. Non-bottleneck steps should operate at the bottleneck's pace, not at their own maximum. +4. **Elevate** the constraint. If exploitation isn't enough, invest in increasing the bottleneck's capacity. +5. **Repeat.** Once the bottleneck is resolved, a new bottleneck appears. Start over. + +### Finding the Bottleneck + +| Method | How | Best For | +|--------|-----|----------| +| **Walk the process** | Stand where the work is. Where is the pile of work-in-progress largest? | Quick assessment, small processes | +| **Capacity analysis** | Calculate maximum throughput of each step. Lowest = bottleneck. | Manufacturing, transactional | +| **Cycle time analysis** | Measure actual time per step. Longest = bottleneck. | Knowledge work, services | +| **Queues** | Where is the longest queue? The step before the queue is the bottleneck. | All processes | + +### Bottleneck Anti-Patterns + +- **Optimizing non-bottlenecks.** Improving a step that isn't the bottleneck increases capacity overall by 0%. It just creates more work-in-progress queued at the bottleneck. +- **Keeping the bottleneck idle.** Lunch breaks, meetings, training. If the bottleneck stops, the whole system loses throughput. Protect the bottleneck's time. +- **Ignoring variability.** A step may not look like the bottleneck on average, but if its variability is high, it causes intermittent bottlenecks. + +--- + +## Workflow Optimization Patterns + +### The Seven Wastes (TIMWOOD) + +| Waste | Description | Example in Knowledge Work | +|-------|-------------|--------------------------| +| **T**ransportation | Unnecessary movement of work | Multiple handoffs between teams | +| **I**nventory | Excess work-in-progress | Too many open tickets | +| **M**otion | Unnecessary movement of people | Context switching, finding information | +| **W**aiting | Idle time between steps | Approval queues, review backlogs | +| **O**ver-processing | Doing more than needed | Excessive documentation, over-engineering | +| **O**ver-production | Doing work before it's needed | Building features without demand | +| **D**efects | Errors requiring rework | Bugs, miscommunication, incorrect data | + +### Process Improvement Heuristics + +| Heuristic | When to Apply | Expected Impact | +|-----------|--------------|-----------------| +| Eliminate handoffs | Process has 5+ handoffs | High — each handoff adds delay and error | +| Parallelize independent steps | Sequential steps that don't depend on each other | Medium-High — reduces lead time significantly | +| Move decisions earlier | Late-stage approvals cause rework | High — fail fast, not late | +| Automate verification | Manual checking is slow and inconsistent | Medium — improves consistency more than speed | +| Standardize exceptions | The same exception gets handled differently every time | High — reduces cognitive load and error | +| Batch size reduction | Large batches increase lead time and variability | Medium — smoother flow, faster feedback | +| Remove sign-off layers | 3+ approvals for routine decisions | High — approvals are delays, not quality | + +### Measuring Improvement + +| Metric | Pre-optimization | Post-optimization | Target | +|--------|-----------------|-------------------|--------| +| Lead time (end-to-end) | | | | +| Value-added time | | | | +| First-pass yield | | | | +| Handoff count | | | | +| Approval steps | | | | +| Rework rate | | | | +| Cost per transaction | | | | diff --git a/operational-design/references/scaling-frameworks.md b/operational-design/references/scaling-frameworks.md new file mode 100644 index 0000000..e17c4f9 --- /dev/null +++ b/operational-design/references/scaling-frameworks.md @@ -0,0 +1,195 @@ +# Scaling Frameworks + +Organizational scaling is a discontinuous function. The operating model that works at 10 people breaks at 50. The model that works at 50 breaks at 200. Each stage requires deliberate redesign. + +## The Scaling Stages + +### Stage 1: Founding (1-10 People) + +**Operating model:** Direct communication. Everyone knows everything. CEO makes most decisions. + +**What works:** +- All-hands meetings, everyone talks to everyone +- CEO approves all hires and major decisions +- Informal processes, no documentation needed +- Generalists preferred over specialists + +**What breaks:** +- Informal communication becomes unreliable +- CEO becomes bottleneck for decisions +- "Everyone does everything" leads to dropped balls +- Hiring starts to need structure (interview process, offer letters) + +**Transition trigger:** CEO can no longer attend every meeting or review every decision. Typically at 8-15 people. + +### Stage 2: The Team Phase (10-50 People) + +**Operating model:** Functional teams with managers. CEO manages via team leads. + +**What changes:** +- Department heads appointed (Engineering, Sales, Marketing) +- Weekly staff meetings with department leads +- Basic processes emerge (hiring, expense approval, customer support) +- First specialist hires (marketing, HR, finance) + +**New challenges:** +- Communication across teams becomes a problem +- CEO is still in most decisions but now through team leads +- Hiring needs process (standardized interviews, offer approval) +- First performance management issues arise + +**Transition trigger:** Team leads can't keep up with coordination. Cross-team projects fail due to poor communication. Typically at 40-60 people. + +### Stage 3: The Department Phase (50-200 People) + +**Operating model:** Functional departments with VP-level leaders. CEO manages the executive team. + +**What changes:** +- VPs hired for each department +- Weekly exec team meeting, monthly all-hands +- Formal processes for hiring, budgeting, performance reviews +- Middle management layer added +- First attempt at OKRs or similar goal-setting + +**New challenges:** +- Silos emerge between departments +- "Over the wall" syndrome — Engineering blames Sales, Sales blames Product +- Decision-making slows as more stakeholders are involved +- Culture dilution — new hires don't know the old ways +- First major process debt — too many processes or not enough + +**Transition trigger:** Cross-functional coordination becomes the primary bottleneck. Silos prevent strategic initiatives. Typically at 150-250 people. + +### Stage 4: The Enterprise Phase (200-1,000+ People) + +**Operating model:** Business units or divisions with P&L responsibility. CEO manages business unit leaders. + +**What changes:** +- Business units with their own P&L +- Shared services (IT, HR, Finance, Legal) as centralized functions +- Formal governance (board meetings, committee structure) +- Strategic planning process (annual + quarterly) +- Professional management systems (compensation bands, leveling, career frameworks) + +**New challenges:** +- Maintaining startup culture at scale +- Bureaucracy and process bloat +- Innovation atrophies — everything requires a business case +- Talent density dilutes — average performers become the norm +- Coordination costs dominate operating expenses + +**Survival strategies:** +- Break into autonomous units when possible +- Maintain small-team dynamics within units +- Invest in internal mobility and talent development +- Fight process creep actively — sunset unnecessary processes + +--- + +## Delegation Patterns at Scale + +### The Delegation Progression + +| Stage | CEO Decisions | Delegated | Mechanisms | +|-------|--------------|-----------|------------| +| 1-10 | All major decisions | Minimal | Direct assign | +| 10-50 | Strategy, hiring, budget, product | Execution | Department leads | +| 50-200 | Strategy, exec hiring, major budget | Operations, product | VP delegation, OKRs | +| 200-1000 | Strategy, capital allocation, culture | Almost everything | Business unit P&L, governance | + +### Scaling the Span of Control + +| Level | Direct Reports | Notes | +|-------|---------------|-------| +| First-line manager | 4-8 | Direct contributors | +| Director/V-Level | 4-6 | Manager of managers | +| C-Suite | 4-8 | Direct reports and functional leaders | +| CEO (early) | 4-6 | Direct reports | +| CEO (scale) | 8-12 | Including functional heads | + +### Span of Control Heuristics +- Technical ICs need more attention → smaller spans (4-6) +- Experienced managers → larger spans (6-10) +- Autonomous/empowered teams → larger spans +- New managers → smaller spans (3-4), grow over time + +--- + +## Organizational Design Patterns + +### Functional Structure + +Pro: Deep expertise, clear career paths, efficient resource use +Con: Silos, slow cross-functional decisions, customer-blind +Best for: 10-200 person companies, stable markets + +### Divisional/BU Structure + +Pro: Customer-focused, fast decisions within unit, clear P&L ownership +Con: Duplication of resources, coordination across units is hard +Best for: 200+ person companies, multiple products/markets + +### Matrix Structure + +Pro: Combines functional expertise with project focus +Con: Dual reporting is confusing, slow decisions, "two bosses" problem +Best for: Project-based organizations (consulting, construction) + +### Team Topologies (Conway's Law Applied) + +Designed to align team structure with communication needs: + +| Team Type | Purpose | Size | Interactions | +|-----------|---------|------|--------------| +| **Stream-aligned** | Owns a full value stream (feature, service, product area) | 6-8 | Collaborates with enabling teams | +| **Enabling** | Helps stream-aligned teams learn and adopt new capabilities | 4-6 | Collaborates, facilitates | +| **Complicated-subsystem** | Owns a domain that requires deep specialized knowledge | 4-8 | Provides, X-as-a-Service | +| **Platform** | Builds internal products that other teams use | 6-10 | Provides, X-as-a-Service | + +--- + +## Conway's Law Applied + +> "Organizations design systems that mirror their communication structure." + +### The Principle + +If you have 3 teams that need to coordinate to ship a feature, the system will have 3 components that need to coordinate to work. The architecture reflects the org chart. + +**Implications:** +- To change the architecture, first change the team structure +- A microservices architecture requires a team structure that supports autonomous services +- A monolith is fine if the team is a monolith (small, colocated) + +### Inverse Conway Maneuver + +Restructure teams to match the desired architecture, then let the architecture follow. + +1. Define the target architecture (e.g., 3 services: payments, inventory, orders) +2. Create 3 stream-aligned teams, each owning one service +3. The architecture will naturally converge on the target because teams can independently deliver + +**Risk:** If the architecture isn't right, you've locked in a bad design. Test the architecture hypothesis before restructuring. + +--- + +## Dunbar's Number + +150 is the theoretical maximum number of stable social relationships a human can maintain. + +### Applications to Organizational Design + +| Number | Social Dynamic | Organizational Implication | +|--------|---------------|---------------------------| +| 5 | Intimate team | Everyone knows everyone deeply. Full trust. | +| 15 | Band | Can maintain shared context without process. | +| 50 | Tribe | Need some structure. Most people know most people. | +| 150 | Clan | Dunbar's number. Start of anonymity. Need formal systems. | +| 500 | Crowd | Cannot know everyone. Need full management hierarchy. | + +### Practical Rules + +- **Keep teams under 10** (preferably 6-8) +- **Keep departments under 150** — once a department exceeds 150, split it +- **All-hands becomes impractical > 150** — use cascading communication +- **Culture is carried by the 150 core** — the first 150 employees define the culture. After that, culture must be actively managed through systems and stories. diff --git a/operational-design/references/vendor-management.md b/operational-design/references/vendor-management.md new file mode 100644 index 0000000..ffe1f71 --- /dev/null +++ b/operational-design/references/vendor-management.md @@ -0,0 +1,161 @@ +# Vendor Management + +Vendors are partners, not transactions. A well-managed vendor relationship reduces risk, improves service quality, and creates leverage for cost negotiations. + +## RFP Process + +Request for Proposal (RFP) is a structured process for evaluating vendor options. Use RFPs for significant investments where comparison across multiple vendors is needed. + +### RFP Lifecycle + +1. **Requirements Definition** — What must the vendor do? What's nice-to-have? +2. **Vendor Shortlist** — 3-5 vendors who can meet requirements +3. **RFP Issuance** — Formal document sent to shortlisted vendors +4. **Vendor Q&A** — Clarify questions, ensure all vendors have same information +5. **Proposal Evaluation** — Score proposals against weighted criteria +6. **Vendor Demos / POCs** — Shortlisted vendors demonstrate capability +7. **Reference Calls** — Check with existing customers +8. **Negotiation & Selection** — Final terms and selection + +### RFP Template + +``` +1. Executive Summary + - Project overview, timeline, budget range + +2. Company Background + - About us, our needs, current state + +3. Scope of Work + - Detailed requirements (must-have, should-have, nice-to-have) + - Deliverables, milestones, success criteria + +4. Vendor Qualification Requirements + - Company size, experience, certifications + - Security and compliance (SOC 2, ISO 27001, GDPR) + - References required + +5. Commercial Terms + - Pricing model requested (per seat, flat, usage-based) + - Contract term, SLA requirements + - Payment terms + +6. Submission Requirements + - Format, deadline, point of contact + - Questions for vendor to answer + +7. Evaluation Criteria + - How proposals will be scored + - Weight for each dimension +``` + +### RFP Evaluation Matrix + +| Criterion | Weight | Vendor A | Vendor B | Vendor C | +|-----------|--------|----------|----------|----------| +| Functional fit | 25% | | | | +| Technical architecture | 15% | | | | +| Security & compliance | 15% | | | | +| Total cost (3-year TCO) | 20% | | | | +| Support & service | 10% | | | | +| Company stability | 10% | | | | +| References | 5% | | | | +| **Total** | **100%** | | | | + +### RFP Anti-Patterns + +- **RFPs for commodity purchases.** An RFP for a $500/month email tool wastes everyone's time. Use RFPs for significant, strategic decisions. +- **Too many vendors.** Evaluating 10 vendors comprehensively is unrealistic. Limit to 3-5. +- **Unequal information.** One vendor gets a question answered, others don't. Share Q&A with all vendors equally. +- **Death by requirements.** A 200-item requirements list buries the important ones. Distinguish must-have from nice-to-have. + +--- + +## SLA Design + +Service Level Agreements define what the vendor guarantees and what happens if they fail. + +### SLA Components + +| Component | Definition | Example | +|-----------|-----------|---------| +| **Service Definition** | What exactly is covered | "Core platform API availability" | +| **Uptime Commitment** | % of time service is available | 99.9% uptime (excluding planned maintenance) | +| **Measurement Period** | How availability is calculated | Monthly average, quarterly true-up | +| **Exclusions** | What's not covered | Scheduled maintenance, force majeure, customer-side issues | +| **Credits** | Penalty for missed SLA | 5% credit per 0.1% below target, max 25% | + +### Credit Structure + +| Uptime % | Credit (Typical) | +|----------|------------------| +| 99.9-100% | No credit | +| 99.0-99.9% | 5% of monthly fee | +| 95.0-99.0% | 10% of monthly fee | +| < 95.0% | 25% of monthly fee + termination rights | + +### Beyond Uptime: Multi-Dimensional SLAs + +| Dimension | Definition | Typical Target | +|-----------|-----------|---------------| +| **Availability** | Service is accessible | 99.9% (standard), 99.99% (critical) | +| **Performance** | Response times within threshold | P95 < 500ms | +| **Support response** | Time to first response | Critical: < 1hr, High: < 4hrs, Normal: < 24hrs | +| **Support resolution** | Time to resolution | Critical: < 4hrs, High: < 8hrs, Normal: < 5 days | + +### SLA Pitfalls + +- **Measuring what's easy, not what matters.** Dashboard uptime is easy to measure. API latency at P95, data freshness, and error rates matter more. +- **No measurement transparency.** If you can't independently verify uptime, the SLA is unenforceable. Require a status page and monthly reports. +- **Credits that don't hurt.** A 5% credit on a $1K/month contract is $50. That's not enough to incentivize performance. +- **Ignoring the exit.** If SLA violations accumulate, there should be a termination-for-cause clause with a data migration assistance obligation. + +--- + +## Vendor Scorecards + +Scorecards provide ongoing evaluation of vendor performance. They should be reviewed quarterly and factored into renewal decisions. + +### Scorecard Template + +| Category | Weight | Metric | Target | Actual | Score | +|----------|--------|--------|--------|--------|-------| +| **Service Quality** | 25% | Uptime SLA | 99.9% | | | +| | | Response time (P95) | < 500ms | | | +| **Support** | 20% | P1 response time | < 1hr | | | +| | | Ticket satisfaction | > 4.0/5.0 | | | +| **Security** | 15% | SOC 2 valid | Yes | | | +| | | Pen test results | No critical findings | | | +| **Value** | 20% | Cost variance vs budget | < 5% | | | +| | | Feature delivery vs roadmap | On track | | | +| **Relationship** | 10% | Executive engagement | Quarterly | | | +| | | Escalation responsiveness | < 24hrs | | | + +### Vendor Tiering + +| Tier | Criticality | Management Cadence | Exit Plan | +|------|-------------|-------------------|-----------| +| **Tier 1: Strategic** | Core to business operations | Monthly review, quarterly business review, annual contract | Maintained, updated semi-annually | +| **Tier 2: Important** | Significant but replaceable | Quarterly performance review | Maintained, reviewed annually | +| **Tier 3: Operational** | Useful but non-critical | Annual review | Documented, not maintained as active plan | +| **Tier 4: Commodity** | Easily replaceable | Monitor via SOW | No formal exit plan needed | + +### Vendor Scorecard Pitfalls + +- **No consequence for poor scores.** If a vendor consistently scores low but keeps the contract, the scorecard is theater. Tie scorecards to renewal decisions. +- **Annual review cadence for everyone.** Annual review is too infrequent for critical vendors and too frequent for commodity vendors. Match cadence to criticality. +- **Scoring without conversation.** Don't just send the scorecard. Review it with the vendor. Their response to the data is as informative as the data itself. + +--- + +## Vendor Management Heuristics + +| Heuristic | Why | +|-----------|-----| +| **Never be a vendor's only customer.** | If they go under, you're stranded. | +| **Always have an exit plan.** | The cost of switching is highest when you can't switch. Know the exit cost before you sign. | +| **Data portability is non-negotiable.** | Ensure you can export your data in a standard format at any time. | +| **Negotiate the renewal before signing.** | Know the renewal process and escalation structure. A vendor that's helpful during sales may be adversarial during renewal. | +| **Vendor concentration is risk.** | If one vendor accounts for > 30% of a capability, you have concentration risk. Identify alternatives. | +| **The cheapest option is rarely the cheapest.** | Hidden costs (integration, training, workarounds) make cheap vendors expensive. Calculate real TCO. | +| **Multi-year contracts need price protection.** | Lock in pricing or maximum annual increases. Without protection, you're at the vendor's pricing mercy. | diff --git a/org-design/README.md b/org-design/README.md new file mode 100644 index 0000000..54d9416 --- /dev/null +++ b/org-design/README.md @@ -0,0 +1,26 @@ +# Org Design + +CHRO methodology — organizational design (team topologies, span of control, reporting structures), talent strategy (make-vs-buy, skill taxonomies, succession planning), compensation frameworks (market benchmarking, equity design, leveling), culture architecture (values codification, rituals, psychological safety), organizational health metrics (eNPS, retention risk, engagement surveys), DEI strategy (inclusive design, equitable systems, belonging). + +## Why Install This Skill + +Your agent reasons about team structure, compensation, and culture with real frameworks — Team Topologies, 9-box grid, comp band structures, psychological safety stages. + +## What You Get + +| Directory | Purpose | +|-----------|---------| +| `SKILL.md` | Core methodology, trigger conditions, reference index | +| `references/` | Deep-dive reference files loaded on demand | + +## Triggers + +Designing team structures, building talent strategy, setting compensation bands, codifying culture, or measuring organizational health. + +## Requirements + +No technical requirements. Frameworks from Skelton/Pais, Clark, and standard HR analytics. + +## Quick Start + +Load SKILL.md for the methodology overview and reference table, then load specific references as needed for the task at hand. diff --git a/org-design/SKILL.md b/org-design/SKILL.md new file mode 100644 index 0000000..7ce4a4c --- /dev/null +++ b/org-design/SKILL.md @@ -0,0 +1,59 @@ +--- +name: org-design +description: CHRO methodology — organizational design (team topologies, span of control, + reporting structures), talent strategy (make-vs-buy, skill taxonomies, succession + planning), compensation frameworks (market benchmarking, equity design, leveling), + culture architecture (values codification, rituals, psychological safety), organizational + health metrics (eNPS, retention risk, engagement surveys), DEI strategy (inclusive + design, equitable systems, belonging). +license: MIT +metadata: + tags: org-design, chro, hr, talent-strategy, compensation, culture, organizational-health, + dei, succession-planning + source_repo: https://github.com/magnus919/hermes-profiles +--- + +# Organizational Design — CHRO Methodology + +CHRO-level methodology for organizational design, talent strategy, compensation, culture, and organizational health. This skill provides the frameworks and reference material for a chief human resources officer profile. + +## When to Load + +| Trigger | What's Needed | +|---------|---------------| +| Design organizational structure | `references/organizational-design.md` — team topologies, span of control, reporting structures, health metrics | +| Develop talent strategy | `references/talent-strategy.md` — make-vs-buy, skill taxonomies, succession planning, 9-box grid | +| Build compensation frameworks | `references/compensation-frameworks.md` — market benchmarking, equity design, leveling bands, variable pay | +| Define culture and values | `references/culture-architecture.md` — values codification, rituals, psychological safety, eNPS, DEI | +| Assess org health | `references/culture-architecture.md` — engagement drivers, retention indicators, DEI maturity | +| Plan succession pipeline | `references/talent-strategy.md` — pipeline coverage, 9-box grid, talent review cadence | + +## Loading Order + +```text +skill_view('org-design') +# Then domain-specific references: +skill_view('org-design', file_path='references/organizational-design.md') +skill_view('org-design', file_path='references/talent-strategy.md') +skill_view('org-design', file_path='references/compensation-frameworks.md') +skill_view('org-design', file_path='references/culture-architecture.md') +``` + +## Reference Files + +| Reference | Purpose | +|-----------|---------| +| `references/organizational-design.md` | Team topologies (stream-aligned, enabling, complicated-subsystem, platform), span of control, reporting structures (functional, matrix, stream-aligned), organizational health metrics | +| `references/talent-strategy.md` | Make-vs-buy decision matrix, skill taxonomies, succession planning (pipeline coverage, 9-box grid), talent review cadence, retention risk indicators | +| `references/compensation-frameworks.md` | Market benchmarking (Radford, Levels.fyi), equity instruments (ISO, NSO, RSU), grant benchmarks by level, vesting schedules, leveling bands, variable pay, comp review cadence | +| `references/culture-architecture.md` | Values codification template, rituals cadence, psychological safety (4 stages, measurement, building), organizational health metrics (eNPS benchmarks, engagement drivers, retention indicators), DEI strategy and maturity model | + +## Output Contract + +The profile using this skill produces artifact pyramids. The response to any caller is the absolute path to `00-index.md`. See `artifact-pyramids` skill for the specification. + +## Related Skills + +- `artifact-pyramids` — output contract +- `implementation-planning` — team-level work breakdown +- `product-strategy` — CPO methodology (complementary for org design around product teams) diff --git a/org-design/references/compensation-frameworks.md b/org-design/references/compensation-frameworks.md new file mode 100644 index 0000000..6a112e5 --- /dev/null +++ b/org-design/references/compensation-frameworks.md @@ -0,0 +1,112 @@ +# Compensation Frameworks + +## Market Benchmarking + +### The Three Sources Approach + +Cross-reference three sources to determine market pay: + +| Source | Pros | Cons | Use for | +|--------|------|------|---------| +| **Radford/Aon** | Comprehensive tech data, global | Expensive, requires subscription | All tech roles | +| **Levels.fyi / Glassdoor** | Transparent, real-time | Self-reported, noisy | Benchmarking reality vs surveys | +| **Payscale / Compensation.com** | Broad coverage, good for non-tech | Less tech-specific | G&A, marketing, operations | +| **Industry-specific** (e.g., Culpepper) | Niche role coverage | Industry-specific only | Biotech, finance | + +### Market Positioning Strategy + +| Position | Definition | When to use | +|----------|------------|-------------| +| **P10** (lagging) | Below market median | Cost-constrained, non-core roles, non-competitive labor market | +| **P50** (market) | At market median | Standard positioning for most roles | +| **P75** (leading) | Above market median | Critical roles, hard-to-fill skills, competitive market | +| **P90** (premium) | Top of market | Executive talent, scarcity-driven roles, strategic hires | + +### Geography Adjustments + +Apply location-based multipliers to base pay: + +| Location Tier | Multiplier (typical) | Examples | +|---------------|---------------------|----------| +| Tier 1 (Premium) | 1.0x (base) | San Francisco, NYC, London | +| Tier 2 (Major metro) | 0.85-0.95x | Seattle, Chicago, Berlin | +| Tier 3 (Secondary) | 0.75-0.85x | Austin, Denver, Dublin | +| Tier 4 (Remote/Rest) | 0.65-0.75x | Rural US, lower-cost regions | + +**Remote-first trend:** Many companies are moving to location-independent pay or a single national band. Tradeoff: hiring efficiency vs compensation equity. + +## Equity Design + +### Equity Instruments + +| Instrument | Description | Typical for | +|------------|-------------|-------------| +| **ISO** (Incentive Stock Options) | Tax-advantaged options, $100K/yr limit per employee | Employees at VC-backed companies | +| **NSO** (Non-Qualified Stock Options) | Standard options, no limit | Employees, contractors, advisors | +| **RSU** (Restricted Stock Units) | Actual shares vest over time | Public companies, late-stage private | +| **SAR** (Stock Appreciation Rights) | Cash payout equal to share appreciation | Private companies wanting option-like economics without dilution | +| **ESPP** (Employee Stock Purchase Plan) | Discounted stock purchase (typically 15%) | Public companies | + +### Equity Grant Benchmarks + +| Level | Grant range (% of pool) | Grant range ($ value, private tech) | +|-------|------------------------|-------------------------------------| +| **Junior IC** | 0.02-0.05% | $10K-$50K | +| **Mid IC** | 0.05-0.15% | $50K-$150K | +| **Senior IC** | 0.15-0.30% | $150K-$350K | +| **Staff IC** | 0.20-0.40% | $350K-$600K | +| **Principal IC** | 0.30-0.60% | $500K-$1M+ | +| **Director** | 0.25-0.40% | $400K-$800K | +| **VP** | 0.50-1.00% | $800K-$2M | +| **C-Suite** | 1.00-3.00% | $2M-$5M+ | + +*Ranges are at Series B stage. Earlier: higher percentages, lower valuations. Later: lower percentages, higher valuations.* + +### Vesting Schedules + +| Schedule | Standard | Alternative | +|----------|----------|-------------| +| **Cliff (initial)** | 1 year | None (rare; attracts risk of early departures) | +| **Vesting (full)** | 4 years | 3 years (faster liquidity for talent), 5 years (longer retention) | +| **Vesting type** | Monthly (standard) | Quarterly (simpler admin), Annual (very rare) | +| **Acceleration** | Single trigger (acquired: unvested accelerates) | Double trigger (acquired + terminated: unvested accelerates) | + +## Leveling & Career Banding + +### Comp Band Structure + +| Level | Title (Engineering example) | Base band | Equity band | Total comp target | +|-------|----------------------------|-----------|-------------|-------------------| +| L3 | Junior Engineer | $80K-$120K | $10K-$30K/yr | $90K-$150K | +| L4 | Engineer | $100K-$150K | $30K-$60K/yr | $130K-$210K | +| L5 | Senior Engineer | $130K-$190K | $60K-$120K/yr | $190K-$310K | +| L6 | Staff Engineer | $160K-$230K | $120K-$200K/yr | $280K-$430K | +| L7 | Principal Engineer | $190K-$280K | $200K-$350K/yr | $390K-$630K | + +### Compensation Philosophy Principles + +1. **Pay for performance, not tenure** — Raises and bonuses correlate with impact, not time in seat +2. **Transparent bands** — Employees know the range for their level, reducing negotiation advantage and bias +3. **Internal equity** — Similar contribution receives similar comp regardless of negotiation skill or tenure +4. **Market-driven** — Bands adjust with market, not by individual request +5. **Total comp mindset** — Base + bonus + equity + benefits = total value. Communicate it that way. + +## Variable Pay & Incentives + +| Type | Who | Structure | Payout | +|------|-----|-----------|--------| +| **Annual bonus** | All employees | % of base (10-20% IC, 20-50% exec) | Based on company + individual performance | +| **Sales commission** | Sales team | % of ARR/bookings (10-20% SMB, 5-10% enterprise) | Paid on closed deals | +| **Spot bonus** | Any | One-time cash award ($500-$5,000) | Immediate recognition | +| **Retention bonus** | Critical at-risk talent | Lump sum with clawback | At anniversary or milestone | +| **Sign-on bonus** | New hires | One-time cash + possible clawback | At start date | + +## Compensation Reviews + +| Review type | Cadence | What changes | +|-------------|---------|-------------| +| **Merit increase** | Annual | Base salary adjustment (3-5% typical, higher for high performers) | +| **Promotion** | As earned | Level change, base increase (10-20% typical), equity refresh | +| **Equity refresh** | Annual or on promotion | Additional equity grant (new vesting schedule) | +| **Market correction** | Ad hoc | Base adjustment when market moves significantly between reviews | +| **Cost of living adjustment** | Annual (if separate from merit) | Across-the-board % increase | diff --git a/org-design/references/culture-architecture.md b/org-design/references/culture-architecture.md new file mode 100644 index 0000000..bc5c6cd --- /dev/null +++ b/org-design/references/culture-architecture.md @@ -0,0 +1,161 @@ +# Culture Architecture + +## Values Codification + +### Writing Meaningful Values + +Values are not aspirational statements — they are behavioral commitments that describe how work gets done. + +| Bad value | Why it fails | Good value | Why it works | +|-----------|-------------|------------|--------------| +| "Integrity" | Everyone claims it; means nothing specific | "Default to Transparency" | Actionable: share information even when it's uncomfortable | +| "Innovation" | Too vague; can't measure | "Bias for Action" | Directional: move fast over perfect | +| "Customer First" | Everyone says it; no tradeoff | "Customer Obsession" | Specific: go to unreasonable lengths for the customer | +| "Teamwork" | No behavioral anchor | "Disagree and Commit" | Normative: debate openly, commit to decision | + +### The Values Template + +Each value should have: + +``` +Name: [One memorable word or phrase] +Catchphrase: [One sentence — the principle itself] +Definition: [2-3 sentences explaining what it means] +Behavioral examples: + - [What it looks like in practice] + - [What it looks like in practice] + - [What it looks like in practice] +Tradeoffs: [What you're willing to sacrifice to uphold this value] +``` + +### Values in Practice + +| Process | How values are embedded | +|---------|------------------------| +| **Hiring** | Behavioral interview questions tied to values | +| **Performance reviews** | Values assessed alongside results | +| **Promotion decisions** | Demonstrated values over time required for advancement | +| **Firing decisions** | Values violations can be grounds regardless of performance | +| **Strategy** | "Does this decision align with our values?" as a strategic filter | + +## Rituals + +### Cadence of Rituals + +| Frequency | Ritual | Purpose | +|-----------|--------|---------| +| **Daily** | Standup | Alignment, blockers, small wins | +| **Weekly** | Team lunch / coffee | Social connection | +| **Bi-weekly** | Demo day | Show progress, celebrate work | +| **Monthly** | All-hands | Transparency, strategy, recognition | +| **Quarterly** | OKR review / planning | Goal alignment, reflection | +| **Semi-annual** | Hackathon / innovation week | Creativity, cross-team connections | +| **Annual** | Company offsite | Strategy, culture building, bonding | + +### Cultural Rituals — Examples + +| Ritual Type | Description | Example | +|-------------|-------------|---------| +| **Recognition** | Public acknowledgment of values-aligned behavior | "Kudos" channel, monthly awards | +| **Learning** | Structured knowledge sharing | Tech talks, book clubs, lunch-and-learn | +| **Debate** | Healthy disagreement structured into process | Red team / blue team reviews, pre-mortems | +| **Onboarding** | Culture immersion for new hires | First-week values deep-dive, buddy system | +| **Retrospective** | Regular process improvement | Sprint retro, incident post-mortem (blameless) | +| **Offboarding** | Knowledge preservation and relationship | Exit interview, knowledge base handoff | + +## Psychological Safety + +### The Four Stages (Timothy Clark) + +| Stage | Description | Team behavior | +|-------|-------------|---------------| +| **1. Inclusion Safety** | I belong here | Members feel welcomed and valued for who they are | +| **2. Learner Safety** | I can learn and grow here | Members feel safe to ask questions, make mistakes, admit gaps | +| **3. Contributor Safety** | I can make a difference here | Members feel safe to contribute ideas and effort | +| **4. Challenger Safety** | I can speak up and challenge here | Members feel safe to disagree, push back, and innovate | + +### Measuring Psychological Safety + +| Signal | Strong team | Weak team | +|--------|-------------|-----------| +| **Meeting participation** | Everyone speaks roughly equally | Dominated by 1-2 voices | +| **Admitting mistakes** | "I was wrong about that" is common | Mistakes are hidden or blamed | +| **Asking for help** | People regularly ask for input | People struggle alone | +| **Disagreement** | Respectful debate is the norm | Disagreement is avoided or escalates | +| **Failure response** | Treated as learning opportunity | Treated as performance failure | + +### Building Psychological Safety + +| Technique | Description | +|-----------|-------------| +| **Leader vulnerability** | Leaders model admitting mistakes and asking for help | +| **Framing work as learning** | "This is uncertain — we'll learn as we go" rather than "Execute perfectly" | +| **Setting explicit permission** | "I want everyone to challenge this plan — that's how we make it better" | +| **Responding productively** | When someone disagrees or admits a mistake, thank them and engage constructively | +| **Blameless post-mortems** | Focus on systems and processes, not individuals | +| **Encouraging questions** | No question is too basic; create dedicated "question time" | + +## Organizational Health Metrics + +### eNPS (Employee Net Promoter Score) + +``` +eNPS = % Promoters - % Detractors + +Question: "How likely are you to recommend [company] as a place to work?" +Scale: 0-10 + Promoters (9-10): Loyal, engaged + Passives (7-8): Satisfied but unenthusiastic + Detractors (0-6): Unhappy, likely to leave +``` + +**Benchmarks:** +| Score | Rating | +|-------|--------| +| >50 | Excellent | +| 30-50 | Good | +| 10-29 | Average | +| <10 | Concerning | + +### Employee Engagement Drivers + +| Driver | Weight (typical) | Measurement | +|--------|------------------|-------------| +| **Meaningful work** | 25% | "My work uses my skills well" | +| **Manager quality** | 20% | "My manager supports my development" | +| **Growth opportunity** | 20% | "I see a path for advancement" | +| **Compensation** | 15% | "I'm paid fairly for my role" | +| **Work environment** | 10% | "I have the tools and resources I need" | +| **Company direction** | 10% | "I believe in where the company is headed" | + +### Retention Risk Indicators + +| Indicator | Risk level | Intervention | +|-----------|------------|--------------| +| Low engagement score | Medium | Targeted manager coaching, skip-level | +| No promotion in 3+ years | Medium | Career pathing conversation | +| Compensation below market | High | Comp adjustment (equity or cash) | +| Manager change | Medium | Check-in after 90 days | +| Org restructuring | High | Communication, engagement pulse | +| High-performer quiet quitting | Critical | Stay interview, root cause analysis | + +## DEI Strategy + +### Inclusive Design Principles + +| Principle | Application | +|-----------|-------------| +| **Representation** | Diverse perspectives in product design, hiring panels, leadership | +| **Equitable access** | Remove barriers: accommodation, accessibility, flexible work | +| **Belonging** | Ensure all groups feel valued and included | +| **Equity over equality** | Different groups need different support to achieve equal outcomes | + +### DEI Maturity Model + +| Stage | Characteristics | Activities | +|-------|-----------------|------------| +| **1. Compliance** | Do the minimum to avoid legal risk | Anti-harassment training, EEO reporting | +| **2. Awareness** | Recognize gaps, start programs | Unconscious bias training, diversity recruiting | +| **3. Integration** | Embed DEI into processes | Diverse slates, pay equity audits, ERGs | +| **4. Accountability** | Measure outcomes, hold leaders responsible | DEI metrics in performance reviews, board diversity targets | +| **5. Systemic** | DEI is part of how the organization operates | Product accessibility, supplier diversity, inclusive design standards | diff --git a/org-design/references/organizational-design.md b/org-design/references/organizational-design.md new file mode 100644 index 0000000..cd30ca2 --- /dev/null +++ b/org-design/references/organizational-design.md @@ -0,0 +1,142 @@ +# Organizational Design + +## Team Topologies + +The four fundamental team types, from Matthew Skelton and Manuel Pais: + +### Stream-Aligned Team + +Aligned to a single, valuable stream of work (product, service, feature, user journey). + +| Characteristic | Description | +|----------------|-------------| +| **Focus** | Delivering value directly to the end customer | +| **Skills** | Cross-functional — development, testing, operations, product | +| **Boundary** | Owns one stream end-to-end | +| **Cognitive load** | Matched to the complexity of the stream | +| **Interaction mode** | Collaborates with enabling teams for capability building | + +**When to use:** Default team type for most modern product development. One team per stream of value. + +### Enabling Team + +Supports stream-aligned teams by building capabilities in specific technical domains. + +| Characteristic | Description | +|----------------|-------------| +| **Focus** | Increasing the effectiveness of other teams | +| **Skills** | Deep expertise in a specific domain (CI/CD, security, testing, observability) | +| **Boundary** | No permanent code ownership — builds capabilities, not products | +| **Lifecycle** | Exists as long as the capability gap exists | +| **Interaction mode** | Collaborates (short bursts) → transitions to facilitating (long-term) | + +**When to use:** When stream-aligned teams lack capability in a critical domain. Enable them to become self-sufficient, then dissolve or move to the next gap. + +### Complicated-Subsystem Team + +Owns a subsystem that requires specialized, deep expertise. + +| Characteristic | Description | +|----------------|-------------| +| **Focus** | Building and maintaining a technically complex component | +| **Skills** | Deep specialization in one domain (video encoding engine, ML model, payment reconciliation) | +| **Boundary** | Clear, well-defined API boundary with the rest of the system | +| **Interaction mode** | X-as-a-Service — stream-aligned teams consume via API | + +**When to use:** When a subsystem's complexity exceeds what a stream-aligned team can reasonably carry. The cognitive load of the subsystem is isolated to one team. + +### Platform Team + +Provides internal services and tools that reduce the cognitive load of stream-aligned teams. + +| Characteristic | Description | +|----------------|-------------| +| **Focus** | Building internal developer platform capabilities | +| **Skills** | Platform engineering, infrastructure, developer experience | +| **Boundary** | Self-service APIs and tools that stream-aligned teams consume | +| **Interaction mode** | X-as-a-Service — platform team provides, stream teams consume | +| **Mindset** | Platform is a product; stream teams are the customers | + +**When to use:** Beyond 5-8 stream-aligned teams, a platform team becomes necessary to prevent fragmentation and toil. + +### Team Topologies Interaction Modes + +| Mode | Description | When to use | +|------|-------------|-------------| +| **Collaboration** | Two teams work together for a limited time on a shared goal | New capability discovery, integration points | +| **X-as-a-Service** | One team consumes another's service via a well-defined interface | Platform → stream, complicated-subsystem → stream | +| **Facilitating** | One team helps another develop a capability, then steps back | Enabling → stream-aligned teams | + +## Span of Control + +| Layer | Ideal span | Notes | +|-------|------------|-------| +| **First-line manager** | 4-8 direct reports | Hands-on coaching, career development, technical guidance | +| **Director** | 3-6 managers | Coordination, strategy translation, resource allocation | +| **VP** | 3-5 directors | Organizational strategy, cross-functional alignment | +| **C-Suite** | 4-8 VPs/C-suite | Enterprise strategy, external representation | + +**Rule of thumb:** The more complex and interdependent the work, the narrower the span. The more standardized and independent, the wider. + +### The Dunbar-Heuristic for Teams + +- **5-8 people**: Optimal for stream-aligned teams (all members can maintain high-bandwidth relationships) +- **15-20 people**: Maximum for a single manager's span in complex work +- **50-150 people**: "Dunbar number" — maximum for a community where everyone can know each other +- **>150 people**: Formal processes, hierarchy, and systems are required + +## Reporting Structures + +### Functional (Traditional) + +``` +CEO +├── Engineering VP +│ ├── Engineering Manager → 5 engineers +│ └── Engineering Manager → 6 engineers +├── Product VP +│ ├── PM Director → 3 PMs +│ └── Design Director → 4 designers +└── Marketing VP + ├── Growth Director → 4 marketers + └── Brand Director → 3 marketers +``` + +**Best for:** Stable organizations, deep specialization, clear career ladders +**Worst for:** Cross-functional collaboration, customer-centricity + +### Matrix + +``` + Product A Product B Product C +Engineering Eng lead A Eng lead B Eng lead C +Design Designer A Designer B Designer C +Product PM A PM B PM C +``` + +**Best for:** Resource sharing across multiple priorities +**Worst for:** Clear decision-making, dual reporting creates tension + +### Product/Stream-Aligned (Modern) + +``` +CEO +├── Stream Alpha (cross-functional team) +├── Stream Beta (cross-functional team) +├── Platform Team +└── Enabling Team (CI/CD) +``` + +**Best for:** Speed, autonomy, customer focus +**Worst for:** Deep specialization (mitigated by enabling teams and communities of practice) + +## Organizational Health Metrics + +| Metric | What it measures | Diagnostic | +|--------|-----------------|------------| +| **Span of control ratio** | Middle management efficiency | >1:8 suggests potential bottlenecks or micromanagement | +| **Org depth** | Levels from IC to CEO | >5 levels slows decision-making | +| **Team size distribution** | How many teams are at optimal size | >8 members per team → likely fragmentation | +| **Dependency count** | Cross-team blockers | High → reorganize to reduce (Conway's Law) | +| **Decision velocity** | Time from proposal to decision | >2 weeks for operational decisions → process is broken | +| **Manager-to-IC ratio** | Proportion of people managers vs individual contributors | >1:3 might indicate too many managers | diff --git a/org-design/references/talent-strategy.md b/org-design/references/talent-strategy.md new file mode 100644 index 0000000..c3e79a2 --- /dev/null +++ b/org-design/references/talent-strategy.md @@ -0,0 +1,114 @@ +# Talent Strategy + +## Make vs Buy + +The fundamental talent decision — build capability internally or acquire it externally. + +### Decision Matrix + +| Factor | Make (Build/Develop) | Buy (Hire/Contract) | +|--------|---------------------|---------------------| +| **Time horizon** | Long-term capability (>2 years) | Immediate need (<6 months) | +| **Strategic importance** | Core competency, competitive advantage | Commodity skill, non-core | +| **Availability** | Plentiful in the market | Scarce or specialized | +| **Development cost** | Low (existing employees who can grow) | High (training new hires or contractors from scratch) | +| **Risk** | Takes time, may not succeed | Immediate, but higher hiring cost | + +### The Build Playbook + +| Phase | Action | Timeline | +|-------|--------|----------| +| **Assess** | Skills gap analysis, career path mapping | 2-4 weeks | +| **Plan** | Individual development plans, stretch assignments | 4 weeks | +| **Execute** | Training, mentorship, rotation programs | 6-24 months | +| **Measure** | Competency assessments, promotion readiness | Quarterly | + +### The Buy Playbook + +| Phase | Action | Timeline | +|-------|--------|----------| +| **Define** | Role scope, must-have vs nice-to-have skills | 1-2 weeks | +| **Source** | Internal referral, external recruiter, direct sourcing | 2-6 weeks | +| **Assess** | Technical interview, behavioral, work sample | 2-4 weeks | +| **Close** | Offer, negotiate, onboard | 2-4 weeks | + +## Skill Taxonomies + +### Building a Skills Framework + +A skills taxonomy organizes capabilities into a hierarchy that maps to career progression, compensation, and learning. + +``` +Level 0: Core competencies (leadership, communication, collaboration) +Level 1: Function-specific skills (engineering: system design, testing) +Level 2: Domain expertise (frontend: React, TypeScript, accessibility) +Level 3: Deep specialization (real-time graphics rendering, distributed consensus) +``` + +### Skills Matrix Example (Engineering) + +| Skill | L1 (Junior) | L2 (Mid) | L3 (Senior) | L4 (Staff) | L5 (Principal) | +|-------|-------------|----------|-------------|------------|-----------------| +| **System design** | Understands components | Designs within a service | Designs across services | Architecture for org | Architecture for industry | +| **Code quality** | Writes functional code | Writes testable code | Writes maintainable code | Sets standards | Advances the craft | +| **Mentorship** | Asks good questions | Helps peers | Mentors juniors | Develops org capability | Grows the industry | +| **Impact** | Owned tasks | Owned features | Owned epics | Owned org strategy | Owned company strategy | + +## Succession Planning + +### Pipeline Coverage + +| Role | Ready Now | Ready in 1-2 years | Ready in 3-5 years | +|------|-----------|--------------------|--------------------| +| CEO | 1 | 2 | 3 | +| CTO | 1 | 1 | 2 | +| VP Engineering | 2 | 2 | 3 | +| Director | 3 | 4 | 4 | + +**Bench strength:** For every critical role, you should have at least one person ready now and 2-3 in the pipeline. + +### Succession Planning Process + +``` +Identify critical roles + ↓ +Define success criteria (competencies, experience, results) + ↓ +Assess current pipeline against criteria + ↓ +Identify gaps → Develop gap-closure plan + ├── Targeted development assignments + ├── Sponsorship (executive visibility) + ├── Formal training / executive education + └── External hire (if gap can't be closed internally) + ↓ +Review quarterly (not annually — pipeline changes fast) +``` + +### The 9-Box Grid (Performance × Potential) + +| | Low Potential | Medium Potential | High Potential | +|---|--------------|-----------------|----------------| +| **High Performance** | Sustainer (support, don't promote) | Rising Star (develop for next level) | High Potential (accelerate development) | +| **Medium Performance** | Low Performer (PIP or exit) | Core Contributor (stable, valued) | Growth Opportunity (stretch assignment) | +| **Low Performance** | Exit Candidate | Underperformer (coach or exit) | Misaligned (role mismatch) | + +## Talent Review Cadence + +| Frequency | Activity | Participants | +|-----------|----------|--------------| +| **Weekly** | 1:1s — coaching, feedback, growth | Manager + direct report | +| **Monthly** | Talent spot check — emerging concerns, flight risks | Manager + skip-level (optional) | +| **Quarterly** | Talent review — performance calibration, promotions, pipeline assessment | Leadership team + HRBP | +| **Annually** | Strategic workforce planning — org design, skill forecasts, succession deep-dive | Executive team + HR leadership | + +## Key Talent Risk Indicators + +| Signal | Risk | Action | +|--------|------|--------| +| Low engagement score (eNPS <10) | Retention risk | Engagement survey follow-up, skip-level meetings | +| Decline in performance review rating | Burnout or misalignment | Check-in with manager, workload review | +| No promotion in 3+ years | Stagnation risk | Career discussion, development plan | +| Counter-offer declined | Exit risk | Stay interview, understand drivers | +| Multiple team members leaving | Manager risk | 360-degree review of the manager | +| Market competitor hiring spree | Talent raid risk | Compensation review, retention bonuses for critical roles | diff --git a/platform-engineering/README.md b/platform-engineering/README.md new file mode 100644 index 0000000..028ea2c --- /dev/null +++ b/platform-engineering/README.md @@ -0,0 +1,26 @@ +# Platform Engineering + +Infrastructure as code, CI/CD, container orchestration, service networking — methodology and reference patterns for building and operating internal developer platforms. + +## Why Install This Skill + +Your agent gets a structured loading order and dense reference patterns for IaC, CI/CD, containers, networking, secrets, and observability — the full platform stack. + +## What You Get + +| Directory | Purpose | +|-----------|---------| +| `SKILL.md` | Core methodology, trigger conditions, reference index | +| `references/` | Deep-dive reference files loaded on demand | + +## Triggers + +Designing CI/CD pipelines, writing Terraform/Pulumi, deploying containers, configuring service networking, managing secrets, or building observability stacks. + +## Requirements + +Platform-agnostic. References cover Terraform, Kubernetes, Helm, ArgoCD, Vault, Prometheus, Traefik, and Tailscale. + +## Quick Start + +Load SKILL.md for the methodology overview and reference table, then load specific references as needed for the task at hand. diff --git a/platform-engineering/SKILL.md b/platform-engineering/SKILL.md new file mode 100644 index 0000000..5690cae --- /dev/null +++ b/platform-engineering/SKILL.md @@ -0,0 +1,88 @@ +--- +name: platform-engineering +description: Infrastructure as code, CI/CD, container orchestration, service networking + — methodology and reference patterns for building and operating internal developer + platforms. +license: MIT +metadata: + tags: '' + source_repo: https://github.com/magnus919/hermes-profiles +--- + +# Platform Engineering + +Core methodology and reference library for platform engineering work. This skill does not execute operations itself — it provides the frameworks, patterns, and reference material that a platform-engineer profile loads on demand. + +## When to Load + +Load this skill when the task involves: + +| Trigger | What's Needed | +|---------|---------------| +| Design a CI/CD pipeline | Pipeline structure, GitOps sync strategies, release automation | +| Infrastructure as code plan | Terraform/OpenTofu module patterns, state management, Pulumi/Ansible patterns | +| Container orchestration design | K8s pod lifecycle, Helm chart conventions, Kustomize overlays, Docker Compose | +| Service networking / mesh | Traefik/nginx/Caddy config, Tailscale/Headscale ACL, WireGuard, service mesh | +| Observability strategy | Prometheus rules, Grafana dashboards-as-code, Loki logging, tracing | +| Secret management design | Vault, SOPS, External Secrets Operator patterns | +| Cloud architecture assessment | Multi-cloud patterns, provider foundations, cost governance | +| Deployment pipeline review | End-to-end delivery pipeline audit, release engineering patterns | + +## Loading Order + +``` +skill_view('platform-engineering') # This — methodology index +skill_view('artifact-pyramids') # Output contract +skill_view('docker-management') # Container lifecycle (if needed) +skill_view('traefik') # Reverse proxy (if needed) +skill_view('tailscale') # Mesh networking (if needed) +skill_view('implementation-planning') # Work breakdown (if needed) +``` + +Then load domain-specific references from this skill: + +``` +skill_view('platform-engineering', file_path='references/ci-cd-pipelines.md') +skill_view('platform-engineering', file_path='references/infrastructure-as-code.md') +# ... etc per domain +``` + +## Reference Files + +| Reference | Purpose | +|-----------|---------| +| `references/ci-cd-pipelines.md` | GitHub Actions, GitLab CI, Forgejo CI, Jenkins, CircleCI; GitOps with ArgoCD/Flux; release automation | +| `references/container-orchestration.md` | K8s/k3s, Helm chart conventions, Kustomize overlays, RBAC patterns, Docker Compose production patterns | +| `references/infrastructure-as-code.md` | Terraform/OpenTofu module design, state backends, Pulumi project structure, Ansible roles, CloudFormation/CDK | +| `references/service-networking.md` | Reverse proxy config (Traefik, nginx, Caddy), Tailscale/Headscale ACL, WireGuard topology, service mesh (Istio, Cilium) | +| `references/observability.md` | Prometheus recording rules/alerting, Grafana dashboards-as-code, Loki log aggregation, OpenTelemetry tracing | +| `references/secret-management.md` | HashiCorp Vault auth/policies, SOPS/age encryption in Git, External Secrets Operator, Sealed Secrets | +| `references/cloud-platforms.md` | AWS/GCP/Azure foundational services, multi-cloud design, cost governance, provider abstraction | +| `references/automation-languages.md` | Go CLI patterns, Python SDK integration, Bash bootstrap/conventions for platform tooling | +| `references/release-engineering.md` | Container image lifecycle, artifact versioning strategies, release gate checklists, Helm chart promotion | + +## Output Contract + +The profile using this skill produces artifact pyramids. The response to any caller is the absolute path to `00-index.md`. See `artifact-pyramids` skill for the specification. + +## Design Principles + +1. **The platform is a product.** Internal developers are your customers. Their productivity, satisfaction, and cognitive load are the primary metrics. +2. **Golden paths, not golden cages.** Provide paved roads for common workflows but allow escape hatches. Make the right thing easy, not the wrong thing impossible. +3. **Reduce cognitive load.** Abstract infrastructure complexity. Developers should not need to understand Kubernetes internals or Terraform state management to deploy their service. +4. **Everything as code.** Infrastructure, configuration, pipelines, and policies are version-controlled, reviewed, and reproducible. Git is the single source of truth. +5. **Self-service over tickets.** Every manual handoff between teams is a bottleneck. If a developer needs another team to deploy, the platform is incomplete. +6. **Automation is the default.** If a process can be automated, it must be. Manual operations are toil — tax on the organization. +7. **Observability is infrastructure.** Logs, metrics, traces, and dashboards are platform contract, not optional extras. Every service gets them by default. +8. **Security is built in, not bolted on.** Supply chain security, secret management, vulnerability scanning, and policy enforcement are platform responsibilities. +9. **API-first design.** Everything the platform does should be accessible via API — enabling automation, self-service portals, and CLI tools. + +## Related Skills + +- `artifact-pyramids` — output contract specification +- `docker-management` — container lifecycle and Compose +- `traefik` — reverse proxy and ingress configuration +- `tailscale` — mesh networking and ACL policies +- `implementation-planning` — work breakdown and dependency ordering +- `mermaid-diagrams` — architecture diagram generation +- `site-reliability-engineering` — sister domain (post-deployment reliability) diff --git a/platform-engineering/references/automation-languages.md b/platform-engineering/references/automation-languages.md new file mode 100644 index 0000000..2d16c38 --- /dev/null +++ b/platform-engineering/references/automation-languages.md @@ -0,0 +1,19 @@ +# Automation Languages — Reference + +## Go + +- **Use in platform engineering:** CLI tools (Cobra/Viper), Kubernetes operators (controller-runtime), Terraform providers (terraform-plugin-framework), ingress controllers, service mesh sidecars, infrastructure agents +- **Key patterns:** `os/exec` for running system commands, `os/signal` for graceful shutdown, `net/http` for API clients, `cobra.Command` for CLI structure, `viper` for config loading, `retry` patterns via backoff +- **Best practices:** Single binary deployment, cross-compilation (`GOOS=linux GOARCH=arm64`), no runtime dependencies, `go vet` + `staticcheck` in CI, readability over cleverness + +## Python + +- **Use in platform engineering:** Automation scripts, cloud SDK clients (boto3, google-cloud, azure-mgmt), CI/CD pipeline scripts, configuration validation, integration testing, internal tools +- **Key patterns:** `argparse`/`click` for CLI, `httpx`/`requests` for API calls, `pydantic` for config validation, `pyyaml` for config parsing, `rich`/`click` for CLI output formatting, `pathlib` for file operations +- **Best practices:** Type hints everywhere (mypy strict), `if __name__ == "__main__":` entry point, installable via `pip install` (entry_points in setup.cfg/pyproject.toml), dependency pinning for reproducibility, `--dry-run` flag on all mutating operations + +## Bash / POSIX Shell + +- **Use in platform engineering:** Bootstrap scripts, CI/CD glue, Dockerfile commands, container entrypoints, developer tool wrappers, provisioning one-shots +- **Key patterns:** `set -euo pipefail` for safety, argument parsing with `getopts` or `while case`, `mktemp` for temp files, `trap cleanup EXIT` for teardown, `${var:-default}` and `${var:?required}` patterns +- **Best practices:** ShellCheck in CI, prefer `[[ ]]` over `[ ]` in Bash, quote all variable expansions, use `printf` over `echo`, keep scripts short (beyond ~100 lines → Python or Go), `set -x` for debugging in development only diff --git a/platform-engineering/references/ci-cd-pipelines.md b/platform-engineering/references/ci-cd-pipelines.md new file mode 100644 index 0000000..ee49ef3 --- /dev/null +++ b/platform-engineering/references/ci-cd-pipelines.md @@ -0,0 +1,62 @@ +# CI/CD Pipelines — Reference + +## Pipeline Platforms + +### GitHub Actions + +- **Workflow structure:** `.github/workflows/*.yml` — triggers, jobs, steps, matrix builds +- **Key patterns:** reusable workflows (`uses:` with `{owner}/{repo}/.github/workflows/{name}@{ref}`), composite actions, OIDC for cloud auth, artifacts/pages for delivery +- **Secrets:** GitHub Actions secrets, environment-level secrets, OIDC as an alternative to static keys +- **Matrix builds:** `strategy.matrix` for cross-platform/testing, `fail-fast` for early exit +- **Self-hosted runners:** scale sets, labels, network isolation, ephemeral runners + +### GitLab CI + +- **Pipeline structure:** `.gitlab-ci.yml` — stages, jobs, needs (DAG), artifacts, cache +- **Key patterns:** multi-project pipelines, parent-child pipelines, merge request pipelines, scheduled pipelines +- **Runners:** shared vs specific, Docker executor, Kubernetes executor, tags, concurrency limits +- **Registry:** GitLab Container Registry integration, dependency proxy + +### Forgejo CI / Gitea Actions + +- **Structure:** `.forgejo/workflows/*.yml` or `.gitea/workflows/*.yml` — compatible with GitHub Actions syntax +- **Runners:** Forgejo Runner (act-based), self-hosted, labels for platform targeting +- **Key differences from GitHub Actions:** Lighter ecosystem, smaller action marketplace, often need to self-host runners +- **Secrets:** Forgejo repository/organization secrets, no OIDC built-in (use manual token exchange) + +### Jenkins + +- **Pipeline structure:** `Jenkinsfile` — declarative (`pipeline { }`) vs scripted (`node { }`) +- **Key concepts:** agents, stages, steps, post-build actions, shared libraries, Blue Ocean +- **Cloud integration:** Jenkins X for Kubernetes, plugin ecosystem, custom agents via Docker + +### CircleCI + +- **Pipeline structure:** `.circleci/config.yml` — orbs, executors, jobs, workflows (DAG) +- **Key concepts:** contexts (env sharing), workspaces/persist-to-workspace, parallelism, test splitting +- **Orbs:** reusable config packages (official and community orbs for AWS, Slack, browsers, etc.) + +## GitOps + +### Argo CD + +- **Core model:** Declarative GitOps — desired state in Git repository, Argo CD syncs to cluster +- **Key concepts:** Applications, Projects, Sync strategies (auto/manual), sync waves, prune policies, health checks +- **Multi-cluster:** Hub-and-spoke, cluster registration, RBAC per cluster +- **Progressive delivery:** Rollouts, canary deployments, blue-green, traffic mirroring (Argo Rollouts add-on) +- **Patterns:** App-of-apps, Kustomize/Helm integration, config management plugins (CMP), ApplicationSets for multi-env/deployment + +### Flux + +- **Core model:** GitOps toolkit — source → kustomize/helm → sync to cluster +- **Key components:** Source Controller, Kustomize Controller, Helm Controller, Notification Controller, Image Automation +- **Key concepts:** GitRepository/Bucket sources, Kustomization/HelmRelease, OCIRepository, ImagePolicy +- **Multi-tenancy:** Namespace isolation, cross-namespace references, access controls + +## Release Automation + +- **Semantic versioning:** `MAJOR.MINOR.PATCH` — breaking changes, features, fixes; pre-release suffixes, build metadata +- **Changelog generation:** Conventional Commits → automated changelog (git-cliff, standard-version, semantic-release) +- **Artifact provenance:** SLSA levels, attestation (in-toto), SBOM generation (Syft, Trivy), signing (Cosign) +- **Release gates:** Manual approvals (GitHub Environments, GitLab Deployments), automatic rollback on health check failure +- **Artifact registries:** Container registries (Docker Hub, GHCR, GitLab Registry, ECR, GAR), package registries (NPM, PyPI, Maven) diff --git a/platform-engineering/references/cloud-platforms.md b/platform-engineering/references/cloud-platforms.md new file mode 100644 index 0000000..0f81bb3 --- /dev/null +++ b/platform-engineering/references/cloud-platforms.md @@ -0,0 +1,23 @@ +# Cloud Platforms — Reference + +## AWS + +- **Core services:** VPC (subnets, route tables, NAT, security groups, NACLs, VPC peering, Transit Gateway), EC2 (instances, AMIs, auto-scaling, launch templates, spot), EKS (managed K8s, node groups, Fargate, IRSA), S3 (buckets, versioning, lifecycle, replication, presigned URLs), IAM (users, roles, policies, instance profiles, OIDC), Route53 (DNS, alias records, health checks, routing policies) +- **Common patterns:** Shared VPC (central networking team), multi-account (Control Tower, Organization, SCPs), IRSA for EKS pod IAM, S3 backend for Terraform state (bucket + DynamoDB lock), CodeBuild/CodePipeline for CI, CloudFront for CDN + +## GCP + +- **Core services:** VPC (subnets, firewall rules, Cloud NAT, VPC peering, Shared VPC), GKE (K8s, node auto-repair/auto-upgrade, Workload Identity for pod IAM), Cloud Storage (buckets, nearline/archive, object lifecycle), IAM (roles, custom roles, service accounts, Workload Identity Federation), Cloud DNS (managed zones, DNS forwarding, policy-based routing) +- **Common patterns:** Shared VPC (host project + service projects), workload identity federation (no static keys), Artifact Registry, Cloud Build CI, Terraform state via Cloud Storage + +## Azure + +- **Core services:** VNet (subnets, NSGs, Azure Bastion, VPN Gateway, VNet peering), AKS (K8s, node pools, managed identity, Azure AD integration), Blob Storage (containers, tiers, lifecycle, Azure Files), RBAC (roles, custom roles, managed identities, service principals), DNS (public/private zones, alias records, Azure DNS Private Resolver) +- **Common patterns:** Hub-and-spoke networking (central firewall), managed identity for pod IAM (AKS with aad-pod-identity), Terraform state via Azure Storage, Azure DevOps pipelines + +## Multi-Cloud and Abstraction + +- **Abstraction layers:** Terraform/OpenTofu providers — write once, target any cloud (with provider-specific variance). Pulumi similarly abstracts. Crossplane for K8s-native cloud resource provisioning +- **Governance cost:** State isolation per cloud, IAM duplication per provider, network egress charges (Free Tier per cloud but real cost at scale), skills distribution across cloud teams +- **When multi-cloud is worth it:** Regulatory (data residency), avoiding single-vendor lock-in for critical few services (object storage, K8s), acquisition integration. It is NOT a cost-savings strategy. +- **Cost governance:** Budget alerts (each cloud), tagging policies (`CostCenter`, `Environment`, `Owner`), right-sizing, reserved instances/committed use discounts, spot/preemptible for batch, storage tier policies diff --git a/platform-engineering/references/container-orchestration.md b/platform-engineering/references/container-orchestration.md new file mode 100644 index 0000000..7a55d6e --- /dev/null +++ b/platform-engineering/references/container-orchestration.md @@ -0,0 +1,29 @@ +# Container Orchestration — Reference + +## Kubernetes and k3s + +- **Core objects:** Pod, Service, Deployment, StatefulSet, DaemonSet, Ingress, ConfigMap, Secret, PersistentVolume/PVC, Namespace, RBAC (Role/ClusterRole/RoleBinding/ClusterRoleBinding), NetworkPolicy +- **Lifecycle:** Rolling updates, recreate, canary via flags, blue-green via Services; readiness/liveness/startup probes, preStop hooks, pod disruption budgets +- **Scheduling:** nodeSelector, node affinity/anti-affinity, pod affinity/anti-affinity, taints and tolerations, topology spread constraints, resource limits/requests, QoS classes (Guaranteed/Burstable/BestEffort) +- **k3s specifics:** Lightweight Kubernetes (single binary), embedded etcd (or SQLite), Traefik as default ingress, servicelb, HelmController, local-path-provisioner, useful for edge/IoT/development + +## Helm + +- **Chart structure:** `Chart.yaml`, `values.yaml`, `templates/`, `charts/` (dependencies), `crds/`, `templates/NOTES.txt`, `templates/tests/` +- **Conventions:** helper templates (helpers.tpl), named templates, required values with `required`, global values for cross-chart values, conditionally-enabled subcharts +- **Lifecycle:** `helm create`, `helm lint`, `helm template`, `helm install --values`, `helm upgrade --install`, `helm rollback`, `helm uninstall`, `helm dependency update` +- **Repos:** ChartMuseum, OCI registries (GHCR, ECR, ACR), `helm repo add/update` +- **Best practices:** Pin dependency versions, set resource limits, use release namespace, separate env values files, enable/disable subcharts via `tags` or `condition` + +## Kustomize + +- **Core model:** Base + overlays — base sets common config, overlays apply env-specific patches +- **Key directives:** `resources`, `patchesStrategicMerge`, `patchesJson6902`, `configMapGenerator`, `secretGenerator`, `namePrefix`, `namespace`, `commonLabels`, `images`, `replicas`, `vars` +- **Patterns:** Multi-environment (dev/staging/prod), multi-cluster, component-based composition, in-line vs file-sourced patches +- **Integration:** `kustomize build` as input to kubectl or ArgoCD, `kustomize edit` for interactive modification + +## Docker Compose (Production Patterns) + +- **Production concerns:** Health checks (`healthcheck:`), restart policies (`unless-stopped/always`), resource limits (`deploy.resources.limits`), logging drivers, network isolation (custom networks), volume management (named volumes, bind mounts, tmpfs) +- **Multi-service patterns:** Depends-on with health check wait, init containers, sidecar containers (nginx reverse proxy, log shipper), environment file separation (`.env`, multiple `--env-file`) +- **Orchestration compatibility:** Compose file can be used directly (single host), converted to K8s via `kompose`, or used as a local dev environment matching production topology diff --git a/platform-engineering/references/infrastructure-as-code.md b/platform-engineering/references/infrastructure-as-code.md new file mode 100644 index 0000000..1f61897 --- /dev/null +++ b/platform-engineering/references/infrastructure-as-code.md @@ -0,0 +1,30 @@ +# Infrastructure as Code — Reference + +## Terraform / OpenTofu + +- **Core concepts:** Resources, data sources, providers, state (local, remote backends), modules, variables, outputs, lifecycle rules (`create_before_destroy`, `prevent_destroy`) +- **State management:** Remote backends (S3 + DynamoDB, GCS, Azure Storage, Terraform Cloud), state locking, state migration, workspaces for env separation, `terraform state` subcommands (mv, rm, pull, push) +- **Module design:** Composition (call smaller modules), version pinning, registry conventions (hashicorp/terraform-google-modules), output minimal surface area, internal vs published modules +- **Advanced patterns:** `for_each`/`count` for dynamic resources, `templatefile` for config injection, file/external data sources for bridge to external systems, provisioners as last resort (remote-exec/local-exec) +- **OpenTofu specifics:** Drop-in Terraform replacement, same HCL syntax, OSS license (no BSL change), `tofu` CLI, supports encryption at rest in state natively, enhanced provider signing + +## Pulumi + +- **Core model:** Infrastructure as real code — Go, Python, TypeScript, .NET, Java, YAML +- **Key concepts:** Programs (stack definitions), stacks (env instances), resources, components (custom abstractions), providers (Pulumi-native, TF bridge), outputs, config/secret management +- **State:** Pulumi Cloud (managed), self-managed backends (S3, GCS, Azure Blob S3-compatible), state encryption +- **Automation API:** Embed Pulumi in applications (CI/CD, self-service platforms), inline updates, preview + deploy in code +- **Bridge to Terraform:** TF bridge adapter wraps existing TF providers as native Pulumi providers — convenient but adds a layer + +## Ansible + +- **Core model:** Agentless — SSH/WinRM transport, push-based, YAML playbooks, Jinja2 templating +- **Key concepts:** Inventory (static, dynamic from cloud APIs), modules (idempotent operations), roles (reusable content packages), playbooks (execution order), variables and facts, handlers (notify-based triggers) +- **Best practices:** Role-based layout, vault for secrets, molecule for testing, ansible-lint, `--check --diff` for dry-run, `--limit` for targeted execution +- **Use case in platform engineering:** Day-2 configuration (post-provisioning), OS hardening, agent installation, but generally less suited than Terraform for cloud resource provisioning + +## CloudFormation / CDK + +- **CloudFormation:** Native AWS IaC — JSON/YAML templates, stacks, nested stacks, change sets, drift detection, stack sets (multi-account, multi-region) +- **CDK (Cloud Development Kit):** CloudFormation as real code (TypeScript, Python, Go, Java, C#) — constructs (L1/L2/L3 abstraction), `cdk synth` → CloudFormation template, `cdk deploy` / `cdk diff`, context, aspects, permissions boundaries +- **CDKTF (CDK for Terraform):** Bridge for Terraform providers in CDK languages — cross-platform between AWS and non-AWS providers diff --git a/platform-engineering/references/observability.md b/platform-engineering/references/observability.md new file mode 100644 index 0000000..56d1d43 --- /dev/null +++ b/platform-engineering/references/observability.md @@ -0,0 +1,30 @@ +# Observability — Reference + +## Metrics: Prometheus + +- **Core model:** Pull-based timeseries — scrape targets, service discovery, metric exposition format (`/metrics` endpoint) +- **Metric types:** Counter (cumulative, only increases), Gauge (up/down), Histogram (bucketed durations/sizes), Summary (quantile-based) +- **Recording rules:** Compute new timeseries from existing ones — `rate(...)[5m]` for per-second averages, `histogram_quantile()` for latency percentiles, aggregation via `sum() by ()` / `topk()`. Stored as new metric in Prometheus, faster than ad-hoc query +- **Alerting rules:** Vector → alert — `for:` duration eliminates flapping, severity labels (critical/warning/info), routing via Alertmanager to PagerDuty, Slack, email, etc. +- **Service discovery:** Kubernetes (pod annotations, kubelet), file_sd (JSON/YAML files), Consul, EC2, DNS +- **Best practices:** Use `rate()` not `irate()`, prefer histograms over summaries (aggregatable), label hygiene (cardinality limits, structured label naming) + +## Dashboards: Grafana + +- **Core model:** Data source abstraction — panel types (time series, bar, stat, table, gauge, logs), variables (interval, datasource filter), dashboard-as-code via JSON provisioning +- **Provisional dashboards:** JSON files in `provisioning/dashboards/` — auto-imported on Grafana startup. YAML datasource config in `provisioning/datasources/`. Version-controlled in Git alongside application config +- **Key patterns:** Template variables for environment switching, repeat panels per label value, annotations from Prometheus alerts, mixed data sources per panel, transformations (merge, group by, rename) +- **Best practices:** Single dashboard per service, row per concern (traffic, errors, latency, saturation), no more than 15 panels per row, dashboard links for navigation, `$__interval` for adaptive time range + +## Logs: Loki + +- **Core model:** Label-based log aggregation — indexes labels (not full text), stores compressed chunks in object storage. Promtail/Alloy/Fluent Bit for log shipping +- **LogQL:** `{label=~"value"} |= "error" \| json` for label matchers + content filters + pipeline stages. `rate()` for log error rates, `count_over_time()` for volume monitoring +- **Best practices:** CRI-O/Docker log format handling, structured logging (JSON), label cardinality limits, retention per storage tier (hot/warm/cold), multi-tenancy via label enforcement + +## Tracing: OpenTelemetry + +- **Core model:** Spans (operation units) → Traces (span DAG) → context propagation via W3C TraceContext headers +- **Signals:** Traces (request flow), Metrics (OTLP), Logs (via OTLP or file export) — unified in OpenTelemetry Collector +- **Components:** SDK (instrumentation libraries for Go, Python, JS, Java, etc.), Collector (receiver → processor → exporter pipeline), sampling (head-based, tail-based for storage cost management) +- **Instrumentation:** Auto-instrumentation (agent injection for Java/Python/.NET/Node), manual instrumentation (create spans, add attributes/events), existing library instrumentation (HTTP, gRPC, DB clients, messaging) diff --git a/platform-engineering/references/release-engineering.md b/platform-engineering/references/release-engineering.md new file mode 100644 index 0000000..94aeade --- /dev/null +++ b/platform-engineering/references/release-engineering.md @@ -0,0 +1,51 @@ +# Release Engineering for Infrastructure Artifacts + +Release engineering for infrastructure artifacts differs from open source releases. Infrastructure artifacts — container images, Helm charts, Terraform modules, Compose stacks — have different publication patterns, versioning strategies, and verification requirements. + +## Artifact Types and Publication Targets + +| Artifact Type | Publication Target | Versioning | Verification | +|--------------|-------------------|------------|--------------| +| Container image | Container registry (GHCR, Docker Hub, ECR) | Git SHA + semver tags | Image scan, size check, smoke test | +| Helm chart | OCI registry, chart repo | Chart.yaml version, semver | `helm template` dry-run, install test | +| Terraform module | Git tag, registry | Semver tag | `terraform validate`, `tflint` | +| Docker Compose stack | Git tag | Semver or date tag | `docker compose config`, smoke test | +| Internal package | Private registry (PyPI, npm) | Semver + pre-release label | Build, install, import test | + +## Container Image Lifecycle + +``` +Build → Tag → Scan → Sign → Push → Verify → Deploy +``` + +| Step | Tooling | Success Criteria | +|------|---------|-----------------| +| Build | `docker build`, `buildkit`, `ko` | Exit code 0, image ID captured | +| Tag | `docker tag` with SHA + semver + "latest" | Tags applied to build image | +| Scan | Trivy, Grype, Snyk | No critical/high CVEs, or exceptions documented | +| Sign | cosign | Signature attached to image | +| Push | `docker push`, `crane` | Registry confirms digest | +| Verify | `cosign verify`, digest comparison | Signature valid, digest matches | +| Deploy | Helm upgrade, kubectl apply, compose up | Pods healthy, endpoint responds | + +## Versioning Strategies + +| Strategy | When to Use | Example | +|----------|-------------|---------| +| Git SHA only | Development, CI-only artifacts | `sha-a1b2c3d` | +| Semver tags | Public-facing releases, API consumers | `v1.2.3` | +| Date-based | Internal rollups without consumer compatibility | `2026-06-01` | +| Semver + pre-release | Release candidates, staging deployments | `v1.2.3-rc.1` | +| Git SHA + semver | Production — both for traceability and semantics | `v1.2.3-sha-a1b2c3d` | + +## Release Gate Checklist + +Before promoting any artifact to production: + +- [ ] All tests pass (unit, integration, E2E) +- [ ] Security scan passes with no unexcepted critical/high CVEs +- [ ] Image is signed (for containerized deployments) +- [ ] Changelog entry exists for the release +- [ ] Rollback plan documented (previous version, restore command) +- [ ] Smoke test passes against staging +- [ ] Release notes drafted for communication diff --git a/platform-engineering/references/secret-management.md b/platform-engineering/references/secret-management.md new file mode 100644 index 0000000..5df546b --- /dev/null +++ b/platform-engineering/references/secret-management.md @@ -0,0 +1,31 @@ +# Secret Management — Reference + +## HashiCorp Vault + +- **Core model:** Dynamic secrets — generated on-demand (database credentials, cloud access keys, PKI certificates), short TTL, automatic revocation +- **Auth methods:** Token, Kubernetes (service account JWT → vault token), LDAP, OIDC, AppRole (machine-to-machine), AWS/GCP/Azure, GitHub +- **Secret engines:** KV (static, versioned), Database (dynamic DB creds), AWS/GCP/Azure (dynamic cloud creds), PKI (leaf certs), Transit (encryption as a service, data never leaves client) +- **Policies:** Path-based access control (`path "secret/data/app/*" { capabilities = ["read", "list"] }`), templating (`{{identity.entity.name}}`), fine-grained CRUD + deny + sudo +- **Patterns:** Sidecar injector (auto-auth, auto-renew), Vault Agent for caching/templating, kubernetes secrets via CSI provider, Terraform Vault provider, ACL templating for per-app secrets + +## SOPS / age + +- **Core model:** Encrypted files in Git — `sops --encrypt` (age or PGP), `sops --decrypt`, encrypted file is valid YAML/JSON with encrypted fields as `ENC[AES256_GCM,...]` +- **Encryption backends:** age (modern, key-based), PGP (traditional), AWS KMS, GCP KMS, Azure Key Vault, HashiCorp Vault +- **Workflow:** `.sops.yaml` config — creation rules per file path, key list for multi-key encryption (dev team key + CI key). Secrets files committed alongside code, CI decrypts at deploy time +- **CI/CD integration:** `sops --decrypt` in pipeline (using CI system's key access), age key securely injected (not in repo), KMS-based for cloud-native CI +- **Limitations:** No secret rotation (re-encrypt manually), no access audit, suitable for static config secrets but not dynamic credentials + +## External Secrets Operator (Kubernetes) + +- **Core model:** CRD-based — ExternalSecret resource syncs from external API to Kubernetes Secret. One-time sync or polling +- **Backends:** AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, HashiCorp Vault, Akeyless, GitLab, SOPS-encrypted files +- **Patterns:** `refreshInterval` for periodic sync, `target` to control created secret name/type, `data` for static key mapping, `dataFrom` for bulk (all keys from remote), `remoteRef` strategies (property, version) +- **Best practices:** Namespace-scoped `ClusterSecretStore` vs `SecretStore`, push secret reconciliation errors to monitoring, avoid over-polling (set realistic `refreshInterval`) + +## Sealed Secrets + +- **Core model:** Encrypt secrets client-side — `SealedSecret` CRD (controller decrypts, creates regular Secret in cluster) +- **Workflow:** Developer creates `SealedSecret` YAML with `kubeseal` using cluster's public cert. Committed to Git. Controller on cluster decrypts and materializes `Secret` when applied. +- **Best for:** GitOps workflows where secrets must be in Git but cannot be stored in plaintext +- **Limitations:** Static only (no rotation), cert management (backup cluster key), no external API integration, per-cluster certs (same sealed secret won't work across clusters) diff --git a/platform-engineering/references/service-networking.md b/platform-engineering/references/service-networking.md new file mode 100644 index 0000000..903122b --- /dev/null +++ b/platform-engineering/references/service-networking.md @@ -0,0 +1,53 @@ +# Service Networking — Reference + +## Reverse Proxy + +### Traefik + +- **Core model:** Dynamic routing — auto-discovers services via providers (Docker, Kubernetes, Consul, file), hot-reloads config +- **Key concepts:** Routers (HTTP/HTTPS/TCP/UDP), middlewares (rate limiting, auth, headers, retries, circuit breakers, compression, redirect), services (load balancing), entrypoints (ports), TLS automation via Let's Encrypt +- **Providers:** Docker provider (labels on containers), Kubernetes provider (IngressRoute CRD, or standard Ingress), file provider (static/dynamic YAML/TOML), Consul, etcd, Redis, ZooKeeper +- **Middleware chains:** Order matters — rate limit → auth → headers → retry; custom middleware via plugins (WebAssembly, Go), pass through ForwardAuth to external services +- **Observability:** Metrics (Prometheus, OpenTelemetry, Datadog, InfluxDB), access logs (in JSON), tracing (Jaeger, Zipkin, OpenTelemetry), dashboard UI + +### nginx + +- **Core model:** Static config (reload on change), high-concurrency event loop, reverse proxy, load balancer, TLS termination, caching +- **Key patterns:** `upstream` blocks for load balancing, `proxy_pass` for forwarding, `location` blocks for path routing, `map` for conditional logic, `limit_req`/`limit_conn` for rate limiting +- **Config management:** Templating (Jinja, envsubst), include directories for modular config, nginx -t for validation, reload via SIGHUP + +### Caddy + +- **Core model:** Automatic HTTPS (ZeroSSL/Lets Encrypt by default), simple `Caddyfile` syntax, JSON API for dynamic config +- **Key features:** HTTP/3 (QUIC) by default, on-demand TLS, HTTP->HTTPS redirects, reverse proxy with health checking, match blocks, matchers +- **Best for:** Simple deployments where Traefik's dynamic service discovery is overkill; excellent developer experience + +## Mesh Networking + +### Tailscale / Headscale + +- **Core model:** WireGuard-based overlay network — nodes get unique Tailscale IP, communicate directly (NAT traversal), ACLs control access +- **ACL policy (huJSON):** `acls` (src/dest/proto/port), `groups` (user groupings), `tags` (device identity), `hosts` (alias mapping), `derpMap` relay configuration, `ssh` for Tailscale SSH +- **Key features:** Subnet routing (advertise routes), exit nodes (traffic to internet), ACL deny rules (refuse trailing), Funnel (allow internet traffic to local), Serve (host services on tailnet), MagicDNS +- **Headscale specifics:** Self-hosted control server, open-source, PostgreSQL/ SQLite backend, OIDC integration, CLI (`headscale users`, `headscale nodes`, `headscale routes`), DERP relay server setup +- **Lifecycle:** Node registration (pre-auth keys, web auth), expiry/node cleanup, key rotation (node keys, auth keys), multi-tailnet federation (Headscale sharing) + +### WireGuard + +- **Core model:** Layer 3 secure tunnel — single UDP port, kernel-level (fast), peer-to-peer, pre-shared or public-key auth +- **Config basics:** Interface (private key, address, listen port, DNS), Peer (public key, allowed IPs, endpoint, persistent keepalive) +- **Topology patterns:** Point-to-point (simple site-to-site), hub-and-spoke (central node routes), mesh (direct peer-to-peer), routed subnet (wg-quick tables, policy routing) + +## Service Mesh + +### Istio + +- **Core model:** Sidecar proxy (Envoy) injected into pods — intercepts all traffic, applies mesh policies +- **Key concepts:** VirtualService (traffic routing, retries, timeouts, mirroring), DestinationRule (load balancing, connection pool, mTLS, circuit breaker), Gateway (ingress/egress), ServiceEntry (external services), PeerAuthentication (mTLS mode), AuthorizationPolicy (RBAC for services) +- **Observability:** Telemetry via Envoy — HTTP/gRPC metrics (Prometheus), distributed tracing (Jaeger/Zipkin/OpenTelemetry), access logs, Kiali for topology visualization + +### Cilium + +- **Core model:** eBPF-based — no sidecar injection, kernel-level networking and security +- **Key capabilities:** NetworkPolicy (identity-based, FQDN-based), Service Mesh (L7 policies, ingress/gateway API, L7 load balancing), Encryption (WireGuard in-kernel), Observability (Hubble: flow logs, metrics, UI, OpenTelemetry), ClusterMesh (multi-cluster networking) +- **Advantage over Istio:** No sidecar overhead, native eBPF performance, integrated with Tetragon for runtime security diff --git a/product-strategy/README.md b/product-strategy/README.md new file mode 100644 index 0000000..d341ea5 --- /dev/null +++ b/product-strategy/README.md @@ -0,0 +1,26 @@ +# Product Strategy + +CPO methodology — product vision and strategy (North Star, product principles), competitive analysis and positioning, roadmap prioritization (RICE, Kano, OST), product-market fit frameworks (Sean Ellis test, retention curves), market sizing (TAM/SAM/SOM), platform strategy, product lifecycle management. + +## Why Install This Skill + +Your agent applies CPO-level frameworks — North Star with anti-patterns, RICE scoring, Porter's Five Forces, PMF signals — instead of generic product advice. + +## What You Get + +| Directory | Purpose | +|-----------|---------| +| `SKILL.md` | Core methodology, trigger conditions, reference index | +| `references/` | Deep-dive reference files loaded on demand | + +## Triggers + +Defining product vision, prioritizing roadmaps, analyzing competitors, sizing markets, evaluating product-market fit, or designing platform strategy. + +## Requirements + +No technical requirements. Frameworks from Dunford, Porter, Kano, Sean Ellis, and standard product analytics. + +## Quick Start + +Load SKILL.md for the methodology overview and reference table, then load specific references as needed for the task at hand. diff --git a/product-strategy/SKILL.md b/product-strategy/SKILL.md new file mode 100644 index 0000000..b85f31c --- /dev/null +++ b/product-strategy/SKILL.md @@ -0,0 +1,57 @@ +--- +name: product-strategy +description: CPO methodology — product vision and strategy (North Star, product principles), + competitive analysis and positioning, roadmap prioritization (RICE, Kano, OST), + product-market fit frameworks (Sean Ellis test, retention curves), market sizing + (TAM/SAM/SOM), platform strategy, product lifecycle management. +license: MIT +metadata: + tags: product-strategy, cpo, product-management, competitive-analysis, market-sizing, + roadmap-prioritization, platform-strategy, product-lifecycle + source_repo: https://github.com/magnus919/hermes-profiles +--- + +# Product Strategy — CPO Methodology + +CPO-level methodology for product strategy, market analysis, competitive positioning, and platform thinking. This skill provides the frameworks and reference material for a chief product officer profile. + +## When to Load + +| Trigger | What's Needed | +|---------|---------------| +| Define product vision and North Star metric | `references/product-strategy.md` — North Star, product principles, vision | +| Analyze competitive landscape | `references/competitive-positioning.md` — Porter's Five Forces, Blue Ocean, positioning | +| Size a market opportunity | `references/market-analysis.md` — TAM/SAM/SOM, PMF, lifecycle | +| Prioritize a roadmap | `references/product-strategy.md` — RICE, Kano, OST | +| Assess product-market fit | `references/market-analysis.md` — Sean Ellis test, retention curves | +| Plan a platform or ecosystem strategy | `references/product-strategy.md` — API-first, marketplace, ecosystem | +| Develop market entry strategy | `references/market-analysis.md` — beachhead, land-and-expand, platform entry | + +## Loading Order + +```text +skill_view('product-strategy') +# Then domain-specific references: +skill_view('product-strategy', file_path='references/product-strategy.md') +skill_view('product-strategy', file_path='references/competitive-positioning.md') +skill_view('product-strategy', file_path='references/market-analysis.md') +``` + +## Reference Files + +| Reference | Purpose | +|-----------|---------| +| `references/product-strategy.md` | North Star, product principles, RICE/Kano/OST, platform strategy, product lifecycle | +| `references/competitive-positioning.md` | Competitive landscape mapping, Porter's Five Forces, April Dunford positioning, differentiation strategies, competitive response playbook | +| `references/market-analysis.md` | TAM/SAM/SOM deep dive, PMF assessment (Sean Ellis test, retention curves), market entry strategy, product lifecycle management | + +## Output Contract + +The profile using this skill produces artifact pyramids. The response to any caller is the absolute path to `00-index.md`. See `artifact-pyramids` skill for the specification. + +## Related Skills + +- `artifact-pyramids` — output contract +- `product-methodology` — tactical product management (RICE, MoSCoW, customer interviews) +- `go-to-market` — CMO methodology (positioning, acquisition, brand, growth modeling) +- `implementation-planning` — work breakdown and dependency ordering diff --git a/product-strategy/references/competitive-positioning.md b/product-strategy/references/competitive-positioning.md new file mode 100644 index 0000000..7b1f8c1 --- /dev/null +++ b/product-strategy/references/competitive-positioning.md @@ -0,0 +1,122 @@ +# Competitive Analysis & Positioning + +## Competitive Analysis Framework + +### The Competitive Landscape Map + +Plot competitors on two axes that matter most to your market: + +| Axis | Description | Example axes | +|------|-------------|-------------| +| **X-axis** | Feature depth vs breadth | Specialized ↔ General purpose | +| **Y-axis** | Price/positioning | Premium ↔ Budget | +| **Bubble size** | Market share or revenue | Indicates scale | + +### Porter's Five Forces + +| Force | Questions | Implication | +|-------|-----------|-------------| +| **Threat of new entrants** | How high are barriers to entry? (Capital, distribution, regulatory, network effects) | Low barriers → constant competition on price | +| **Bargaining power of suppliers** | Are key inputs concentrated? Can suppliers forward-integrate? | High power → margin pressure | +| **Bargaining power of buyers** | How easy is switching? Are buyers price-sensitive? | High power → need to differentiate | +| **Threat of substitutes** | What adjacent solutions solve the same need differently? | Many substitutes → price ceiling | +| **Industry rivalry** | How intense is current competition? | Intense → zero-sum market share battles | + +### Blue Ocean vs Red Ocean + +| Dimension | Red Ocean (Competing) | Blue Ocean (Creating) | +|-----------|----------------------|----------------------| +| Market | Compete in existing market space | Create uncontested market space | +| Demand | Beat the competition | Make competition irrelevant | +| Strategy | Exploit existing demand | Create and capture new demand | +| Tradeoffs | Value-cost tradeoff | Pursues differentiation and low cost simultaneously | +| System | Align whole system with differentiation or low cost | Align whole system in pursuit of differentiation and low cost | + +## Positioning Methodology (April Dunford's Framework) + +Positioning is not your tagline — it's the context you create for your product in the market. It defines *who* your product is for, *what* it does, and *why* that matters. + +### The 10-Step Positioning Process + +1. **Understand the customers who love you** — Who are your best-fit customers? What do they all have in common? +2. **Identify your competitive alternatives** — What do customers use *instead* of your product? (Could be a competitor, a manual process, or doing nothing) +3. **Determine your unique capabilities** — What can you do that alternatives can't? Be specific. +4. **Identify the value of those capabilities** — What outcome do those capabilities unlock for the customer? +5. **Find your best market** — Which customer segment cares most about this value? This is your beachhead. +6. **Map the market alternatives** — How are customers currently solving this? Position yourself *relative* to them. +7. **Create a positioning concept** — One sentence that captures who you are, what you do, and why it matters. +8. **Build your positioning validation** — Test it with customers. Do they recognize themselves? Do they believe the claims? +9. **Craft your message** — Now write the tagline, value prop, and supporting narrative. +10. **Operationalize** — Embed positioning in product, sales, marketing, and support. + +### The Positioning Diagnostic + +When positioning is wrong, you see symptoms: + +| Symptom | Likely root cause | +|---------|-------------------| +| "I don't understand what you do" | Value proposition is unclear or too generic | +| "How are you different from X?" | Insufficient differentiation | +| "Who is this for?" | Market definition is too broad | +| "It sounds like [competitor]" | You're positioning against the wrong alternative | +| "That's interesting, but we're not interested" | The market segment doesn't need what you uniquely do | + +### The Positioning Statement Template + +``` +For [target customer segment] +who [current alternative for solving the problem], +[product name] is a [category/new category] +that provides [key capability]. +Unlike [alternative], +[product name] [key differentiation]. +``` + +## Differentiation Strategies + +### Types of Differentiation + +| Type | Description | Example | +|------|-------------|---------| +| **Product** | Features, performance, design | Apple's UX polish | +| **Price** | Cost leadership or premium | Walmart (low cost) vs Patagonia (premium) | +| **Experience** | Service, onboarding, support | Zappos' customer service | +| **Distribution** | Channels, availability | Coca-Cola's shelf presence | +| **Brand** | Trust, identity, status | Nike's brand equity | +| **Ecosystem** | Integration, network effects | Salesforce AppExchange | + +### The Differentiation Test + +For each claimed differentiator, ask: + +1. **Relevant** — Does the customer care about this? +2. **Credible** — Can we prove it? (Data, case studies, benchmarks) +3. **Unique** — Can competitors credibly claim the same thing? +4. **Durable** — How long before competitors catch up? + +If any answer is "no," it's not a real differentiator. + +## Competitive Response Playbook + +### When a Competitor Launches Against You + +| Situation | Response | +|-----------|----------| +| They match your core feature | Don't panic. Move upmarket or deepen your differentiator. Feature parity without context is not a threat. | +| They undercut on price | If you can't compete on price, don't. Compete on value, switching costs, or service. | +| They copy your positioning | If your positioning is well-connected to actual product experience, copying the positioning reveals their weakness. Let them be seen as an imitator. | +| They enter your beachhead segment | Double down on your best customers. Often, the new entrant's initial users are your weakest users who would churn anyway. | + +### Defensive Moves + +- **Strengthen switching costs** — Integrations, data portability, workflows users can't easily recreate +- **Raise barriers** — Patents, network effects, exclusive partnerships, brand equity +- **Segment retreat** — Cede low-value segments to competitors; reinforce defensible high-value segments +- **Platform lock-in** — Make your product more valuable as more of the customer's workflow lives in it + +### Offensive Moves + +- **Flanking** — Attack competitors' weaker segments +- **Encirclement** — Surround a competitor with a broader platform play +- **Guerrilla** — Targeted price cuts, feature releases timed to competitor launches, comparison marketing +- **Disruption** — Change the basis of competition entirely (new business model, technology, or channel) diff --git a/product-strategy/references/market-analysis.md b/product-strategy/references/market-analysis.md new file mode 100644 index 0000000..df91414 --- /dev/null +++ b/product-strategy/references/market-analysis.md @@ -0,0 +1,155 @@ +# Market Analysis + +## TAM / SAM / SOM — Deep Dive + +### TAM (Total Addressable Market) + +The total revenue opportunity if your product achieved 100% market share in a defined market. + +**Top-down method:** +``` +TAM = Number of potential customers × Average revenue per customer +``` + +Sources: Industry analyst reports (Gartner, Forrester, IDC), government statistics (NAICS codes), trade associations, public company filings (10-Ks). + +**Bottom-up method:** +``` +TAM = Price × number of units the entire market buys +``` + +More credible than top-down because it's grounded in transactional reality. + +**Pitfall:** TAM inflation. If you define your market as "every company that uses software," the TAM is meaningless. Narrow to the specific use case your product addresses. + +### SAM (Serviceable Addressable Market) + +The portion of TAM your product and distribution channels can realistically reach. + +**Filters:** +- Geographic — "We only sell in the US and EU" +- Channel — "We sell via Shopify App Store, not enterprise sales" +- Product fit — "Our product requires a modern tech stack" +- Segment — "SMBs only (under 500 employees)" + +``` +SAM = TAM × (% of market reachable via our channels) +``` + +### SOM (Serviceable Obtainable Market) + +The portion of SAM you can realistically capture given your team, capital, and competitive position — typically over a 12-36 month horizon. + +**Bottom-up (most credible):** +``` +SOM = Sales capacity × Average deal size × (1 - churn rate) × Time +``` + +- Sales capacity: Number of reps × deals per rep per month +- Marketing capacity: CAC × budget → number of new customers +- Time horizon: Usually 12-24 months for startup fundraising, 36 months for strategic planning + +## Product-Market Fit Assessment + +### The PMF Pyramid + +``` + ┌─────────────┐ + │ Retention │ ← Core indicator of PMF + ├─────────────┤ + │ Activation │ ← Do users get value in the first session? + ├─────────────┤ + │ Acquisition │ ← Can you reach users efficiently? + ├─────────────┤ + │ Product │ ← Does the product solve a real need? + └─────────────┘ +``` + +PMF is layered. You can't have retention without activation, or activation without acquisition, or acquisition without a product-market need. + +### PMF Signals + +| Signal | What to look for | Red flag | +|--------|------------------|----------| +| **Organic growth** | W-o-M referrals, inbound requests | Zero organic — all paid | +| **Retention** | Flattening retention curve at 30-60-90 days | Continuous downward slope | +| **Usage depth** | Power users who use >5x per week | Everyone uses sporadically | +| **Paying willingness** | High conversion from free to paid | Users love it but won't pay | +| **Churn reason** | "Couldn't get value" (bad PMF) vs "went out of business" (sales problem) | "Didn't see the need anymore" | +| **NPS at scale** | >40 with high response rate | Low response rate or <10 NPS | + +### When PMF Is Strong vs Weak + +| Strong PMF | Weak PMF | +|------------|----------| +| Users who hit the "aha moment" stay for months | Users try, plateau, and leave | +| 40%+ would be "very disappointed" if you disappeared | <20% would be disappointed | +| Net revenue retention >100% (existing customers spend more over time) | Net revenue retention <80% | +| Hard to scale because demand overwhelms capacity | Hard to scale because nobody cares | +| Sales-led growth works because users bring the product to their org | Sales-led growth fails because there's no bottom-up pull | + +## Market Entry Strategy + +### Beachhead Selection + +Choose your first market segment using these criteria: + +| Criterion | Weight | Question | +|-----------|--------|----------| +| **Need** | High | Does this segment urgently need what you build? | +| **Access** | High | Can you reach them cost-effectively? | +| **Competition** | Medium | Is the segment underserved by incumbents? | +| **Budget** | High | Do they have money to spend? | +| **Reference value** | Medium | Will winning here help you win adjacent segments? | +| **Scale** | Low | Is the segment large enough to matter? | + +**The beachhead test:** Pick a segment where you can achieve market leadership with a focused offering. If you can't own this segment, pick a narrower one. + +### Land-and-Expand + +| Phase | Goal | Strategy | +|-------|------|----------| +| **Land** | Get one foot in the door | Solve a specific, painful problem for one team/department | +| **Prove** | Demonstrate measurable ROI | Track adoption, usage, and outcomes rigorously | +| **Expand** | Grow within the account | Add users, teams, use cases, integrations | +| **Entrench** | Become infrastructure | Embed in workflows, data, and processes | + +### Platform Entry Strategy + +When entering as a platform (not a point solution): + +1. **Start with a killer app** — The platform needs one iconic use case that drives adoption +2. **Then open the API** — Once users are on the platform, let them extend it +3. **Then build an ecosystem** — Third-party developers extend reach +4. **Then move up the stack** — Add higher-value capabilities on top of the platform + +## Product Lifecycle Management + +### The PLC Stages + +| Stage | Revenue | Profit | Cash flow | Team focus | +|-------|---------|--------|-----------|------------| +| **Introduction** | Low, growing slowly | Negative | Heavy burn | Product-market fit | +| **Growth** | Rapid growth | Breakeven → positive | Moderate burn → cash-flow-positive | Scale | +| **Maturity** | Slowing growth | Peak | Strong generator | Efficiency, segmentation | +| **Decline** | Declining | Declining | Decreasing | Harvest or pivot | + +### Lifecycle Decisions + +| Decision point | Introduction | Growth | Maturity | Decline | +|----------------|-------------|--------|----------|---------| +| **Investment** | Heavy (finding PMF) | Heavy (scaling) | Selective (segments) | Minimal (harvesting) | +| **Pricing** | Skim or penetrate | Maintain or optimize | Defend or bundle | Cut | +| **Distribution** | Direct, selective | Broaden channels | Optimize channels | Reduce | +| **Competition** | Few early entrants | More entrants | Many competitors | Consolidation | +| **Product** | Core features | Expand horizontally | Segment-specific variants | Reduce SKUs | + +### The S-Curve + +Products follow S-curves: slow adoption → rapid growth → plateau. The strategic challenge is to invest in the **next S-curve** while the current one still generates cash. This is the Innovator's Dilemma: + +- By the time the current S-curve plateaus, the next one is already growing +- Incumbents underinvest in the next S-curve because the current one is profitable +- New entrants ride the next S-curve up and displace incumbents + +**Strategic response:** Run parallel innovation tracks — one optimized for the current curve (extract value), one for the next curve (explore). diff --git a/product-strategy/references/product-strategy.md b/product-strategy/references/product-strategy.md new file mode 100644 index 0000000..9dc81a5 --- /dev/null +++ b/product-strategy/references/product-strategy.md @@ -0,0 +1,201 @@ +# Product Strategy & Vision + +## North Star Framework + +The North Star is the single, leading metric that captures the value your product delivers to customers. It aligns the entire organization around outcome over output. + +### Criteria for a Good North Star + +| Criterion | Description | Example (Spotify) | +|-----------|-------------|-------------------| +| **Leading indicator** | Predicts long-term business outcomes | Time spent listening → predicts retention | +| **Customer-centric** | Reflects value delivered, not internal activity | Not "features shipped" | +| **Actionable** | Teams can directly influence it | Not "brand awareness" | +| **Measurable** | Can be instrumented and tracked | DAU/MAU, sessions, engagement depth | + +### North Star Anti-Patterns + +- **Revenue masquerading as a North Star** — Revenue is a lagging indicator. The North Star should be a leading indicator that *drives* revenue. +- **Composite metrics** — A score that combines 5 sub-metrics into one number isn't actionable. No team knows how to move it. +- **Vanity metrics** — Total registered users, not active users. Downloads, not engagement. +- **Annual planning cycle** — The North Star doesn't change quarterly. If you're reevaluating it every planning cycle, you don't have a North Star. + +### Formulating a North Star + +```text +[Product/Feature] helps [customer segment] achieve [core outcome] by [core capability]. +We measure success by [North Star metric]. +``` + +Example (Airbnb): +> Airbnb helps travelers belong anywhere by connecting them with local hosts. +> We measure success by **nights booked**. + +## Product Principles + +Product principles are decision-making heuristics that encode your product philosophy. They don't tell teams *what* to build — they tell them *how to decide* what to build. + +### Anatomy of a Good Principle + +| Component | Description | Example | +|-----------|-------------|---------| +| **Name** | Memorable label | "Default to Open" | +| **One-liner** | The rule itself | "Share data by default; only restrict when there's a clear privacy or security reason." | +| **Tradeoff** | What you're willing to sacrifice | "This may mean competitors see our usage patterns. That's acceptable because it builds trust faster." | +| **Boundaries** | When the principle doesn't apply | "Not applicable to PII or billing data." | + +### Example Product Principles + +- **Solve for the 80%** — Build the path most users take; power users get their edge cases through APIs and extensibility. *Tradeoff: power users may feel underserved.* +- **Progress, not perfection** — Ship the 80% solution this quarter rather than polishing to 95% for two quarters. *Tradeoff: early versions will have rough edges.* +- **Platform over point solution** — Build capabilities that multiple features can leverage, not single-use features. *Tradeoff: slower initial delivery.* +- **Default to simplicity** — When a decision adds complexity without a measurable improvement in the primary metric, reject it. *Tradeoff: some elegant-but-complex solutions won't ship.* + +## Roadmap Prioritization + +### RICE Scoring + +The RICE framework scores feature proposals across four dimensions: + +``` +RICE Score = (Reach × Impact × Confidence) / Effort +``` + +- **Reach**: How many users per unit time this affects +- **Impact**: How much it matters to those users (0.25× to 3× multiplier) +- **Confidence**: How sure you are about estimates (20%–100%) +- **Effort**: Total team-weeks required + +See `rrice-framework.md` in the product-methodology skill for the full treatment. + +### Kano Model + +Categorizes features by their relationship to customer satisfaction: + +| Category | Description | Example | Saturation risk | +|----------|-------------|---------|-----------------| +| **Threshold** (Must-be) | Expected; absence causes dissatisfaction | Login, payment processing | None — required to compete | +| **Performance** (One-dimensional) | More is better; directly correlates with satisfaction | Battery life, page load speed | Yes — diminishing returns | +| **Delight** (Attractive) | Unexpected; absence doesn't hurt, presence delights | Confetti animation on first purchase | Yes — becomes threshold over time | +| **Indifferent** | No impact on satisfaction | Unused settings options | N/A | +| **Reverse** | More is worse | Excessive notifications | N/A | + +**Strategic implication:** Don't invest in delighters at the expense of threshold features. Threshold features are table stakes — you can't win on them, but you can lose without them. Invest in performance features for competitive differentiation, and sprinkle delighters as surprise-and-delight moments. + +### Opportunity Solution Trees (OST) + +A framework for connecting customer needs to build decisions without jumping to solutions: + +``` +Opportunity (customer need) +├── Solution option A +│ └── Assumption to test +├── Solution option B +│ └── Assumption to test +└── Solution option C + └── Assumption to test +``` + +**Key rules:** +1. Start with an **opportunity** (something the customer needs to achieve), not a solution. +2. Branch into **possible solutions** — multiple options for each opportunity. +3. Each solution has **testable assumptions** — what must be true for it to work. +4. Test the riskiest assumption first. If it fails, move to the next solution. + +**Pitfall:** Teams jump to a single solution because it's obvious. OST forces you to consider alternatives before committing. + +## Product-Market Fit (PMF) + +### Sean Ellis Test + +Ask active users: "How would you feel if you could no longer use [product]?" + +| Response | Interpretation | +|----------|---------------| +| **Very disappointed** | Core engaged users — 40%+ indicates PMF | +| **Somewhat disappointed** | Users who find value but aren't hooked | +| **Not disappointed** | Low value — no PMF | +| **N/A — no longer use** | Churned | + +**Threshold:** If ≥40% answer "Very disappointed," you have product-market fit. + +### Retention Curves + +Plot % of users retained over time from their signup date: + +- **Flattening curve** → PMF hit. Users who get the value stay. +- **Downward slope** → No PMF. Users try the product and leave. +- **Flattening then dropping** → Weak PMF for a specific segment. Find the cohort that sticks and focus on it. + +**Segmentation tip:** Don't look at aggregate retention. Segment by acquisition channel, user persona, feature adoption, and behavior. The right cohort tells you where PMF exists; the wrong cohort hides it. + +## Market Sizing: TAM / SAM / SOM + +| Term | Definition | Question it answers | +|------|------------|---------------------| +| **TAM** (Total Addressable Market) | Total revenue opportunity if 100% market share | "How big is the pie?" | +| **SAM** (Serviceable Addressable Market) | Segment of TAM your product/service can reach | "How much of the pie can we actually serve?" | +| **SOM** (Serviceable Obtainable Market) | Share of SAM you can realistically capture | "How much will we actually eat?" | + +### Top-Down Approach + +Start with industry analyst data and apply filters: + +``` +TAM = Global market revenue for category X +SAM = TAM × % accessible via our distribution channels +SOM = SAM × % we can capture given our team/capital/competitive position +``` + +**Pitfall:** Top-down tends to overestimate. Analysts define markets broadly; your actual reach is narrower. + +### Bottom-Up Approach + +Start with your unit economics and scale up: + +``` +SOM = (Sales capacity × conversion rate × average deal size) × time horizon +SAM = SOM × (theoretical max sales capacity / current capacity) +TAM = Bottom-up SAM × (your price / average category price) +``` + +**Bottom-up is more credible** because it's grounded in operational reality. + +## Platform Strategy + +### API-First Design + +Build the API before the UI. This forces contract clarity, enables multiple clients (web, mobile, partner API), and creates a foundation for ecosystem growth. + +**Key decisions:** +- REST vs GraphQL vs gRPC — each has different tradeoffs for discoverability, performance, and client complexity +- Versioning strategy — URL-based (`/v1/`), header-based, or contract-based (GraphQL) +- Authentication — API keys (simple), OAuth2 (scoped), JWTs (stateless) + +### Marketplace Strategy + +Two-sided network effects create defensible moats: + +| Phase | Supply side | Demand side | Mechanics | +|-------|-------------|-------------|-----------| +| **Cold start** | Recruit supply manually | Seed demand through marketing | Both sides need value before the other exists — hardest phase | +| **Growth** | Supply grows with demand | Demand grows with supply | Each new supply unit attracts demand, and vice versa | +| **Moat** | High switching costs | Deep inventory | Competitors can't replicate the liquidity | + +### Ecosystem Strategy + +- **Platform extensibility** — APIs, plugins, webhooks, embeddable widgets +- **Developer experience** — Documentation, SDKs, sandbox environments, SLAs +- **Governance** — What third parties can/cannot build, revenue share, certification +- **Control points** — Where you retain control vs open up + +## Product Lifecycle Management + +| Stage | Characteristics | Strategy | Metrics | +|-------|----------------|----------|---------| +| **Introduction** | Low revenue, high investment | Build awareness, find PMF | Activation rate, early retention | +| **Growth** | Rapid revenue growth | Scale acquisition, expand features | Net revenue retention, market share | +| **Maturity** | Slowing growth, stable revenue | Extract profits, segment positioning | Profit margin, customer lifetime value | +| **Decline** | Revenue declining | Harvest or pivot | Cash flow, cost of maintenance | + +**Key insight:** The worst product strategy mistake is treating a mature product like a growth product (over-investing) or a growth product like a mature product (under-investing for short-term profit). diff --git a/qa-methodology/README.md b/qa-methodology/README.md new file mode 100644 index 0000000..024f5a1 --- /dev/null +++ b/qa-methodology/README.md @@ -0,0 +1,26 @@ +# Qa Methodology + +Quality assurance methodology — test strategy design, test automation patterns, regression testing, CI quality gates, test data management, and quality metrics. Grounded in practical patterns for teams that want confident shipping. + +## Why Install This Skill + +Your agent designs test strategies, manages test data, runs performance tests, and integrates security testing — the full QA lifecycle, not just 'write more tests'. + +## What You Get + +| Directory | Purpose | +|-----------|---------| +| `SKILL.md` | Core methodology, trigger conditions, reference index | +| `references/` | Deep-dive reference files loaded on demand | + +## Triggers + +Designing test strategy, selecting automation frameworks, building regression suites, managing test data, performance testing, or adding security testing to CI. + +## Requirements + +Platform-agnostic. References cover pytest, Playwright, k6, Semgrep, Trivy, and factory_boy. + +## Quick Start + +Load SKILL.md for the methodology overview and reference table, then load specific references as needed for the task at hand. diff --git a/qa-methodology/SKILL.md b/qa-methodology/SKILL.md new file mode 100644 index 0000000..e26c764 --- /dev/null +++ b/qa-methodology/SKILL.md @@ -0,0 +1,49 @@ +--- +name: qa-methodology +description: Quality assurance methodology — test strategy design, test automation + patterns, regression testing, CI quality gates, test data management, and quality + metrics. Grounded in practical patterns for teams that want confident shipping. +license: MIT +metadata: + tags: qa, testing, quality-assurance, test-automation, regression, CI, quality-gates, + flaky-tests, quality-metrics + source_repo: https://github.com/magnus919/hermes-profiles +--- + +# QA Methodology + +Quality assurance is the practice of making confident shipping routine. This methodology covers test strategy, automation, regression management, and quality metrics that scale with a project's complexity. + +## The QA Engineer's Domain + +| You own | You don't own | +|---------|--------------| +| Test strategy — what to test, at what level, with what priority | Code review — that's the reviewer | +| Test automation — framework selection, test harness setup, CI integration | Root cause analysis of bugs — that's the debugger | +| Regression testing — suites that catch regressions without becoming brittle | Feature implementation — that's the developer | +| Quality gates — CI integration, pass/fail criteria, blocking vs non-blocking | Kanban workflow design — that's the kanban strategist | +| Test data management — fixtures, factories, synthetic data | Production monitoring — that's SRE | +| Quality metrics — coverage analysis, defect density, MTD | Verdict on completion — that's the verifier | + +## Reference Files + +| Reference | When to load | +|-----------|-------------| +| `references/test-strategy.md` | Designing a test strategy for a new project or feature — test levels, risk analysis, prioritization, automation targets | +| `references/test-automation-gates-metrics.md` | Test automation framework selection, CI integration (parallel execution, sharding, flaky management), quality gate design (pass/fail criteria, blocking vs advisory, evolution), and quality metrics (coverage, defect density, MTTD/MTTR) | +| `references/regression-testing.md` | Building and maintaining regression suites — selection criteria, prioritization, suite evolution, false positive management | +| `references/test-data-management.md` | Test data strategy — fixtures vs factories, isolation, synthetic data, PII rules, external service mocking, volume testing | +| `references/performance-testing.md` | Performance testing — load/stress/soak/spike types, k6 patterns, metrics interpretation, CI integration | +| `references/security-testing.md` | Security testing — SAST/DAST/dependency audit, OWASP Top 10 test patterns, container scanning, CI gates | + +## Core Principles + +**If it isn't tested, it's broken** — Untested code is not working code; it's code whose failure mode hasn't been discovered yet. + +**Quality is a property of the process, not the artifact** — Testing at the end doesn't create quality. Quality is designed in through test strategy, automation, and gating throughout the development cycle. + +**Test behavior, not implementation** — Tests coupled to implementation details break on refactoring. Tests coupled to behavior survive it. Prefer testing what the system does, not how it does it. + +**Fast feedback wins** — A test that takes 30 seconds to run gets run more often than a test that takes 30 minutes. Invest in test speed proportional to feedback frequency. + +**Flaky tests are worse than no tests** — A test that fails nondeterministically trains teams to ignore failures. Fix or remove flaky tests on detection. diff --git a/qa-methodology/references/performance-testing.md b/qa-methodology/references/performance-testing.md new file mode 100644 index 0000000..bf13167 --- /dev/null +++ b/qa-methodology/references/performance-testing.md @@ -0,0 +1,76 @@ +# Performance Testing + +## Types + +| Type | Question Answered | Tool Examples | +|------|-------------------|---------------| +| Load test | Does it handle expected traffic? | k6, Locust, Gatling | +| Stress test | Where does it break? | k6 (ramping VUs), wrk | +| Soak test | Does it degrade over time? | k6 (constant load, 4–24h) | +| Spike test | Does it survive sudden bursts? | k6 (spike scenario) | +| Benchmark | What's the raw throughput/latency? | wrk, hey, ab, pytest-benchmark | + +## When to Performance Test + +- Before launch (baseline) +- After architectural changes (new DB, new cache layer, new service boundary) +- After dependency upgrades (ORM version, driver changes) +- When latency SLO is at risk (p99 trending up over 2+ sprints) + +**Not** on every PR — that's what unit/integration tests are for. + +## k6 Pattern (Load Test) + +```javascript +import http from 'k6/http'; +import { check, sleep } from 'k6'; + +export const options = { + stages: [ + { duration: '30s', target: 20 }, // ramp up + { duration: '1m', target: 20 }, // steady state + { duration: '10s', target: 0 }, // ramp down + ], + thresholds: { + http_req_duration: ['p(95)<500', 'p(99)<1000'], + http_req_failed: ['rate<0.01'], + }, +}; + +export default function () { + const res = http.get('https://staging.example.com/api/items'); + check(res, { + 'status 200': (r) => r.status === 200, + 'response time < 200ms': (r) => r.timings.duration < 200, + }); + sleep(1); +} +``` + +## Key Metrics + +| Metric | Definition | Target Guidance | +|--------|-----------|-----------------| +| p50 latency | Median response time | User-perceived "normal" | +| p95 latency | 95th percentile | SLO boundary for most APIs | +| p99 latency | 99th percentile | Tail latency — catches GC pauses, cold starts | +| Throughput | Requests/sec sustained | Compare against capacity plan | +| Error rate | 5xx / total | < 0.1% under load | +| Saturation | CPU/memory/connections at peak | < 80% = headroom | + +## Interpreting Results + +| Symptom | Likely Cause | Next Step | +|---------|-------------|-----------| +| Latency climbs linearly with VUs | Single-threaded bottleneck or lock contention | Profile CPU, check for global locks | +| Latency flat then sudden cliff | Resource exhaustion (connections, memory, file descriptors) | Check pool sizes, `ulimit`, OOM killer | +| Throughput plateaus early | Downstream dependency is the bottleneck | Test the dependency in isolation | +| Errors only at high concurrency | Race condition or timeout misconfiguration | Check connection pool, retry storms | +| Memory grows during soak | Leak — unclosed connections, unbounded cache | Heap dump at intervals, diff allocations | + +## CI Integration + +- Run a **smoke benchmark** (10 VUs, 30s) on PRs that touch hot paths — fast, catches 10× regressions +- Run **full load test** nightly against staging +- Alert if p95 regresses > 20% vs 7-day baseline +- Store results in time-series (k6 Cloud, Grafana, or CSV + script) for trend detection diff --git a/qa-methodology/references/regression-testing.md b/qa-methodology/references/regression-testing.md new file mode 100644 index 0000000..85c7159 --- /dev/null +++ b/qa-methodology/references/regression-testing.md @@ -0,0 +1,34 @@ +# Regression Testing + +## What Belongs in a Regression Suite + +| Include | Exclude | +|---------|---------| +| Every fixed bug (as a test) | Tests that haven't failed in 6+ months (archive) | +| Critical user paths | Tests for deprecated features | +| Known failure patterns | Tests that overlap with lower-level coverage | +| API contract checks | Visual regression tests for work-in-progress UI | +| Data integrity assertions | Performance tests (separate suite) | + +## Suite Hygiene + +| Condition | Action | +|-----------|--------| +| Test hasn't failed in 6 months | Consider archiving — it may not be testing anything meaningful | +| Test flakes > 1% over 10 runs | Investigate and fix or remove immediately | +| Test takes > 5s (unit) / > 30s (integration) | Optimize or promote to slower tier | +| Test depends on another test's state | Fix isolation — tests must be independent | +| Test runs against production data | Switch to synthetic fixtures | + +## Change-Based Test Selection + +Not every change needs the full regression suite. For targeted changes: + +| Change Type | Required Tests | +|-------------|---------------| +| Bug fix | Test that reproduces the bug + related unit tests | +| Feature addition | New feature tests + smoke tests on adjacent areas | +| Refactoring | Full unit suite + integration smoke tests | +| Configuration change | Smoke tests on affected components | +| Dependency update | Full regression suite | +| Infrastructure change | Integration + E2E suite | diff --git a/qa-methodology/references/security-testing.md b/qa-methodology/references/security-testing.md new file mode 100644 index 0000000..7e94565 --- /dev/null +++ b/qa-methodology/references/security-testing.md @@ -0,0 +1,69 @@ +# Security Testing + +## Test Types by Phase + +| Phase | Test Type | Tool | Frequency | +|-------|-----------|------|-----------| +| Pre-commit | Secret scanning | gitleaks, trufflehog | Every commit | +| CI | SAST (static analysis) | Semgrep, CodeQL, bandit (Python), eslint-security | Every PR | +| CI | Dependency audit | `npm audit`, `pip-audit`, Dependabot, Snyk | Every PR + daily | +| CI | Container scanning | Trivy, Grype | Every image build | +| Staging | DAST (dynamic) | OWASP ZAP, Burp Suite | Nightly or pre-release | +| Staging | API fuzzing | Schemathesis (OpenAPI), RESTler | Weekly | +| Pre-release | Pen test (manual) | External firm or red team | Quarterly / major release | + +## OWASP Top 10 — Test Patterns + +| Risk | What to Test | How | +|------|-------------|-----| +| Injection (SQL, command, LDAP) | All user input reaches DB/shell | Parameterized query audit; fuzz with `' OR 1=1`, `; rm -rf` | +| Broken auth | Session fixation, token expiry, brute force | Attempt reuse of expired tokens; test rate limiting | +| Sensitive data exposure | PII in logs, unencrypted transit | Grep logs for email/SSN patterns; verify TLS everywhere | +| XXE | XML parsers accept external entities | Send `]>` | +| Broken access control | IDOR, privilege escalation | Access other users' resources by ID; test admin endpoints as regular user | +| Security misconfiguration | Default creds, verbose errors, open ports | Banner grab; check `/debug`, `/admin`, `.env` exposure | +| XSS | Reflected/stored/DOM | Inject `` in every input field | +| Insecure deserialization | Pickle, YAML.load, Java ObjectInputStream | Audit all `loads()`/`unserialize()` calls for untrusted input | +| Known vulnerabilities | CVEs in dependencies | `pip-audit`, `npm audit`, Trivy | +| SSRF | Server fetches user-supplied URLs | Provide `http://169.254.169.254/` (cloud metadata), `http://localhost:6379` | + +## SAST in CI (Semgrep Example) + +```yaml +# .github/workflows/security.yml +- name: Semgrep + uses: semgrep/semgrep-action@v1 + with: + config: >- + p/owasp-top-ten + p/python + p/security-audit +``` + +## Dependency Audit Discipline + +- Block merges on **critical** and **high** severity findings +- **Medium**: create issue, fix within sprint +- **Low**: batch into maintenance window +- Pin transitive deps with lockfiles (`package-lock.json`, `uv.lock`, `Gemfile.lock`) +- Review Dependabot PRs weekly — don't let them accumulate + +## Container Security + +```bash +# Scan image for OS + language CVEs +trivy image --severity HIGH,CRITICAL myapp:latest + +# Fail CI on critical findings +trivy image --exit-code 1 --severity CRITICAL myapp:latest +``` + +- Use distroless or alpine base images (smaller attack surface) +- Run as non-root (`USER 1000` in Dockerfile) +- No secrets in image layers — use runtime injection (Vault, SSM, k8s secrets) + +## Security Test Data + +- Never test with real PII — use synthetic data (see test-data-management.md) +- Credential testing uses obviously-fake values: `AKIAIOSFODNN7EXAMPLE` +- If a test discovers a real vulnerability, stop and report — don't commit exploit code diff --git a/qa-methodology/references/test-automation-gates-metrics.md b/qa-methodology/references/test-automation-gates-metrics.md new file mode 100644 index 0000000..e12ea93 --- /dev/null +++ b/qa-methodology/references/test-automation-gates-metrics.md @@ -0,0 +1,756 @@ +# QA Methodology Reference: Test Automation, Quality Gates & Quality Metrics + +> A comprehensive reference for QA engineers, covering test automation patterns, quality gate design, and quality metrics. Researched June 2026. + +--- + +## Table of Contents + +1. [Test Framework Selection](#1-test-framework-selection) +2. [CI Integration: Parallel Execution, Sharding & Test Splitting](#2-ci-integration-parallel-execution-sharding--test-splitting) +3. [Flaky Test Management](#3-flaky-test-management) +4. [Quality Gates](#4-quality-gates) +5. [Quality Metrics](#5-quality-metrics) + +--- + +## 1. Test Framework Selection + +### 1.1 Framework Decision Matrix + +| Criteria | pytest | Playwright | Vitest | Cypress | +|---|---|---|---|---| +| **Language** | Python | JS/TS, Python, .NET, Java | JS/TS (Vite-based) | JS/TS (bundled) | +| **Primary Domain** | Unit, integration, API | E2E browser, mobile (WebKit) | Unit, component, E2E | E2E browser, component | +| **Browser Support** | N/A | Chromium, Firefox, WebKit, Edge | Via Playwright/WDIO browser mode | Chromium, Firefox, Edge, WebKit | +| **Parallelism** | pytest-xdist | Built-in workers + sharding | Built-in worker pool + sharding | Dashboard parallelization (paid) | +| **Auto-wait** | N/A | Yes (built-in) | N/A (VDOM assertions) | Yes (built-in, retry-ability) | +| **Network Mocking** | responses / pytest-httpx | route() API | vi.mock / msw | cy.intercept() | +| **Debugging** | pdb / --pdb | Trace viewer, screenshots, video | Browser DevTools | Time-travel, snapshots | +| **CI-first?** | Yes | Yes (blob reports, sharding) | Yes (sharding, pool) | Dashboard-based | +| **Community** | Mature, 10k+ plugins | Rapidly growing, MS-backed | Growing, Vite ecosystem | Large, mature | +| **Best For** | Python projects, data/API testing | Multi-browser E2E, cross-platform | Vite/React/Vue component & unit | Dev-integrated E2E, component testing | + +### 1.2 When to Use Which Framework + +**pytest** +- Python projects of any size +- API/integration testing against backends +- Data pipeline validation, DB testing +- Parameterized testing at scale (built-in) +- When you need 500+ plugins (django, mock, cov, xdist, splinter) +- Architecture: `conftest.py` hierarchy with scoped fixtures + +**Playwright** +- Cross-browser E2E (Chromium + Firefox + WebKit) +- Mobile Web testing (emulation) +- Network interception and mocking +- When CI speed matters (native sharding + workers) +- Trace viewer for debugging flaky tests +- API testing alongside browser tests (request context) + +```typescript +// Playwright sharding in CI +// npx playwright test --shard=1/4 +// npx playwright test --shard=2/4 +``` + +**Vitest** +- Vite-based projects (React, Vue, Svelte) +- Component tests with HMR (instant feedback during dev) +- Unit tests needing ES module support +- When you want Jest-compatible API but faster +- Browser mode (experimental) for limited E2E + +```typescript +// vitest.config.ts +import { defineConfig } from 'vitest/config' +export default defineConfig({ + test: { + globals: true, + pool: 'forks', // or 'threads' + poolOptions: { threads: { singleThread: true } }, + }, +}) +``` + +**Cypress** +- Developer-focused E2E where debugging UX matters +- Teams already in the JS ecosystem +- Component tests for React/Vue (experimental) +- When time-travel debugging is essential +- Note: limited cross-browser (no Safari) without paid plan + +### 1.3 Framework Selection Decision Flow + +``` +Is the project Python? + |-- YES --> Use pytest (with xdist for speed) + |-- NO --> Is it a Vite-based JS/TS project? + |-- YES --> Unit/component: Vitest + | E2E: Playwright or Cypress + |-- NO --> JS/TS non-Vite: Playwright (E2E) + Jest/Vitest (unit) +``` + +--- + +## 2. CI Integration: Parallel Execution, Sharding & Test Splitting + +### 2.1 Three Levels of Parallelism + +| Level | What It Does | Tooling | +|---|---|---| +| **Within a job (multi-worker)** | Multiple tests run on the same machine in parallel | `pytest -n auto` (xdist), Playwright workers, Vitest pool | +| **Across jobs (sharding)** | Test suite split into N groups, each on its own CI runner | `--shard=x/y`, matrix strategy | +| **Across suites** | Different test types (unit, integration, E2E) run in separate CI jobs | CI matrix, workflow orchestration | + +### 2.2 Pytest Parallelism + +**pytest-xdist (within node)** +```bash +# Auto-detect CPU count +pytest -n auto + +# Fixed number of workers +pytest -n 4 + +# Distribute by scope: each worker gets a subset of tests +pytest -n 4 --dist loadscope # tests in same module stay together +pytest -n 4 --dist loadfile # tests in same file stay together +pytest -n 4 --dist worksteal # dynamic rebalancing (pytest-xdist 3.x+) +``` + +**pytest-split (across CI jobs)** +```bash +# Job 1 +pytest --splits 4 --group 1 + +# Job 2 +pytest --splits 4 --group 2 + +# Uses timing data from --store-durations to balance groups +``` + +```yaml +# GitHub Actions: pytest-split with matrix +jobs: + test: + strategy: + matrix: + group: [1, 2, 3, 4] + steps: + - uses: actions/checkout@v4 + - run: pip install pytest pytest-split + - run: pytest --splits 4 --group ${{ matrix.group }} +``` + +### 2.3 Playwright Sharding + +```yaml +# playwright.config.ts +export default defineConfig({ + fullyParallel: true, // split at test level, not file level + workers: process.env.CI ? 2 : undefined, + reporter: process.env.CI ? 'blob' : 'html', +}) + +# GitHub Actions with matrix +# npx playwright test --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }} +jobs: + test: + strategy: + matrix: + shardIndex: [1, 2, 3, 4] + shardTotal: [4] + steps: + - run: npx playwright test --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }} + - uses: actions/upload-artifact@v4 + if: always() + with: + name: blob-report-${{ matrix.shardIndex }} + path: blob-report + + merge-reports: + if: always() + needs: [test] + steps: + - uses: actions/download-artifact@v4 + - run: npx playwright merge-reports --reporter html ./all-blob-reports +``` + +**Shard Balancing Tips** +- `fullyParallel: true` splits at the individual test level for even distribution +- Without `fullyParallel`, shards split at the file level (files with many tests can unbalance) +- Use `blob` reporter in CI to capture results across shards +- Merge reports into a single HTML for aggregate viewing + +### 2.4 Vitest Sharding + +```bash +# CLI sharding +vitest --shard=1/4 +vitest --shard=2/4 + +# GitHub Actions with matrix +jobs: + test: + strategy: + matrix: + shard: [1, 2, 3, 4] + steps: + - run: npx vitest --reporter=junit --shard=${{ matrix.shard }}/4 +``` + +**Vitest Pool Options** +| Pool | Description | Best For | +|---|---|---| +| `threads` (default) | Uses worker_threads, fastest | Pure unit tests | +| `forks` | Uses child_process, better isolation | Tests with side effects | +| `vmThreads` | Uses vm module within threads | Tests needing module sandboxing | + +```typescript +// vitest.config.ts +export default defineConfig({ + test: { + pool: 'forks', + poolOptions: { + forks: { + singleFork: false, + maxForks: 4, + minForks: 1, + }, + }, + }, +}) +``` + +### 2.5 Cypress Parallelization + +Cypress requires the **Dashboard** service (paid) for native parallelization: + +```bash +# Autoscales across available CI machines +cypress run --record --parallel +``` + +Alternatively, use **cypress-split** (open source) for manual splitting: +```bash +# Job 1 +cypress run --env split=1,of=4 + +# Job 2 +cypress run --env split=2,of=4 +``` + +### 2.6 Best Practices for CI Test Distribution + +1. **Log historical timing data** — tools like pytest-split use previous run durations to balance groups. Run `--store-durations` on a baseline first. +2. **Set job-level timeouts** — prevent hung workers from blocking the pipeline. +3. **Fail fast on critical failures** — separate critical (blocking) tests from advisory tests so critical failures halt early. +4. **Use dependency caching** — cache node_modules, .cache, and pip packages between shards. +5. **Shard by timing, not alphabetically** — alphabetically splitting tests creates unbalanced groups. + +--- + +## 3. Flaky Test Management + +### 3.1 Definition & Impact + +> A flaky (or flakey) test is an automated test that produces inconsistent outcomes — green one run, red the next — with no changes to code or test environment. + +**Impact Summary** +- **Delayed PRs** — developers rerun jobs or request overrides +- **Higher CI costs** — repeated runs consume compute +- **Loss of trust** — teams bypass automation for manual checks +- **Customer risk** — real defects masked by noise +- **Quantifiable**: 5% flakiness on 10,000 daily tests = 500 false failures/day (~40+ hours lost/week) + +### 3.2 Root Causes + +| Category | Examples | Fix Pattern | +|---|---|---| +| **Async / Race Conditions** | Click before element ready, DOM not updated | Explicit waits, auto-waiting frameworks | +| **External Dependencies** | API latency, DB connection flakiness | Mock/stub external services | +| **Uncontrolled Test Data** | Random data, data collisions | Seeded randomness, idempotent setup | +| **Environment Issues** | CI resource contention, clock skew | Increase resources, isolate test environments | +| **State Leakage** | Shared mutable state, bad teardown | Isolated state per test, fixture cleanup | +| **Test Interdependence** | Test B depends on Test A's state | Fully independent tests, random execution order | + +### 3.3 Detection Strategies + +```bash +# Repeat test to reproduce flakiness +pytest --repeat 20 test_flaky.py +npx playwright test --repeat-each=20 +npx vitest --repeats 10 +``` + +**Automated Detection** +1. **Retry analysis** — track which tests fail on first attempt but pass on retry +2. **Cross-environment comparison** — compare pass/fail across branches and CI runners +3. **Statistical trending** — track flakiness rate per test over time (failures / total runs) +4. **Burn-in** — run new tests 100+ times in CI before allowing into the main suite + +### 3.4 Management Framework + +``` +Detection --> Measurement --> Prioritization --> Resolution --> Prevention +``` + +| Phase | Actions | +|---|---| +| **Detection** | Retries (2-3x), repeat-each runs, cross-branch analysis | +| **Measurement** | Flakiness Rate = failures / runs; set threshold (<1-2%) | +| **Prioritization** | Fix core workflow / critical-path tests first; quarantine non-critical | +| **Resolution** | Reproduce locally, inspect traces/logs/videos, stabilize waits/selectors | +| **Prevention** | Ownership assignment, flakiness budgets, dashboards + alerts | + +### 3.5 Auto-Retry vs Quarantine + +| Strategy | How It Works | When to Use | +|---|---|---| +| **Auto-retry** | Test that fails on first attempt is retried 1-3x before reporting failure | Smoke tests, CI where 1-2% flakiness is tolerated | +| **Quarantine** | Flaky test moved to separate suite; runs but doesn't block pipeline | Tests > 5% flaky, or blocking PRs | +| **Blocking** | Test must pass every time without retry | Critical-path tests, security, payment flows | + +```yaml +# Playwright: auto-retry with limits +# playwright.config.ts +export default defineConfig({ + retries: process.env.CI ? 2 : 0, +}) +``` + +### 3.6 Stabilization Patterns + +**Stable selectors and waits (Playwright)** +```typescript +// BAD: blind sleep +await page.waitForTimeout(3000) + +// GOOD: await observable state +await expect(page.locator('[data-testid="submit"]')).toBeVisible({ timeout: 5000 }) +await page.waitForLoadState('networkidle') +await page.waitForResponse(resp => resp.url().includes('/api/login') && resp.status() === 200) +``` + +**Controlled test data (pytest)** +```python +import uuid + +def test_create_user(db_session): + unique_email = f"test-{uuid.uuid4()}@example.com" + # use unique_email to prevent collision +``` + +**Isolated fixtures (pytest)** +```python +@pytest.fixture(autouse=True) +def clean_state(db_session): + yield + db_session.rollback() # never leak state between tests +``` + +**Mocking external services** +```python +# pytest with responses library +import responses + +@responses.activate +def test_api_call(): + responses.get("https://api.example.com/data", json={"key": "value"}) + # test now has 0 flakiness from network +``` + +### 3.7 Burn-in Protocol + +New tests should prove their reliability before entering the main suite: + +1. **Commit test** → run in PR pipeline +2. **Burn-in period** → run 100+ times in CI (bg job or nightly) +3. **Stability check** → if flakiness rate > threshold, quarantine until fixed +4. **Promotion** → move to main test suite once proven stable + +--- + +## 4. Quality Gates + +### 4.1 Definition & Core Principles + +> A quality gate is an enforced measure built into your pipeline that must be satisfied before the software can proceed to the next stage or be released. + +**Core Principles** + +| Principle | Description | +|---|---| +| **Shift left** | Catch issues as early as possible — static analysis before unit tests, unit tests before integration | +| **Automate by default** | Prefer automated gates; only use manual approvals where regulatory compliance requires them | +| **Fast feedback** | Quick gates run first (linting, unit tests); slower gates (E2E, performance) run later | +| **Make pass/fail unambiguous** | Criteria must be binary — no "mostly passes" | +| **Allow manual override** | Emergency bypass must exist with accountability (multi-party approval, audit trail) | +| **Evolve over time** | Gate thresholds should tighten as the project matures | + +### 4.2 Gate Types: Blocking vs Advisory + +| Type | Behavior | CI Signal | Example | +|---|---|---|---| +| **Blocking** | Pipeline stops. Artifact is not promoted. Release is blocked. | ❌ Red | Unit test failure, security vulnerability | +| **Advisory** | Pipeline continues. Warning is logged. Team notified. | ⚠️ Warn-only | Code coverage dropped under threshold, linting warnings | +| **Informational** | No pipeline impact. Metric recorded for dashboard. | ℹ️ Info | Test execution time trend, flakiness rate | + +**Mixing blocking and advisory gates by stage:** + +``` +Commit --> [Lint (advisory)] --> [Unit tests (blocking)] + --> [Build (blocking)] --> [Static analysis (advisory)] + --> [Integration tests (blocking)] --> [E2E tests (blocking)] + --> [Performance/load (advisory)] --> [Security scan (blocking)] + --> [Manual approval (blocking)] --> Release +``` + +### 4.3 Common Quality Gates by Pipeline Stage + +| Stage | Gate | Type | Threshold | +|---|---|---|---| +| **Code commit** | Linting / formatting | Advisory | 0 errors; formatting warnings logged | +| **Build** | Compilation | Blocking | 0 compile errors | +| **Unit tests** | Pass rate | Blocking | 100% pass (known fails = 0) | +| **Code coverage** | Coverage threshold | Advisory → Blocking | Unit ≥ 80%, Integration ≥ 60% | +| **Static analysis** | Bugs / smells | Blocking | 0 critical/blocker bugs | +| **Security** | SAST / dependency check | Blocking | 0 known CVEs above threshold | +| **Integration tests** | Pass rate | Blocking | 100% pass | +| **E2E tests** | Pass rate | Blocking | 100% pass (with flake retry budget) | +| **Performance** | Response time / throughput | Advisory | p95 < 500ms, no regression > 10% | +| **Flaky tests** | Quarantine count | Advisory | < N flaky tests in suite | + +### 4.4 Implementing Quality Gates in CI + +**GitHub Actions example — staged gates** + +```yaml +jobs: + lint: + runs-on: ubuntu-latest + steps: + - run: npm run lint + continue-on-error: true # advisory: warn but continue + + unit-tests: + needs: [lint] + runs-on: ubuntu-latest + steps: + - run: npm test -- --coverage + - run: | + # Advisory: coverage check (warns but doesn't fail) + npx istanbul check-coverage --statement=80 + + integration: + needs: [unit-tests] + runs-on: ubuntu-latest + steps: + - run: npm run test:integration + # Blocking: integration tests must all pass + + e2e: + needs: [integration] + runs-on: ubuntu-latest + steps: + - run: npx playwright test + # Blocking: E2E must pass + + security-scan: + needs: [unit-tests] + runs-on: ubuntu-latest + if: github.ref == 'refs/heads/main' + steps: + - run: npm audit --audit-level=high +``` + +**Jenkins Pipeline — declarative gates** + +```groovy +stage('Quality Gates') { + steps { + // Blocking: test pass + sh 'pytest tests/unit --junitxml=unit-results.xml' + + // Advisory: coverage + sh ''' + coverage=$(python -c "import json; d=json.load(open('coverage.json')); print(d['totals']['percent_covered'])") + if (( $(echo "$coverage < 80" | bc -l) )); then + echo "WARNING: Coverage ${coverage}% below 80% threshold" + # Don't fail the build + fi + ''' + + // Blocking: SonarQube quality gate + withSonarQubeEnv('SonarQube') { + sh 'mvn sonar:sonar' + } + timeout(time: 5, unit: 'MINUTES') { + waitForQualityGate abortPipeline: true + } + } +} +``` + +### 4.5 Security Gates + +| Check | Tooling | Gate Behavior | +|---|---|---| +| **SAST** | SonarQube, Semgrep, CodeQL | Block on critical/high findings | +| **SCA (dependency vulns)** | Dependabot, Snyk, OWASP DC | Block on CVSS ≥ 7.0 | +| **Secrets detection** | GitLeaks, TruffleHog | Block on any hardcoded secret | +| **Container scanning** | Trivy, Grype | Block on critical OS-level CVEs | +| **License compliance** | FOSSA, LicenseFinder | Block on GPL/AFL in commercial project | + +### 4.6 Quality Gate Evolution + +Gates should tighten as the project and team mature: + +``` +Phase 1 (Starting out): + - Blocking: tests must compile and pass + - Advisory: coverage > 50% + +Phase 2 (Growing): + - Blocking: unit tests pass, coverage > 70%, 0 critical SonarQube issues + - Advisory: coverage > 80%, security scan clean + +Phase 3 (Maturing): + - Blocking: all tests pass, coverage > 80%, 0 blocker/critical SonarQube, 0 high CVEs + - Advisory: coverage > 85%, flakiness < 2% + +Phase 4 (High performance): + - Blocking: all tests pass, coverage > 85%, 0 SonarQube bugs, 0 CVEs above threshold + - Advisory: coverage > 90%, performance regression < 5%, flakiness < 1% +``` + +**Gate review cadence:** Re-evaluate gate thresholds quarterly. If a gate never fires (all PRs pass trivially), consider tightening it. If a gate fires too frequently (50%+ of PRs blocked), loosen it temporarily while the team improves code quality. + +### 4.7 Common Anti-Patterns + +| Anti-pattern | Problem | Fix | +|---|---|---| +| **Too many blocking gates** | Developers bypass or game the pipeline | Keep blocking gates to critical checks; use advisory for everything else | +| **Gates that never change** | Thresholds become irrelevant as project evolves | Review quarterly, tighten gradually | +| **Measuring coverage without quality** | 90% coverage of untested logic is misleading | Combine coverage with mutation testing or code review | +| **Single flaky test blocks the entire pipeline** | Loss of trust in CI | Quarantine flaky tests automatically; separate blocking vs advisory suites | +| **Manual gates for everything** | Pipeline becomes the bottleneck | Automate everything that can be scripted; keep manual for regulatory sign-offs only | + +--- + +## 5. Quality Metrics + +### 5.1 The 5-10 Rule + +Track only **5-10 core metrics** that directly inform decisions. Fewer than 5 misses signals; more than 10 causes analysis paralysis and actionability drops. + +### 5.2 Metric Taxonomy + +| Type | Definition | Examples | +|---|---|---| +| **Absolute** | Raw counts | Tests run, bugs found, lines of code | +| **Derived** | Ratios / percentages | Coverage %, pass rate %, defect density | +| **Leading** | Predicts future quality | Test coverage, execution status, code complexity | +| **Lagging** | Validates past outcomes | Production escapes, customer complaints, MTTR | +| **Effectiveness** | Does testing catch bugs? | Defect detection percentage, test effectiveness ratio | +| **Efficiency** | How fast is testing? | Test execution time, time to test bug fix | + +### 5.3 Core Metrics — Formulas & Guidance + +#### 5.3.1 Test Coverage + +```text +Line Coverage = (Lines executed / Total lines) × 100 +Branch Coverage = (Branches executed / Total branches) × 100 +Function Coverage = (Functions called / Total functions) × 100 +``` + +**Practical guidance:** +- 100% coverage is a trap — diminishing returns after ~80% +- Focus coverage on high-risk areas: payment flows, auth, data transformations +- Combine **line coverage** + **branch coverage** (line only misses `if` branches) +- Mutation testing (pitest, mutmut) validates coverage quality — high line coverage but low mutation score means tests don't actually assert behavior + +```bash +# pytest coverage +pytest --cov=src --cov-report=term-missing --cov-fail-under=80 + +# vitest coverage +npx vitest --coverage --coverage.thresholds.lines 80 +``` + +#### 5.3.2 Defect Density + +```text +Defect Density = Total confirmed defects / Software size (LOC or function points) +``` + +**Interpretation:** +- Lower is better, but context matters (complex modules naturally have higher density) +- **Stratify by severity**: critical: 0.5/KLOC, minor: 2/KLOC → same density, different risk profiles +- **Stratify by module**: find modules with abnormally high density for targeted refactoring +- **Combine with coverage**: high defect density + low coverage = urgent improvement needed + +**Common pitfalls:** +- Comparing defect density across projects of different types (libraries vs applications) +- Including duplicate/won't-fix bugs in the count +- Not normalizing for code complexity (a simple CRUD module vs complex algorithm) + +#### 5.3.3 Test Effectiveness + +```text +Test Effectiveness = (Defects found by testing / Total defects found) × 100 + +Or more practically: +DDP (Defect Detection Percentage) = (Bugs found before release / (Bugs found before + after release)) × 100 +``` + +**Goal:** DDP > 95% means fewer than 5% of bugs reach production. + +**Measuring what tests catch:** +- Track which tests actually find bugs (linked in bug tracker) +- Identify tests that never fail → candidates for removal or rewriting +- **Regression test effectiveness**: of bugs fixed, how many had a regression test added? Target > 80%. + +#### 5.3.4 MTTD (Mean Time to Detect) + +```text +MTTD = Sum of (Detection time - Introduction time) / Total defects found +``` + +Where detection time = when the bug was first observed (not when reported). + +| Scenario | Typical MTTD | Interpretation | +|---|---|---| +| Automated test catches bug in PR | Minutes | Excellent — shift-left detection | +| Caught in staging CI | Hours | Good | +| Caught in QA period | Days | Needs faster feedback | +| Caught in production by monitoring | Hours–days | Acceptable for edge cases | +| Caught in production by customer report | Days–weeks | Poor — invest in monitoring | + +**Reducing MTTD:** +- Expand automated test coverage +- Improve production monitoring (APM, error tracking) +- Feature flags for gradual rollouts +- Real user monitoring (RUM) and session replay + +#### 5.3.5 MTTR (Mean Time to Resolve / Repair) + +```text +MTTR = Sum of (Resolution time - Detection time) / Total defects fixed +``` + +MTTR includes: triage → debug → fix → test → deploy + +**Targets by severity:** +| Severity | Target MTTR | +|---|---| +| Critical (P0) | < 1 hour | +| High (P1) | < 4 hours | +| Medium (P2) | < 24 hours | +| Low (P3) | < 1 week | + +**Reducing MTTR:** +- Automated rollback (fast revert) +- Feature flags to disable problematic code without redeploy +- Structured debugging tools (Playwright traces, log correlation) +- Post-incident reviews to eliminate process bottlenecks + +#### 5.3.6 Defect Escape Rate + +```text +Defect Escape Rate = Production defects / (Pre-production defects + Production defects) × 100 +``` + +**Interpretation:** +- < 5%: Strong QA process +- 5-15%: Average — room for improvement +- > 15%: Significant escape pattern — invest in shift-left testing + +### 5.4 Visualizing Metrics + +**Recommended dashboard structure:** + +``` +╔══════════════════════════════════════╗ +║ Quality Dashboard — Sprint 24 ║ +╠══════════════════════════════════════╣ +║ PASS RATE │ COVERAGE │ DEFECTS ║ +║ 98.5% ✓ │ 83% ⚠️ │ 12 (3 P1)║ +╠══════════════════════════════════════╣ +║ DDP │ MTTD │ MTTR ║ +║ 94% ✓ │ 2.1h ✓ │ 4.5h ⚠️ ║ +╠══════════════════════════════════════╣ +║ FLAKINESS │ SUITE DUR │ BUDGET ║ +║ 1.2% ✓ │ 14m ✓ │ 62% ▓██ ║ +╚══════════════════════════════════════╝ +``` + +### 5.5 Metric Selection Per Methodology + +| Methodology | Priority Metrics | +|---|---| +| **Agile / Scrum** | Sprint pass rate, defect escape rate, test execution status, velocity-adjusted coverage | +| **Kanban** | Lead time to test, cycle time per fix, flow efficiency, WIP limits | +| **CI/CD** | Build stability, deployment frequency, change failure rate, MTTD, MTTR | +| **Waterfall** | Requirements coverage, phase-wise defect density, test case effectiveness | + +### 5.6 Data-Driven Quality Culture + +**Implementation principles:** +1. **Can you act on it?** — If a metric changes, do you know what to do next? If not, don't track it. +2. **Can you update it regularly?** — Match refresh frequency to decision cadence (daily for CI, sprintly for coverage, monthly for trends). +3. **Align with goals** — Faster releases = track speed + quality together (change failure rate). +4. **Avoid vanity metrics** — "Tests executed" is activity; "Tests that caught a real bug" is value. +5. **Share broadly** — Developers, PMs, and executives all get relevant slices of the same data. + +--- + +## Quick Reference Summary + +### Test Framework Cheatsheet + +| Need | Pick | +|---|---| +| Python API/unit tests | pytest + xdist + pytest-cov | +| Multi-browser E2E | Playwright (sharding, trace viewer) | +| Vite/React component tests | Vitest (HMR, browser mode) | +| Dev-focused E2E debugging | Cypress (time-travel, retry-ability) | + +### CI Parallelism Cheatsheet + +| Tool | Within Node | Across CI Jobs | Timing Balance | +|---|---|---|---| +| pytest | `-n auto` (xdist) | pytest-split (`--splits N --group X`) | `--store-durations` | +| Playwright | `workers: N` | `--shard=x/y` | `fullyParallel: true` | +| Vitest | `poolOptions.forks.maxForks` | `--shard=x/y` | Pool-level balancing | +| Cypress | Auto (Dashboard) | `--parallel` (Dashboard) or cypress-split | Dashboard-managed | + +### Quality Gates Cheatsheet + +| Gate | Stage | Type | Threshold | +|---|---|---|---| +| Pass unit tests | Build → Integration | Blocking | 100% | +| Coverage ≥ 80% | After unit tests | Advisory → Blocking | Line + branch | +| No critical SonarQube | Static analysis | Blocking | 0 blocker + critical | +| No high CVEs | Security scan | Blocking | CVSS ≥ 7.0 | +| E2E tests pass | Pre-deployment | Blocking | 100% (retry budget) | +| Performance < 10% regression | Load test | Advisory | p95, throughput | + +### Quality Metrics Cheatsheet + +| Metric | Target | Formula | +|---|---|---| +| Defect Detection Percentage | > 95% | pre-release / (pre + post) × 100 | +| Defect Density | < 1/KLOC (critical), < 5/KLOC (all) | defects / LOC × 1000 | +| Line Coverage | > 80% | executed lines / total lines × 100 | +| MTTD | < 2 hours | sum(detection time) / defects | +| MTTR (critical) | < 1 hour | sum(resolution time) / fixes | +| Flakiness Rate | < 2% | flaky failures / total runs × 100 | +| Defect Escape Rate | < 5% | production defects / total defects × 100 | + +--- + +*Document produced June 2026. Sources include Playwright docs, Vitest docs, SonarSource, TestRail, Currents, Information Week, MinimumCD Practice Guide, and industry patterns from leading QA teams.* diff --git a/qa-methodology/references/test-data-management.md b/qa-methodology/references/test-data-management.md new file mode 100644 index 0000000..5af9726 --- /dev/null +++ b/qa-methodology/references/test-data-management.md @@ -0,0 +1,81 @@ +# Test Data Management + +## Fixtures vs Factories + +| Approach | When | Trade-off | +|----------|------|-----------| +| Static fixtures (JSON/YAML files) | Small, stable datasets; API contract tests | Brittle to schema changes, easy to read | +| Factory functions (factory_boy, fishery) | Relational data, many-to-many, randomized | Setup complexity, harder to debug | +| Builder pattern | Complex objects with many optional fields | Verbose but explicit | +| Inline construction | One-off tests, 1–3 fields | Doesn't scale, but zero indirection | + +### Factory Pattern (Python) + +```python +# factories.py +import factory +from myapp.models import User, Order + +class UserFactory(factory.django.DjangoModelFactory): + class Meta: + model = User + email = factory.Sequence(lambda n: f"user{n}@test.dev") + name = factory.Faker("name") + +class OrderFactory(factory.django.DjangoModelFactory): + class Meta: + model = Order + user = factory.SubFactory(UserFactory) + total = factory.Faker("pydecimal", min_value=1, max_value=500, right_digits=2) +``` + +## Test Isolation + +| Strategy | Mechanism | Speed | Safety | +|----------|-----------|-------|--------| +| Transaction rollback | Wrap test in transaction, rollback after | Fast | High — no cross-test leakage | +| Database per test | Create/drop schema per test | Slow | Highest — full isolation | +| Truncate between tests | `TRUNCATE ... CASCADE` after each | Medium | High | +| Unique prefixes | Each test uses `test-{uuid}-` prefixed data | Fast | Medium — relies on discipline | + +**Rule:** Prefer transaction rollback (pytest-django `@pytest.mark.django_db`, Rails `use_transactional_tests`). Fall back to truncation only when tests need committed state (e.g., testing triggers, background jobs). + +## Synthetic Data Generation + +| Tool | Use Case | +|------|----------| +| Faker | Names, emails, addresses, dates — realistic but fake | +| Presidio + Faker | Generate PII-shaped data that passes validation without real PII | +| SDV (Synthetic Data Vault) | Statistical replicas of production tables — preserves distributions | +| dbt seed + Jinja | Version-controlled CSV fixtures with templated expansion | + +### PII Rules + +- **Never** use production PII in test databases +- Synthetic data must pass the same validation rules as real data (format, length, checksums) +- If a test needs a specific edge case (e.g., unicode name, 255-char email), construct it explicitly — don't rely on random generation hitting it + +## External Service Data + +| Service | Test Strategy | +|---------|---------------| +| Payment (Stripe) | Test-mode API keys + recorded fixtures (VCR.py / Polly.js) | +| Email (SendGrid) | Mock at transport layer; assert on message content | +| S3 / object storage | MinIO or `moto` (AWS mock); never hit real buckets | +| Third-party APIs | Contract tests (Pact) + recorded responses; rotate recordings quarterly | + +## Data Volume Testing + +| Scenario | Approach | +|----------|----------| +| Pagination | Seed exactly `page_size + 1` records | +| Performance under load | `generate_series()` in SQL or bulk factory (10K–100K rows) | +| Edge cases | Empty table, single row, max-length fields, unicode, nulls | +| Time-dependent | Freeze time (`freezegun`, `timecop`) — never `sleep()` | + +## Migration Testing + +- Run migrations against a copy of production schema (anonymized) in CI +- Test both forward and rollback paths +- Data migrations: seed before-state, run migration, assert after-state +- Never test migrations against a schema that diverges from production (see: systematic-debugging Phase 1 step 5a) diff --git a/qa-methodology/references/test-strategy.md b/qa-methodology/references/test-strategy.md new file mode 100644 index 0000000..eeb296f --- /dev/null +++ b/qa-methodology/references/test-strategy.md @@ -0,0 +1,40 @@ +# Test Strategy Design + +## Test Pyramid + +``` + ╱╲ + ╱ E2E ╲ Few — critical user journeys + ╱────────╲ + ╱ ╲ + ╱ Integration ╲ Some — contract tests, API tests, service boundaries + ╱────────────────╲ + ╱ ╲ + ╱ Unit / Component ╲ Many — isolated, fast, deterministic +╱────────────────────────╲ +``` + +| Level | Speed | Cost to maintain | What it catches | +|-------|-------|-----------------|-----------------| +| Unit | ms | Low | Logic errors, edge cases, invariants | +| Integration | seconds | Medium | Contract mismatches, data flow errors | +| E2E | minutes | High | System behavior, user-visible regressions | + +## Risk-Based Prioritization + +| Risk Level | Test Coverage Required | Example | +|------------|----------------------|---------| +| Critical (P0) | 100% — every path, every edge case | Payment processing, auth, data integrity | +| High (P1) | 90%+ — all happy paths, known failure modes | Core business logic, API contracts | +| Medium (P2) | 70%+ — happy paths, common failure modes | Secondary features, non-critical APIs | +| Low (P3) | Smoke test only | UI polish, optional features, debug tooling | + +## Test Selection by Project Type + +| Project Type | Recommended Test Mix | +|-------------|---------------------| +| Library / SDK | 80% unit, 15% integration, 5% E2E | +| Web API | 40% unit, 40% integration, 20% E2E | +| Web application | 30% unit, 40% integration, 30% E2E | +| CLI tool | 60% unit, 30% integration (output comparison), 10% E2E | +| Data pipeline | 50% unit, 40% integration (data flow), 10% E2E | diff --git a/verification-methodology/SKILL.md b/verification-methodology/SKILL.md index ba1090d..0ba449a 100644 --- a/verification-methodology/SKILL.md +++ b/verification-methodology/SKILL.md @@ -29,8 +29,56 @@ Stop when every criterion has direct evidence or an explicit blocked/not-applica |-----------|-------------| | `references/criteria-assessment.md` | You need to evaluate whether work meets completion criteria | | `references/evidence-standards.md` | You need to judge whether evidence supports the claims made | +| `references/magnus919-refine-to-ship-gate.md` | You are running the Magnus919 Refine-to-Ship verifier gate — 12 criteria, editorial change verification, output structure | | `references/verdict-template.md` | You need to produce a structured pass/fail/hold verdict | +## Magnus919 Refine-to-Ship Gate Criteria + +12 criteria for the verifier profile. Each criterion maps to an observable, reproducible check. + +### All 12 Criteria + +| # | Criterion | How to Verify | +|---|-----------|---------------| +| 1 | **Dash scan** — zero em dash (U+2014), en dash (U+2013), horizontal bar (U+2015), or visible prose double-hyphen | `search_files` for `[\u2014\u2013\u2015]` and `\-\-`. Double-hyphens in YAML frontmatter delimiters are OK. | +| 2 | **Fact-check** — all methodology claims map to source; no fabricated numbers, chronology, or universal claims | Cross-reference article claims to source document sections. Search for `\d+%`, `percent`, `average of`, `illustrative`. Search for `research proves`, `studies demonstrate`. | +| 3 | **Voice-check** — Magnus fingerprint: conversational first-person, contractions, "But" pivots (not formal transitions), colons over semicolons, no consultant cadence | Search for `Furthermore`, `Moreover`, `Nevertheless`, `Consequently`, `Therefore`, `not only.*but also`, triplet parallelism. Count colons vs semicolons (should skew heavily toward colons). | +| 4 | **Oxford commas, spelling, grammar** — American English, Oxford commas in series, no spelling errors | Manual read of series. Check for consistent formatting. | +| 5 | **No formulaic AI closing** — zero "In conclusion", "To summarize", "In this article", generic motivational advice | `search_files` for `In conclusion`, `Ultimately,`, `To summarize`, `In this article`, `In this post`. | +| 6 | **Methodology-first** — personal frame ≤ ~10% of article; rest is methodology | Count paragraphs in frame vs body. | +| 7 | **Human stake integrated** — cognitive burden, expertise formation, transferred work, anti-surveillance, accountable authority | Verify dedicated section or dispersed coverage of all dimensions. | +| 8 | **Privacy/anonymization** — zero company identifiers, role titles, named people, source filename, proprietary domain examples | `search_files` for company name, product names, domain-specific terminology from source. | +| 9 | **Frontmatter** — title, slug, date, byline correct and value-identical to specification | `read_file` lines 1–11. | +| 10 | **Links resolve** — each distinct URL appears once at first meaningful mention; all return 200 | `curl -s -o /dev/null -w "%{http_code}"` each URL. Verify link text is at first meaningful mention. | +| 11 | **Hugo build + routes** — build exit 0; new route returns 200; old take-home-title route returns 404 | `hugo --quiet && echo EXIT:$?`. `curl` both routes. | +| 12 | **No duplicate source bundle** — single directory, single `index.md`; no stale `*take-home*` directories | `ls` the page bundle directory. `find` in content/posts for duplicate slug patterns. | + +### Parent-Requested Editorial Changes + +When the parent profile specifies editorial changes during gate recovery, verify each one is present before proceeding with the full criteria scan: + +| Change Type | Verification Method | +|-------------|-------------------| +| Fabricated illustrative numbers removed | `search_files` for `\d+%`, `percent`, `PRs? per`, `average of` → zero hits | +| Tense correction | `search_files` for the exact parent-specified phrase | +| Closing replacement | `search_files` for the first and last sentence of the parent-specified closing | + +### Verdict Rules + +- **PASS:** All 12 criteria met. Produce 00-index.md, 01-summary/verdict.md, 02-analysis/per-criteria-results.md. +- **BLOCK:** Any criterion fails. Produce gap-details.md with specific fix instructions. See `verifier-gate-recovery` skill for remediation patterns. + +### Output Structure + +``` +/private/tmp/verifier-gate/-refine/ + 00-index.md — verdict, links to artifacts + 01-summary/verdict.md — per-criterion pass/fail table + 02-analysis/ + per-criteria-results.md — detailed evidence per criterion + gap-details.md — only if BLOCK, with remediation instructions +``` + ## Portability -This skill is intentionally host-neutral. Use your agent's normal mechanisms to load the references, templates, and scripts listed here. Do not assume a particular profile system, task orchestrator, memory service, or response-handoff format. +This skill is intentionally host-neutral. Use your agent's normal mechanisms to load the references, templates, and scripts listed here. Do not assume a particular profile system, task orchestrator, memory service, or response-handoff format. \ No newline at end of file