mirror of
https://github.com/magnus919/agent-skills.git
synced 2026-09-11 19:47:12 +03:00
feat: add data-architect skill
A virtual data architect for teams without one. Includes: - 8 reference files covering architecture patterns, cloud platforms, governance maturity, anti-patterns, compliance, vendor evaluation, case studies, and discovery frameworks - Interactive governance maturity assessment script - ADR template for capturing architecture decisions - 4 Mermaid decision trees for common architecture choices - QuickScan and proactive discovery flow for users who don't know where to start No personal or identifying information included.
This commit is contained in:
Magnus Hedemark
@@ -14,6 +14,10 @@ Each skill lives in its own directory with a `SKILL.md` as the entry point, opti
|
||||
|
||||
Design and build CLI tools for AI agent consumption. 10 universal patterns (non-interactive, `--json`, `--dry-run`, idempotent, lazy auth, progressive help), an agent-compatibility test suite, and a bash scaffold template. Principles grounded in real failures from building 15+ agent-facing CLIs.
|
||||
|
||||
### [data-architect](data-architect/SKILL.md)
|
||||
|
||||
A virtual data architect for teams who don't have one. Consult on architecture decisions, review data models and pipelines, evaluate tradeoffs (warehouse vs lakehouse, Kimball vs Inmon, batch vs streaming), assess governance maturity, and plan data strategy. Includes 8 reference files, a governance maturity assessment script, and an ADR template. Designed for data engineers working without a human architect — includes a QuickScan and proactive discovery flow for when you don't know where to start.
|
||||
|
||||
### [agent-skills](agent-skills/SKILL.md)
|
||||
|
||||
Reference for the Agent Skills open format itself — directory structure, frontmatter schema, naming conventions, and progressive disclosure model. Use this meta-skill when creating or reviewing any other skill in this repository.
|
||||
|
||||
@@ -0,0 +1,207 @@
|
||||
---
|
||||
name: data-architect
|
||||
description: >-
|
||||
A virtual data architect for teams who don't have one. If your data pipelines
|
||||
are growing faster than your team, nobody agrees on what 'customer' means,
|
||||
your cloud bill is climbing without clear reason, or you're about to choose a
|
||||
data platform and need someone who's seen this before — load this skill. I'll
|
||||
help you spot problems you didn't know you had, ask questions you didn't know
|
||||
to ask, and give you a path forward even when you're not sure where to start.
|
||||
compatibility: >-
|
||||
Designed for agentic AI assistants (Hermes Agent, Claude Code, similar
|
||||
coding agents). No special system requirements.
|
||||
metadata:
|
||||
author: data-architect contributors
|
||||
version: "1.0.0"
|
||||
topics: data-architecture, data-modeling, data-warehouse, data-governance, data-platform, etl, streaming, cloud-data
|
||||
---
|
||||
|
||||
# Data Architect — Virtual Expert
|
||||
|
||||
When this skill is loaded, I become a **virtual data architect** — someone who's seen enough data platforms go wrong to recognize the patterns early. I don't wait for you to know the right questions. If you're not sure where to start, tell me and I'll run a discovery.
|
||||
|
||||
## Do You Need an Architect? (Recognizing the Symptoms)
|
||||
|
||||
Load this skill if any of these sound familiar — even if you're not sure what to do about them:
|
||||
|
||||
**Pain signals:**
|
||||
- Your data team is 3-5 people and growing, and you're starting to trip over each other
|
||||
- Different teams have different definitions for the same business terms ("what does 'active customer' even mean?")
|
||||
- You're about to pick a data platform and everyone has a strong opinion but no clear criteria
|
||||
- Your cloud data bill keeps climbing and nobody can explain which pipeline is driving it
|
||||
- Data pipelines break regularly and the root cause is hard to trace
|
||||
- You're building your third pipeline that does basically the same thing as the first two
|
||||
- Someone just asked "should we use Data Mesh?" and you need a sanity check
|
||||
- You're migrating from an on-prem warehouse to the cloud and don't know the right sequence
|
||||
|
||||
**Ambient anxiety signals:**
|
||||
- "I feel like we should have a data catalog but I'm not sure"
|
||||
- "We have data quality issues that keep surfacing in production"
|
||||
- "I think we need better governance but nobody wants to be the one to slow things down"
|
||||
- "We're growing fast and I'm worried our current setup won't scale"
|
||||
|
||||
Not sure if you need help? Say "I don't know where to start" and I'll run a quick discovery.
|
||||
|
||||
## QuickScan — Five Minutes to Spot Common Gaps
|
||||
|
||||
If you're not sure what problems you have, answer these yes/no questions. I'll use your answers to identify where to focus. You don't need to know anything about data architecture to answer them.
|
||||
|
||||
**Q1: Data inventory.** Can you list every system that produces data your team consumes? Do you know what's in each one?
|
||||
- If no → we should start with data source discovery (`references/discovery-framework.md`)
|
||||
|
||||
**Q2: Data definitions.** If two teams use the term "active customer" or "revenue," would they get the same answer?
|
||||
- If no → you have a semantic alignment problem. Let's talk about business glossary and data contracts.
|
||||
|
||||
**Q3: Data ownership.** For each important dataset, is there a named person responsible for its quality?
|
||||
- If no → we should design a data ownership model. This is a governance maturity gap.
|
||||
|
||||
**Q4: Pipeline observability.** When a pipeline breaks, can you trace which source caused it and which reports are affected?
|
||||
- If no → you need column-level lineage. Let's look at data catalogs and lineage tooling.
|
||||
|
||||
**Q5: Platform selection criteria.** If you had to pick between Snowflake, BigQuery, Redshift, and Databricks today, would you have a structured way to decide?
|
||||
- If no → load `references/cloud-platform-comparison.md` and `references/architecture-patterns.md`.
|
||||
|
||||
**Q6: Data quality SLAs.** Do you know the accuracy and freshness of your most critical datasets?
|
||||
- If no → governance maturity gap. See `references/governance-maturity.md`.
|
||||
|
||||
**Q7: Cost attribution.** Can you explain this month's cloud data bill? Do you know which pipelines, queries, or storage consume the most?
|
||||
- If no → you need cost observability. This is a FinOps for data problem.
|
||||
|
||||
**Q8 : Schema management.** When a source system changes its schema, does anything automatically detect and flag the change?
|
||||
- If no → you need schema registry or contract testing. Let's look at data contracts.
|
||||
|
||||
**Scoring:**
|
||||
- **0-2 no's:** You're in decent shape. Pick the specific area that bothers you most.
|
||||
- **3-5 no's:** Classic growing-pain territory. Say "I don't know where to start" and I'll prioritize.
|
||||
- **6-8 no's:** You've been flying without instruments. This is exactly the right time to bring in architectural thinking.
|
||||
|
||||
I embody these traits when consulting:
|
||||
|
||||
**I push back on premature solutions.** Before any technology recommendation, I need to understand the business problem, the actual scale, the consumers, and the team's capability.
|
||||
|
||||
**I make tradeoffs explicit.** Every decision is a set of tradeoffs — I frame them clearly rather than giving a single right answer.
|
||||
|
||||
**I think in systems, not components.** I trace data from source to consumption, identifying where quality degrades, latency accumulates, governance gaps exist, and costs blow up.
|
||||
|
||||
**I design for the team that will maintain it.** A clever architecture is a liability if the team can't operate it. I factor in team size, skill level, and organizational context.
|
||||
|
||||
**I teach as I go.** If you don't know what a term means or why I'm asking a question, say so. I'll explain the concept and why it matters before we move on. The goal is not just to give you answers — it's to help you recognize these patterns yourself next time.
|
||||
|
||||
**I'm honest about uncertainty.** If your context needs something I'm not sure about, I'll tell you and suggest how to validate it.
|
||||
|
||||
## Consulting Patterns
|
||||
|
||||
### Architecture Review
|
||||
When you present a design for review:
|
||||
1. Ask clarifying questions about constraints (scale, budget, team, timeline)
|
||||
2. Identify implicit assumptions that may be wrong
|
||||
3. Trace failure modes — what breaks and how
|
||||
4. Suggest alternatives with clear tradeoff language
|
||||
5. Prioritize findings by impact
|
||||
6. **Produce an ADR** — Say "capture that as an ADR" and I'll generate a structured Architecture Decision Record using `templates/adr-template.md`
|
||||
|
||||
### Decision Framework
|
||||
When asked "X vs Y", I structure the answer:
|
||||
- Core difference in architectural philosophy
|
||||
- What problem each solves best
|
||||
- What context tilts the decision
|
||||
- Migration cost if you pick wrong
|
||||
- Operational complexity of each
|
||||
|
||||
### Strategy & Roadmap
|
||||
When planning multi-quarter evolution:
|
||||
1. Current-state assessment — what you have, what hurts
|
||||
2. Identify quick wins with high impact-to-effort ratio
|
||||
3. Sequence investments so each phase enables the next
|
||||
4. Flag organizational dependencies (hiring, skill building, governance maturity)
|
||||
5. Define success criteria for each phase
|
||||
|
||||
## Proactive Discovery — When You Don't Know Where to Start
|
||||
|
||||
If you load this skill and say "I don't know where to start" or "just help me figure out what I need," here's what I'll do. You don't need to prepare anything.
|
||||
|
||||
**Step 1: Context grab (2 minutes)**
|
||||
I'll ask a few quick things:
|
||||
- How big is your data team? (1-2 people? 3-10? 10+?)
|
||||
- How many data sources do you have?
|
||||
- What's the #1 thing that's bothering you right now? (cost, reliability, speed, confusion)
|
||||
- Are you on a cloud platform already, and which one?
|
||||
|
||||
**Step 2: QuickScan (covered above)**
|
||||
I'll walk through the 8 questions. Just answer yes/no — I'll track the score.
|
||||
|
||||
**Step 3: Prioritize**
|
||||
Based on your answers, I'll tell you:
|
||||
- The one thing I'd fix first (highest impact, lowest effort)
|
||||
- The one thing I'd plan for but not act on yet (emerging risk)
|
||||
- What to ignore for now (it can wait)
|
||||
|
||||
**Step 4: Next action**
|
||||
I'll give you a concrete next step — something you can do today, in this session, that will produce value. Maybe it's "let's sketch your current data flow" or "let me help you define what 'customer' means so both teams align."
|
||||
|
||||
**To trigger this:** Just say "I don't know where to start." I'll take it from there.
|
||||
|
||||
## Core Expertise Areas
|
||||
|
||||
I have deep knowledge across these domains. Each has a reference file with decision guides — load them on demand when the topic comes up:
|
||||
|
||||
- **Data modeling** — Kimball, Inmon, Data Vault, lakehouse, star vs snowflake. → `references/architecture-patterns.md`
|
||||
- **Data warehousing & lakehouse** — Medallion architecture, cloud warehouse design, cost optimization. → `references/architecture-patterns.md`
|
||||
- **Cloud data platforms** — Snowflake, BigQuery, Redshift, Databricks. → `references/cloud-platform-comparison.md`
|
||||
- **Data governance** — Frameworks, maturity model, quality dimensions, metadata management. → `references/governance-maturity.md`
|
||||
- **Compliance & regulated environments** — GDPR, HIPAA, CCPA, SOX, PCI DSS, BCBS 239. → `references/compliance-by-framework.md`
|
||||
- **Vendor evaluation** — Data catalogs, ETL/ELT tools, orchestration platforms. → `references/vendor-evaluation.md`
|
||||
- **Data integration & ETL/ELT** — Batch vs streaming, CDC, dbt patterns, data contracts
|
||||
- **Streaming & real-time** — Kafka architecture, Kappa vs Lambda, when streaming is worth it
|
||||
- **AI/ML data infrastructure** — Feature stores, RAG architecture, training data pipelines
|
||||
- **Tools ecosystem** — Modeling, warehouse, integration, governance, storage, observability tools
|
||||
- **Real-world case studies** — Lakehouse migrations, Data Vault implementations, hybrid architectures. → `references/case-studies.md`
|
||||
|
||||
## Reference Files
|
||||
|
||||
Load these on demand when the topic comes up:
|
||||
|
||||
- `references/architecture-patterns.md` — Decision framework for Kimball vs Inmon vs Data Vault vs Lakehouse, including strengths, weaknesses, and when to choose each. Also covers streaming vs batch, star vs snowflake, Medallion architecture.
|
||||
- `references/anti-patterns.md` — 13 named anti-patterns with symptoms, root causes, and remediations. Load when doing design review or incident post-mortem.
|
||||
- `references/discovery-framework.md` — Structured discovery questions and consulting session flow. Load at the start of a new architecture engagement.
|
||||
- `references/cloud-platform-comparison.md` — Snowflake vs BigQuery vs Redshift vs Databricks: architecture, pricing, scaling, lock-in vectors, and decision framework. Load when doing platform selection or migration planning.
|
||||
- `references/governance-maturity.md` — Staged data governance maturity model (Level 0-5) with DAMA-DMBOK framework, what each stage looks like in practice, and progression paths. Load when designing or assessing a governance program.
|
||||
- `references/vendor-evaluation.md` — Structured evaluation criteria for data catalogs (Atlan, Alation, Collibra, DataHub, etc.), ETL/ELT tools (Fivetran, Airbyte, dbt), and orchestration (Airflow, Dagster, Prefect). Load during vendor selection.
|
||||
- `references/compliance-by-framework.md` — What GDPR, HIPAA, CCPA, SOX, PCI DSS, and BCBS 239 require from a data architecture perspective. Design patterns for each. Load when designing for regulated environments.
|
||||
- `references/case-studies.md` — Real-world architecture transformations: Data Vault at a commercial bank, lakehouse at Avant/Insulet/7-Eleven, hybrid Snowflake+Databricks at Janus Henderson. Load when you want concrete examples to ground a recommendation.
|
||||
|
||||
## Scripts & Templates
|
||||
|
||||
The skill includes tools I can run during a session:
|
||||
|
||||
- `scripts/governance-assessment.py` — Interactive governance maturity assessment. Asks 15 scored questions across 5 dimensions, produces a maturity level, dimension scores, and prioritized recommendations. Run when someone asks "how mature is our governance?"
|
||||
- `templates/adr-template.md` — Architecture Decision Record template. I'll fill this in when you say "capture that as an ADR" during a consulting session.
|
||||
|
||||
Usage:
|
||||
```bash
|
||||
# Interactive assessment
|
||||
python3 scripts/governance-assessment.py
|
||||
|
||||
# Planned: maturity report in JSON for programmatic use
|
||||
python3 scripts/governance-assessment.py --json
|
||||
```
|
||||
|
||||
## When NOT to Load This Skill
|
||||
|
||||
This skill is for data architecture strategy, design, and governance. Don't load it for:
|
||||
|
||||
- **Real-time pipeline debugging** — If a Kafka consumer is falling behind or an Airflow DAG keeps failing, you need an SRE or data engineer, not an architect.
|
||||
- **SQL optimization** — Slow query? That's a tuning problem. I can point you to the right performance patterns, but I won't write your query plans.
|
||||
- **Specific tool configuration** — "How do I set up RBAC in Snowflake?" / "What's the dbt YAML syntax for tests?" These are implementation details, not architecture decisions.
|
||||
- **Data science model development** — Feature selection, hyperparameter tuning, model evaluation — that's the data scientist's domain. I handle the infrastructure that serves the data to them, not the modeling itself.
|
||||
|
||||
## Common Anti-Patterns (Quick Reference)
|
||||
|
||||
The most frequent issues I flag:
|
||||
|
||||
- **Silver bullet thinking** — Adopting Data Mesh because it's trendy, not because your org is ready for domain ownership
|
||||
- **Governance as an afterthought** — "We'll add governance later" (you won't, and it'll cost 10x)
|
||||
- **SoR vs SSoT confusion** — Treating a transactional System of Record (e.g. ERP) as the enterprise Single Source of Truth, creating a bottleneck
|
||||
- **Neglecting the team** — Designing a system nobody can operate or troubleshoot
|
||||
|
||||
See all 13 with full remediations in `references/anti-patterns.md`.
|
||||
@@ -0,0 +1,112 @@
|
||||
# Data Architecture Anti-Patterns Catalog
|
||||
|
||||
A catalog of named anti-patterns with symptoms, root causes, and remediations.
|
||||
Sources: Sunil Prakash (Enterprise Data Architecture), Kengie Ho (Common Enterprise Architectural Pitfalls), DataForest, and industry experience.
|
||||
|
||||
## 1. Silver Bullet Thinking
|
||||
|
||||
**Symptom:** Adopting Data Mesh, Data Fabric, or another pattern because it's trendy, without organizational readiness.
|
||||
|
||||
**Root cause:** Believing an architectural pattern alone solves what are fundamentally organizational problems (ownership, trust, skills).
|
||||
|
||||
**Remediation:** Before adopting Data Mesh, verify you have: domain teams willing to own data products, a platform team to build self-serve infra, and executive sponsorship for federated governance. If any is missing, start smaller.
|
||||
|
||||
## 2. SoR vs SSoT Confusion
|
||||
|
||||
**Symptom:** Business leaders point to a legacy 20-year-old billing system and say "That's our Single Source of Truth." Modern apps are forced to dumb down their data to fit the legacy system's narrow schema.
|
||||
|
||||
**Root cause:** Confusing a transactional System of Record (SoR) — where data is originally authored — with a Single Source of Truth (SSoT) — an aggregated, reconciled, governed view across the enterprise.
|
||||
|
||||
**Remediation:** Decouple the two domains. Let operational systems be localized SoRs for their domain. Build a dedicated layer (MDM platform, data lakehouse, or data mesh) above them that ingests from every SoR, resolves conflicts, applies quality rules, and serves as the true SSoT.
|
||||
|
||||
## 3. High-Fidelity / Lossy Sync Destruction
|
||||
|
||||
**Symptom:** A modern CRM with rich, structured data is synced bidirectionally with a legacy ERP. The ERP truncates names, flattens addresses, drops extra fields — then syncs that flattened data back to the CRM, permanently overwriting the high-fidelity records.
|
||||
|
||||
**Root cause:** Allowing a low-fidelity system to blindly update a high-fidelity system through lossy synchronization. Known formally as semantic heterogeneity, information loss in schema mapping, and destructive overwrites.
|
||||
|
||||
**Remediation:** Place an integration middleware or MDM platform at the system boundary with **metadata-driven field-level survivorship**. Apply hierarchical rules: "The ERP always wins financial fields, but the CRM always wins customer contact info." Build an Anti-Corruption Layer (from Domain-Driven Design) to protect modern systems from legacy constraints.
|
||||
|
||||
## 4. Fragmented Identity / "Customer 720"
|
||||
|
||||
**Symptom:** Customer logs in with email (System A), calls support as "Jane Doe" (System B). Without a shared primary key, the integration layer creates a duplicate record. Instead of Customer 360, you get "Customer 720."
|
||||
|
||||
**Root cause:** Relying on simple SQL JOINs or exact-match keys for entity resolution across systems that have no common identifier.
|
||||
|
||||
**Remediation:** Introduce an Identity Resolution Engine at the data ingestion layer. Use two-stage matching:
|
||||
- **Deterministic:** exact matches on hashed email, SSN, phone
|
||||
- **Probabilistic:** ML-based matching (Levenshtein distance, shared device IPs, behavioral patterns) with confidence scoring
|
||||
|
||||
Treat identity resolution as a foundational requirement, not a downstream analytics problem.
|
||||
|
||||
## 5. Schema Evolution Mismatch
|
||||
|
||||
**Symptom:** A modern microservices team splits `FullName` into `FirstName`/`LastName`. A downstream legacy app consuming the data stream crashes because it expects the old schema.
|
||||
|
||||
**Root cause:** Speed mismatch between agile schema evolution and rigid legacy consumers. Tightly coupled interfaces without mediation.
|
||||
|
||||
**Remediation:** Introduce an Anti-Corruption Layer (ACL) between modern and legacy systems. The ACL translates schemas at the boundary — the modern system evolves freely, the ACL provides backward-compatible views to legacy consumers.
|
||||
|
||||
## 6. The Ivory Tower
|
||||
|
||||
**Symptom:** Architecture team produces standards, roadmaps, and reference architectures in isolation, without engaging delivery teams or business stakeholders. Artifacts are technically correct but irrelevant to actual problems.
|
||||
|
||||
**Root cause:** Architecture practiced as a detached design discipline rather than an embedded consulting function.
|
||||
|
||||
**Remediation:** Embed architects with delivery teams. Architecture decisions should emerge from concrete problems, not abstract frameworks. The architect who writes a standard without understanding the team's actual constraints is building shelfware.
|
||||
|
||||
## 7. Tool-First Architecture
|
||||
|
||||
**Symptom:** Organization picks Snowflake (or Databricks, or Kafka) and then goes looking for problems to solve with it.
|
||||
|
||||
**Root cause:** Technology selection driven by vendor marketing, hiring availability, or executive preference rather than requirements analysis.
|
||||
|
||||
**Remediation:** Start with business requirements, data profiles, and consumer needs. Let the architecture drive the tool selection, not the other way around. Run POCs against real workloads before committing.
|
||||
|
||||
## 8. Governance as an Afterthought
|
||||
|
||||
**Symptom:** "We'll add governance later." Data proliferates uncontrolled for 18 months. When governance finally arrives, it requires a multi-quarter cleanup effort and meets resistance from teams accustomed to full autonomy.
|
||||
|
||||
**Root cause:** Treating governance as a phase rather than a design constraint.
|
||||
|
||||
**Remediation:** Establish minimum viable governance from day one: data classification, ownership tagging, basic lineage. Add rigor incrementally as the platform matures. Governance that grows with the system is adopted; governance imposed after the fact is rejected.
|
||||
|
||||
## 9. Premature Petabyte
|
||||
|
||||
**Symptom:** Designing for "petabyte scale" when the organization has 2TB of data. Complex, expensive architecture that delivers no marginal benefit for years.
|
||||
|
||||
**Root cause:** Architects designing for resumes rather than actual problems. Future-proofing without understanding the cost of complexity.
|
||||
|
||||
**Remediation:** Design for 10x growth, not 1000x. Use architectures that scale horizontally without requiring upfront complexity. The cloud warehouse patterns (Snowflake, BigQuery) handle growth without premature optimization.
|
||||
|
||||
## 10. Streaming Everything
|
||||
|
||||
**Symptom:** Kafka for every data movement, even daily batch reports. Streaming infrastructure cost exceeds the value of real-time data.
|
||||
|
||||
**Root cause:** Assuming "real-time is better" without quantifying the latency requirement. Conflating event-driven architecture with streaming infrastructure.
|
||||
|
||||
**Remediation:** Require a concrete latency SLA before introducing streaming. If the business can tolerate 15-minute latency, micro-batch is sufficient. If it can tolerate hourly, batch is fine. Keep streaming infrastructure for the subset of pipelines that genuinely need sub-second delivery.
|
||||
|
||||
## 11. Copy-Paste FAANG Architecture
|
||||
|
||||
**Symptom:** Replicating the data infrastructure patterns of Google, Netflix, or Uber without accounting for your organization's actual scale, budget, team capability, or problem complexity.
|
||||
|
||||
**Root cause:** Assuming that what works at hyperscale is optimal at any scale. FAANG patterns evolved to solve FAANG-scale problems — not general problems.
|
||||
|
||||
**Remediation:** Choose the simplest architecture that meets your actual requirements. Managed services (Snowflake, BigQuery, Fivetran, dbt Cloud) are often the right choice for teams of 5-50, even if they wouldn't scale to FAANG levels. Complexity should be earned, not inherited.
|
||||
|
||||
## 12. Design by Committee
|
||||
|
||||
**Symptom:** Architecture that tries to accommodate every stakeholder's preference and ends up satisfying no one. Compromised designs that mix incompatible patterns.
|
||||
|
||||
**Root cause:** Architecture by consensus rather than by clear ownership and decision rights.
|
||||
|
||||
**Remediation:** Assign clear decision authority for architecture decisions. Solicit input from stakeholders, but the architect makes the final call with documented rationale. ADRs (Architecture Decision Records) make this explicit: "We considered X, Y, Z. We chose X because... The tradeoffs are..."
|
||||
|
||||
## 13. Neglecting the Team
|
||||
|
||||
**Symptom:** Designing a technically elegant system that the team cannot operate, troubleshoot, or extend. High "bus factor" on the architecture itself.
|
||||
|
||||
**Root cause:** Designing for technical purity rather than organizational reality.
|
||||
|
||||
**Remediation:** Factor in team size, skill level, and operational maturity. A simple architecture operated well is better than a clever architecture nobody understands. Build runbooks, invest in training, and design for debuggability.
|
||||
@@ -0,0 +1,229 @@
|
||||
# Data Architecture Patterns — Decision Framework
|
||||
|
||||
A practical guide to choosing between the major data architecture patterns. Based on research from Ryan Kirsch, TalkingSchema, LinkedIn data architecture practitioners, and industry reference architectures.
|
||||
|
||||
## Decision Tree — Which Pattern to Use?
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Q1["What's your primary use case?"] --> Q1a{Analytics/BI<br>or ML/AI?}
|
||||
Q1a -->|Analytics & BI| Q2["What's your organizational context?"]
|
||||
Q1a -->|ML/AI + Analytics| Q5["Do you have a dedicated<br>infrastructure engineering team?"]
|
||||
|
||||
Q2 -->|Startup or small team| A[Kimball + dbt<br>on managed cloud warehouse]
|
||||
Q2 -->|Enterprise, complex<br>cross-domain integration| Q3["Can you invest in upfront design<br>and wait longer for value?"]
|
||||
Q2 -->|Regulated industry<br>banking, insurance, healthcare| B[Data Vault 2.0]
|
||||
|
||||
Q3 -->|Yes, we have time<br>and budget| C[Inmon: normalized EDW<br>+ derived data marts]
|
||||
Q3 -->|No, need iterative<br>delivery| D[Hybrid: Inmon staging layer<br>+ Kimball marts]
|
||||
|
||||
Q5 -->|Yes → we have infra engineers| E[Lakehouse: open formats,<br>multiple compute engines]
|
||||
Q5 -->|No → we want managed| F[Cloud warehouse + dbt<br>Snowflake / BigQuery / Redshift]
|
||||
|
||||
style A fill:#e3f2fd,stroke:#1565c0
|
||||
style B fill:#fce4ec,stroke:#c62828
|
||||
style C fill:#f3e5f5,stroke:#6a1b9a
|
||||
style D fill:#e8f5e9,stroke:#2e7d32
|
||||
style E fill:#fff3e0,stroke:#e65100
|
||||
style F fill:#e0f7fa,stroke:#00695c
|
||||
```
|
||||
|
||||
**Quick summary:**
|
||||
- **Regulated, audit-heavy** → Data Vault 2.0
|
||||
- **Startup, need speed** → Kimball + dbt
|
||||
- **Enterprise, big investment** → Inmon or hybrid
|
||||
- **ML + analytics from same data** → Lakehouse
|
||||
- **Frequent source schema changes** → Data Vault
|
||||
|
||||
## Kimball (Dimensional / Bottom-Up)
|
||||
|
||||
**Philosophy:** Build data marts for individual business processes first. The enterprise warehouse emerges from integrating marts via conformed dimensions.
|
||||
|
||||
**Core artifacts:** Fact tables (measurements: revenue, clicks, orders) + Dimension tables (context: customers, products, dates) arranged in star schemas.
|
||||
|
||||
**Strengths:**
|
||||
- Business users can query marts directly with minimal SQL
|
||||
- Fast delivery — build one domain at a time
|
||||
- Maps naturally to BI tool concepts (dimensions as filters, facts as measures)
|
||||
|
||||
**Weaknesses:**
|
||||
- Conformed dimensions are hard to maintain as marts proliferate
|
||||
- Inconsistent grain definitions across marts create data discrepancies
|
||||
- Bottom-up can produce marts that are hard to integrate later for cross-domain questions
|
||||
|
||||
**When to choose:**
|
||||
- Startup or small team building the first warehouse
|
||||
- Business intelligence and reporting are the primary use case
|
||||
- Need fast time-to-value per business domain
|
||||
- Using modern tooling (dbt on Snowflake/BigQuery/Redshift)
|
||||
|
||||
## Inmon (3NF / Top-Down / Corporate Information Factory)
|
||||
|
||||
**Philosophy:** Design a normalized, subject-oriented, integrated enterprise warehouse first, then derive data marts as views or aggregates on top.
|
||||
|
||||
**Core structure:** Third normal form (3NF) — normalized to eliminate redundancy, subject-oriented (customer, product, transaction rather than source-system-oriented), integrated across all source systems.
|
||||
|
||||
**Strengths:**
|
||||
- Genuine single source of truth enforced at the schema level
|
||||
- Flexible — can answer questions not anticipated at design time
|
||||
- Data quality and integration enforced centrally
|
||||
|
||||
**Weaknesses:**
|
||||
- Upfront design effort is substantial
|
||||
- Time-to-first-insight is much longer than Kimball
|
||||
- Normalized schemas require complex queries for business users
|
||||
- Iterative delivery is harder when everything flows from a central schema
|
||||
|
||||
**When to choose:**
|
||||
- Enterprise with complex cross-domain integration requirements
|
||||
- Data consistency is the highest priority
|
||||
- Organization has the patience and resources for upfront design
|
||||
- Regulatory environment demands strict auditability of data lineage
|
||||
|
||||
## Data Vault 2.0
|
||||
|
||||
**Philosophy:** Decompose every data entity into three types: hubs (business keys only), links (relationships between hubs), and satellites (descriptive attributes with full history).
|
||||
|
||||
**Core structure:**
|
||||
```
|
||||
hub_customer (customer_hk, customer_id, load_date, record_source)
|
||||
hub_order (order_hk, order_id, load_date, record_source)
|
||||
link_customer_order (customer_order_hk, customer_hk, order_hk, load_date)
|
||||
sat_customer_details (customer_hk, load_date, name, email, segment, hash_diff)
|
||||
sat_order_details (order_hk, load_date, status, amount, hash_diff)
|
||||
```
|
||||
|
||||
**Strengths:**
|
||||
- Extremely flexible to schema changes in source systems
|
||||
- Complete auditability (every record has load date and source)
|
||||
- Supports parallel loading (hubs, links, satellites load independently)
|
||||
- Scales to massive heterogeneous data environments
|
||||
|
||||
**Weaknesses:**
|
||||
- High structural complexity — raw vault is not queryable by analysts
|
||||
- Most consumers need a "business vault" or information mart layer on top
|
||||
- Tooling and expertise are less common than Kimball or Inmon
|
||||
- Generally overkill outside regulated enterprise environments
|
||||
|
||||
**When to choose:**
|
||||
- Regulated industry (banking, insurance, healthcare) with strict audit requirements
|
||||
- Multiple heterogeneous source systems that change frequently
|
||||
- Historical tracking and data provenance are legal requirements
|
||||
- Organization has the expertise to operate it
|
||||
|
||||
## Lakehouse (Modern Synthesis)
|
||||
|
||||
**Philosophy:** Separate storage from compute using open table formats (Apache Iceberg, Delta Lake, Apache Hudi) on object storage (S3, GCS). Multiple compute engines (Spark, Trino, DuckDB, Snowflake, BigQuery) read from and write to the same storage.
|
||||
|
||||
**Layer structure:**
|
||||
```
|
||||
s3://data-lake/
|
||||
bronze/ # Raw, source-aligned (Inmon influence)
|
||||
silver/ # Cleaned, validated, integrated
|
||||
gold/ # Business-ready, query-optimized (Kimball influence)
|
||||
```
|
||||
|
||||
**Strengths:**
|
||||
- Decoupled storage and compute — pay for compute only when querying
|
||||
- Multiple engines serve different use cases from the same data
|
||||
- Open formats avoid vendor lock-in
|
||||
- Layer structure combines Inmon's integration discipline with Kimball's query performance
|
||||
- Serves both SQL analytics and ML pipelines from the same storage
|
||||
|
||||
**Weaknesses:**
|
||||
- More moving parts than a managed warehouse (Snowflake, BigQuery)
|
||||
- Operational complexity of managing object storage + table format metadata + multiple engines
|
||||
- "Best of both worlds" marketing often undersells the engineering work required
|
||||
|
||||
**When to choose:**
|
||||
- Team with significant unstructured/semi-structured data
|
||||
- ML workloads alongside analytics from the same data
|
||||
- Team has dedicated infrastructure engineers
|
||||
- Need to avoid vendor lock-in at the storage layer
|
||||
|
||||
## Batch vs Streaming
|
||||
|
||||
### Decision Tree — When to Stream?
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
Q1["What latency does<br>the business need?"] --> Q1a{Sub-second?}
|
||||
Q1a -->|Yes| Q2["Is the data volume<br>high and continuous?"]
|
||||
Q1a -->|No| Q3{"< 1 minute?"}
|
||||
|
||||
Q2 -->|Yes| A["Streaming<br>Kafka + Flink"]
|
||||
Q2 -->|No| B["Micro-batch<br>Spark Streaming"]
|
||||
|
||||
Q3 -->|Yes| B
|
||||
Q3 -->|No| C{"< 15 minutes?"}
|
||||
|
||||
C -->|Yes| D["Micro-batch<br>Spark / Kafka Connect"]
|
||||
C -->|No| E["Batch<br>Airflow / Dagster"]
|
||||
|
||||
style A fill:#fce4ec,stroke:#c62828
|
||||
style E fill:#e8f5e9,stroke:#2e7d32
|
||||
```
|
||||
|
||||
**Rule of thumb:** Start with batch. Only move to streaming when you have a concrete latency requirement that batch can't meet. "Real-time" is rarely worth the complexity premium.
|
||||
|
||||
### Batch (Airflow/Dagster scheduled jobs)
|
||||
- **Strengths:** Simpler, cheaper, easier to reprocess, well-understood failure modes
|
||||
- **Weaknesses:** Higher latency (minutes to hours), stale data between runs
|
||||
- **When:** Reporting, ML training data, any scenario where sub-minute freshness isn't required
|
||||
|
||||
### Streaming (Kafka/Flink)
|
||||
- **Strengths:** Real-time (sub-second), event-driven, supports reactive systems
|
||||
- **Weaknesses:** Significantly more complex, harder to reprocess, state management challenges, expensive
|
||||
- **When:** Fraud detection, real-time dashboards, operational alerts, event-driven microservices
|
||||
|
||||
### Micro-batch (Spark Streaming, Kafka Connect)
|
||||
- **Strengths:** Sweet spot for most use cases — seconds to minutes latency without full streaming complexity
|
||||
- **Weaknesses:** Not truly real-time, batch windows create artificial latency
|
||||
- **When:** Most enterprise real-time use cases that don't need sub-second
|
||||
|
||||
**Rule of thumb:** Start with batch. Only move to streaming when you have a concrete latency requirement that batch can't meet. "Real-time" is rarely worth the complexity premium.
|
||||
|
||||
## Star Schema vs Snowflake Schema
|
||||
|
||||
### Star Schema
|
||||
- Denormalized dimensions, fewer joins, simpler queries
|
||||
- Better BI tool performance
|
||||
- Preferred for analytics
|
||||
|
||||
### Snowflake Schema
|
||||
- Normalized dimensions, saves storage, enforces integrity
|
||||
- More complex queries, worse query performance
|
||||
- Generally not worth the maintenance cost — star is almost always the right choice for analytics
|
||||
|
||||
**Verdict:** "Never! — I strongly believe the high maintenance of this outweighs any benefits compared to other methods." Use star schema for analytics. Use 3NF for operational/transactional systems.
|
||||
|
||||
## Decision Flow
|
||||
|
||||
1. **What's the primary use case?**
|
||||
- Analytics/BI → go to 2
|
||||
- ML/AI + analytics → lean Lakehouse
|
||||
- Transactional/operational → this is OLTP, not data warehousing
|
||||
|
||||
2. **What's the organizational context?**
|
||||
- Startup/small team → Kimball + dbt on managed cloud warehouse
|
||||
- Enterprise, complex integration, can invest upfront → Inmon or hybrid Inmon staging + Kimball marts
|
||||
- Regulated, audit-heavy → Data Vault 2.0
|
||||
- Has dedicated infra team, ML workloads → Lakehouse
|
||||
|
||||
3. **What's the data profile?**
|
||||
- Structured, predictable → any pattern works
|
||||
- Semi-structured, schema-on-read important → Lakehouse
|
||||
- Frequent source schema changes → Data Vault
|
||||
|
||||
4. **What's the team capability?**
|
||||
- Generalist data team → managed warehouse + Kimball
|
||||
- Strong engineering team with infra skills → Lakehouse or Data Vault
|
||||
- Small team, need speed → Kimball + dbt
|
||||
|
||||
## Source References
|
||||
|
||||
- Ryan Kirsch, "Data Warehouse Architecture Patterns: Kimball, Inmon, and the Modern Lakehouse" — practical comparison with code examples
|
||||
- TalkingSchema, "Kimball vs Inmon vs Data Vault 2.0: Choose Like an Architect, Not a Fanboy" — decision framework with debunked myths
|
||||
- Benjamin Tabares Jr., "Data Modelling Frameworks: Understanding Inmon, Kimball, and Data Vault"
|
||||
- Blockmill, "OBT vs Star Schema vs Data Vault vs Inmon and more" — hybrid approach patterns
|
||||
- LinkedIn Data Warehousing, "Comparing and Contrasting Three Data Warehouse Design Frameworks"
|
||||
@@ -0,0 +1,101 @@
|
||||
# Data Architecture Case Studies
|
||||
|
||||
Real-world architecture transformations. Sources: Hiflylabs, Datalere, Databricks customer stories, Qubika, and Atlan.
|
||||
|
||||
## Commercial Bank: Data Vault 2.0 for Payment Processing
|
||||
|
||||
**Source:** Hiflylabs case study
|
||||
**Context:** A commercial bank needed to modernize its payment processing data warehouse without a full system overhaul.
|
||||
|
||||
**Problem:** Legacy payment processing architecture was slow and inflexible. Core payment data processing took 4-5 minutes per run, with the full process chain taking 15-20 minutes. The legacy system couldn't support the bank's evolving business rules.
|
||||
|
||||
**Approach:** Rather than rebuilding everything, narrowed focus to two key data sources and three downstream processes. Hybrid approach — bridging the legacy system with Data Vault 2.0 methodology through 15 views and a new hub-link-satellite model on Snowflake.
|
||||
|
||||
**Results:**
|
||||
- Core payment data processing: **4-5 minutes → 10 seconds** (30x improvement)
|
||||
- Overall process chain: **15-20 min → 8-10 min**
|
||||
- Enabled parallel processing of satellite tables
|
||||
- Automation toolkit for Data Vault component generation
|
||||
- Reverse ETL for historical data migration, legacy systems undisturbed
|
||||
|
||||
**Key lesson:** Data Vault doesn't require a big bang. Targeted application to the highest-pain area delivered in two sprints (one month). Automation toolkit meant they could extend to other domains.
|
||||
|
||||
## Avant: Lakehouse Modernization (Fintech)
|
||||
|
||||
**Source:** Qubika / Databricks
|
||||
**Context:** Fintech company needed faster credit decisions, smarter marketing, and automated dispute resolution.
|
||||
|
||||
**Approach:** Migrated to Databricks lakehouse with Delta Lake, MLflow, Unity Catalog. Built end-to-end ML pipelines.
|
||||
|
||||
**Results:**
|
||||
- **56% increase** in delivery velocity
|
||||
- **60% reduction** in data initiative costs
|
||||
- **15+ production ML models**
|
||||
- **10% faster** model predictions
|
||||
- Lower default rates, improved cash flow
|
||||
|
||||
**Key lesson:** Lakehouse served both analytics and ML from the same platform, eliminating data duplication. Cost reduction came from retiring the legacy stack, not optimizing it.
|
||||
|
||||
## Janus Henderson: Hybrid Snowflake + Databricks
|
||||
|
||||
**Source:** Datalere (Mark Goodwin, Data Architect at Janus Henderson)
|
||||
**Context:** Investment firm with both BI/reporting and data science/streaming needs.
|
||||
|
||||
**Problem:** Adopted both platforms independently, creating duplication and inconsistent data.
|
||||
|
||||
**Approach:** Designed unified architecture with clear ownership boundaries:
|
||||
- **Databricks** → complex transformations, data science, streaming
|
||||
- **Snowflake** → BI, reporting, governed analytics
|
||||
|
||||
Data flows from Databricks engineering → Snowflake consumption.
|
||||
|
||||
**Results:**
|
||||
- Eliminated data duplication
|
||||
- Clear ownership per platform
|
||||
- BI teams got governed, consistent data
|
||||
- Engineering teams kept flexibility
|
||||
|
||||
**Key lesson:** Hybrid works with explicit boundaries and data lifecycle governance. Without those, it's worse than picking one.
|
||||
|
||||
## Insulet: Lakehouse for Medical Manufacturing
|
||||
|
||||
**Source:** Databricks Data + AI Summit
|
||||
**Context:** Medical device manufacturer needed to unify Salesforce, SAP, and other data.
|
||||
|
||||
**Approach:** Replaced outdated ETL with Lakeflow, Delta Lake for ACID on the lake.
|
||||
|
||||
**Results:**
|
||||
- **12x faster** real-time data processing
|
||||
- **83% fewer SQL queries** after replacing ETL
|
||||
- **97% lower TCO** by eliminating third-party ETL tools
|
||||
|
||||
**Key lesson:** Biggest win was eliminating expensive middleware entirely, not optimizing it.
|
||||
|
||||
## 7-Eleven: AI at 13,000+ Stores
|
||||
|
||||
**Source:** Databricks Data + AI Summit
|
||||
**Context:** Retailer needed AI-driven store insights across a massive footprint.
|
||||
|
||||
**Approach:** Multi-agent marketing assistant on Databricks. RAG for maintenance knowledge retrieval. Unity Catalog for governance at scale.
|
||||
|
||||
**Results:**
|
||||
- AI-powered search across all stores
|
||||
- Technician productivity improved via RAG
|
||||
- Streamlined governance migration
|
||||
|
||||
**Key lesson:** At 13K+ stores, AI isn't optional — it's how you keep per-store costs from growing linearly.
|
||||
|
||||
## Additional References
|
||||
|
||||
- **Delivery Hero** — Data mesh for multi-market scale. Each market = domain. Result: faster onboarding, required significant platform investment.
|
||||
- **Intuit** — Data mesh across QuickBooks, TurboTax, Mint. Platform treated as product with its own roadmap.
|
||||
- **Dr. Martens** (via Atlan) — Impact analysis from 4-6 weeks to under 30 min via data catalog.
|
||||
- **Kiwi.com** (via Atlan) — 53% engineering workload reduction in 90 days.
|
||||
|
||||
## Sources
|
||||
|
||||
- Hiflylabs, "Commercial Bank Data Warehouse Case Study"
|
||||
- Datalere, "Using Snowflake and Databricks Together: A Unified Architecture"
|
||||
- Databricks, "Data Intelligence in Action: 100+ Data and AI Use Cases"
|
||||
- Qubika, "Avant and Qubika" case study
|
||||
- Atlan, "Data Mesh: Architecture, Principles, and Case Studies"
|
||||
@@ -0,0 +1,170 @@
|
||||
# Cloud Data Platform Comparison
|
||||
|
||||
A practical comparison of Snowflake, Google BigQuery, Amazon Redshift, and Databricks. Based on practitioner research, vendor documentation, and comparison guides from Reintech, Eidosoft, Improvado, Flexera, and Lakshmanan (ex-Google Cloud).
|
||||
|
||||
## Decision Tree — Which Platform to Choose?
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Q1["What's your primary workload?"] --> Q1a{SQL Analytics<br>or ML / Data Science?}
|
||||
Q1a -->|SQL Analytics, BI, Reporting| Q2["What's your cloud strategy?"]
|
||||
Q1a -->|Data Science, ML,<br>complex transformations| Q5["Do you have a team with<br>Spark / Python skills?"]
|
||||
|
||||
Q2 -->|Multi-cloud or no preference| A[Snowflake]
|
||||
Q2 -->|GCP-native / startup| B[BigQuery]
|
||||
Q2 -->|AWS-native / existing AWS| Q3["Do you have DBA expertise<br>for tuning?"]
|
||||
|
||||
Q3 -->|Yes, we can tune| C[Redshift provisioned]
|
||||
Q3 -->|No, want managed| D[Redshift Serverless<br>or Snowflake on AWS]
|
||||
|
||||
Q5 -->|Yes → engineering team| E[Databricks]
|
||||
Q5 -->|No → SQL-only team| F[Snowflake Cortex AI<br>or BigQuery + Vertex AI]
|
||||
|
||||
style A fill:#e3f2fd,stroke:#1565c0
|
||||
style B fill:#e8f5e9,stroke:#2e7d32
|
||||
style C fill:#fce4ec,stroke:#c62828
|
||||
style E fill:#fff3e0,stroke:#e65100
|
||||
```
|
||||
|
||||
**Quick summary:**
|
||||
- **Don't know where to start?** → Snowflake. It's the safest default for most organizations
|
||||
- **Already on GCP / startup** → BigQuery (zero ops, AI included)
|
||||
- **Already on AWS with DBA staff** → Redshift (best price/performance at scale)
|
||||
- **ML is as important as analytics** → Databricks (lakehouse, Spark-native)
|
||||
- **Small team, want managed** → BigQuery or Snowflake
|
||||
|
||||
## Platform Comparison
|
||||
|
||||
### Snowflake
|
||||
|
||||
**Architecture:** Cloud-agnostic (AWS, Azure, GCP). Separates storage and compute completely — multiple virtual warehouses access the same data without copying. Proprietary storage layer.
|
||||
|
||||
**Pricing model:** Credit-based. Compute ($2-$6/credit depending on edition/region). Storage ($40/TB/month on-demand, $23/TB on capacity). Cloud services (free up to 10% of compute). Per-second billing with 60s minimum. Capacity contracts cut credits 20-30%.
|
||||
|
||||
**Scaling:** Spin up multiple warehouses of different sizes (XS through 6XL). Independent scaling for ETL vs analytics workloads. Auto-suspend and auto-resume. Multi-cluster warehouses for concurrency.
|
||||
|
||||
**Performance strengths:**
|
||||
- Concurrent mixed workloads (100+ concurrent users)
|
||||
- Data sharing across organizations (no data movement)
|
||||
- Query performance on structured analytics
|
||||
- Materialized views, search optimization, clustering
|
||||
|
||||
**Weaknesses:**
|
||||
- Cost unpredictable — hidden costs from serverless features (materialized views, Search Optimization, Snowpipe), Cortex AI, Time Travel storage
|
||||
- No native ML training (limited to SQL-based ML)
|
||||
- Proprietary storage format (though now supports Iceberg writes)
|
||||
- Frozen data costs for high-update tables
|
||||
|
||||
**Vendor lock-in vectors:** Proprietary storage format, unique SQL dialect, marketplace ecosystem dependency, data egress costs to migrate out. Iceberg support (2025) reduces lock-in.
|
||||
|
||||
**Best for:** BI-heavy teams, data sharing across business units, multi-cloud strategy, organizations that want managed service with minimal operational burden.
|
||||
|
||||
### Google BigQuery
|
||||
|
||||
**Architecture:** Fully serverless — no clusters to provision. Google's Dremel query engine with columnar Capacitor storage. Tightly integrated with GCP ecosystem.
|
||||
|
||||
**Pricing model:** On-demand ($5/TB queried, 1TB free/month) or flat-rate (slot-based reservations). Storage ($20/TB/month for active, $10/TB for long-term). Committed Use Discounts (CUDs) for predictable workloads. Gemini AI bundled at no extra cost.
|
||||
|
||||
**Scaling:** True auto-scaling — no manual sizing. Resources allocated per query. Slot reservations for predictable performance. Max number of concurrent queries based on slot pool.
|
||||
|
||||
**Performance strengths:**
|
||||
- Ad-hoc analytics on petabyte-scale datasets
|
||||
- Automatic optimization — no tuning required
|
||||
- Streaming ingestion and real-time analytics
|
||||
- Gemini AI (natural-language pipeline creation, visual data exploration)
|
||||
- Multi-region replication
|
||||
|
||||
**Weaknesses:**
|
||||
- Less control over resource allocation for complex queries
|
||||
- No separation of compute for ETL vs analytics (everything shares slot pool)
|
||||
- Query costs unpredictable for exploratory workloads
|
||||
- Multi-region data transfer fees (starting Feb 2026)
|
||||
- Weakest ecosystem outside GCP
|
||||
|
||||
**Vendor lock-in vectors:** Proprietary query engine, storage format coupling, strict GCP dependency for native integration. BigQuery Omni provides limited multi-cloud. Open-source connector ecosystems reduce but don't eliminate dependency.
|
||||
|
||||
**Best for:** GCP-native organizations, ad-hoc analytics, startups that want zero ops, ML/AI teams using Vertex AI integration, variable workloads that benefit from serverless.
|
||||
|
||||
### Amazon Redshift
|
||||
|
||||
**Architecture:** Columnar warehouse on AWS. Traditional provisioned clusters + Redshift Serverless (matured by 2026). RA3 nodes with managed storage separate from compute. Deep AWS integration.
|
||||
|
||||
**Pricing model:** Provisioned (pay per node-hour) or Serverless (pay per RPU-hour). RA3 pricing separates compute and storage billing. Spectrum queries data in S3 without loading. Reserved Instances for 1-3 year terms.
|
||||
|
||||
**Scaling:** Elastic resize (minutes) or classic resize (hours). Concurrency scaling for burst. Redshift Serverless auto-scales. Multi-dimensional data layouts (10x better price/performance on qualifying workloads).
|
||||
|
||||
**Performance strengths:**
|
||||
- Massively parallel processing on huge datasets
|
||||
- AQUA (Advanced Query Accelerator) for S3-based queries
|
||||
- Zero-ETL integrations with 23 sources (PostgreSQL, DynamoDB, Salesforce)
|
||||
- Materialized views, sort keys, distribution keys
|
||||
- Redshift ML for SQL-based model creation
|
||||
|
||||
**Weaknesses:**
|
||||
- Historically required significant tuning (sort keys, distribution styles, compression encodings)
|
||||
- Workload management requires manual configuration
|
||||
- Ecosystem less developed for ML/AI compared to Databricks
|
||||
- Serverless is newer (less proven at extreme scale)
|
||||
- Concurrency scaling costs add up
|
||||
|
||||
**Vendor lock-in vectors:** Deep AWS ecosystem integration, unique workload management model, Spectrum ties to S3. Migration-friendly via open formats (Parquet, Iceberg). AWS ecosystem lock-in is the real concern, not Redshift itself.
|
||||
|
||||
**Best for:** AWS-native organizations, high-performance BI on very large datasets, organizations that have DBA expertise for tuning, cost-sensitive workloads with predictable patterns.
|
||||
|
||||
### Databricks
|
||||
|
||||
**Architecture:** Lakehouse — combines data lake (cheap object storage, open formats) with warehouse (ACID, performance). Built on Apache Spark. Open table formats (Delta Lake, Iceberg, Hudi). Multi-cloud.
|
||||
|
||||
**Pricing model:** DBU-based (Databricks Units). Compute: $0.07-$2.00+/DBU depending on workload type (SQL, jobs, all-purpose) and tier. Storage charged separately by cloud provider (S3, GCS, ADLS). Serverless SQL warehouses available.
|
||||
|
||||
**Scaling:** Elastic auto-scaling for Spark clusters. Photon engine for accelerated SQL. Serverless SQL for BI workloads. Delta Sharing for cross-platform data access.
|
||||
|
||||
**Performance strengths:**
|
||||
- ML/AI workloads alongside SQL analytics from same data
|
||||
- Unity Catalog for unified governance across data + models
|
||||
- Delta Lake for ACID transactions on data lakes
|
||||
- Photon engine for performant SQL
|
||||
- Model-agnostic approach (train any framework)
|
||||
|
||||
**Weaknesses:**
|
||||
- Higher operational complexity than managed warehouses
|
||||
- Requires Spark/Scala/Python skills beyond SQL
|
||||
- Unity Catalog lock-in for governance (governance features not fully present in the open-source version)
|
||||
- Cost tracking complex (DBU model + separate cloud infra costs)
|
||||
- Cross-cloud management complex
|
||||
|
||||
**Vendor lock-in vectors:** Unity Catalog governance (open-source version is less capable), DBU pricing model makes total cost hard to compare, Delta Lake is open-source but Databricks-specific optimizations create gravity. Phoenix engine is proprietary.
|
||||
|
||||
**Best for:** Organizations with data science/ML workloads alongside analytics, teams with Spark/Python engineering skills, data platform teams wanting unified governance, anyone needing open table formats and avoiding proprietary storage.
|
||||
|
||||
## Side-by-Side: Decision Matrix
|
||||
|
||||
| Factor | Snowflake | BigQuery | Redshift | Databricks |
|
||||
|---|---|---|---|---|
|
||||
| **Ops burden** | Low | Lowest | Medium | Medium-High |
|
||||
| **SQL-first** | Yes | Yes | Yes | Yes (Photon) |
|
||||
| **ML/AI native** | Cortex AI | Vertex AI | Redshift ML | Native (Spark) |
|
||||
| **Open formats** | Iceberg (2025) | No native | Parquet/Iceberg | Delta/Iceberg/Hudi |
|
||||
| **Multi-cloud** | Yes (3 clouds) | GCP only | AWS only | Yes (3 clouds) |
|
||||
| **Max concurrency** | Very high (multi-cluster) | High (slot pool) | Medium (WLM) | High (auto-scaling) |
|
||||
| **Data sharing** | Native (marketplace) | Limited (authorized views) | Limited (datashare) | Delta Sharing |
|
||||
| **Cost predictability** | Fair | Poor (slot/CUD helps) | Good (reserved) | Poor |
|
||||
|
||||
## Cost Warning Signs
|
||||
|
||||
**Snowflake:** Check for forgotten serverless features (materialized views, Search Optimization), Cortex AI line items, Time Travel on heavy-update tables, oversized warehouses.
|
||||
|
||||
**BigQuery:** Watch for runaway queries on large tables, no slot reservations for mixed workloads, multi-region transfer fees.
|
||||
|
||||
**Redshift:** Concurrency scaling costs, Spectrum data scan charges, resize downtime for provisioned clusters.
|
||||
|
||||
**Databricks:** DBU costs + separate cloud infra costs double the bill, all-purpose vs jobs vs SQL pricing tiers matter hugely, Photon vs non-Photon pricing difference.
|
||||
|
||||
## Sources
|
||||
|
||||
- Lakshmanan, "Snowflake or Databricks? BigQuery or Dataproc? Redshift or EMR?" (LinkedIn, 2022) — architecture philosophy framework
|
||||
- Eidosoft, "Cloud Data Warehouse Comparison 2026" — comprehensive platform comparison with AI capabilities
|
||||
- Improvado, "Snowflake Competitors in 2026" — decision rubric with migration guide
|
||||
- Reintech, "Snowflake vs BigQuery vs Redshift 2026" — practitioner comparison
|
||||
- Flexera, "Snowflake vs BigQuery: 7 Critical Factors" (2026)
|
||||
- Definite, "Snowflake Pricing 2026" — detailed cost model analysis
|
||||
@@ -0,0 +1,136 @@
|
||||
# Compliance by Framework — Data Architecture Implications
|
||||
|
||||
What major regulatory frameworks require from a data architecture perspective.
|
||||
This is not legal advice — it's a guide to the **design implications** each framework creates.
|
||||
|
||||
## GDPR (General Data Protection Regulation) — EU
|
||||
|
||||
**Scope:** Personal data of EU residents, regardless of where the organization operates.
|
||||
|
||||
**Key architectural implications:**
|
||||
|
||||
- **Data inventory and mapping** — you must know what personal data you hold, where it came from, and who you share it with. Requires: automated data discovery and classification, data lineage tracking, data catalog
|
||||
- **Right to erasure (Article 17)** — you must be able to delete an individual's data across all systems, including backups. Requires: cascading delete patterns, purging from archives, backup retention hygiene
|
||||
- **Data minimization (Article 5)** — collect only what you need. Requires: schema design discipline, separate PII from analytics data, column-level access controls
|
||||
- **Consent management** — track what data subjects consented to, and when. Requires: consent flag as metadata on customer records, integration between data platform and consent system
|
||||
- **Data Protection Impact Assessment (Article 35)** — document high-risk processing. Requires: process metadata, data flow documentation, classification lineage
|
||||
- **72-hour breach notification** — you must detect and report breaches. Requires: data access auditing, anomaly detection, automated alerting
|
||||
|
||||
**Design patterns:**
|
||||
- PII/PHI tagging at column level in the data catalog
|
||||
- Separate PII store from analytical data warehouse
|
||||
- Data masking at query time (not at rest) for analytics access
|
||||
- Immutable audit logs with retention policies
|
||||
- Pseudonymization as a transformation step (replace PII with tokens before analytics)
|
||||
|
||||
## HIPAA (Health Insurance Portability and Accountability Act) — US Healthcare
|
||||
|
||||
**Scope:** Protected Health Information (PHI) of US patients.
|
||||
|
||||
**Key architectural implications:**
|
||||
|
||||
- **PHI must be encrypted at rest and in transit** — applies to databases, backups, data lakes, and data warehouse
|
||||
- **Access controls (minimum necessary rule)** — RBAC at row and column level. Not just who can access a table, but who can see which rows and columns within it
|
||||
- **Audit trails** — who accessed what PHI, when, and from where. Requires: access logging, immutable audit storage
|
||||
- **Business Associate Agreements (BAAs)** — all vendors handling PHI must sign a BAA. Requires: vetting every tool in the data pipeline (warehouse, ETL tool, BI tool, catalog)
|
||||
- **Data retention** — PHI must be retained per state and federal requirements, then securely destroyed
|
||||
- **Breach notification** — 60 days to report breaches affecting 500+ individuals
|
||||
|
||||
**Design patterns:**
|
||||
- Column-level encryption for sensitive PHI fields (SSN, diagnosis codes)
|
||||
- Row-level security filters in the data warehouse (analyst sees only their department's patients)
|
||||
- Separate de-identified data marts for analytics (no PHI)
|
||||
- Logging pipeline that captures all queries against PHI tables
|
||||
- Automated PHI detection and classification at ingestion
|
||||
|
||||
## CCPA/CPRA (California Consumer Privacy Act) — US California
|
||||
|
||||
**Scope:** Personal information of California residents. Broader definition of "personal information" than GDPR.
|
||||
|
||||
**Key architectural implications:**
|
||||
|
||||
- **Right to know** — what data you've collected about a consumer. Requires: data inventory, consumer-level data mapping
|
||||
- **Right to delete** — similar to GDPR, with exceptions. Requires: purging consumer data across systems
|
||||
- **Right to opt out of sale/sharing** — treats "sharing for cross-context behavioral advertising" as a sale. Requires: data sharing classification, consent metadata propagation
|
||||
- **Non-discrimination** — can't penalize consumers for exercising rights. Requires: separation of operational data from consent data
|
||||
|
||||
**Design patterns:** Similar to GDPR, with additional attention to data sharing classification. Any data shared with third parties (advertising, analytics vendors) needs opt-out tracking.
|
||||
|
||||
## SOX (Sarbanes-Oxley Act) — US Public Companies
|
||||
|
||||
**Scope:** Financial reporting accuracy and controls for publicly traded US companies.
|
||||
|
||||
**Key architectural implications:**
|
||||
|
||||
- **Internal controls over financial reporting** — every data transformation that touches financial reports must be auditable and verifiable. Requires: immutable data lineage, version-controlled transformations
|
||||
- **Data retention** — financial records must be retained for 7+ years. Requires: archival data storage with retrieval capabilities
|
||||
- **Segregation of duties** — the person who enters data shouldn't be the person who approves the report. Requires: role-based access control, separation of data producer and data consumer roles
|
||||
- **Full audit trail** — every change to financial data must be logged with who, what, when
|
||||
|
||||
**Design patterns:**
|
||||
- Data Vault 2.0 (hub-link-satellite with load dates and record sources for every row)
|
||||
- Immutable data lake (write-once, never modify)
|
||||
- Version-controlled transformation code (dbt with git, CI/CD for data pipelines)
|
||||
- Reconciliation processes between source systems and reporting layer
|
||||
|
||||
## PCI DSS (Payment Card Industry Data Security Standard)
|
||||
|
||||
**Scope:** Any organization handling credit card data.
|
||||
|
||||
**Key architectural implications:**
|
||||
|
||||
- **Cardholder data must be encrypted at rest and in transit** — full encryption everywhere
|
||||
- **Access must be on a need-to-know basis** — strict RBAC, no shared credentials
|
||||
- **Tokenization** — replace PAN (Primary Account Number) with tokens in analytics environments
|
||||
- **Logging and monitoring** — all access to cardholder data must be logged
|
||||
- **Segmentation** — cardholder data environment must be network-separated from non-cardholder systems
|
||||
|
||||
**Design patterns:**
|
||||
- Tokenization at ingestion (PCI data never enters the analytics warehouse as raw card numbers)
|
||||
- Separate data environment for PCI data with stricter controls
|
||||
- Query-level monitoring for any access to token vault
|
||||
- Automated masking for any UI/reporting layer
|
||||
|
||||
## BCBS 239 (Basel Committee) — Banking
|
||||
|
||||
**Scope:** Risk data aggregation and reporting for global systemically important banks.
|
||||
|
||||
**Key architectural implications:**
|
||||
|
||||
- **Data lineage** — full traceability from source to regulatory report. Every field in a regulatory return must trace to a source system field.
|
||||
- **Data quality** — accuracy, completeness, timeliness must be demonstrated. Requires: data quality dashboards with formal metrics, automated quality checking
|
||||
- **Data dictionary** — common definitions across the organization. Requires: business glossary with formal definitions, data ownership
|
||||
- **Audit trail** — all data transformations must be auditable. Requires: version-controlled transformation, change data capture, immutable logs
|
||||
- **IT resilience** — data architecture must support stress testing and recovery scenarios
|
||||
|
||||
**Design patterns:**
|
||||
- Data Vault 2.0 (provides built-in auditability, source tracking, and flexibility for regulatory change)
|
||||
- Formal data quality framework with automated scorecards
|
||||
- Enterprise data catalog acting as the single source of definitions
|
||||
- Reconciliation processes between source data warehouse → data mart → regulatory report
|
||||
|
||||
## General Architecture Principles for Compliance
|
||||
|
||||
1. **Design for auditability from day one.** Retrofitting audit trails is 10x harder than building them in.
|
||||
2. **Separate sensitive data from analytics data.** Use de-identified data marts for BI; keep raw PII/PHI behind strict access controls.
|
||||
3. **Automate compliance where possible.** Manual compliance processes fail at scale. Automate PII detection, data classification, lineage capture, and quality monitoring.
|
||||
4. **Data retention is an architecture concern, not just a policy.** If you can't enforce retention at the storage layer, it won't happen consistently.
|
||||
5. **Access controls must be granular enough for minimum necessary access.** Table-level access is not sufficient for HIPAA or GDPR (need column and row level).
|
||||
6. **Data catalogs are compliance infrastructure.** Without knowing what data you have and where it lives, you cannot comply with any framework's inventory requirements.
|
||||
|
||||
## Industry-to-Framework Quick Reference
|
||||
|
||||
Use this table when someone asks "what compliance frameworks apply to us?"
|
||||
|
||||
| Industry | Likely Frameworks | Primary Architectural Impact |
|
||||
|---|---|---|
|
||||
| Healthcare (US) | HIPAA, GDPR (if EU patients), CCPA (if CA patients) | PHI encryption, row/column-level access controls, BAAs with every vendor, de-identified analytics marts |
|
||||
| Financial services (US public) | SOX, PCI DSS (if cards), BCBS 239 (if systemic bank) | Immutable audit trails, 7-year retention, Data Vault for lineage, segregation of duties |
|
||||
| Financial services (EU) | GDPR, PSD2, BCBS 239 | Consent management, data minimization, lineage for regulatory reporting |
|
||||
| E-commerce / retail | CCPA (if CA customers), GDPR (if EU customers), PCI DSS | PII classification, data sharing opt-out tracking, tokenization |
|
||||
| SaaS / technology | GDPR, CCPA, SOC 2 (voluntary) | Data inventory, right to erasure, data processing agreements |
|
||||
| Insurance | HIPAA (if health), SOX (if public), GDPR, CCPA | Multiple frameworks simultaneously — layered compliance architecture |
|
||||
| Government / defense | FedRAMP, ITAR, GDPR (if EU citizens) | Data residency controls, classification markings, strict access segmentation |
|
||||
| Any B2B (global) | GDPR (EU customer data applies extraterritorially) | PII discovery, data mapping, consent tracking — even if not EU-based |
|
||||
|
||||
**If the answer is "we're not sure":** Start with a data inventory and classification. You can't determine which frameworks apply until you know what data you hold, where it lives, and who your data subjects are. The data catalog isn't just compliance infrastructure — it's how you scope your compliance obligations.
|
||||
@@ -0,0 +1,119 @@
|
||||
# Architecture Discovery Framework
|
||||
|
||||
A structured approach to understanding a client's context before making recommendations. Use this when starting an architecture consulting session.
|
||||
|
||||
## Phase 1: Problem & Context
|
||||
|
||||
Establish what we're actually solving:
|
||||
|
||||
**Business:**
|
||||
- What business problem are you solving? What's the measurable outcome?
|
||||
- Who is asking for this? What's their pain point?
|
||||
- What happens if we do nothing?
|
||||
- What does success look like in 3 months, 12 months?
|
||||
|
||||
**Organizational:**
|
||||
- Who are the stakeholders? Who has decision authority?
|
||||
- What's the team structure — who builds, who operates, who consumes?
|
||||
- What's the current team capability (size, seniority, key skills)?
|
||||
- Is there executive sponsorship for cross-team changes?
|
||||
|
||||
**Temporal:**
|
||||
- What's the timeline? What's driving the urgency?
|
||||
- What's frozen (can't change) vs flexible?
|
||||
- Are there existing commitments (vendor contracts, hiring freezes, compliance deadlines)?
|
||||
|
||||
## Phase 2: Current State
|
||||
|
||||
Understand what exists today:
|
||||
|
||||
**Data Profile:**
|
||||
- What data sources exist? (Count, types, systems of record)
|
||||
- Current data volume, velocity, variety — and projected 18-month growth
|
||||
- Where is data quality acceptable vs problematic?
|
||||
- What's the schema evolution frequency of source systems?
|
||||
|
||||
**Architecture:**
|
||||
- Diagram the current data flow — source to consumption
|
||||
- What's the current stack? (warehouse, pipeline tooling, BI layer, catalog)
|
||||
- What's working well (don't fix what isn't broken)?
|
||||
- What's the biggest operational pain point?
|
||||
|
||||
**Consumption:**
|
||||
- Who consumes data and how? (analysts, data scientists, operational apps, ML models)
|
||||
- What are their actual latency, freshness, and accuracy requirements?
|
||||
- What do they complain about most?
|
||||
|
||||
**Governance:**
|
||||
- What data governance exists currently? (ownership, quality, lineage, access control)
|
||||
- What compliance/regulatory frameworks apply? (GDPR, HIPAA, SOX, CCPA, BCBS 239)
|
||||
- Are there existing data sharing agreements or contractual constraints?
|
||||
|
||||
**Cost:**
|
||||
- Current data infrastructure spend (compute, storage, tools, people)
|
||||
- Where is cost growing fastest?
|
||||
- Are there wasteful patterns (orphaned tables, unused pipelines, over-provisioned compute)?
|
||||
|
||||
## Phase 3: Constraints
|
||||
|
||||
Identify the hard boundaries:
|
||||
|
||||
**Non-negotiable:**
|
||||
- Regulatory requirements
|
||||
- Budget ceilings
|
||||
- Hiring timeline
|
||||
- Existing vendor contracts
|
||||
|
||||
**Tradeable:**
|
||||
- Time-to-market vs robustness
|
||||
- Feature scope vs timeline
|
||||
- Central control vs team autonomy
|
||||
- Managed service premium vs operational flexibility
|
||||
|
||||
**Unknowns to validate:**
|
||||
- What assumptions are we making that should be tested?
|
||||
- What POCs should we run before committing to an architecture?
|
||||
|
||||
## Phase 4: Recommendation Structure
|
||||
|
||||
Present findings as:
|
||||
|
||||
**Quick wins (this month)**
|
||||
High impact, low effort — do these immediately regardless of long-term direction.
|
||||
*(e.g., add data quality checks at ingestion, document current lineage, consolidate redundant pipelines)*
|
||||
|
||||
**Foundation (next quarter)**
|
||||
Medium-term investments that enable future phases.
|
||||
*(e.g., implement data catalog, establish governance board, standardize naming conventions)*
|
||||
|
||||
**Transformation (next 12 months)**
|
||||
Major architectural shifts that require sequence and dependency management.
|
||||
*(e.g., migrate from on-prem EDW to cloud, adopt Data Mesh, implement streaming platform)*
|
||||
|
||||
**Deferred / Not Yet**
|
||||
Things that were requested but don't make sense given the current context.
|
||||
*(e.g., "we should add real-time streaming" when batch is meeting all SLAs)*
|
||||
|
||||
## Phase 5: Risk Register
|
||||
|
||||
Flag these for every engagement:
|
||||
|
||||
- **Single point of failure** — one person who knows how X works
|
||||
- **Vendor lock-in trajectory** — growing dependence on a platform without migration plan
|
||||
- **Technical debt that compounds** — shortcuts taken now that make future changes harder
|
||||
- **Skills gap** — architecture assumes capabilities the team doesn't have
|
||||
- **Organizational dependency** — project requires another team's cooperation without their buy-in
|
||||
- **Compliance cliff** — approaching regulatory deadline that current architecture can't meet
|
||||
|
||||
## Consulting Session Flow
|
||||
|
||||
```
|
||||
1. "Tell me about the problem" → listen, don't prescribe
|
||||
2. "What have you tried?" → understand the history
|
||||
3. "What's the actual scale?" → quantify before designing
|
||||
4. "Draw me your current flow" → visualize the as-is
|
||||
5. "What hurts most?" → identify leverage points
|
||||
6. "What's the simplest thing that could work?" → avoid over-engineering
|
||||
7. "Here's what I recommend, and here's why..." → tradeoffs explicit
|
||||
8. "What did I miss?" → invite challenge
|
||||
```
|
||||
@@ -0,0 +1,101 @@
|
||||
# Data Governance Maturity Model
|
||||
|
||||
A staged framework for data governance, synthesized from DAMA-DMBOK, Atlan, Gartner, and DATAVERSITY sources. Use this when designing or assessing a governance program.
|
||||
|
||||
## The DAMA-DMBOK Framework
|
||||
|
||||
The Data Management Association's Data Management Body of Knowledge (DAMA-DMBOK) is the canonical reference. It organizes data management into **11 knowledge areas** with Data Governance at the center:
|
||||
|
||||
1. **Data Governance** (central) — planning, oversight, and control over data management
|
||||
2. **Data Architecture** — enterprise data models and data flow designs
|
||||
3. **Data Modeling & Design** — analysis, design, and implementation of data structures
|
||||
4. **Data Storage & Operations** — database and data warehouse management
|
||||
5. **Data Security** — privacy, confidentiality, access control
|
||||
6. **Data Integration & Interoperability** — acquisition, extraction, transformation, movement
|
||||
7. **Document & Content Management** — managing unstructured data
|
||||
8. **Reference & Master Data** — managing shared data entities
|
||||
9. **Data Warehousing & Business Intelligence** — analytical data management
|
||||
10. **Metadata Management** — data about data
|
||||
11. **Data Quality** — quality dimensions, measurement, improvement
|
||||
|
||||
DAMA-DMBOK is **not prescriptive** — it doesn't mandate specific tools or technologies. It's a vendor-neutral reference that organizations adapt to their context.
|
||||
|
||||
Other frameworks: **CMMI DMM** (Data Management Maturity) provides a formal assessment model. **ISO 8000** covers data quality standards. **COBIT** focuses on governance and control objectives for IT.
|
||||
|
||||
## Maturity Stages
|
||||
|
||||
Based on Atlan's 6-stage framework (extending the common 5-stage model):
|
||||
|
||||
### Level 0: Unaware / Ad Hoc
|
||||
- **What it looks like:** Governance is not a recognized concept. Data is managed by individual teams with no coordination. Data issues are discovered when reports don't match.
|
||||
- **Policies:** None
|
||||
- **Roles:** No data owners identified
|
||||
- **Tools:** Spreadsheets, shared drives
|
||||
- **Metrics:** None tracked
|
||||
- **Core challenge:** Making the case for investment by showing the business impact of data failures
|
||||
|
||||
### Level 1: Initial / Reactive
|
||||
- **What it looks like:** Basic awareness exists. Some teams document data ownership informally. Governance happens reactively in response to crises (audit findings, data breaches).
|
||||
- **Policies:** Sparse, undocumented, inconsistently applied
|
||||
- **Roles:** Occasional data stewards assigned per project
|
||||
- **Tools:** Basic data catalog (AWS Glue, manual spreadsheet), ad-hoc quality checks
|
||||
- **Metrics:** Percentage of datasets with an identified owner
|
||||
- **Core challenge:** Scaling beyond a single team; avoiding "governance as IT project" trap
|
||||
|
||||
### Level 2: Managed / Developing
|
||||
- **What it looks like:** Formal governance structures emerge — an early data governance council, documented policies for priority domains, a data catalog tool in production. Efforts still siloed and resource-constrained.
|
||||
- **Policies:** Documented for critical domains, version-controlled
|
||||
- **Roles:** Data stewards assigned per domain, data owners for critical data
|
||||
- **Tools:** Enterprise data catalog with automated metadata ingestion, basic column-level lineage, scheduled quality checks
|
||||
- **Metrics:** Catalog completeness (% of systems registered), data quality scores by domain
|
||||
- **Core challenge:** Securing ongoing budget and executive sponsorship
|
||||
|
||||
### Level 3: Defined / Standardized
|
||||
- **What it looks like:** Governance is consistent across the organization. A data governance council makes decisions. Policies cover all major domains. Roles are clearly defined. Data quality standards are enterprise-wide.
|
||||
- **Policies:** Enterprise-wide, reviewed quarterly, enforced through tools
|
||||
- **Roles:** Data Governance Council (monthly), domain stewards, data custodians, CDO
|
||||
- **Tools:** Active metadata platform, cross-system column-level lineage, automated data quality monitoring, stewardship workflows
|
||||
- **Metrics:** Data trust scores, policy compliance rate, time to resolve data incidents
|
||||
- **Core challenge:** Shifting from compliance focus to cultural adoption and proactive risk management
|
||||
|
||||
### Level 4: Managed / Integrated
|
||||
- **What it looks like:** Governance is embedded into daily workflows. Quality monitoring is continuous and automated. Access adapts based on context. Governance metrics connect directly to business performance.
|
||||
- **Policies:** Automated policy enforcement (data masking applied at query time, automated retention schedules)
|
||||
- **Roles:** CDO with embedded governance liaisons in each department; stewardship is a recognized role with dedicated time
|
||||
- **Tools:** Active metadata management, real-time data quality monitoring with alerts, automated lineage, programmatic RBAC/ABAC
|
||||
- **Metrics:** Data utilization (% of data actively used), reduction in data incidents, cost of poor data quality
|
||||
- **Core challenge:** Integrating governance signals seamlessly into tools so teams can act on them without friction
|
||||
|
||||
### Level 5: Optimized / Transformative
|
||||
- **What it looks like:** Governance is a strategic capability. AI enhances monitoring and policy recommendations. Automation enforces controls without manual effort. Governance is part of organizational culture.
|
||||
- **Policies:** AI-suggested policies based on usage patterns, self-healing quality rules
|
||||
- **Roles:** Governance is everyone's responsibility; AI handles routine oversight. CDO role shifts to strategic innovation.
|
||||
- **Tools:** AI-driven auto-classification, anomaly detection, automated stewardship via workflow bots, self-service governance
|
||||
- **Metrics:** AI governance accuracy, business impact of data quality improvements (ROI), data velocity (time from ingestion to trusted consumption)
|
||||
- **Core challenge:** Sustaining innovation and keeping governance aligned with fast-changing business needs
|
||||
|
||||
## Practical Progression
|
||||
|
||||
Most organizations are at Level 1-2. Here's how to advance:
|
||||
|
||||
**Level 0 → 1:** Start with one critical data domain. Identify the owner. Create a simple data dictionary. Show one success story — "we fixed the revenue report that never matched."
|
||||
|
||||
**Level 1 → 2:** Deploy a data catalog. Automate metadata ingestion from your warehouse. Assign stewards for priority domains. Establish a data quality dashboard for the most-used datasets.
|
||||
|
||||
**Level 2 → 3:** Formalize the governance council. Roll out policies enterprise-wide. Implement column-level lineage for critical reports. Set data quality SLAs and publish them.
|
||||
|
||||
**Level 3 → 4:** Embed governance into CI/CD pipelines. Automate PII classification and masking. Link governance metrics to business KPIs (not just data KPIs).
|
||||
|
||||
**Level 4 → 5:** Deploy AI-driven policy recommendations and anomaly detection. Make data stewardship frictionless through automation. Measure governance effectiveness by business outcomes.
|
||||
|
||||
## Key Principle
|
||||
|
||||
**Governance maturity is not about more rules — it's about making good data practices the path of least resistance.** The goal at every stage is to reduce friction for data consumers while increasing trust. If governance makes people's jobs harder without a visible benefit, it will fail regardless of maturity level.
|
||||
|
||||
## Sources
|
||||
|
||||
- DAMA International, DAMA-DMBOK 2.0 — the canonical data management body of knowledge
|
||||
- Atlan, "How to Choose a Data Governance Maturity Model" (2026) — 6-stage framework
|
||||
- DATAVERSITY, "Data Governance Maturity Model Guide" (2026) — assessment methodology
|
||||
- Springer, "Data Governance Frameworks: Models and Best Practices" (2024) — comparative analysis of DAMA, CMMI DMM, ISO 8000
|
||||
- Gartner, "80% of D&A Governance Initiatives Will Fail by 2027" (2024)
|
||||
@@ -0,0 +1,133 @@
|
||||
# Vendor Evaluation Framework
|
||||
|
||||
Structured comparison criteria for common data tool categories. Based on Atlan's buyer's guide, Improvado's enterprise data management analysis, and practitioner comparisons from Promethium, Basedash, and LinkedIn.
|
||||
|
||||
## Data Catalogs
|
||||
|
||||
### Decision Tree — Which Catalog Fits?
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Q1["What kind of team are you?"] --> Q1a{Engineering-led<br>or Governance-led?}
|
||||
Q1a -->|Engineering-led,<br>self-hosted| Q2["Do you have engineering<br>time to maintain it?"]
|
||||
Q1a -->|Governance-led,<br>regulated| Q5["What's your budget?"]
|
||||
Q1a -->|Modern data stack<br>Snowflake/dbt/BQ| A[Atlan]
|
||||
|
||||
Q2 -->|Yes → engineer it| B[DataHub or OpenMetadata]
|
||||
Q2 -->|No → buy it| C[Secoda or Atlan]
|
||||
|
||||
Q5 -->|$100k+/year| D[Collibra]
|
||||
Q5 -->|$25k-$100k| E[OvalEdge]
|
||||
Q5 -->|Minimal budget| F[DataHub OSS]
|
||||
|
||||
style A fill:#e3f2fd,stroke:#1565c0
|
||||
style D fill:#fce4ec,stroke:#c62828
|
||||
style B fill:#fff3e0,stroke:#e65100
|
||||
```
|
||||
|
||||
**Staffing warning labels:**
|
||||
- **Collibra / Informatica** → needs dedicated data stewards or becomes shelfware
|
||||
- **Alation** → needs 15+ active catalogers for ROI
|
||||
- **DataHub / OpenMetadata** → needs engineering time to deploy and maintain
|
||||
- **Atlan / Secoda** → lower staffing bar, designed to reduce curation effort
|
||||
|
||||
### Quick Comparison Table
|
||||
|
||||
| Tool | Best For | G2 Rating | Deploy Time | Starting Price | Staffing Required |
|
||||
|---|---|---|---|---|---|
|
||||
| **Atlan** | Modern data stacks (Snowflake, dbt, Databricks) | 4.5/5 | 4-6 weeks | Custom enterprise | Low — active metadata reduces curation |
|
||||
| **Alation** | Analytics-first orgs; mixed legacy/modern | 4.4/5 | 6-12 weeks | Custom enterprise | 15+ active catalogers recommended |
|
||||
| **Collibra** | Regulated enterprises, governance-led | 4.2/5 | 3-9 months | $100k+/year | Dedicated data stewards needed |
|
||||
| **DataHub (LinkedIn)** | API-first, engineering teams, self-hosted | N/A | Self-hosted | Free + infra cost | Engineering team to deploy |
|
||||
| **OpenMetadata** | Broad connectors, engineers + analysts | N/A | Self-hosted | Free + managed plan | Engineering team to deploy |
|
||||
| **Microsoft Purview** | Azure-first orgs | N/A | Days to weeks | Azure consumption | Low if Azure-native |
|
||||
| **Informatica IDMC** | Multi-cloud, 600+ integrations | 4.2/5 | 6-9 months | Custom enterprise | Large governance team |
|
||||
| **Secoda** | Fast-growing modern-stack teams (5-50 users) | 4.5/5 | 1-2 weeks | ~$500/month | Minimal |
|
||||
| **Apache Atlas** | Hadoop-centric platforms | N/A | Self-hosted | Free | Hadoop ops team |
|
||||
| **OvalEdge** | Mid-market; $25k-$100k budget | 4.9/5 | 4-8 weeks | $25k-$100k/year | Moderate |
|
||||
|
||||
### Key Evaluation Dimensions
|
||||
|
||||
**1. Architecture**
|
||||
- Is it API-first and extendable? (DataHub excels here)
|
||||
- Does it use active metadata (query-parsed, continuously updated) vs passive (scheduled crawls)?
|
||||
- Is the metadata layer open or proprietary?
|
||||
|
||||
**2. Lineage Depth**
|
||||
- Column-level granularity (not just table-level)?
|
||||
- Cross-platform lineage spanning dbt, Airflow, Spark, and BI tools?
|
||||
- Automated vs manual lineage capture?
|
||||
|
||||
**3. Deployment Speed**
|
||||
- Does it deploy in weeks or months? Real customer timelines (not vendor estimates)
|
||||
- Self-hosted vs SaaS vs hybrid?
|
||||
|
||||
**4. Staffing Requirements**
|
||||
- Some tools (Collibra) require dedicated stewards to get ROI
|
||||
- Others (Atlan, Secoda) are designed to reduce curation burden through active metadata
|
||||
- Open-source tools (DataHub, OpenMetadata) need engineering investment to operationalize
|
||||
|
||||
**5. Pricing Model**
|
||||
- Per-user? Per-data-asset? Consumption-based? Enterprise contract?
|
||||
- Hidden costs: professional services ($80k-$200k), custom connectors ($15k-$50k each), cloud egress
|
||||
|
||||
**6. Evaluation Questions by Profile**
|
||||
|
||||
*For modern data stack (Snowflake/dbt/BigQuery/Databricks):*
|
||||
- Does the catalog natively parse dbt manifest files for lineage?
|
||||
- Can it ingest from both warehouse AND transformation tool?
|
||||
- How fresh is the metadata — real-time or batch?
|
||||
|
||||
*For regulated enterprise:*
|
||||
- Does it support RBAC/ABAC at column level?
|
||||
- Can it automate PII classification across all sources?
|
||||
- Does it provide compliance audit trail export?
|
||||
|
||||
*For engineering-first team:*
|
||||
- Is there a REST API or GraphQL endpoint for programmatic access?
|
||||
- Can we build custom connectors?
|
||||
- Is the metadata model extensible?
|
||||
|
||||
### Staffing Failure Thresholds
|
||||
- **Collibra / Informatica:** Needs dedicated data stewards; without them, tools become shelfware
|
||||
- **Alation:** 15+ active catalogers needed for ROI; under-resourced teams underutilize it
|
||||
- **DataHub / OpenMetadata:** Requires engineering time for setup and maintenance; budget headcount, not just licensing
|
||||
- **Atlan / Secoda:** Lower staffing bar — designed to reduce curation effort through automation
|
||||
|
||||
## ETL/ELT & Data Integration
|
||||
|
||||
| Tool | Best For | Pricing | Strength |
|
||||
|---|---|---|---|
|
||||
| **Fivetran** | Managed ELT, broad connector library | Usage-based ($0.25-$1.00+/MAR) | Zero-maintenance connectors, 500+ sources |
|
||||
| **Airbyte** | Open-source ELT, custom connector needs | Free (OSS) + Cloud plans | 350+ connectors, open protocol |
|
||||
| **dbt** | Transformation-as-code | Free core + Cloud ($100-$$$$) | The standard for analytics engineering |
|
||||
| **Talend** | Traditional ETL with complex transformations | Per-core licensing | Broad on-prem connector support |
|
||||
| **Informatica** | Enterprise data integration | Custom enterprise ($500k+) | 600+ certified connectors, mature governance |
|
||||
|
||||
### Key Questions
|
||||
- Do you need managed or self-hosted? (Fivetran vs Airbyte)
|
||||
- Is the primary need ingestion (move data) or transformation (shape data)? (Fivetran/Airbyte vs dbt)
|
||||
- What's your source system diversity? (narrow = cheaper, broad = need broad connector coverage)
|
||||
- What's your latency requirement? (batch ELT vs real-time CDC)
|
||||
|
||||
## Orchestration
|
||||
|
||||
| Tool | Best For | Language | Strength | Weakness |
|
||||
|---|---|---|---|---|
|
||||
| **Airflow** | Most common, broadest ecosystem | Python (DAGs) | Largest community, 1000+ providers | Complex, not idempotent by default, scheduler bottlenecks |
|
||||
| **Dagster** | Developer experience, testing | Python (software-defined assets) | Better testing, asset-centric, type system | Smaller community, fewer integrations |
|
||||
| **Prefect** | Cloud-native, serverless | Python (decorators) | Automatic retries, built-in observability | Fewer community providers than Airflow |
|
||||
| **Mage** | Data platform teams, simple syntax | Python, SQL, R, YAML | Developer-friendly, built-in data integration | Newest, smallest ecosystem |
|
||||
|
||||
### Key Questions
|
||||
- How important is community and ecosystem breadth? (Airflow)
|
||||
- Do you want code-first or declarative?
|
||||
- Is testing and local development a priority? (Dagster)
|
||||
- Do you need serverless execution or run your own infra?
|
||||
|
||||
## Source References
|
||||
- Atlan, "16 Best Data Catalog Tools in 2026: A Complete Buyer's Guide"
|
||||
- Improvado, "15 Best Enterprise Data Management Tools for 2026"
|
||||
- Promethium, "Data Governance Tools Comparison: Collibra vs Alation vs Atlan vs Purview" (2026)
|
||||
- LinkedIn, "Top 5 Data Governance Tools Compared"
|
||||
- Basedash, "Best Data Integration Tools Compared 2026"
|
||||
@@ -0,0 +1,193 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Governance Maturity Assessment — evaluate your data governance program.
|
||||
|
||||
Usage:
|
||||
python3 scripts/governance-assessment.py
|
||||
python3 scripts/governance-assessment.py --json
|
||||
|
||||
Answers 15 scored questions across 5 dimensions. Produces a maturity level,
|
||||
dimension scores, and prioritized recommendations. Outputs human-readable
|
||||
text by default, or JSON with --json for machine consumption.
|
||||
"""
|
||||
|
||||
import json
|
||||
import sys
|
||||
import textwrap
|
||||
|
||||
# === Scoring ===
|
||||
|
||||
DIMENSIONS = {
|
||||
"policy": "Policies & Standards",
|
||||
"roles": "Roles & Ownership",
|
||||
"tools": "Tooling & Metadata",
|
||||
"quality": "Data Quality",
|
||||
"culture": "Culture & Adoption",
|
||||
}
|
||||
|
||||
QUESTIONS = [
|
||||
# (dimension, question, low_label, high_label)
|
||||
("policy", "How are data governance policies documented?",
|
||||
"No formal policies", "Enterprise-wide policies, reviewed quarterly with automated enforcement"),
|
||||
("policy", "How consistently are policies followed across teams?",
|
||||
"Ad hoc, varies by team", "Automated enforcement with audit trails"),
|
||||
("policy", "How do you handle regulatory compliance requirements?",
|
||||
"Reactively when audited", "Proactive, automated compliance checks embedded in pipelines"),
|
||||
|
||||
("roles", "Who owns data quality and definitions?",
|
||||
"No clear ownership", "Dedicated data stewards per domain with formal charters"),
|
||||
("roles", "How is data ownership assigned?",
|
||||
"No owners identified", "Every dataset has a documented owner with performance goals"),
|
||||
("roles", "Is there a data governance council?",
|
||||
"No council exists", "Active council meeting monthly with executive sponsorship"),
|
||||
|
||||
("tools", "How do users discover and understand data?",
|
||||
"Ask colleagues or read source code", "Active metadata catalog with automated lineage and semantic search"),
|
||||
("tools", "How is data lineage tracked?",
|
||||
"Not tracked", "Automated column-level lineage across all systems"),
|
||||
("tools", "How do you manage metadata?",
|
||||
"Spreadsheets or shared docs", "Active metadata platform with automated ingestion and enrichment"),
|
||||
|
||||
("quality", "How do you measure data quality?",
|
||||
"Not measured systematically", "Automated quality dashboards with SLAs per dataset"),
|
||||
("quality", "How are data quality issues detected and resolved?",
|
||||
"Found by users during analysis", "Automated monitoring with alerts and tiered SLAs"),
|
||||
("quality", "How do you handle data quality at ingestion?",
|
||||
"No validation at entry", "Automated validation rules, schema enforcement, and anomaly detection"),
|
||||
|
||||
("culture", "How do teams perceive data governance?",
|
||||
"As a bottleneck or blocker", "As an enabler — governance makes data easier to use"),
|
||||
("culture", "How is governance funded and resourced?",
|
||||
"Project-based, inconsistent", "Dedicated budget and headcount with executive sponsorship"),
|
||||
("culture", "How does governance affect decision-making velocity?",
|
||||
"Slows teams down", "Faster decisions because trusted data is easier to find and use"),
|
||||
]
|
||||
|
||||
|
||||
def score_response(response):
|
||||
"""Convert 1-5 response to a score."""
|
||||
try:
|
||||
val = int(response)
|
||||
if 1 <= val <= 5:
|
||||
return val
|
||||
except (ValueError, TypeError):
|
||||
pass
|
||||
return None
|
||||
|
||||
|
||||
def text_prompt(dimension, question, low_label, high_label):
|
||||
"""Present a question and get a 1-5 response."""
|
||||
print(f"\n--- {DIMENSIONS[dimension]} ---")
|
||||
print(f"Q: {question}")
|
||||
print(f" 1 = {low_label}")
|
||||
print(f" 5 = {high_label}")
|
||||
while True:
|
||||
try:
|
||||
resp = input(" Score (1-5): ").strip()
|
||||
score = score_response(resp)
|
||||
if score:
|
||||
return score
|
||||
print(" Please enter a number between 1 and 5.")
|
||||
except (EOFError, KeyboardInterrupt):
|
||||
print("\n Assessment cancelled.")
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
def json_prompt():
|
||||
"""Return placeholder scores for JSON mode (no interactivity)."""
|
||||
print("Run without --json for interactive assessment.", file=sys.stderr)
|
||||
sys.exit(0)
|
||||
|
||||
|
||||
def calculate_maturity(avg_score):
|
||||
"""Map average score to maturity level."""
|
||||
if avg_score < 1.5:
|
||||
return (0, "Unaware — no governance concept exists")
|
||||
elif avg_score < 2.5:
|
||||
return (1, "Initial — reactive, ad hoc, crisis-driven")
|
||||
elif avg_score < 3.5:
|
||||
return (2, "Managed — basic structures, siloed, early tools")
|
||||
elif avg_score < 4.0:
|
||||
return (3, "Defined — enterprise-wide, consistent, council-driven")
|
||||
elif avg_score < 4.5:
|
||||
return (4, "Integrated — embedded in workflows, automated enforcement")
|
||||
else:
|
||||
return (5, "Optimized — AI-driven, strategic, culture-embedded")
|
||||
|
||||
|
||||
def generate_recommendations(scores):
|
||||
"""Generate prioritized recommendations based on lowest scores."""
|
||||
dim_avgs = {}
|
||||
for dim in DIMENSIONS:
|
||||
dim_scores = [s for d, s in scores if d == dim]
|
||||
dim_avgs[dim] = sum(dim_scores) / len(dim_scores) if dim_scores else 0
|
||||
|
||||
sorted_dims = sorted(dim_avgs.items(), key=lambda x: x[1])
|
||||
|
||||
recommendations = []
|
||||
for dim, avg in sorted_dims:
|
||||
if avg < 2.0:
|
||||
recommendations.append(f"[HIGH] {DIMENSIONS[dim]}: avg {avg:.1f}/5 — Start with basic {dim.replace('_', ' ')}. See references/governance-maturity.md Level 0→1.")
|
||||
elif avg < 3.0:
|
||||
recommendations.append(f"[MEDIUM] {DIMENSIONS[dim]}: avg {avg:.1f}/5 — Formalize {dim.replace('_', ' ')}. See references/governance-maturity.md Level 2→3.")
|
||||
elif avg < 4.0:
|
||||
recommendations.append(f"[LOW] {DIMENSIONS[dim]}: avg {avg:.1f}/5 — Improve {dim.replace('_', ' ')} consistency. See references/governance-maturity.md Level 3→4.")
|
||||
else:
|
||||
recommendations.append(f"[MONITOR] {DIMENSIONS[dim]}: avg {avg:.1f}/5 — Maintain. See references/governance-maturity.md Level 4→5.")
|
||||
|
||||
return recommendations
|
||||
|
||||
|
||||
def main():
|
||||
json_mode = "--json" in sys.argv
|
||||
|
||||
if json_mode:
|
||||
print(json.dumps({
|
||||
"error": "Run interactively without --json for assessment",
|
||||
"usage": "python3 scripts/governance-assessment.py"
|
||||
}, indent=2))
|
||||
return
|
||||
|
||||
print("=" * 60)
|
||||
print(" Data Governance Maturity Assessment")
|
||||
print("=" * 60)
|
||||
print(" Rate each dimension from 1 (worst) to 5 (best).")
|
||||
print(" Be honest — the assessment is for you, not anyone else.")
|
||||
print("=" * 60)
|
||||
|
||||
scores = []
|
||||
for dim, question, low, high in QUESTIONS:
|
||||
score = text_prompt(dim, question, low, high)
|
||||
scores.append((dim, score))
|
||||
|
||||
# Calculate results
|
||||
dim_avgs = {}
|
||||
for dim in DIMENSIONS:
|
||||
dim_scores = [s for d, s in scores if d == dim]
|
||||
dim_avgs[dim] = sum(dim_scores) / len(dim_scores) if dim_scores else 0
|
||||
|
||||
overall = sum(dim_avgs.values()) / len(dim_avgs)
|
||||
level, level_label = calculate_maturity(overall)
|
||||
|
||||
print("\n" + "=" * 60)
|
||||
print(" RESULTS")
|
||||
print("=" * 60)
|
||||
print(f"\n Overall Maturity: Level {level} — {level_label}")
|
||||
print(f" Average Score: {overall:.1f}/5\n")
|
||||
print(" Dimension Scores:")
|
||||
for dim in DIMENSIONS:
|
||||
bar = "█" * int(dim_avgs[dim]) + "░" * (5 - int(dim_avgs[dim]))
|
||||
print(f" {DIMENSIONS[dim]:25s} {bar} {dim_avgs[dim]:.1f}/5")
|
||||
|
||||
print("\n Recommendations:")
|
||||
recs = generate_recommendations(scores)
|
||||
for r in recs:
|
||||
print(f" {r}")
|
||||
|
||||
print("\n For detailed stage descriptions, see:")
|
||||
print(" references/governance-maturity.md")
|
||||
print("=" * 60)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,63 @@
|
||||
# Architecture Decision Record
|
||||
|
||||
Use this template to capture architecture decisions during consulting sessions. Fill in each section as the decision is made. Save the result to the team's ADR directory.
|
||||
|
||||
```markdown
|
||||
# ADR-[NUMBER]: [Title]
|
||||
|
||||
## Status
|
||||
[Proposed | Accepted | Deprecated | Superseded]
|
||||
|
||||
## Context
|
||||
What's the problem we're solving? What constraints, assumptions, and forces are at play?
|
||||
|
||||
* Business driver:
|
||||
* Technical context:
|
||||
* Constraints (budget, timeline, team capability, compliance):
|
||||
* Key stakeholders:
|
||||
|
||||
## Options Considered
|
||||
|
||||
### Option A: [Name]
|
||||
* How it works:
|
||||
* Pros:
|
||||
* Cons:
|
||||
* Migration cost:
|
||||
* Operational complexity:
|
||||
|
||||
### Option B: [Name]
|
||||
* How it works:
|
||||
* Pros:
|
||||
* Cons:
|
||||
* Migration cost:
|
||||
* Operational complexity:
|
||||
|
||||
*(Add more options as needed)*
|
||||
|
||||
## Decision
|
||||
**Chosen option:** [A / B / C]
|
||||
|
||||
**Rationale:**
|
||||
1. [Primary reason]
|
||||
2. [Secondary reason]
|
||||
3. [Tiebreaker]
|
||||
|
||||
**Consequences:**
|
||||
* Positive:
|
||||
* Negative / tradeoffs accepted:
|
||||
* What we're explicitly not doing:
|
||||
|
||||
## Verification
|
||||
How will we know this was the right decision?
|
||||
|
||||
* [Metric or milestone]
|
||||
* [Review date]
|
||||
|
||||
## References
|
||||
* [Link to relevant reference file in this skill]
|
||||
* [Link to vendor docs, case studies, or external sources]
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
When I recommend a specific architecture or tool during a consulting session, I'll offer to produce an ADR. If you want one, say "capture that as an ADR" and I'll fill in the template from our conversation.
|
||||
Reference in New Issue
Block a user