magnus919_agent-skills/data-architect/references/anti-patterns.md

Data Architecture Anti-Patterns Catalog

A catalog of named anti-patterns with symptoms, root causes, and remediations. Sources: Sunil Prakash (Enterprise Data Architecture), Kengie Ho (Common Enterprise Architectural Pitfalls), DataForest, and industry experience.

1. Silver Bullet Thinking

Symptom: Adopting Data Mesh, Data Fabric, or another pattern because it's trendy, without organizational readiness.

Root cause: Believing an architectural pattern alone solves what are fundamentally organizational problems (ownership, trust, skills).

Remediation: Before adopting Data Mesh, verify you have: domain teams willing to own data products, a platform team to build self-serve infra, and executive sponsorship for federated governance. If any is missing, start smaller.

2. SoR vs SSoT Confusion

Symptom: Business leaders point to a legacy 20-year-old billing system and say "That's our Single Source of Truth." Modern apps are forced to dumb down their data to fit the legacy system's narrow schema.

Root cause: Confusing a transactional System of Record (SoR) — where data is originally authored — with a Single Source of Truth (SSoT) — an aggregated, reconciled, governed view across the enterprise.

Remediation: Decouple the two domains. Let operational systems be localized SoRs for their domain. Build a dedicated layer (MDM platform, data lakehouse, or data mesh) above them that ingests from every SoR, resolves conflicts, applies quality rules, and serves as the true SSoT.

3. High-Fidelity / Lossy Sync Destruction

Symptom: A modern CRM with rich, structured data is synced bidirectionally with a legacy ERP. The ERP truncates names, flattens addresses, drops extra fields — then syncs that flattened data back to the CRM, permanently overwriting the high-fidelity records.

Root cause: Allowing a low-fidelity system to blindly update a high-fidelity system through lossy synchronization. Known formally as semantic heterogeneity, information loss in schema mapping, and destructive overwrites.

Remediation: Place an integration middleware or MDM platform at the system boundary with metadata-driven field-level survivorship. Apply hierarchical rules: "The ERP always wins financial fields, but the CRM always wins customer contact info." Build an Anti-Corruption Layer (from Domain-Driven Design) to protect modern systems from legacy constraints.

4. Fragmented Identity / "Customer 720"

Symptom: Customer logs in with email (System A), calls support as "Jane Doe" (System B). Without a shared primary key, the integration layer creates a duplicate record. Instead of Customer 360, you get "Customer 720."

Root cause: Relying on simple SQL JOINs or exact-match keys for entity resolution across systems that have no common identifier.

Remediation: Introduce an Identity Resolution Engine at the data ingestion layer. Use two-stage matching:

• Deterministic: exact matches on hashed email, SSN, phone
• Probabilistic: ML-based matching (Levenshtein distance, shared device IPs, behavioral patterns) with confidence scoring

Treat identity resolution as a foundational requirement, not a downstream analytics problem.

5. Schema Evolution Mismatch

Symptom: A modern microservices team splits FullName into FirstName/LastName. A downstream legacy app consuming the data stream crashes because it expects the old schema.

Root cause: Speed mismatch between agile schema evolution and rigid legacy consumers. Tightly coupled interfaces without mediation.

Remediation: Introduce an Anti-Corruption Layer (ACL) between modern and legacy systems. The ACL translates schemas at the boundary — the modern system evolves freely, the ACL provides backward-compatible views to legacy consumers.

6. The Ivory Tower

Symptom: Architecture team produces standards, roadmaps, and reference architectures in isolation, without engaging delivery teams or business stakeholders. Artifacts are technically correct but irrelevant to actual problems.

Root cause: Architecture practiced as a detached design discipline rather than an embedded consulting function.

Remediation: Embed architects with delivery teams. Architecture decisions should emerge from concrete problems, not abstract frameworks. The architect who writes a standard without understanding the team's actual constraints is building shelfware.

7. Tool-First Architecture

Symptom: Organization picks Snowflake (or Databricks, or Kafka) and then goes looking for problems to solve with it.

Root cause: Technology selection driven by vendor marketing, hiring availability, or executive preference rather than requirements analysis.

Remediation: Start with business requirements, data profiles, and consumer needs. Let the architecture drive the tool selection, not the other way around. Run POCs against real workloads before committing.

8. Governance as an Afterthought

Symptom: "We'll add governance later." Data proliferates uncontrolled for 18 months. When governance finally arrives, it requires a multi-quarter cleanup effort and meets resistance from teams accustomed to full autonomy.

Root cause: Treating governance as a phase rather than a design constraint.

Remediation: Establish minimum viable governance from day one: data classification, ownership tagging, basic lineage. Add rigor incrementally as the platform matures. Governance that grows with the system is adopted; governance imposed after the fact is rejected.

9. Premature Petabyte

Symptom: Designing for "petabyte scale" when the organization has 2TB of data. Complex, expensive architecture that delivers no marginal benefit for years.

Root cause: Architects designing for resumes rather than actual problems. Future-proofing without understanding the cost of complexity.

Remediation: Design for 10x growth, not 1000x. Use architectures that scale horizontally without requiring upfront complexity. The cloud warehouse patterns (Snowflake, BigQuery) handle growth without premature optimization.

10. Streaming Everything

Symptom: Kafka for every data movement, even daily batch reports. Streaming infrastructure cost exceeds the value of real-time data.

Root cause: Assuming "real-time is better" without quantifying the latency requirement. Conflating event-driven architecture with streaming infrastructure.

Remediation: Require a concrete latency SLA before introducing streaming. If the business can tolerate 15-minute latency, micro-batch is sufficient. If it can tolerate hourly, batch is fine. Keep streaming infrastructure for the subset of pipelines that genuinely need sub-second delivery.

11. Copy-Paste FAANG Architecture

Symptom: Replicating the data infrastructure patterns of Google, Netflix, or Uber without accounting for your organization's actual scale, budget, team capability, or problem complexity.

Root cause: Assuming that what works at hyperscale is optimal at any scale. FAANG patterns evolved to solve FAANG-scale problems — not general problems.

Remediation: Choose the simplest architecture that meets your actual requirements. Managed services (Snowflake, BigQuery, Fivetran, dbt Cloud) are often the right choice for teams of 5-50, even if they wouldn't scale to FAANG levels. Complexity should be earned, not inherited.

12. Design by Committee

Symptom: Architecture that tries to accommodate every stakeholder's preference and ends up satisfying no one. Compromised designs that mix incompatible patterns.

Root cause: Architecture by consensus rather than by clear ownership and decision rights.

Remediation: Assign clear decision authority for architecture decisions. Solicit input from stakeholders, but the architect makes the final call with documented rationale. ADRs (Architecture Decision Records) make this explicit: "We considered X, Y, Z. We chose X because... The tradeoffs are..."

13. Neglecting the Team

Symptom: Designing a technically elegant system that the team cannot operate, troubleshoot, or extend. High "bus factor" on the architecture itself.

Root cause: Designing for technical purity rather than organizational reality.

Remediation: Factor in team size, skill level, and operational maturity. A simple architecture operated well is better than a clever architecture nobody understands. Build runbooks, invest in training, and design for debuggability.