mirror of
https://github.com/magnus919/agent-skills.git
synced 2026-09-12 12:06:29 +03:00
Magnus HedemarkandGitHub
c7c4d3b74f
Engineering: backend-engineering, frontend-engineering, data-engineering, ml-engineering, platform-engineering, qa-methodology Executive: go-to-market, legal-strategy, operational-design, org-design, product-strategy ml-engineering: added missing training-infrastructure.md reference qa-methodology: added test-data-management, performance-testing, security-testing references All frontmatter converted to agent-skills convention. Source: https://github.com/magnus919/hermes-profiles
1.6 KiB
1.6 KiB
Vector Database Operations
Collection Lifecycle
| Phase | Activities |
|---|---|
| Design | Schema definition, dimension selection, distance metric (cosine, euclidean, dot), index type selection |
| Create | Collection provisioning, index creation, partition configuration, alias setup |
| Ingest | Batch loading, streaming ingestion, data validation, consistency verification |
| Maintain | Index rebuilding, compaction, collection health monitoring, performance tuning |
| Migrate | Dimension changes, index type changes, cluster migration, data reindexing |
| Decommission | Data archival, collection backup, alias reassignment, collection drop |
Index Type Selection
| Index Type | Best for | Tradeoffs |
|---|---|---|
| IVF_FLAT | Balanced accuracy/speed | Higher memory, good recall |
| HNSW | High-recall, large datasets | Higher memory, slower build |
| IVF_SQ8 | Memory-efficient | Lower recall than IVF_FLAT |
| FLAT | Exact search, small datasets | O(n) search, exact recall |
Migration Patterns
| Scenario | Approach |
|---|---|
| Dimension change | Create new collection with target dimension, dual-write during migration, batch reindex old data, swap alias |
| Index type change | Online index rebuild if supported, otherwise parallel collection with dual-write |
| Cluster migration | Backup → restore on target, validate row counts and sample queries, cut over via alias |
| Embedding model change | Full reindex: read source → generate new embeddings → write to new collection → verify → swap |