mirror of
https://github.com/magnus919/agent-skills.git
synced 2026-09-19 07:27:04 +03:00
Magnus HedemarkandGitHub
c7c4d3b74f
Engineering: backend-engineering, frontend-engineering, data-engineering, ml-engineering, platform-engineering, qa-methodology Executive: go-to-market, legal-strategy, operational-design, org-design, product-strategy ml-engineering: added missing training-infrastructure.md reference qa-methodology: added test-data-management, performance-testing, security-testing references All frontmatter converted to agent-skills convention. Source: https://github.com/magnus919/hermes-profiles
103 lines
3.9 KiB
Markdown
103 lines
3.9 KiB
Markdown
# Training Infrastructure
|
||
|
||
## GPU Selection
|
||
|
||
| GPU | VRAM | Best For | Notes |
|
||
|-----|------|----------|-------|
|
||
| RTX 4090 | 24 GB | LoRA/QLoRA 7B–13B, inference | Consumer, no NVLink |
|
||
| RTX 5090 | 32 GB | LoRA 13B–30B, QLoRA 70B | Consumer, no NVLink |
|
||
| A6000 | 48 GB | Full fine-tune 7B, LoRA 30B–70B | Prosumer, NVLink pair |
|
||
| A100 80GB | 80 GB | Full fine-tune 13B–30B, multi-GPU | Datacenter, NVLink |
|
||
| H100 80GB | 80 GB | Full fine-tune 30B–70B, RLHF | Datacenter, NVLink, FP8 |
|
||
| H200 141GB | 141 GB | Full fine-tune 70B+, long context | Datacenter, NVLink |
|
||
|
||
## VRAM Budgeting
|
||
|
||
Rule of thumb for training memory (mixed precision, AdamW):
|
||
|
||
```
|
||
VRAM ≈ params × (2 + 2 + 4 + 4) bytes [weights + grads + optimizer states]
|
||
+ activation memory (batch-dependent)
|
||
```
|
||
|
||
| Technique | VRAM multiplier | Example: 7B model |
|
||
|-----------|----------------|-------------------|
|
||
| Full fine-tune (FP16 + Adam) | ~16× params | ~112 GB |
|
||
| LoRA (rank 64) | ~2.5× params | ~18 GB |
|
||
| QLoRA (4-bit + LoRA) | ~0.8× params | ~6 GB |
|
||
| Inference only (FP16) | ~2× params | ~14 GB |
|
||
| Inference only (4-bit) | ~0.5× params | ~4 GB |
|
||
|
||
## Multi-GPU Training
|
||
|
||
| Strategy | When | Framework |
|
||
|----------|------|-----------|
|
||
| DataParallel (DP) | Single node, quick experiments | PyTorch native |
|
||
| DistributedDataParallel (DDP) | Single node, production training | `torchrun --nproc_per_node=N` |
|
||
| FSDP / DeepSpeed ZeRO | Model doesn't fit one GPU | `accelerate`, DeepSpeed config |
|
||
| Pipeline parallelism | Very large models (>70B) | DeepSpeed, Megatron-LM |
|
||
| Tensor parallelism | Latency-critical inference | vLLM, TensorRT-LLM |
|
||
|
||
### DDP Launch Pattern
|
||
|
||
```bash
|
||
torchrun --nproc_per_node=4 --master_port=29500 train.py \
|
||
--model_name meta-llama/Llama-3-8B \
|
||
--per_device_train_batch_size 4 \
|
||
--gradient_accumulation_steps 8 \
|
||
--bf16 true
|
||
```
|
||
|
||
Effective batch size = `per_device × nproc × grad_accum` = 4 × 4 × 8 = 128.
|
||
|
||
### FSDP Config (accelerate)
|
||
|
||
```yaml
|
||
# accelerate_config.yaml
|
||
compute_environment: LOCAL_MACHINE
|
||
distributed_type: FSDP
|
||
fsdp_config:
|
||
fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
|
||
fsdp_sharding_strategy: FULL_SHARD # ZeRO-3 equivalent
|
||
fsdp_state_dict_type: SHARDED_STATE_DICT
|
||
fsdp_cpu_ram_efficient_loading: true
|
||
mixed_precision: bf16
|
||
num_processes: 4
|
||
```
|
||
|
||
## Cloud vs On-Prem Decision
|
||
|
||
| Factor | Cloud (Lambda, RunPod, Vast) | On-Prem |
|
||
|--------|------------------------------|---------|
|
||
| Utilization < 30% | ✅ Pay per hour | ❌ Idle hardware |
|
||
| Utilization > 60% | ❌ Expensive at scale | ✅ Amortizes in ~8 months |
|
||
| Data sensitivity | ❌ Data leaves premises | ✅ Stays local |
|
||
| Burst capacity | ✅ Scale to 8×H100 on demand | ❌ Fixed ceiling |
|
||
| Ops burden | ❌ Zero (managed) | ✅ You maintain cooling, power, drivers |
|
||
| Experiment velocity | ✅ Spin up, tear down | ⚠️ Queue contention on shared cluster |
|
||
|
||
### Cloud Cost Reference (spot/on-demand, 2025)
|
||
|
||
| GPU | On-demand $/hr | Spot $/hr |
|
||
|-----|----------------|-----------|
|
||
| A100 80GB | $1.80–$2.50 | $0.90–$1.40 |
|
||
| H100 80GB | $3.50–$5.00 | $2.00–$3.00 |
|
||
| RTX 4090 | $0.40–$0.70 | $0.25–$0.45 |
|
||
|
||
## Storage and Data Pipeline
|
||
|
||
- Training data on NVMe or tmpfs — network storage stalls GPUs
|
||
- Checkpoints to object storage (S3/GCS) or NAS — never only local disk
|
||
- Use `safetensors` format — faster load, no pickle security risk
|
||
- Pre-tokenize datasets for large corpora — tokenization at load time wastes GPU-hours
|
||
|
||
## Monitoring Training
|
||
|
||
| Metric | Healthy Range | Red Flag |
|
||
|--------|--------------|----------|
|
||
| GPU utilization | > 90% | < 70% = data pipeline bottleneck |
|
||
| GPU memory | Stable after warmup | Growing = leak (check grad accumulation) |
|
||
| Loss curve | Smooth decrease | Spikes = LR too high; plateau = converged or stuck |
|
||
| Grad norm | Stable or decreasing | Exploding = reduce LR or add clipping |
|
||
| Throughput (samples/sec) | Consistent | Degrading = thermal throttle or I/O |
|