mirror of https://github.com/magnus919/agent-skills.git synced 2026-09-15 05:26:28 +03:00

Files

T

Magnus HedemarkandGitHub c7c4d3b74f Port 11 methodology skills from hermes-profiles (#69 )

Engineering: backend-engineering, frontend-engineering, data-engineering,
ml-engineering, platform-engineering, qa-methodology

Executive: go-to-market, legal-strategy, operational-design, org-design,
product-strategy

ml-engineering: added missing training-infrastructure.md reference
qa-methodology: added test-data-management, performance-testing,
security-testing references

All frontmatter converted to agent-skills convention.
Source: https://github.com/magnus919/hermes-profiles

2026-07-21 00:58:26 -04:00

3.9 KiB

Raw Blame History

Training Infrastructure

GPU Selection

GPU	VRAM	Best For	Notes
RTX 4090	24 GB	LoRA/QLoRA 7B–13B, inference	Consumer, no NVLink
RTX 5090	32 GB	LoRA 13B–30B, QLoRA 70B	Consumer, no NVLink
A6000	48 GB	Full fine-tune 7B, LoRA 30B–70B	Prosumer, NVLink pair
A100 80GB	80 GB	Full fine-tune 13B–30B, multi-GPU	Datacenter, NVLink
H100 80GB	80 GB	Full fine-tune 30B–70B, RLHF	Datacenter, NVLink, FP8
H200 141GB	141 GB	Full fine-tune 70B+, long context	Datacenter, NVLink

VRAM Budgeting

Rule of thumb for training memory (mixed precision, AdamW):

VRAM ≈ params × (2 + 2 + 4 + 4) bytes  [weights + grads + optimizer states]
     + activation memory (batch-dependent)

Technique	VRAM multiplier	Example: 7B model
Full fine-tune (FP16 + Adam)	~16× params	~112 GB
LoRA (rank 64)	~2.5× params	~18 GB
QLoRA (4-bit + LoRA)	~0.8× params	~6 GB
Inference only (FP16)	~2× params	~14 GB
Inference only (4-bit)	~0.5× params	~4 GB

Multi-GPU Training

Strategy	When	Framework
DataParallel (DP)	Single node, quick experiments	PyTorch native
DistributedDataParallel (DDP)	Single node, production training	`torchrun --nproc_per_node=N`
FSDP / DeepSpeed ZeRO	Model doesn't fit one GPU	`accelerate`, DeepSpeed config
Pipeline parallelism	Very large models (>70B)	DeepSpeed, Megatron-LM
Tensor parallelism	Latency-critical inference	vLLM, TensorRT-LLM

DDP Launch Pattern

torchrun --nproc_per_node=4 --master_port=29500 train.py \
  --model_name meta-llama/Llama-3-8B \
  --per_device_train_batch_size 4 \
  --gradient_accumulation_steps 8 \
  --bf16 true

Effective batch size = per_device × nproc × grad_accum = 4 × 4 × 8 = 128.

FSDP Config (accelerate)

# accelerate_config.yaml
compute_environment: LOCAL_MACHINE
distributed_type: FSDP
fsdp_config:
  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
  fsdp_sharding_strategy: FULL_SHARD  # ZeRO-3 equivalent
  fsdp_state_dict_type: SHARDED_STATE_DICT
  fsdp_cpu_ram_efficient_loading: true
mixed_precision: bf16
num_processes: 4

Cloud vs On-Prem Decision

Factor	Cloud (Lambda, RunPod, Vast)	On-Prem
Utilization < 30%	✅ Pay per hour	❌ Idle hardware
Utilization > 60%	❌ Expensive at scale	✅ Amortizes in ~8 months
Data sensitivity	❌ Data leaves premises	✅ Stays local
Burst capacity	✅ Scale to 8×H100 on demand	❌ Fixed ceiling
Ops burden	❌ Zero (managed)	✅ You maintain cooling, power, drivers
Experiment velocity	✅ Spin up, tear down	⚠️ Queue contention on shared cluster

Cloud Cost Reference (spot/on-demand, 2025)

GPU	On-demand $/hr	Spot $/hr
A100 80GB	$1.80–$2.50	$0.90–$1.40
H100 80GB	$3.50–$5.00	$2.00–$3.00
RTX 4090	$0.40–$0.70	$0.25–$0.45

Storage and Data Pipeline

Training data on NVMe or tmpfs — network storage stalls GPUs
Checkpoints to object storage (S3/GCS) or NAS — never only local disk
Use safetensors format — faster load, no pickle security risk
Pre-tokenize datasets for large corpora — tokenization at load time wastes GPU-hours

Monitoring Training

Metric	Healthy Range	Red Flag
GPU utilization	> 90%	< 70% = data pipeline bottleneck
GPU memory	Stable after warmup	Growing = leak (check grad accumulation)
Loss curve	Smooth decrease	Spikes = LR too high; plateau = converged or stuck
Grad norm	Stable or decreasing	Exploding = reduce LR or add clipping
Throughput (samples/sec)	Consistent	Degrading = thermal throttle or I/O

3.9 KiB Raw Blame History Unescape Escape