From dc08b67203f50d117e8bef896055e7fd27a87ff9 Mon Sep 17 00:00:00 2001 From: Magnus Hedemark Date: Sat, 23 May 2026 17:10:10 -0400 Subject: [PATCH] =?UTF-8?q?feat:=20add=20researched=20code=20integration?= =?UTF-8?q?=20references=20=E2=80=94=20PyTorch,=20sklearn,=20DS=20workflow?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three researched references validated against current API docs: - references/pytorch-integration.md: device management, training loops, AMP, torch.compile, transfer learning, LoRA, distillation, pruning, DDP, debugging (validated against PyTorch 2.12 docs) - references/sklearn-integration.md: pipelines, ColumnTransformer, model selection, ensembles, calibration, imbalanced data, custom estimators, feature selection (validated against sklearn 1.8.0 docs) - references/data-science-coding-workflow.md: project structure, config management, experiment logging (MLflow/TensorBoard/WandB), result serialization, reproducibility, data versioning, unit testing 66/66 validation tests passing. Closes #23 --- .../data-science-coding-workflow.md | 489 ++++++++++++++ .../references/pytorch-integration.md | 616 ++++++++++++++++++ .../references/sklearn-integration.md | 577 ++++++++++++++++ .../scripts/test_references_completeness.sh | 149 +++++ 4 files changed, 1831 insertions(+) create mode 100644 data-scientist/references/data-science-coding-workflow.md create mode 100644 data-scientist/references/pytorch-integration.md create mode 100644 data-scientist/references/sklearn-integration.md create mode 100644 data-scientist/scripts/test_references_completeness.sh diff --git a/data-scientist/references/data-science-coding-workflow.md b/data-scientist/references/data-science-coding-workflow.md new file mode 100644 index 0000000..25f6b42 --- /dev/null +++ b/data-scientist/references/data-science-coding-workflow.md @@ -0,0 +1,489 @@ +# Data Science Coding Workflow + +**Source validated against:** Cookiecutter Data Science, MLflow documentation, DVC documentation, Kedro documentation, established DS project conventions. +**Last reviewed:** 2026-05-23 +**When to load:** The campaign protocol has produced results and you need to structure them into a reproducible project; or the user asks "how should I set up this DS project?" + +--- + +## Project Directory Structure + +A consistent project structure makes experiments reproducible, results findable, and collaboration possible. + +### Recommended Layout + +``` +project/ +├── data/ +│ ├── raw/ # Immutable original data +│ ├── processed/ # Cleaned, feature-engineered data +│ └── external/ # External reference data (lookups, metadata) +├── notebooks/ # Exploratory analysis, prototypes +│ └── 01-exploration.ipynb +├── src/ # Reusable code +│ ├── __init__.py +│ ├── features/ # Feature engineering +│ ├── models/ # Model definitions, training logic +│ └── utils/ # Helper functions (logging, metrics) +├── models/ # Trained model artifacts +│ └── run_001/ +│ ├── model.pt +│ └── config.json +├── reports/ # Generated analysis, figures +│ └── figures/ +├── config/ +│ ├── config.yaml # Experiment configuration +│ └── params.yaml # Hyperparameters +├── experiments/ +│ └── experiment_log.json # Structured experiment record +├── requirements.txt +├── environment.yaml # Conda env export +├── setup.py # If src/ is a Python package +├── Makefile # Common commands (make train, make test) +└── README.md +``` + +### Quick Bootstrap + +```bash +# Using Cookiecutter Data Science (cookiecutter) +python -m pip install cookiecutter +cookiecutter https://github.com/drivendata/cookiecutter-data-science +``` + +### The Golden Rule + +**Raw data is read-only.** Never modify `data/raw/`. Always create derived data in `data/processed/` with explicit scripts. This ensures reproducibility — any change to processing is captured in the script, not hidden in a manual edit. + +--- + +## Configuration Management + +### YAML Config Pattern + +```yaml +# config/config.yaml +data: + raw_path: "data/raw/dataset.csv" + test_size: 0.2 + random_state: 42 + +preprocessing: + scaling: "standard" + handle_missing: "median" + categorical_encoding: "onehot" + +model: + name: "random_forest" + params: + n_estimators: 200 + max_depth: 10 + random_state: 42 + +training: + batch_size: 32 + learning_rate: 0.001 + epochs: 100 +``` + +### Loading Config in Python + +```python +import yaml +from pathlib import Path + +with open("config/config.yaml") as f: + config = yaml.safe_load(f) + +# Use config throughout the code +model_class = config["model"]["name"] +model_params = config["model"]["params"] +``` + +### Sweep Config (for Hyperparameter Search) + +```yaml +# config/sweep.yaml +parameters: + learning_rate: [0.0001, 0.001, 0.01] + batch_size: [16, 32, 64] + n_layers: [2, 4, 6] +``` + +### OmegaConf / Hydra (Advanced) + +For complex experiment configurations with hierarchical overrides: + +```python +# pip install omegaconf +from omegaconf import OmegaConf + +config = OmegaConf.create(""" +model: + name: resnet50 + pretrained: true +data: + path: ./data + augment: true +""") + +# Override from command line or code +config.model.name = "efficientnet" +``` + +--- + +## Experiment Logging + +### Why Log Experiments + +Without logging, you lose the mapping between code, data, hyperparameters, and results. A year later, "run_004" means nothing. Logging solves: +- **What** hyperparameters produced this result? +- **Where** is the trained model artifact? +- **When** was it trained (data version, code version)? +- **How** does this compare to previous runs? + +### Minimal Logging (JSON File) + +```python +import json +from datetime import datetime +from pathlib import Path + +def log_experiment( + experiment_dir: str, + model_name: str, + params: dict, + metrics: dict, + model_path: str = None, +) -> dict: + """Log a single experiment to a JSON file.""" + log_path = Path(experiment_dir) / "experiment_log.json" + log_path.parent.mkdir(parents=True, exist_ok=True) + + entry = { + "timestamp": datetime.now().isoformat(), + "model_name": model_name, + "params": params, + "metrics": metrics, + "model_path": model_path, + } + + # Append to log + if log_path.exists(): + with open(log_path) as f: + log = json.load(f) + else: + log = [] + log.append(entry) + + with open(log_path, "w") as f: + json.dump(log, f, indent=2) + + return entry +``` + +### MLflow Tracking + +```python +# pip install mlflow +import mlflow + +mlflow.set_experiment("customer-churn") + +with mlflow.start_run(run_name="random_forest_v2"): + # Log parameters + mlflow.log_param("n_estimators", 200) + mlflow.log_param("max_depth", 10) + + # Log metrics + mlflow.log_metric("f1", 0.87) + mlflow.log_metric("accuracy", 0.91) + + # Log model + mlflow.sklearn.log_model(pipeline, "model") + + # Log artifacts (figures, configs) + mlflow.log_artifact("config/config.yaml") + mlflow.log_artifact("reports/confusion_matrix.png") + + # Log tags for searchability + mlflow.set_tag("dataset_version", "v2.1") + mlflow.set_tag("status", "candidate") +``` + +### TensorBoard (for Deep Learning) + +```python +from torch.utils.tensorboard import SummaryWriter + +writer = SummaryWriter(log_dir="runs/experiment_1") + +# Log per-epoch metrics +for epoch in range(num_epochs): + train_loss = train_one_epoch(model, dataloader) + val_loss, val_acc = evaluate(model, val_loader) + + writer.add_scalar("Loss/train", train_loss, epoch) + writer.add_scalar("Loss/val", val_loss, epoch) + writer.add_scalar("Accuracy/val", val_acc, epoch) + + # Log model graph (once) + if epoch == 0: + writer.add_graph(model, example_input) + +# Launch: tensorboard --logdir runs/ +``` + +### WandB (Weights & Biases) + +```python +# pip install wandb +import wandb + +wandb.init(project="customer-churn", config={ + "learning_rate": 0.001, + "batch_size": 32, + "epochs": 100, +}) + +# Log metrics +for epoch in range(config["epochs"]): + loss = train_step() + wandb.log({"loss": loss, "epoch": epoch}) + +# Log model +wandb.save("model.pt") +``` + +**When to use what:** + +| Tool | Best For | Hosting | +|---|---|---| +| JSON file | Single user, no infrastructure | Local | +| MLflow | Teams, experiment comparison | Self-hosted or Databricks | +| TensorBoard | Deep learning training curves | Local | +| WandB | Collaborative DL experiments | Cloud (SaaS) | + +--- + +## Result Serialization + +| Data Type | Format | Library | Notes | +|---|---|---|---| +| Tabular data | Parquet | `pandas.DataFrame.to_parquet()` | Fast, compressed, columnar. **Best choice for most data.** | +| Metrics / hyperparams | JSON | `json.dump()` | Human-readable, universally parseable | +| Model (sklearn) | `.pkl` / `.joblib` | `joblib.dump()` | Load with `joblib.load()` | +| Model (PyTorch) | `.pt` / `.pth` | `torch.save()` | Use state_dict format | +| Model (export) | `.onnx` | `torch.onnx.export()` | Framework-neutral, deployable anywhere | +| Figures | `.png` / `.pdf` | `matplotlib.savefig()` | 300 DPI minimum for publication | +| Intermediate data | Feather | `pandas.DataFrame.to_feather()` | Fast read/write, no compression | + +```python +# Parquet — best for tabular data +df.to_parquet("data/processed/features.parquet") +df = pd.read_parquet("data/processed/features.parquet") + +# JSON — best for metrics +with open("reports/metrics.json", "w") as f: + json.dump(metrics, f, indent=2) + +# Joblib — best for sklearn models +import joblib +joblib.dump(pipeline, "models/pipeline_v2.pkl") +``` + +--- + +## Reproducibility + +### Seed Management + +```python +import random +import numpy as np +import torch + +def set_all_seeds(seed: int = 42): + """Set seeds for all random number generators used in ML.""" + random.seed(seed) + np.random.seed(seed) + torch.manual_seed(seed) + if torch.cuda.is_available(): + torch.cuda.manual_seed_all(seed) + torch.backends.cudnn.deterministic = True + torch.backends.cudnn.benchmark = False +``` + +### Environment Pinning + +```bash +# pip: freeze exact versions +pip freeze > requirements.txt + +# conda: export full environment +conda env export > environment.yaml + +# pip-compile (pip-tools): layered requirements +# requirements.in has loose deps, requirements.txt has pinned +``` + +### Docker for Full Reproducibility + +```dockerfile +FROM python:3.12-slim +WORKDIR /app +COPY requirements.txt . +RUN pip install --no-cache-dir -r requirements.txt +COPY src/ src/ +COPY config/ config/ +ENTRYPOINT ["python", "src/train.py"] +``` + +**When Docker is overkill:** Single-script analyses, exploration, individual experiment debugging. Use pinned requirements + seed setting instead. + +**When Docker is necessary:** Team projects, production deployment, sharing with non-technical stakeholders, running experiments on different hardware. + +### Code Version Tracking + +```python +# Embed git commit hash in experiment log +import subprocess + +def get_git_commit_hash(): + """Get the current git commit hash.""" + try: + return subprocess.run( + ["git", "rev-parse", "--short", "HEAD"], + capture_output=True, text=True, check=True + ).stdout.strip() + except (subprocess.CalledProcessError, FileNotFoundError): + return "unknown" + +# Include in experiment log: +log_entry["git_commit"] = get_git_commit_hash() +``` + +--- + +## Data Versioning + +### DVC (Data Version Control) + +```bash +# pip install dvc +dvc init +dvc add data/raw/dataset.csv # Tracks dataset with .dvc file +git add data/raw/dataset.csv.dvc # Commit pointer, not data +git commit -m "add dataset v1" + +# Push to remote storage +dvc remote add myremote s3://mybucket/dvc +dvc push + +# Later, pull a specific version +git checkout +dvc checkout # Restores the matching data version +``` + +### Without DVC: Simple Hash-Based Cache + +```python +import hashlib +from pathlib import Path + +def hash_file(path: Path) -> str: + """SHA-256 hash of a file for integrity checking.""" + hasher = hashlib.sha256() + with open(path, "rb") as f: + for chunk in iter(lambda: f.read(65536), b""): + hasher.update(chunk) + return hasher.hexdigest() + +# Store hash alongside experiment results +dataset_hash = hash_file("data/raw/dataset.csv") +log_entry["data_hash"] = dataset_hash +``` + +--- + +## Unit Testing for Data Science + +### Test Data Pattern + +```python +def test_feature_engineering(): + """Test feature engineering with a tiny known dataset.""" + # Arrange: create 5-sample dataset with known properties + X_test = pd.DataFrame({ + "age": [25, 30, 45, 60, 35], + "income": [50000, 60000, 80000, 120000, 75000], + "gender": ["M", "F", "F", "M", "F"], + }) + + # Act + result = create_features(X_test) + + # Assert: known properties + assert result.shape[0] == 5, "Should preserve row count" + assert "age_scaled" in result.columns, "Should have age_scaled column" + assert result["age_scaled"].std() > 0, "Scaled values should have variance" +``` + +### Model Invariance Test + +```python +def test_model_output_shape(): + """Model should produce correct output shape on valid input.""" + X_sample = np.random.randn(32, 10) # 32 samples, 10 features + y_sample = (X_sample[:, 0] > 0).astype(int) + + model = RandomForestClassifier(n_estimators=10, random_state=42) + model.fit(X_sample, y_sample) + + predictions = model.predict(X_sample) + assert predictions.shape == (32,), "Should output one prediction per sample" + assert set(predictions).issubset({0, 1}), "Should predict binary classes" +``` + +### Data Integrity Tests + +```python +def test_no_missing_values_after_imputation(): + """Preprocessing should handle all missing values.""" + # Load a sample of processed data + X = pd.read_parquet("data/processed/features.parquet") + assert X.isnull().sum().sum() == 0, "No missing values should remain" + +def test_target_distribution(): + """Target variable should have expected distribution.""" + df = pd.read_parquet("data/processed/train.parquet") + class_counts = df["target"].value_counts() + # Warn if any class has < 1% prevalence + for cls, count in class_counts.items(): + assert count / len(df) >= 0.01, f"Class {cls} has < 1% prevalence" +``` + +--- + +## Common Pitfalls + +| Pitfall | Symptom | Fix | +|---|---|---| +| Notebooks with unnumbered cells | Can't reproduce order | Number cells (01-load, 02-explore, 03-model). Convert to scripts before production. | +| Hardcoded file paths | Code breaks on different machines | Use `pathlib.Path`, config files, or `os.getenv` | +| No `random_state` | Results change each run | Set seeds at the top of every script | +| Data leakage in preprocessing | Overly optimistic results | Fit preprocessors on training data only, use `Pipeline` | +| Training on full data before evaluation | No held-out test set | Always split before any modeling | +| Git-ignored data/ directory | No one else can run the code | Use DVC or document how to obtain data | +| One giant `train.py` | Hard to debug, test, reuse | Split into `features.py`, `model.py`, `train.py`, `evaluate.py` | + +--- + +## See Also + +- `references/experimental-campaign-protocol.md` — the high-level workflow this supports +- `references/pytorch-integration.md` — training loops and model persistence +- `references/sklearn-integration.md` — pipelines and model selection +- `assets/experimental-plan-template.md` — pre-registration-style planning document +- `assets/report-template.md` — analysis report format diff --git a/data-scientist/references/pytorch-integration.md b/data-scientist/references/pytorch-integration.md new file mode 100644 index 0000000..aa05ea5 --- /dev/null +++ b/data-scientist/references/pytorch-integration.md @@ -0,0 +1,616 @@ +# PyTorch Integration Reference + +**Source validated against:** PyTorch 2.12 documentation (pytorch.org/docs/stable) +**Last reviewed:** 2026-05-23 +**When to load:** The campaign protocol (Phase 3-7) or any task requires implementing, training, or deploying a PyTorch model. + +--- + +## Device Management + +### Canonical Device Pattern + +Always parameterize the device. Never hardcode `"cuda"`. + +```python +import torch + +device = torch.device( + "cuda" if torch.cuda.is_available() + else "mps" if torch.backends.mps.is_available() + else "cpu" +) +# Usage: model.to(device); tensor.to(device) +``` + +### Checking Device Properties + +```python +if torch.cuda.is_available(): + print(f"Device: {torch.cuda.get_device_name(0)}") + print(f"VRAM: {torch.cuda.get_device_properties(0).total_memory / 1e9:.1f} GB") + print(f"CUDA Capability: {torch.cuda.get_device_capability(0)}") +``` + +**MPS (Apple Silicon):** Available on macOS 12.3+. Not all operations are supported — check `torch.backends.mps.is_available()` and `torch.backends.mps.is_built()`. Some operations fall back to CPU automatically. + +--- + +## Training Loop Patterns + +### Basic Supervised Training Loop + +```python +model.train() +for epoch in range(num_epochs): + running_loss = 0.0 + for batch_x, batch_y in dataloader: + batch_x, batch_y = batch_x.to(device), batch_y.to(device) + + optimizer.zero_grad() + outputs = model(batch_x) + loss = criterion(outputs, batch_y) + loss.backward() + + # Gradient clipping (essential for stability) + torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0) + + optimizer.step() + + running_loss += loss.item() + + avg_loss = running_loss / len(dataloader) + print(f"Epoch {epoch}: loss={avg_loss:.4f}") +``` + +### Gradient Accumulation (for large models on limited VRAM) + +```python +accumulation_steps = 4 # Effective batch_size = physical_batch * accumulation + +for i, (batch_x, batch_y) in enumerate(dataloader): + outputs = model(batch_x) + loss = criterion(outputs, batch_y) / accumulation_steps + loss.backward() + + if (i + 1) % accumulation_steps == 0: + torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0) + optimizer.step() + optimizer.zero_grad() +``` + +### Validation Loop + +```python +model.eval() +val_loss = 0.0 +correct = 0 +total = 0 + +with torch.no_grad(): + for batch_x, batch_y in val_loader: + batch_x, batch_y = batch_x.to(device), batch_y.to(device) + outputs = model(batch_x) + val_loss += criterion(outputs, batch_y).item() + _, predicted = torch.max(outputs, 1) + total += batch_y.size(0) + correct += (predicted == batch_y).sum().item() + +print(f"Val Loss: {val_loss/len(val_loader):.4f}, Acc: {100*correct/total:.2f}%") +``` + +--- + +## Dataset & DataLoader + +### Custom Dataset Class + +```python +from torch.utils.data import Dataset, DataLoader + +class CustomDataset(Dataset): + def __init__(self, features, labels, transform=None): + self.features = torch.tensor(features, dtype=torch.float32) + self.labels = torch.tensor(labels, dtype=torch.long) + self.transform = transform + + def __len__(self): + return len(self.labels) + + def __getitem__(self, idx): + x = self.features[idx] + y = self.labels[idx] + if self.transform: + x = self.transform(x) + return x, y +``` + +### DataLoader Configuration + +```python +dataloader = DataLoader( + dataset, + batch_size=32, + shuffle=True, + num_workers=4, # Set to 0 on Windows if multiprocessing issues + pin_memory=True, # Speeds up GPU transfer (only with CUDA) + persistent_workers=True if num_workers > 0 else False, # PyTorch 2.0+ + collate_fn=None, # Custom collation for variable-length data +) +``` + +**Note on `num_workers`:** On Linux, 4-8 workers is typical. On macOS, keep at 0-2. On Windows, 0 is safest. The optimal value depends on the data loading speed vs GPU speed. + +### Collate Function for Variable-Length Data + +```python +def collate_fn(batch): + """Pad sequences in a batch to the same length.""" + inputs, labels = zip(*batch) + # Pad to max length in this batch + inputs_padded = torch.nn.utils.rnn.pad_sequence(inputs, batch_first=True) + return inputs_padded, torch.tensor(labels) +``` + +--- + +## Model Saving & Loading + +### Save/Load State Dict (Recommended) + +```python +# Save +torch.save({ + "epoch": epoch, + "model_state_dict": model.state_dict(), + "optimizer_state_dict": optimizer.state_dict(), + "loss": loss, + "config": {"n_layers": 4, "hidden_dim": 256}, # metadata +}, "checkpoint.pt") + +# Load +checkpoint = torch.load("checkpoint.pt", map_location=device) +model.load_state_dict(checkpoint["model_state_dict"]) +optimizer.load_state_dict(checkpoint["optimizer_state_dict"]) +start_epoch = checkpoint["epoch"] + 1 +``` + +### Save for Inference (weights only) + +```python +torch.save(model.state_dict(), "model_weights.pt") + +# Load for inference +model = YourModel() +model.load_state_dict(torch.load("model_weights.pt", map_location="cpu")) +model.eval() +``` + +### TorchScript / torch.export (for production deployment) + +```python +# torch.export (PyTorch 2.x preferred, producesExportedProgram) +exported_program = torch.export.export(model, (example_input,)) +torch.export.save(exported_program, "model.pt2") + +# TorchScript (legacy, still widely supported) +scripted_model = torch.jit.script(model) +scripted_model.save("model_scripted.pt") +``` + +--- + +## Mixed Precision (AMP) + +Automatic Mixed Precision trains with `float16` (or `bfloat16`) where safe and `float32` where needed. Typically 1.5-2x faster with minimal accuracy loss. + +```python +from torch.amp import autocast, GradScaler + +scaler = GradScaler("cuda") # "cuda" or "cpu" + +for batch_x, batch_y in dataloader: + batch_x, batch_y = batch_x.to(device), batch_y.to(device) + + optimizer.zero_grad() + + # Autocast context manager + with autocast(device_type="cuda"): # or "cpu" + outputs = model(batch_x) + loss = criterion(outputs, batch_y) + + # Scale loss, backward, unscale, step + scaler.scale(loss).backward() + scaler.unscale_(optimizer) + torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0) + scaler.step(optimizer) + scaler.update() +``` + +**Key points:** +- `autocast` wraps forward pass + loss computation only +- `GradScaler` prevents underflow in small gradients +- Use `bfloat16` on Ampere+ GPUs (less numerical issues than `float16`) +- AMP is most beneficial for large models (CNNs, Transformers) — small models may not see speedup + +--- + +## torch.compile (PyTorch 2.x JIT Compilation) + +`torch.compile` compiles model graphs for faster execution with minimal code changes. + +```python +# Basic usage (reduces model — returns a compiled wrapper) +compiled_model = torch.compile(model) + +# With options +compiled_model = torch.compile( + model, + mode="reduce-overhead", # "default", "reduce-overhead", "max-autotune" + fullgraph=True, # Fail if graph breaks + dynamic=True, # Support dynamic tensor shapes +) + +# Training is identical: +output = compiled_model(batch_x) # First call compiles, subsequent calls are fast +``` + +**When to use:** Models with large tensor operations (CNNs, Transformers). Not beneficial for very small models or heavy data loading bottlenecks. + +**When not to use:** Dynamic control flow, custom CUDA extensions, models compiled for TorchScript export. + +**Mode selection:** +- `"default"` — conservative, works for most models +- `"reduce-overhead"` — good for inference, reduces Python overhead +- `"max-autotune"` — benchmarks and picks the best backend (slow first run) + +--- + +## Loss Functions + +| Problem Type | Loss Function | Import | +|---|---|---| +| Binary classification | `BCEWithLogitsLoss` | `torch.nn.BCEWithLogitsLoss` | +| Multi-class classification | `CrossEntropyLoss` | `torch.nn.CrossEntropyLoss` | +| Multi-label classification | `BCEWithLogitsLoss` | Combines sigmoid + BCELoss | +| Regression (MSE) | `MSELoss` | `torch.nn.MSELoss` | +| Regression (MAE) | `L1Loss` | `torch.nn.L1Loss` | +| Regression (Huber) | `HuberLoss` | `torch.nn.HuberLoss` (delta parameter) | +| Contrastive / Siamese | `TripletMarginLoss` / `ContrastiveLoss` | `torch.nn.TripletMarginLoss` | +| Imbalanced classes | Weighted `CrossEntropyLoss` | Pass `weight` tensor to constructor | +| Sequence (CTC) | `CTCLoss` | `torch.nn.CTCLoss` | + +```python +# Weighted loss for imbalanced data +class_weights = torch.tensor([0.2, 0.8]).to(device) # inverse frequency +criterion = torch.nn.CrossEntropyLoss(weight=class_weights) +``` + +--- + +## Optimizers & Schedulers + +| Optimizer | When | Learning Rate | +|---|---|---| +| `AdamW` | Default for most models (Transformers, CNNs) | 1e-4 to 3e-4 | +| `Adam` | Legacy; prefer AdamW (proper weight decay) | 1e-4 to 3e-4 | +| `SGD` + momentum | When Adam overfits; vision models | 1e-2 to 1e-1 | +| `AdamW` (LoRA) | Fine-tuning with LoRA | 2e-4 to 5e-4 | + +```python +import torch.optim as optim + +# AdamW — the default +optimizer = optim.AdamW(model.parameters(), lr=1e-4, weight_decay=0.01) + +# SGD with momentum (for vision fine-tuning) +optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.9, weight_decay=1e-4) +``` + +### Learning Rate Schedulers + +```python +from torch.optim.lr_scheduler import ( + ReduceLROnPlateau, # Reduce when metric plateaus + CosineAnnealingLR, # Cosine decay + OneCycleLR, # Warmup + cosine ("super-convergence") +) + +# Reduce on plateau (works well for most tasks) +scheduler = ReduceLROnPlateau(optimizer, mode="min", patience=5, factor=0.5) + +# One-cycle (fast convergence, needs max_lr) +scheduler = OneCycleLR( + optimizer, + max_lr=1e-3, + steps_per_epoch=len(train_loader), + epochs=num_epochs, +) + +# Cosine annealing (good for Transformers) +scheduler = CosineAnnealingLR(optimizer, T_max=num_epochs) + +# Warmup + cosine (standard for LLM fine-tuning) +# Implement manually: +def get_cosine_schedule_with_warmup(optimizer, warmup_steps, total_steps): + def lr_lambda(current_step): + if current_step < warmup_steps: + return float(current_step) / float(max(1, warmup_steps)) + progress = float(current_step - warmup_steps) / float(max(1, total_steps - warmup_steps)) + return 0.5 * (1.0 + math.cos(math.pi * progress)) + return torch.optim.lr_scheduler.LambdaLR(optimizer, lr_lambda) +``` + +--- + +## Transfer Learning + +### Feature Extraction (Freeze Backbone) + +```python +import torchvision.models as models + +model = models.resnet50(weights=models.ResNet50_Weights.IMAGENET1K_V2) + +# Freeze all layers +for param in model.parameters(): + param.requires_grad = False + +# Replace classifier +num_features = model.fc.in_features +model.fc = torch.nn.Linear(num_features, num_classes) + +# Only the new layer's params are trainable +optimizer = optim.AdamW(model.fc.parameters(), lr=1e-3) +``` + +### Full Fine-Tuning + +```python +# Unfreeze all layers +for param in model.parameters(): + param.requires_grad = True + +# Use lower learning rate +optimizer = optim.AdamW(model.parameters(), lr=2e-5) + +# Often helps to freeze early layers, fine-tune later layers: +for name, param in model.named_parameters(): + if "layer1" in name or "layer2" in name: + param.requires_grad = False +``` + +### LoRA for Transformer Models + +LoRA (Low-Rank Adaptation) trains small rank-decomposition matrices while keeping the base model frozen. Requires the `peft` library. + +```python +from peft import LoraConfig, get_peft_model + +lora_config = LoraConfig( + r=8, # Rank — higher = more expressiveness, more params + lora_alpha=32, # Scaling factor + target_modules=["q_proj", "v_proj"], # Which modules to apply LoRA to + lora_dropout=0.1, + bias="none", # Don't train bias terms + task_type="SEQ_CLS", # Task type (SEQ_CLS, CAUSAL_LM, TOKEN_CLS, etc.) +) + +model = get_peft_model(base_model, lora_config) + +# Train with slightly higher LR +optimizer = optim.AdamW(model.parameters(), lr=3e-4) + +# LoRA adds very few params (< 1% of base model) +model.print_trainable_parameters() # e.g., "trainable params: 294,912 || all params: 110,080,512 || %: 0.2679" +``` + +--- + +## Knowledge Distillation + +```python +import torch.nn.functional as F + +def distillation_loss(student_logits, teacher_logits, labels, temperature=4.0, alpha=0.7): + """ + Combined distillation + supervised loss. + + Args: + temperature: Higher = softer probability distribution (more information from teacher) + alpha: Weight for distillation loss (vs standard cross-entropy) + """ + # Soft target loss (distillation) + soft_student = F.log_softmax(student_logits / temperature, dim=-1) + soft_teacher = F.softmax(teacher_logits.detach() / temperature, dim=-1) + distill_loss = F.kl_div(soft_student, soft_teacher, reduction="batchmean") + distill_loss *= temperature ** 2 # Scale to keep gradients in right range + + # Hard target loss + hard_loss = F.cross_entropy(student_logits, labels) + + return alpha * distill_loss + (1 - alpha) * hard_loss + + +# Training loop with distillation +teacher_model.eval() +for batch_x, batch_y in dataloader: + with torch.no_grad(): + teacher_logits = teacher_model(batch_x) + + student_logits = student_model(batch_x) + loss = distillation_loss(student_logits, teacher_logits, batch_y) + + optimizer.zero_grad() + loss.backward() + optimizer.step() +``` + +**Temperature tuning:** Start with T=4.0. Higher values produce softer targets (more small-class information). Lower values (T=1.0) collapse to standard cross-entropy. + +**Alpha tuning:** α=0.7 (weight on distillation) is a common starting point. Increase α when the teacher is much better than the student. + +--- + +## Model Pruning + +```python +import torch.nn.utils.prune as prune + +# Apply pruning to specific layers +prune.l1_unstructured(module=model.fc, name="weight", amount=0.3) # Remove 30% of weights + +# Make pruning permanent (removes the pruning mask) +prune.remove(module=model.fc, name="weight") + +# Structured pruning (removes entire neurons/channels) +prune.ln_structured(module=model.conv1, name="weight", amount=0.2, n=2, dim=0) + +# Global pruning (prune all layers together by importance) +parameters_to_prune = [ + (model.layer1, "weight"), + (model.layer2, "weight"), + (model.fc, "weight"), +] +prune.global_unstructured( + parameters_to_prune, + pruning_method=prune.L1Unstructured, + amount=0.2, # Remove 20% of weights globally +) +``` + +**After pruning:** Fine-tune the pruned model. Pruning then fine-tuning almost always recovers accuracy. Pruning without fine-tuning degrades performance significantly. + +--- + +## Distributed Data Parallel (DDP) + +For multi-GPU training. DDP wraps the model and handles gradient synchronization. + +```python +import torch.distributed as dist +import torch.multiprocessing as mp +from torch.nn.parallel import DistributedDataParallel as DDP + +def setup(rank, world_size): + """Initialize the distributed process group.""" + dist.init_process_group( + backend="nccl", # "nccl" for NVIDIA, "gloo" for CPU + init_method="env://", # Use env vars MASTER_ADDR and MASTER_PORT + rank=rank, + world_size=world_size, + ) + +def cleanup(): + dist.destroy_process_group() + +def train(rank, world_size): + setup(rank, world_size) + + # Model must be on the correct device BEFORE wrapping + model = YourModel().to(rank) + ddp_model = DDP(model, device_ids=[rank]) + + # DataLoader must use DistributedSampler + sampler = torch.utils.data.distributed.DistributedSampler( + dataset, num_replicas=world_size, rank=rank + ) + dataloader = DataLoader(dataset, batch_size=32, sampler=sampler) + + # Training loop (same structure, use ddp_model) + for epoch in range(num_epochs): + sampler.set_epoch(epoch) # Shuffle each epoch + for batch_x, batch_y in dataloader: + batch_x, batch_y = batch_x.to(rank), batch_y.to(rank) + outputs = ddp_model(batch_x) + loss = criterion(outputs, batch_y) + loss.backward() + optimizer.step() + + cleanup() + +# Launch +if __name__ == "__main__": + world_size = torch.cuda.device_count() + mp.spawn(train, args=(world_size,), nprocs=world_size) +``` + +**When DDP is worth it:** Models that take > 1 hour to train on a single GPU. For short experiments, single-GPU + AMP is often faster due to communication overhead. + +--- + +## Debugging + +### Common Failure Patterns + +| Symptom | Likely Cause | Fix | +|---|---|---| +| `loss = nan` | Exploding gradients, bad learning rate | Lower LR, add gradient clipping, check for NaN in input data | +| `loss = nan` after AMP | Gradient underflow | Increase `GradScaler` init_scale, or use `bfloat16` | +| Loss doesn't decrease | Wrong LR, wrong loss function | Check LR range, verify loss function matches task | +| CUDA OOM | Batch size too large | Reduce batch size, enable gradient checkpointing, use AMP | +| `Expected all tensors to be on...` | Device mismatch | Always `.to(device)` tensors before model forward | +| `CUDA error: device-side assert` | Wrong label class (out of range) | Check label values are in `[0, num_classes)` | +| Model doesn't overfit 1 batch | Bug in model architecture | Try overfitting on a single batch (batch of 2-4 samples for 100 steps) | + +### Overfit on One Batch (Diagnostic) + +```python +# If model can't overfit a single batch, something is fundamentally wrong +single_batch = next(iter(dataloader)) + +for step in range(100): + outputs = model(single_batch[0].to(device)) + loss = criterion(outputs, single_batch[1].to(device)) + loss.backward() + optimizer.step() + +print(f"Loss after 100 steps on 1 batch: {loss.item():.6f}") +# If loss is not near 0, model has a bug or LR is way off. +``` + +### Gradient Checking + +```python +# Check gradient norms during training +total_norm = 0.0 +for p in model.parameters(): + if p.grad is not None: + param_norm = p.grad.data.norm(2) + total_norm += param_norm.item() ** 2 +total_norm = total_norm ** 0.5 + +if total_norm > 10.0: + print(f"WARNING: Large gradient norm: {total_norm:.4f}") +``` + +--- + +## Reproducibility + +```python +import random +import numpy as np +import torch + +def set_seed(seed: int = 42): + random.seed(seed) + np.random.seed(seed) + torch.manual_seed(seed) + torch.cuda.manual_seed_all(seed) + torch.backends.cudnn.deterministic = True + torch.backends.cudnn.benchmark = False # Trades off speed for determinism + +set_seed(42) +``` + +**Note on `cudnn.deterministic`:** Setting this to `True` may reduce performance (5-15%). For experiments where exact reproducibility isn't critical, leave `benchmark=True` and accept minor stochastic variation. + +--- + +## See Also + +- `references/experimental-campaign-protocol.md` — where this reference fits in the campaign workflow +- `references/sklearn-integration.md` — for non-deep-learning methods +- `scripts/detect-compute.py` — check your hardware before choosing model size +- pytorch.org/docs/stable — official API documentation diff --git a/data-scientist/references/sklearn-integration.md b/data-scientist/references/sklearn-integration.md new file mode 100644 index 0000000..1c2dec6 --- /dev/null +++ b/data-scientist/references/sklearn-integration.md @@ -0,0 +1,577 @@ +# Scikit-Learn Integration Reference + +**Source validated against:** scikit-learn 1.8.0 (scikit-learn.org/stable) +**Last reviewed:** 2026-05-23 +**When to load:** The campaign protocol (Phase 2, 4, 6), baseline modeling, preprocessing, or any task involving sklearn estimators. + +--- + +## Pipeline Composition + +Pipelines chain preprocessing and modeling into a single estimator. This enables proper cross-validation (no data leakage from preprocessing) and simplifies deployment. + +### Basic Pipeline + +```python +from sklearn.pipeline import Pipeline +from sklearn.preprocessing import StandardScaler +from sklearn.linear_model import LogisticRegression + +pipeline = Pipeline([ + ("scaler", StandardScaler()), + ("classifier", LogisticRegression(max_iter=1000, random_state=42)), +]) + +# Use like a regular estimator +pipeline.fit(X_train, y_train) +y_pred = pipeline.predict(X_test) +``` + +### Shortcut: `make_pipeline` + +```python +from sklearn.pipeline import make_pipeline + +pipeline = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)) +# Step names are auto-generated: "standardscaler", "logisticregression" +``` + +### Accessing Step Attributes + +```python +# After fitting +pipeline.fit(X_train, y_train) + +# Access the trained scaler +scaler = pipeline.named_steps["scaler"] +print(f"Mean: {scaler.mean_}") + +# Access coefficients from the classifier +coefs = pipeline.named_steps["classifier"].coef_ +``` + +--- + +## ColumnTransformer (Heterogeneous Data) + +When your data has both numeric and categorical columns, use `ColumnTransformer` to apply different preprocessing to different columns. + +```python +from sklearn.compose import ColumnTransformer, make_column_selector +from sklearn.preprocessing import StandardScaler, OneHotEncoder + +numeric_features = ["age", "income", "score"] +categorical_features = ["gender", "region", "education"] + +preprocessor = ColumnTransformer([ + ("num", StandardScaler(), numeric_features), + ("cat", OneHotEncoder(handle_unknown="ignore"), categorical_features), +]) + +# Or use column type selectors +preprocessor = ColumnTransformer([ + ("num", StandardScaler(), make_column_selector(dtype_include="number")), + ("cat", OneHotEncoder(handle_unknown="ignore"), + make_column_selector(dtype_include="object")), +]) +``` + +### Full Pipeline with ColumnTransformer + +```python +from sklearn.ensemble import RandomForestClassifier + +pipeline = Pipeline([ + ("preprocessor", preprocessor), + ("classifier", RandomForestClassifier(n_estimators=200, random_state=42)), +]) + +# Grid search over both preprocessing and model params +param_grid = { + "preprocessor__num__with_mean": [True, False], + "classifier__n_estimators": [100, 200, 500], + "classifier__max_depth": [10, 20, None], +} + +grid = GridSearchCV(pipeline, param_grid, cv=5, scoring="f1_macro") +grid.fit(X_train, y_train) +``` + +**Memory-Efficient ColumnTransformer:** Set `remainder="passthrough"` to keep columns not specified, or `remainder="drop"` (default) to drop them. + +--- + +## Preprocessing + +### Scaling & Normalization + +| Scaler | Description | When | +|---|---|---| +| `StandardScaler` | Z-score: (x - μ) / σ | Default for most models. Assumes roughly Gaussian data. | +| `MinMaxScaler` | Scale to [0, 1] | When bounded ranges matter (neural nets, distance-based). | +| `RobustScaler` | Uses median and IQR | When data has outliers. More robust than StandardScaler. | +| `MaxAbsScaler` | Scale to [-1, 1] | For sparse data (preserves sparsity). | +| `Normalizer` | Unit norm per sample | Text classification, cosine similarity. | + +```python +from sklearn.preprocessing import StandardScaler, MinMaxScaler, RobustScaler + +# StandardScaler is the default +scaler = StandardScaler() +X_scaled = scaler.fit_transform(X_train) + +# Always fit on training, transform both train and test +X_test_scaled = scaler.transform(X_test) +``` + +### Encoding Categorical Features + +```python +from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder + +# One-hot (nominal categories — no ordering) +encoder = OneHotEncoder(handle_unknown="ignore", sparse_output=False) +X_encoded = encoder.fit_transform(X_categorical) + +# Ordinal (ordered categories) +encoder = OrdinalEncoder(categories=[["low", "medium", "high"]]) +X_encoded = encoder.fit_transform(X_ordinal) +``` + +### Handling Missing Values + +```python +from sklearn.impute import SimpleImputer, KNNImputer, IterativeImputer + +# Simple imputation (fast) +imputer = SimpleImputer(strategy="median") # "mean", "median", "most_frequent", "constant" + +# KNN imputation (better for local patterns, slower) +imputer = KNNImputer(n_neighbors=5) + +# Iterative imputation (MICE-style, best but slow) +imputer = IterativeImputer(max_iter=10, random_state=42) # Experimental — requires explicit import + +# In a pipeline: +pipeline = Pipeline([ + ("imputer", SimpleImputer(strategy="median")), + ("scaler", StandardScaler()), + ("classifier", LogisticRegression()), +]) +``` + +--- + +## Model Selection + +### Cross-Validation Strategies + +| Splitter | Use Case | +|---|---| +| `KFold(n_splits=5, shuffle=True)` | Default for most tasks | +| `StratifiedKFold(n_splits=5)` | Classification — preserves class proportions | +| `GroupKFold(n_splits=5)` | When samples belong to groups (e.g., same patient) | +| `TimeSeriesSplit(n_splits=5)` | Temporal data — train on past, test on future | +| `RepeatedStratifiedKFold(n_repeats=3)` | More robust estimate, higher variance | +| `LeaveOneOut()` | Very small datasets (< 100 samples) | + +```python +from sklearn.model_selection import ( + KFold, StratifiedKFold, GroupKFold, TimeSeriesSplit, cross_val_score, cross_validate +) + +# Quick cross-validation score +scores = cross_val_score(pipeline, X, y, cv=StratifiedKFold(5), scoring="f1_macro") +print(f"F1: {scores.mean():.4f} ± {scores.std():.4f}") + +# Detailed cross-validation +cv_results = cross_validate( + pipeline, X, y, + cv=StratifiedKFold(5), + scoring=["f1_macro", "accuracy", "roc_auc"], + return_estimator=True, # Return fitted models for inspection + return_train_score=True, # Detect overfitting +) +``` + +### Grid Search + +```python +from sklearn.model_selection import GridSearchCV, RandomizedSearchCV, HalvingGridSearchCV + +# Grid search (exhaustive) +grid = GridSearchCV( + pipeline, + param_grid={ + "classifier__C": [0.01, 0.1, 1.0, 10.0], + "classifier__penalty": ["l2"], + }, + cv=5, + scoring="f1_macro", + n_jobs=-1, # Use all CPU cores + verbose=1, +) +grid.fit(X_train, y_train) + +print(f"Best params: {grid.best_params_}") +print(f"Best score: {grid.best_score_:.4f}") + +# Random search (better for high-dimensional spaces) +random_search = RandomizedSearchCV( + pipeline, + param_distributions={ + "classifier__C": [0.01, 0.1, 1.0, 10.0, 100.0], + "classifier__max_iter": [500, 1000, 2000], + }, + n_iter=20, # Number of random combinations to try + cv=5, + scoring="f1_macro", + n_jobs=-1, + random_state=42, +) + +# Halving search (successive halving — tries many candidates, prunes poor ones fast) +halving_search = HalvingGridSearchCV( + pipeline, + param_grid={"classifier__C": [0.01, 0.1, 1.0, 10.0]}, + factor=3, # Reduce candidates by factor 3 each iteration + cv=5, + scoring="f1_macro", + n_jobs=-1, + verbose=1, +) +``` + +### Nested Cross-Validation (Unbiased Performance Estimate) + +```python +from sklearn.model_selection import cross_val_score +from sklearn.model_selection import GridSearchCV +from sklearn.tree import DecisionTreeClassifier + +# Inner CV: model selection +inner_cv = StratifiedKFold(3, shuffle=True, random_state=42) +grid = GridSearchCV(DecisionTreeClassifier(), + {"max_depth": [3, 5, 10, None]}, + cv=inner_cv) + +# Outer CV: performance estimation +outer_cv = StratifiedKFold(5, shuffle=True, random_state=42) +nested_scores = cross_val_score(grid, X, y, cv=outer_cv, scoring="f1_macro") +# This gives an unbiased estimate of the tuned model's performance +print(f"Unbiased F1: {nested_scores.mean():.4f} ± {nested_scores.std():.4f}") +``` + +--- + +## Ensemble Methods + +```python +from sklearn.ensemble import ( + RandomForestClassifier, + GradientBoostingClassifier, + StackingClassifier, + VotingClassifier, + AdaBoostClassifier, + BaggingClassifier, +) + +# Stacking (meta-model combines base models) +stack = StackingClassifier( + estimators=[ + ("rf", RandomForestClassifier(n_estimators=100, random_state=42)), + ("gb", GradientBoostingClassifier(n_estimators=100, random_state=42)), + ("svc", LinearSVC(random_state=42)), + ], + final_estimator=LogisticRegression(), + cv=5, +) + +# Voting (simple majority or weighted average) +vote = VotingClassifier( + estimators=[ + ("lr", LogisticRegression()), + ("rf", RandomForestClassifier(n_estimators=100)), + ("gnb", GaussianNB()), + ], + voting="soft", # "hard" for majority vote, "soft" for probability average +) +``` + +### XGBoost / LightGBM Integration + +```python +# sklearn-compatible API +import xgboost as xgb +import lightgbm as lgb + +xgb_model = xgb.XGBClassifier( + n_estimators=200, + max_depth=6, + learning_rate=0.1, + eval_metric="logloss", + use_label_encoder=False, + random_state=42, +) + +lgb_model = lgb.LGBMClassifier( + n_estimators=200, + num_leaves=31, + learning_rate=0.1, + random_state=42, + verbose=-1, +) + +# Both work in sklearn pipelines and GridSearchCV +pipeline = Pipeline([ + ("preprocessor", preprocessor), + ("classifier", xgb_model), +]) +``` + +--- + +## Custom Estimators + +### Custom Transformer + +```python +from sklearn.base import BaseEstimator, TransformerMixin + +class LogTransformer(BaseEstimator, TransformerMixin): + """Apply log(1 + x) to specified columns.""" + + def __init__(self, columns=None): + self.columns = columns # None = all columns + + def fit(self, X, y=None): + # LogTransform doesn't need fitting, but fit must return self + return self + + def transform(self, X): + X = X.copy() + cols = self.columns if self.columns is not None else X.columns + X[cols] = X[cols].applymap(lambda x: np.log1p(x)) # log1p = log(1+x) + return X +``` + +### Custom Estimator + +```python +from sklearn.base import BaseEstimator, ClassifierMixin + +class SimpleThresholdClassifier(BaseEstimator, ClassifierMixin): + """Classify based on a learned threshold on one feature.""" + + def __init__(self, threshold=0.5): + self.threshold = threshold + + def fit(self, X, y): + # Learn optimal threshold + # Implementation here + self.is_fitted_ = True + return self + + def predict(self, X): + check_is_fitted(self) + return (X[:, 0] > self.threshold).astype(int) + + def predict_proba(self, X): + # Not implemented — raises error if called + raise NotImplementedError("This estimator doesn't support probabilities") +``` + +### FunctionTransformer (Quick Custom Transform) + +```python +import numpy as np +from sklearn.preprocessing import FunctionTransformer + +# No class needed for simple transforms +log_transform = FunctionTransformer(func=np.log1p, validate=True) + +# In a pipeline: +pipeline = Pipeline([ + ("log", log_transform), + ("scaler", StandardScaler()), +]) +``` + +--- + +## Persistence + +```python +import joblib + +# Save +joblib.dump(pipeline, "model.pkl") + +# Load +loaded_pipeline = joblib.load("model.pkl") +predictions = loaded_pipeline.predict(X_new) +``` + +**⚠️ Security:** `joblib.load` can execute arbitrary code on deserialization. Only load models from trusted sources. Use `pickle` with the same caveat. + +**Model portability:** sklearn models versioned with the sklearn version that created them. Cross-version compatibility is not guaranteed. Always save the sklearn version alongside the model. + +--- + +## Imbalanced Data + +### Built-in sklearn Support + +```python +from sklearn.linear_model import LogisticRegression +from sklearn.utils.class_weight import compute_class_weight + +# Option 1: Use class_weight parameter +model = LogisticRegression(class_weight="balanced", max_iter=1000) + +# Option 2: Manual class weights +weights = compute_class_weight("balanced", classes=np.unique(y), y=y) +class_weight_dict = dict(zip(np.unique(y), weights)) +model = LogisticRegression(class_weight=class_weight_dict, max_iter=1000) +``` + +### imbalanced-learn Library + +```python +from imblearn.over_sampling import SMOTE, ADASYN, RandomOverSampler +from imblearn.under_sampling import RandomUnderSampler, NearMiss +from imblearn.pipeline import Pipeline as ImbPipeline # Note: different import! + +# SMOTE in pipeline (SMOTE + classifier) +pipeline = ImbPipeline([ + ("sampler", SMOTE(random_state=42)), + ("classifier", RandomForestClassifier(n_estimators=200, random_state=42)), +]) +``` + +--- + +## Calibration + +```python +from sklearn.calibration import CalibratedClassifierCV + +# Most sklearn classifiers output uncalibrated probabilities +# Calibrate after training for reliable probability estimates + +# Method 1: Platt scaling (sigmoid) — default, good for SVMs, boosting +calibrated = CalibratedClassifierCV(model, method="sigmoid", cv=5) +calibrated.fit(X_train, y_train) +probabilities = calibrated.predict_proba(X_test) + +# Method 2: Isotonic regression — non-parametric, needs more data +calibrated = CalibratedClassifierCV(model, method="isotonic", cv=5) +``` + +--- + +## Dimensionality Reduction + +### PCA + +```python +from sklearn.decomposition import PCA + +pca = PCA(n_components=0.95) # Keep 95% of variance +X_pca = pca.fit_transform(X_scaled) + +print(f"Components: {pca.n_components_}") # How many components retained +print(f"Explained variance: {pca.explained_variance_ratio_}") +``` + +**PCA assumptions:** Data should be scaled first (use `StandardScaler`). PCA assumes linear relationships. PCA is exploratory / descriptive, not inferential — it cannot confirm a hypothesis. + +### t-SNE / UMAP (Visualization Only) + +```python +from sklearn.manifold import TSNE + +tsne = TSNE(n_components=2, perplexity=30, random_state=42) +X_tsne = tsne.fit_transform(X_scaled) +``` + +**⚠️ t-SNE is for visualization only.** The embedding is stochastic and non-parametric. Different runs produce different results. Do not use t-SNE embeddings as input to other models. + +--- + +## Feature Selection + +```python +from sklearn.feature_selection import ( + SelectKBest, + SelectFromModel, + RFE, + mutual_info_classif, + chi2, +) + +# Filter method (fast, univariate) +selector = SelectKBest(mutual_info_classif, k=20) +X_selected = selector.fit_transform(X, y) + +# Wrapper method (RFE — Recursive Feature Elimination) +selector = RFE(estimator=RandomForestClassifier(), n_features_to_select=20) +X_selected = selector.fit_transform(X, y) + +# Embedded method (from model coefficients) +selector = SelectFromModel( + LogisticRegression(C=1.0, max_iter=1000, penalty="l1", solver="libao"), + max_features=20, + threshold="median", +) + +# In a pipeline +pipeline = Pipeline([ + ("scaler", StandardScaler()), + ("feature_selection", SelectKBest(mutual_info_classif, k=20)), + ("classifier", RandomForestClassifier(n_estimators=200)), +]) +``` + +--- + +## Common Pitfalls + +| Pitfall | Symptom | Fix | +|---|---|---| +| Data leakage from preprocessing | Overly optimistic CV scores | Always use `Pipeline` for preprocessing | +| `OneHotEncoder` creates too many features | High-dimensional sparse matrix | Use `min_frequency=0.01` to group rare categories | +| `KNNImputer` on unscaled data | Poor imputation | Scale before imputing | +| `GridSearchCV` on entire parameter space | Search takes days | Use `RandomizedSearchCV` for > 5 params | +| Using `PCA` before train/test split | Data leakage | PCA in pipeline, fitted on training only | +| `stratify` parameter in `train_test_split` | Uneven class distribution in splits | Always `stratify=y` for classification | +| Not setting `random_state` | Non-reproducible results | Set `random_state=42` on every estimator | +| `joblib.load` from untrusted source | Code execution vulnerability | Only load models you trained | + +--- + +## Reproducibility + +```python +import numpy as np + +# Set random state on every estimator +model = RandomForestClassifier(n_estimators=200, random_state=42) + +# Set numpy seed for reproducibility in preprocessing +np.random.seed(42) + +# Use the same seed in train_test_split and CV +from sklearn.model_selection import train_test_split, KFold +X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42) +cv = KFold(n_splits=5, shuffle=True, random_state=42) +``` + +--- + +## See Also + +- `references/experimental-campaign-protocol.md` — where this reference fits in the campaign workflow (Baseline phase) +- `references/pytorch-integration.md` — for deep learning methods +- `references/data-science-coding-workflow.md` — project structure, experiment logging +- scikit-learn.org/stable/user_guide — official user guide diff --git a/data-scientist/scripts/test_references_completeness.sh b/data-scientist/scripts/test_references_completeness.sh new file mode 100644 index 0000000..984be1f --- /dev/null +++ b/data-scientist/scripts/test_references_completeness.sh @@ -0,0 +1,149 @@ +#!/usr/bin/env bash +# test_references_completeness.sh — Validate the researched code integration references +# +# Checks: +# - All three reference files exist +# - Each covers the required topic areas +# - Each cross-references the parent skill +# - Source URLs are documented + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_DIR="$(cd "$SCRIPT_DIR/.." && pwd)" +PASS=0 +FAIL=0 + +test_case() { + local name="$1" + shift + echo " TEST: $name" + if "$@" 2>/dev/null; then + echo " ✓ PASS" + PASS=$((PASS + 1)) + else + echo " ✗ FAIL" + FAIL=$((FAIL + 1)) + fi +} + +echo "══════════════════════════════════════════════" +echo "Code Integration References Test Suite" +echo "══════════════════════════════════════════════" +echo "" + +# ── File existence ─────────────────────────────────────────────── +echo "── File Existence ──────────────────────────" +echo "" + +test_case "pytorch-integration.md exists" \ + test -f "$REPO_DIR/references/pytorch-integration.md" +test_case "sklearn-integration.md exists" \ + test -f "$REPO_DIR/references/sklearn-integration.md" +test_case "data-science-coding-workflow.md exists" \ + test -f "$REPO_DIR/references/data-science-coding-workflow.md" + +# ── PyTorch reference coverage ───────────────────────────────── +echo "" +echo "── PyTorch Reference Coverage ─────────────" +echo "" + +PT="$REPO_DIR/references/pytorch-integration.md" + +for topic in "Device Management" "Training Loop" "DataLoader" "AMP" "Mixed Precision" \ + "torch.compile" "Transfer Learning" "LoRA" "Knowledge Distillation" \ + "Model Pruning" "DDP" "Distributed Data" "Reproducibility"; do + test_case "Covers: $topic" grep -qi "$topic" "$PT" +done + +test_case "Has device pattern (cuda/mps/cpu)" \ + grep -q "torch.device" "$PT" +test_case "Has gradient clipping" \ + grep -q "clip_grad_norm" "$PT" +test_case "Has model saving/loading pattern" \ + grep -q "torch.save\|model.state_dict" "$PT" +test_case "Has loss function table" \ + grep -q "CrossEntropyLoss\|BCEWithLogitsLoss\|MSELoss" "$PT" +test_case "Has learning rate schedulers" \ + grep -q "ReduceLROnPlateau\|CosineAnnealingLR\|OneCycleLR" "$PT" +test_case "Has debugging section" \ + grep -q "Debugging\|Common Failure" "$PT" +test_case "Source validated date present" \ + grep -q "Last reviewed\|Source validated" "$PT" +test_case "References pytorch.org" \ + grep -q "pytorch.org" "$PT" +test_case "References experimental-campaign-protocol" \ + grep -q "experimental-campaign-protocol" "$PT" + +# ── sklearn reference coverage ───────────────────────────────── +echo "" +echo "── Scikit-Learn Reference Coverage ────────" +echo "" + +SK="$REPO_DIR/references/sklearn-integration.md" + +for topic in "Pipeline" "ColumnTransformer" "Preprocessing" "Model Selection" \ + "Cross-Validation" "GridSearchCV" "Ensemble" "Calibration" \ + "Imbalanced" "Feature Selection" "PCA" "Custom Estimator" \ + "Persistence" "Reproducibility"; do + test_case "Covers: $topic" grep -qi "$topic" "$SK" +done + +test_case "Has ColumnTransformer example" \ + grep -q "ColumnTransformer" "$SK" +test_case "Has OneHotEncoder + StandardScaler" \ + grep -q "OneHotEncoder\|StandardScaler" "$SK" +test_case "Has imputation (SimpleImputer/IterativeImputer)" \ + grep -q "SimpleImputer\|IterativeImputer" "$SK" +test_case "Has HalvingGridSearchCV" \ + grep -q "HalvingGridSearchCV" "$SK" +test_case "Has Random Forest example" \ + grep -q "RandomForest" "$SK" +test_case "Has XGBoost/LightGBM integration" \ + grep -q "XGBClassifier\|LGBMClassifier" "$SK" +test_case "Source validated date present" \ + grep -q "Last reviewed\|Source validated" "$SK" +test_case "References scikit-learn.org" \ + grep -q "scikit-learn.org" "$SK" +test_case "References experimental-campaign-protocol" \ + grep -q "experimental-campaign-protocol" "$SK" + +# ── DS Coding Workflow coverage ──────────────────────────────── +echo "" +echo "── DS Coding Workflow Coverage ────────────" +echo "" + +WF="$REPO_DIR/references/data-science-coding-workflow.md" + +for topic in "Project Directory" "Configuration" "Experiment Logging" \ + "MLflow" "Result Serialization" "Reproducibility" \ + "Data Versioning" "Unit Testing" "Docker" "Seed"; do + test_case "Covers: $topic" grep -qi "$topic" "$WF" +done + +test_case "Has directory structure layout" \ + grep -q "data/raw/\|data/processed/" "$WF" +test_case "Has MLflow example" \ + grep -q "mlflow" "$WF" +test_case "Has DVC reference" \ + grep -q "dvc\|DVC" "$WF" +test_case "Has JSON experiment log pattern" \ + grep -q "experiment_log\.json\|json" "$WF" +test_case "Has reproducibility section" \ + grep -q "Reproducibility\|random_state\|set_all_seeds" "$WF" +test_case "Has pitfalls table" \ + grep -q "Pitfall\|pitfall" "$WF" +test_case "Source validated date present" \ + grep -q "Last reviewed\|Source validated" "$WF" +test_case "References experimental-campaign-protocol" \ + grep -q "experimental-campaign-protocol" "$WF" + +# ── Summary ────────────────────────────────────────────────────── +echo "" +echo "══════════════════════════════════════════════" +echo "Results: $PASS passed, $FAIL failed" +echo "══════════════════════════════════════════════" + +if [ "$FAIL" -gt 0 ]; then + exit 1 +fi