magnus919_agent-skills/data-scientist/references/bayesian-workflow.md

# Bayesian Workflow Reference

## The Bayesian Workflow (Gelman et al. 2020)

Bayesian analysis is not "put a prior on everything and sample." It's an iterative process:

```
1. Model building
   │
   ├─ 2. Prior predictive check (do priors produce plausible data?)
   │  └─ If not, revise priors
   │
   ├─ 3. Inference (MCMC, VI, or exact)
   │
   ├─ 4. Posterior predictive check (does model reproduce observed data?)
   │  └─ If not, revise model
   │
   ├─ 5. Model comparison (WAIC, LOO, cross-validation)
   │
   └─ 6. Sensitivity analysis (try different priors, different likelihoods)
```

---

## 1. Prior Elicitation

### Types of Priors

| Prior Type | Description | When to Use |
|-----------|-------------|-------------|
| **Flat / Improper** | No information, ∝ 1 | Default reference, rarely recommended |
| **Weakly Informative** | Wide but finite: Normal(0, 10), Cauchy(0, 5) | Default for most parameters; shrink extreme values |
| **Regularizing** | Shrink toward zero: Normal(0, 1), Laplace(0, 1) | Many predictors, prevents overfitting |
| **Informative** | Narrowly centered on known value | Strong prior knowledge (previous studies, physical constraints) |
| **Hierarchical** | Prior on prior parameters (hyperpriors) | Multiple groups sharing information |

### Prior Recommendation Cheat Sheet

| Parameter Type | Default Prior | Why |
|---------------|--------------|-----|
| **Regression coefficient** (continuous predictor) | Normal(0, 1) or Normal(0, 2.5) | 95% of coefficients within ±2-5 units on logit/probit scale |
| **Intercept** | Normal(0, 10) | Wide enough for most scales |
| **Standard deviation** (positive) | Half-Cauchy(0, 2.5), Exponential(1), or Inverse-Gamma(3, 3) | Half-Cauchy preferred; avoids near-zero mass of inverse-gamma |
| **Correlation** (between -1 and 1) | LKJ(2) | Slightly regularizes toward zero correlation |
| **Variance parameter** (hierarchical) | Exponential(1) or Half-Cauchy(0, 2) | Robust to scale differences |
| **Proportion / probability** | Beta(1, 1) or Beta(2, 2) | Uniform vs slightly central |
| **Log-odds** (logistic) | Normal(0, 1.5) or Student-t(3, 0, 2.5) | Gelman et al. 2008: default for logistic regression |

### Prior Predictive Check Protocol

1. Sample from the prior distribution
2. Generate synthetic data from the prior-predictive distribution
3. Plot the distribution of synthetic data
4. Ask: "Could data generated this way actually occur in my problem?"
5. If the synthetic data implies impossible values (negative variances, 150-year-old humans, etc.): tighten the prior
6. If the synthetic data is too constrained (never allows realistic values): widen the prior

---

## 2. Computation

### MCMC Methods

| Method | Software | When to Use |
|--------|----------|-------------|
| **Hamiltonian Monte Carlo (HMC)** | Stan (via PyStan, CmdStanPy), PyMC (NUTS), NumPyro (NUTS) | Default for continuous parameters. Efficient exploration of posterior. |
| **Variational Inference (VI)** | PyMC (ADVI), NumPyro (SVI), Stan (VB) | Large datasets, fast approximate inference. Check fit against HMC on subset. |
| **Gibbs Sampling** | JAGS, BUGS | Legacy. Only when HMC is impractical. |
| **SMC / ABC** | PyMC (SMC), custom | Discrete parameters, likelihood-free inference |
| **Laplace Approximation** | INLA, statsmodels | Fast, accurate for latent Gaussian models (GLMM, spatial) |

### MCMC Diagnostics

| Diagnostic | What It Checks | Threshold | How to Fix |
|-----------|---------------|-----------|------------|
| **R-hat (Ř)** | Convergence across chains | < 1.01 (modern threshold, Vehtari et al. 2021) | Run longer, reparameterize, increase warmup |
| **Effective Sample Size (ESS)** | Number of independent samples | bulk-ESS > 400, tail-ESS > 400 | Run longer, thinner |
| **Trace plot** | Mixing, stationarity | No trends, good overlap between chains | Re-run with more iterations |
| **Autocorrelation** | Sampling efficiency | Lag-1 autocorrelation < 0.5 | Re-parameterize (centered vs non-centered) |
| **Divergent transitions** | HMC-specific: geometry problems | 0 divergences | Increase adapt_delta (0.80 → 0.95-0.99), reparameterize |
| **BFMI (Bayesian Fraction of Missing Information)** | Energy transition efficiency | BFMI > 0.2-0.3 | Re-parameterize non-centered |

### Reparameterization: Centered vs Non-Centered

| Type | Formula | When |
|------|---------|------|
| **Centered** | μ_normal = α + βX, σ ~ Half-Cauchy | Weak data, strong priors |
| **Non-centered** | μ_raw ~ Normal(0, 1), μ = α + βX + σ·μ_raw | Strong data, weak priors, hierarchical models |

Non-centered breaks the dependence between group-level mean and variance parameters, improving HMC geometry. Try non-centered when you see divergent transitions.

---

## 3. Model Comparison

### Information Criteria

| Criterion | What It Measures | When to Use | Formula |
|-----------|-----------------|-------------|---------|
| **WAIC** (Watanabe-Akaike) | Expected log pointwise predictive density + correction | Bayesian models, MCMC | WAIC = lppd − p_waic |
| **LOO-CV** (Leave-One-Out) | Same as WAIC but with Pareto-smoothed importance sampling | Bayesian models, MCMC (via `loo()` in ArviZ, loo package) | Computed via PSIS-LOO |
| **AIC** | −2 logL + 2k | Frequentist models, no prior | — |
| **BIC** | −2 logL + k log n | Frequentist models, nested comparison | — |
| **DIC** | Deviance + effective parameters | Bayesian models, MCMC (less stable than WAIC) | — |

### Model Comparison Decision Tree

```
Compare candidate models:
│
├─ Same likelihood, same data, nested?
│  ├─ Use information criteria (WAIC/LOO) for predictive comparison
│  └─ Use cross-validation for predictive accuracy
│
├─ Different likelihoods (e.g., Poisson vs Negative Binomial)?
│  └─ Use WAIC/LOO on same data, or cross-validation of predictive score
│
├─ Need to quantify evidence for H₀ vs H₁?
│  └─ Use Bayes Factor (BF₁₀) — but sensitive to prior choice. Report sensitivity.
│
└─ Which model is more useful given decision context?
    └─ Consider predictive performance (CV), interpretability, computational cost
```

### Bayes Factor Guidelines

| BF₁₀ | Evidence for H₁ |
|------|----------------|
| 1-3 | Weak / Anecdotal |
| 3-10 | Moderate |
| 10-30 | Strong |
| 30-100 | Very Strong |
| >100 | Extreme |

**Warning:** Bayes factors are very sensitive to prior choice. A diffuse prior can arbitrarily deflate the BF in favor of H₁. Always report sensitivity to prior width.

### Cross-Validation in Bayesian Models

- **Approximate LOO-CV** via PSIS (Pareto-smoothed importance sampling) — built into ArviZ
- **k-fold CV** — run inference on k-1 folds, evaluate on held-out fold. More robust but more expensive
- **Diagnostic:** Pareto k parameter. k < 0.5: reliable. 0.5 < k < 0.7: ok but watch. k > 0.7: unreliable (use k-fold instead)

---

## 4. Posterior Predictive Checking

### Core Idea
If the model is adequate, data generated from the posterior predictive distribution should resemble the observed data.

### Checks

| Check | What It Tests | Visualization |
|-------|--------------|--------------|
| **Mean / variance** | First and second moments | Overlay observed statistic on posterior of statistic from replicated datasets |
| **Extreme values** | Tail behavior | Probability of max(y_rep) ≥ max(y) |
| **Distribution shape** | Whole-distribution fit | Density overlay of y_rep and y |
| **Discrepancy measures** | Specific features you care about | Bayesian p-value (χ², skewness, proportion of zeros) |
| **Residual vs fitted** | Model structure violations | Plot standardized residuals from posterior mean vs fitted values |

### Bayesian p-values

A Bayesian p-value near 0.5 indicates good fit. Values near 0 or 1 indicate systematic discrepancy between model and data. Because it's the posterior probability that the test statistic is more extreme in the replicated data than in the observed, it doesn't have the same issues as frequentist p-values.

---

## 5. Hierarchical / Multilevel Bayesian Models

### Shrinkage

Hierarchical models borrow strength across groups. Group-level estimates are "shrunk" toward the population mean, with the amount of shrinkage determined by the group-level variance:

```
Groups with small N → more shrinkage (less reliable estimates pulled toward mean)
Groups with large N → less shrinkage (more data to estimate their own parameter)
```

### The 8 Schools Example (classic)

The canonical hierarchical Bayesian example: 8 schools each ran an experiment. Estimates vary widely due to small samples. Hierarchical model produces more realistic, shrunken estimates with proper uncertainty.

### When to Go Hierarchical:
- Multi-level data structure (students in classes in schools)
- You want to estimate group-level parameters while sharing information
- Small sample sizes in some groups (partial pooling)
- You need to model variation across groups explicitly

---

## 6. Software Quick Reference

| Task | PyMC | NumPyro | Stan |
|------|------|---------|------|
| Basic regression | `pm.Model()` with `pm.Normal()` | `numpyro.sample()` with `dist.Normal()` | `model { ... }` block |
| MCMC sampling | `pm.sample(1000)` | `mcmc.run(rng_key, ...)` | `./sample` via cmdstan |
| HMC NUTS | Default | `NUTS` | Default |
| Variational inference | `pm.fit(method='advi')` | `SVI` with `AutoDiagonalNormal` | `stan optimize` (VB deprecated) |
| Prior predictive | `pm.sample_prior_predictive()` | `Predictive()` with prior_samples | `generated quantities` block |
| Posterior predictive | `pm.sample_posterior_predictive()` | `Predictive()` with posterior_samples | `generated quantities` block |
| LOO/WAIC | `az.loo()`, `az.waic()` | `az.loo()`, `az.waic()` | `loo::loo()` in R |
| Visualization | ArviZ (`az.plot_trace`, etc.) | ArviZ | `bayesplot`, `shinystan` |

### Installation Notes

- **PyMC:** `pip install pymc arviz`
- **NumPyro:** `pip install numpyro arviz jax jaxlib`
- **Stan (Python):** `pip install pystan` (v3) or `cmdstanpy` (must install CmdStan separately: `install_cmdstan()`)
- **Stan (R):** `install.packages("rstan")`, `brms` for formula-based interface
- **R with Python:** use `rpy2` bridge or `reticulate` in R