magnus919_agent-skills/data-scientist/references/statistical-methodology.md

Statistical Methodology Reference

Test Selection Decision Tree

What type of outcome (dependent variable)?
│
├─ CONTINUOUS (interval/ratio)
│  ├─ 1 group → One-sample t-test (normal) or Wilcoxon signed-rank (non-normal)
│  ├─ 2 independent groups → Independent t-test (normal, equal var) or Welch's t-test (normal, unequal var) or Mann-Whitney U (non-normal)
│  ├─ 2 paired groups → Paired t-test (normal) or Wilcoxon signed-rank (non-normal)
│  ├─ 3+ independent groups → One-way ANOVA (normal, equal var, independent) or Kruskal-Wallis (non-normal)
│  ├─ 3+ paired groups → Repeated measures ANOVA (normal, sphericity) or Friedman test (non-normal)
│  ├─ Controlling for covariates → ANCOVA (normal, equal slopes, independent)
│  └─ Multiple predictors → Linear regression / GLM
│
├─ BINARY (yes/no, success/failure)
│  ├─ 1 group / compare to known proportion → Binomial test / One-proportion z-test
│  ├─ 2 independent groups → Chi-square test of independence or Fisher's exact (small cells) or Two-proportion z-test
│  ├─ 2 paired groups → McNemar's test
│  ├─ 3+ groups → Chi-square test
│  ├─ Multiple predictors → Logistic regression
│  └─ Rare outcome → Logistic regression with Firth correction or exact methods
│
├─ COUNT (0, 1, 2, 3...)
│  ├─ Unbounded → Poisson regression (mean = variance) or Negative binomial (variance > mean)
│  ├─ Excess zeros → Zero-inflated Poisson/NB or Hurdle model
│  └─ Over time → Poisson/NB with offset for exposure
│
├─ ORDINAL (Likert, ranked)
│  ├─ Compare 2 groups → Mann-Whitney U (independent) or Wilcoxon signed-rank (paired)
│  ├─ Compare 3+ groups → Kruskal-Wallis
│  └─ Multiple predictors → Ordinal logistic regression (proportional odds)
│
├─ TIME-TO-EVENT (survival)
│  ├─ Compare 2 groups → Log-rank test
│  ├─ Multiple predictors → Cox proportional hazards
│  └─ Proportional hazards violated → Accelerated failure time models
│
├─ CATEGORICAL (nominal, 3+ levels)
│  ├─ Outcome is categorical → Chi-square test of independence or Multinomial logistic regression
│  └─ Agreement between raters → Cohen's kappa (2 raters) or Fleiss' kappa (3+)
│
├─ TIME SERIES (repeated over time)
│  ├─ Single series forecasting → ARIMA, Exponential smoothing, Prophet
│  ├─ Multiple series comparison → Structural time series, Dynamic regression
│  ├─ Seasonal patterns → Seasonal decomposition (STL), SARIMA
│  └─ Anomaly detection → Change point detection, Twitter/AnomalyDetection
│
└─ MULTIVARIATE (multiple outcomes)
    ├─ Dimensionality reduction → PCA, t-SNE, UMAP, Factor analysis
    ├─ Group structure → Cluster analysis (k-means, hierarchical, DBSCAN)
    └─ Multiple DVs by group → MANOVA (normal, equal cov matrices)

---

Assumptions Reference Table

Method	Assumptions	How to Check	What If Violated
One-sample t-test	Independence, normality	Q-Q plot, Shapiro-Wilk test (n < 50), Anderson-Darling	Wilcoxon signed-rank test
Independent t-test	Independence, normality, equal variance	Levene's test, F-test of variances, Q-Q plot	Welch's t-test (unequal var), Mann-Whitney (non-normal)
Paired t-test	Normality of differences, independence of pairs	Q-Q plot of differences	Wilcoxon signed-rank test
One-way ANOVA	Independence, normality (within groups), equal variance (homoscedasticity)	Levene's test, Shapiro-Wilk per group, Q-Q plot, residual plot	Welch's ANOVA (unequal var), Kruskal-Wallis (non-normal)
Repeated measures ANOVA	Normality, sphericity (equal variances of differences), independence between subjects	Mauchly's test for sphericity, ε correction (Greenhouse-Geisser, Huynh-Feldt)	Friedman test, or use mixed effects model with unstructured covariance
Linear regression	Linearity, independence of errors, homoscedasticity, normality of residuals, no multicollinearity	Residuals vs fitted plot, Q-Q plot, Breusch-Pagan test, VIF (< 5-10), Durbin-Watson, Cook's distance for influential points	Robust SEs (heteroscedasticity), weighted least squares, transformations (non-linearity), GLS (correlated errors)
Logistic regression	Linearity in logit (continuous predictors independent of log-odds), independence	Box-Tidwell test (linearity), Hosmer-Lemeshow goodness-of-fit, AUC-ROC, residual plots	Splines or polynomials (non-linearity), Firth regression (rare events/separation)
Chi-square test	Expected frequency ≥ 5 in each cell, independence of observations	Check expected counts	Fisher's exact test (2x2), Monte Carlo simulation (larger tables)
Cox proportional hazards	Proportional hazards, independent censoring	Schoenfeld residuals test (global + per covariate), log-log plots	Time-dependent covariates, stratified Cox, AFT models
ANCOVA	Normality, equal variance, independence, linear relationship with covariate, equal slopes assumption	Homogeneity of slopes test (covariate × group interaction)	Add interaction term, use nonparametric ANCOVA (Quade)
MANOVA	Multivariate normality, equal covariance matrices, independence	Box's M-test, Q-Q plots per variable	Separate ANOVAs with Bonferroni, PERMANOVA

---

Effect Size Guide

Test / Design	Effect Size Measure	Interpretation	CI Available
t-test (independent)	Cohen's d = (M₁−M₂)/s_pooled	0.2=small, 0.5=medium, 0.8=large	Yes (non-central t)
t-test (paired)	Cohen's d_z = t/√n	Same conventions	Yes
ANOVA (one-way)	η² = SS_between/SS_total	0.01=small, 0.06=medium, 0.14=large	Yes
ANOVA	Partial η²	Same (for multifactor designs)	Yes
ANOVA (fixed effects)	Cohen's f = √(η²/(1−η²))	0.10=small, 0.25=medium, 0.40=large	Yes
Chi-square	Cramér's V = √(χ²/(n·min(r−1,c−1)))	0.1=small, 0.3=medium, 0.5=large	Yes (bootstrap)
2×2 tables	Odds ratio	OR=1 no effect, OR>1 increased odds	Yes (Woolf)
2×2 tables	Risk ratio / Relative risk	RR=1 no effect	Yes (Katz)
2×2 tables	Risk difference	Absolute difference in proportions	Yes (Newcombe)
Correlation	r (Pearson)	0.1=small, 0.3=medium, 0.5=large	Yes (Fisher z)
Correlation	r_s (Spearman)	Same conventions	Yes
Regression	R²	Variance explained	Yes
Regression	Cohen's f² = R²/(1−R²)	0.02=small, 0.15=medium, 0.35=large	Yes
Bayesian	Bayes factor BF₁₀	1-3=weak, 3-10=moderate, 10-30=strong, 30-100=very strong, >100=extreme evidence for H₁	Yes (HDI)

Reporting Examples

• "The mean difference was 3.2 points (95% CI [1.8, 4.6]), t(58) = 3.41, p = 0.001, Cohen's d = 0.87."
• "The treatment group had 2.3× the odds of recovery (OR 2.3, 95% CI [1.4, 3.8], χ²(1) = 12.4, p < 0.001)."
• "The model explained 34% of variance in outcome (R² = 0.34, F(3, 96) = 16.5, p < 0.001, Cohen's f² = 0.52)."

---

Multiple Testing Corrections

Correction	Use When	How It Works	Power
Bonferroni	Small number of planned comparisons	α/m	Low — conservative
Holm-Bonferroni	Same, slightly less conservative	Sequential rejective	Better than Bonferroni
Benjamini-Hochberg (BH)	Exploratory analysis, many tests	Controls FDR	Higher — recommended for omics/exploratory
Benjamini-Yekutieli (BY)	Dependent tests, same as BH	Controls FDR under dependency	Lower than BH
Tukey HSD	All pairwise comparisons after ANOVA	Studentized range distribution	Good, designed for this case
Dunnett	Multiple comparisons vs a single control	Comparison-specific critical values	Good for this case
Scheffé	Post-hoc contrasts not planned in advance	Most flexible, most conservative	Low
FDR (q-value)	Thousands of tests (genomics, fMRI)	Estimates proportion of false discoveries	High

Rule of thumb: For 2-5 planned comparisons, use Bonferroni or Holm. For dozens of exploratory tests, use BH at FDR=0.05. For pairwise post-ANOVA, use Tukey HSD. Never cherry-pick which p-values to correct.

---

Bayesian Alternatives for Common Frequentist Tests

Frequentist	Bayesian Alternative	Key Benefit
One-sample t-test	Bayesian one-sample t-test (BEST)	Can quantify evidence for H₀ via BF
Two-sample t-test	Bayesian two-sample t-test (BEST)	Robust to outliers via heavy-tailed likelihood
ANOVA	Bayesian ANOVA (BANOVA)	Model comparison via BFs, no sphericity assumption
Linear regression	Bayesian linear regression	Prior regularization, full posterior for coefficients
Logistic regression	Bayesian logistic regression (with priors)	Handles separation, shrinks extreme estimates
Chi-square test	Beta-Binomial model, contingency table Bayes factor	More intuitive: what's the posterior difference in proportions?
Correlation	Bayesian correlation (beta* prior on ρ)	Posterior distribution of ρ
t-test with non-inferiority	Bayesian region of practical equivalence (ROPE)	Direct probability of clinically meaningful difference
Meta-analysis	Bayesian hierarchical meta-analysis	Handles heterogeneity, small studies better

---

Best Practices & Red Flags

Green Flags (good analysis practices)

• Pre-registered analysis plan (when possible)
• Effect sizes with CIs reported alongside p-values
• Assumption checks documented (and violations addressed)
• Sensitivity analyses reported (different specifications, outlier exclusion)
• Code and data available for reproduction
• Multiple testing corrections applied where appropriate
• Missing data mechanism discussed (MCAR, MAR, MNAR)
• Power analysis conducted before data collection

Red Flags (poor analysis practices)

• p-values without effect sizes
• Stepwise variable selection (forward/backward)
• P-hacking: trying tests until significance
• Ignoring violations of assumptions
• No missing data handling (or claiming "no missing data" unrealistically)
• Over-interpreting non-significant results as "no effect" without equivalence testing
• Reporting only significant results (cherry-picking)
• Using parametric tests on clearly non-normal data without justification
• "p = 0.06 is marginally significant" (it's not — p = 0.06 is non-significant)