A/B Testing for ML Systems: Design, Statistical Rigor & Production Pitfalls

A/B testing for ML models is not the same as testing a UI change. ML model A/B tests have unique failure modes that invalidate conclusions at almost every mature ML team.

The four ways ML A/B tests mislead:

1. Wrong randomization unit: Randomizing by session when measuring by user creates Simpson's Paradox. Randomizing by user when effects are social/collaborative creates contamination. The randomization unit must match the unit of analysis AND avoid interference between groups.

2. Novelty effect: New model shows different content → users click more in weeks 1–2 simply because content is fresh. The underlying quality hasn't improved. Insufficient test duration (< 3 weeks) ships models based on novelty, not quality.

3. Multiple comparisons: A team running 20 metrics with α=0.05 has a 64% chance of at least one false positive even if the model does nothing. Without correction, teams cherry-pick the one significant metric.

4. Insufficient sample size: Pre-computing sample size is skipped. Test runs for 'a few days' until it 'looks good.' This is p-hacking — stopping when you see significance inflates false positive rate dramatically.

Knowing these failure modes and how to prevent them is what separates candidates who've worked with real experimental infrastructure from those who've only read about A/B testing.

Sample size calculation is the step that most candidates skip and most practitioners get wrong. Here's how to do it correctly for common ML metrics:

For proportions (CTR, conversion rate): n = (z_α/2 + z_β)² × 2p(1-p) / δ²

Where δ = absolute MDE. For α=0.05, power=80%: (z_α/2 + z_β)² = (1.96 + 0.84)² = 7.84.

Practical examples:

• Baseline CTR 5%, MDE 0.5% absolute: n ≈ 20,000 per variant
• Baseline CTR 5%, MDE 0.1% absolute: n ≈ 500,000 per variant
• Baseline conversion rate 2%, MDE 0.1% absolute: n ≈ 2.4M per variant

For continuous metrics (revenue, session length): n = 2σ²(z_α/2 + z_β)² / δ²

Where σ is the standard deviation of the metric. Revenue is typically highly skewed (long tail from high-value users) → high σ → very large n required. CUPED (see below) reduces σ and thus reduces required n.

Key insight: MDE drives sample size quadratically. Halving MDE (detecting 0.5% lift instead of 1%) requires 4× more users. This is the fundamental tension between experiment sensitivity and speed.

CUPED (Controlled-experiment Using Pre-Experiment Data) is Microsoft's technique for dramatically reducing the variance of metric estimates without changing sample size — effectively making experiments 2–4× more sensitive (Deng, Xu, Kohavi, Walker, 2013 — https://exp-platform.com/Documents/2013-02-CUPED-ImprovingSensitivityOfControlledExperiments.pdf).

The insight: Much of the variance in a metric like revenue/session comes from pre-existing user heterogeneity — some users are always high spenders, others always low. This variance is not related to the treatment effect. We can 'subtract out' this pre-existing variance using pre-experiment covariate data.

CUPED adjustment: Y_CUPED = Y - θ × X_pre

Where:

• Y = observed metric during experiment (revenue)
• X_pre = same user's metric in a control period before the experiment
• θ = Cov(Y, X_pre) / Var(X_pre) (regression coefficient)

Variance of Y_CUPED = Var(Y) × (1 - ρ²) where ρ is the correlation between Y and X_pre.

In practice: if a user's past-7-day revenue correlates with their in-experiment revenue at ρ=0.8, CUPED reduces variance by 1 - 0.64 = 36%. Same as running an experiment with 1.6× more users. At scale, CUPED lets teams run experiments 30–60% faster or detect smaller effects.

Implementation: compute X_pre for every user (their metric value in the 7 days before the experiment starts). Compute θ using the holdout set. Apply adjustment per user. Run standard two-sample t-test on Y_CUPED values.

CUPED is used in production at Microsoft, Airbnb (DiD), Netflix, and LinkedIn. It's a credible signal in interviews that you've worked with serious experimentation infrastructure.