Preview — Pro guide
You are seeing a portion of this guide. Sign in and upgrade to unlock the full article, quizzes, and interview answers.
Sequential Testing & the Peeking Problem: Alpha Spending, SPRT, and Always-Valid Inference
Product teams peek at A/B tests daily — but naive repeated significance testing inflates Type I error from ~5% to ~30% or higher. This topic covers alpha spending functions, group sequential designs, SPRT intuition, and production platforms (Optimizely Stats Engine, Statsig, Eppo) that deliver always-valid confidence sequences so you can monitor experiments without lying about significance.
Why Peeking Breaks Fixed-Horizon Inference
In a classical fixed-horizon test you pre-commit to sample size and a single analysis at the end. Under that contract, rejecting when controls the false positive rate at *when you actually wait until *. The moment you compute -values on accumulating data every day — or stop the experiment the first time — you have changed the stopping rule. The test statistic is no longer evaluated at one random time; it is the **minimum** over many random times. That distribution is stochastically smaller than the nominal under the null, so the **effective** explodes. Interviewers at Meta, Stripe, and Airbnb routinely probe whether you understand that **monitoring is not free**. The fix is not "never look" (business needs velocity) but **sequential inference**: procedures whose validity holds under optional stopping.