Preview — Pro guide
You are seeing a portion of this guide. Sign in and upgrade to unlock the full article, quizzes, and interview answers.
Bayesian A/B Testing vs Frequentist: Priors, Posteriors, Probability of Superiority, and Expected Loss
Bayesian experimentation reports P(treatment beats control | data) and expected regret — intuitive for executives — but priors, ROPE, and MCMC diagnostics create new failure modes. This guide contrasts Thompson sampling, Beta-Binomial conjugate updates, decision rules based on expected loss, and when frequentist fixed-n tests remain the compliance-safe choice.
Why Teams Reach for Bayesian Language in A/B Tests
Frequentist outputs: -value, confidence interval, reject/retain at fixed . These answer **procedure** questions: "How surprising is this data if the null were true?" Bayesian outputs: **Posterior** , **probability treatment is better**, **expected regret** if you ship the wrong arm. These align with how PMs think: "What is the chance B beats A, *given what we have seen so far*?" The tradeoff: Bayesian answers depend on **priors** and (for non-conjugate models) computation. Interviewers test whether you can articulate **what changes** in ship decisions — not just relabeling a -test "Bayesian."