Preview — Pro guide
You are seeing a portion of this guide. Sign in and upgrade to unlock the full article, quizzes, and interview answers.
Practical vs Statistical Significance: MDE, Cohen's d, Confidence Intervals, and Business Loss
At large n, trivial lifts become p less than 0.001. Interviewers expect you to separate statistical evidence from business value using minimum detectable effect (MDE), Cohen's d, absolute vs relative lifts, and confidence interval width. This topic ties power analysis to engineering cost and revenue translation — the bar senior DS candidates clear at Stripe, Uber, and DoorDash.
Large n Makes Everything Statistically Significant
**Statistical significance** answers: is the signal larger than sampling noise would typically produce under the null? **Practical significance** answers: is the signal large enough to **change a decision** — ship code, hire headcount, reprice SKUs? With millions of users, a **0.02 percentage point** CTR lift can yield . The lift is real; it may still be **smaller than measurement error** in downstream revenue models or **smaller than the cost** of maintaining the feature. Mature teams pre-define **MDE** — the smallest absolute effect that would justify the change — before power analysis. After the test, they compare the **confidence interval** to MDE, not just the point estimate to zero.