Preview — Pro guide
You are seeing a portion of this guide. Sign in and upgrade to unlock the full article, quizzes, and interview answers.
Sections
Related Guides
How to Approach Data & Product Scenario Questions
Scenarios
Scenario Walkthrough: Engagement vs Revenue — Guardrails & Horizon
Scenarios
A/B Testing & Experimentation at Scale
Machine Learning
Statistics & Probability Foundations
Machine Learning
A/B Test Critique: Finding Flaws in Experiment Designs
Production Engineering
Scenario Walkthrough: Post-Launch — Was This Feature a Success?
Scenarios
Scenario Walkthrough: Marketplace Supply–Demand Imbalance — Liquidity First
Scenarios
Scenario Walkthrough: The A/B Test Went Wrong — SRM, Peeking, and Interference
When the experiment is lying but the slide deck is green. Walk the failure modes that dominate production: sample ratio mismatch, peeking and early stopping, novelty and learning effects, network interference, wrong randomization unit, and thin-event metrics read too early. Teaches the remediation playbook: invalidate, debug assignment, or redesign — and how to say that without panicking the room. Grounded in standard OCE practice, chi-square SRM, and the Kohavi, Tang, and Xu body of work on trustworthy experiments.
The Question Under the Question
A broken test in interviews is usually one of a few classes. (1) Statistics — peeking, optional stopping, many metrics without multiplicity control, or reading a thin revenue line on a short window. (2) Engineering — SRM from bad assignment, eligibility applied after the bucket, redirect drops, or treatment-induced dropout. (3) Product — novelty in week one that inverts later. (4) Interference — shared inventory, ad auctions, or feeds so user-level SUTVA fails.
Weak candidates treat a small p-value as a ship bit. Strong candidates stop when identification breaks, run the SRM check before hero metrics, and separate monitoring from a pre-registered launch readout. They price a re-run or redesign against a bad OKR that can live for quarters, not against looking busy in SQL.