Scenario Walkthrough: Engagement vs Revenue — Guardrails & Horizon

Engagement (time-on-site, sessions, DAU) is a manipulable input on short horizons. Revenue and profit are often lagging and carry variance that a 7-day A/B is underpowered to pin down. A feed team can maximize watch-time in ways that destroy ad inventory quality, push down RPM, and increase refunds—sometimes while raising a naive CTR.

Strong interview answers refuse a single-hero-metric story. They define an OEC (overall evaluation criterion) with explicit guardrails—a maximum acceptable regression on ARPU, refund rate, trust-and-safety incidents, and latency—and they talk about horizon: the experiment duration needed before you would even expect to see certain harms. Weak answers say "we will monitor revenue" without a stopping rule, a duration, and an owner.

Goodhart's law is not philosophy club—it is a release-review reality. If PMs or ML teams are bonused on a single engagement scalar, the system (human + model) will find degenerate equilibria: endless notifications, low-quality recency bias, or dark-pattern friction removal that increases short clicks and refunds.

Production orgs answer with a stack: a primary (often a growth or experience objective), 2–4 guardrails with redlines ("do not launch if..."), and diagnostic metrics to explain mechanism (ad CTR, ad depth, time-to-first-ad, return rate, chargeback rate, CS contacts). The interview wants you to name a mechanism when engagement and money diverge—composition (low-value time), auction pressure, or ad relevance—not vibes.

A single-surface win can be a portfolio loss: more time in one tab that steals from higher-RPM moments, a checkout experiment that steers bad margin SKUs, or a ranking change that overweights cheap UGC and hollows the premium supply your ads target.

Staff candidates say cannibalization out loud and ask whether the A/B is isolated or portfolio-interfering. If the same user sees multiple experiments or multiple ranking objectives, the SUTVA (stable unit treatment value assumption) story breaks; you need stricter readouts, switchback-style or stratified designs for network-heavy products, and longer washout when treatment effects are sticky.

CUPED (Deng, Xu, et al. at Microsoft) reduces variance in A/B by using a pre-period covariate; it is standard vocabulary for senior DS. The interview point is not the formula—it is: "We should not pretend a noisy 7d ARPU with wide CIs is a license to ship if the guardrail is a safety constraint."

What impresses: you separate statistical significance on a short window from business sufficiency. If ARPU is a hard guardrail, you either extend the test, raise power with CUPED/covariates, or reframe the launch as staged with automatic rollback. You do not hand-wave "we will watch monthly ARPU" without an owner and a trigger.