How to Approach Data & Product Scenario Questions

Scenario questions give you an open-ended business situation and ask how you'd respond. Unlike SQL or ML theory questions, there's no single correct answer. The interviewer is evaluating your reasoning process, not just your conclusion.

Common scenario formats:

• Metric drop investigation: "Our DAU dropped 15% overnight. Walk me through your investigation."
• Model degradation: "The fraud model's precision fell from 92% to 71% in production. What do you do?"
• Ship-or-not decision: "The A/B test shows a 3% lift in checkout completions but a 1% drop in NPS. Do you ship?"
• Incident triage: "We have a P0 incident. The cart service is returning 500s. Walk me through your response."
• Feature design: "Design a metric system for the content recommendation team."

What the interviewer is evaluating:

1. Structure: do you have a systematic approach, or do you guess randomly?
2. Hypotheses before data: do you form a hypothesis tree before pulling queries?
3. Breadth: do you consider multiple explanations (data pipeline, external factors, product change, fraud)?
4. Business judgment: do you connect your findings to a business recommendation?
5. Communication: can you narrate your thinking out loud clearly?

Scenario questions are one of the highest-signal interview formats because they can't be memorized. They test judgment developed through real analytical experience.

Before investigating causes, get precise about what changed.

Define the signal (D):

• What metric dropped/spiked? (DAU, revenue, conversion rate, model precision)
• By how much? Is 15% drop large or small for this metric's normal variance?
• When did it start? Gradually over days, or a step change at a specific time?
• Which segments are affected? All users or a subset? All platforms or one?
• What is the normal baseline? Is this a seasonal trough or a genuine drop?

Check the instrument (I) — always do this first: Approximately 30–40% of "metric drops" in production are actually data pipeline failures, not real product changes. Check this before investigating business causes.

Instrument checks to run:

1. Is the logging system healthy? Check event ingestion lag, error rates in the pipeline.
2. Compare across data sources: does BigQuery show the same drop as the Kafka consumer?
3. Check recent deployments to the logging/analytics layer specifically.
4. Look at upstream dependencies: did the app stop sending events or did events start failing?
5. Check if the issue is time-zone localized (common with UTC vs local time bugs).

If the instrument check passes (data is trustworthy), proceed. If it fails, the "incident" is a data pipeline bug — escalate to data engineering and scope the data gap.

Form a complete hypothesis tree before writing a single query. This is the most differentiating step — it shows you think before you act.

Standard hypothesis tree for a metric drop:

Internal causes (things you control): Product changes: new feature, A/B test launched, UI change, paywall added Engineering changes: deployment, config change, feature flag flipped Data/instrumentation: logging bug, pipeline outage, schema change Algorithm changes: ranking model update, recommendation change

External causes (things outside your product): Seasonality: day of week, holiday, back-to-school, product cycle Competitor action: competitor launched a feature, had an outage (your users migrating back) Platform change: iOS/Android OS update, App Store policy, browser update Macroeconomic: news event, regulatory change, market disruption

Prioritize hypotheses before querying:

• "We deployed a new checkout flow 3 hours before the drop" → investigate that first
• "It's a Sunday in Q4" → check if this matches historical Sunday patterns first
• "The logging team had a config change yesterday" → check instrument first

Generating diagnostic queries:

• Segment by platform (iOS vs Android vs web) — if drop is iOS-only, look at iOS deployment
• Segment by user type (new vs returning) — if drop is new-user-only, look at acquisition funnel
• Segment by geography — if drop is US-only, look for US-specific events
• Compare to same time last week/month — is this seasonal?
• Funnel analysis: where in the conversion path are users dropping off?

Narrowing to root cause: use a process of elimination, not trial and error.

If segmentation shows: iOS drop = 40%, Android drop = 2%, Web drop = 1% → This is iOS-specific. Investigate iOS deployment, iOS version update, App Store listing.

If funnel analysis shows: signups unchanged, but checkout completions dropped 20% → The drop is in the checkout step. Investigate checkout changes, payment provider.

If cohort comparison shows: new users dropped, returning users unchanged → The drop is in acquisition or first-session experience. Investigate ad campaigns, onboarding flow, landing pages.

Once you have the root cause, quantify the impact:

• How many users are affected? (X% of DAU × user base size)
• What is the revenue impact? ($Y per affected user × affected user count)
• Is this P0 (system down), P1 (major regression), or P2 (minor regression)?

Own a recommendation:

• If it's a bug: "Roll back the deployment immediately. Estimated fix time is X hours. Estimated impact at current rate is $Y/hour of continued outage."
• If it's a feature regression: "Disable the feature flag. Scope the full user impact before re-enabling. Schedule post-mortem."
• If it's external/seasonal: "No action required. Set up an alert for the same pattern next season. Update the DAU forecast model."
• If it's still unclear: "I need 2 more days of data to confirm. In the meantime, I'd monitor hourly and escalate if the drop accelerates."