Offline vs Online Evaluation: Why Metrics Disagree and What to Do About It

The most common failure pattern in production ML: a model improves AUC by 0.02 in offline evaluation, gets deployed, and CTR drops 5%. Or the opposite: offline metrics are flat, but the A/B test shows a significant positive impact on revenue.

Why offline and online metrics diverge:

1. Distribution shift: Offline evaluation is on historical data collected by the previous model. The new model will see different distributions because it serves different items. The historical data doesn't reflect what the new model will actually show.

2. Position bias: Historical data records which items users clicked, but doesn't record that items in position 1 get ~3× more clicks than position 3, regardless of quality. An offline evaluation dataset contains this bias. A model that memorizes 'items the previous model showed in position 1 tend to get clicked' will score high offline but not improve the actual user experience.

3. Feedback loops: The training data was collected by a model that already had its own biases — popular items were shown more, so they got more clicks, so they became stronger training signals. Offline evaluation on this biased data doesn't measure what users genuinely prefer.

4. Novelty effect: When users see a new recommendation style, engagement spikes for 1–2 weeks (everything is fresh and novel) then returns to baseline or below. Offline metrics cannot capture this time-varying effect.

5. Goodhart's Law: When a metric becomes a target, it ceases to be a good measure. Optimizing NDCG@10 hard enough will produce a model that ranks items with historically high CTR at the top, regardless of whether they satisfy the current user's intent.

Position bias is why offline metrics systematically mislead you about ranking model quality.

The problem: in your training/evaluation data, items that were shown in position 1 received 3× more clicks than the same items shown in position 5 — not because they were more relevant, but because users are more likely to click whatever is at the top. A model trained on this data learns: 'whatever the previous model put in position 1 tends to get clicked.' This is positional knowledge, not relevance knowledge.

How to detect it: Split your offline evaluation dataset by position. If items at position 1 have consistently higher offline metric scores than items at position 5 (even when controlling for item quality), you have position bias in your training data.

Inverse Propensity Scoring (IPS) — the standard debiasing fix:

For each (user, item, position) training example, weight the example by 1/P(click | position). Items shown at position 1 (high P(click | position)) are down-weighted. Items shown at position 5 (low P(click | position)) are up-weighted. This corrects for the measurement bias introduced by the position-dependent click probability.

# IPS debiasing in training loss
click_probability_by_position = {1: 0.15, 2: 0.09, 3: 0.07, 4: 0.05, 5: 0.04, ...}
for position, label, model_score in training_examples:
    propensity = click_probability_by_position[position]
    ips_weight = 1.0 / propensity  # Inverse of propensity
    loss += ips_weight * cross_entropy(model_score, label)

Alternatively: add position as an explicit training feature (position embedding or position index as input). Zero it out at serving time. The model learns the positional click bias during training but serves position-agnostic predictions.

Counterfactual evaluation answers: 'How would a new policy have performed on historical data, corrected for the exploration policy used to collect that data?'

The standard approach is Doubly Robust (DR) estimation or IPS-based off-policy evaluation (OPE):

DR estimator: V_DR = (1/N) Σ [r(a, x) / π(a|x) - (r̂(a, x) / π(a|x)) + v̂(x)]

Where:

• r(a, x): observed reward (click) for action a (item shown) in context x
• π(a|x): probability that the logging policy showed item a in context x (must be known)
• r̂(a, x): reward model's prediction for this (action, context) pair
• v̂(x): direct model estimate of expected reward

This is doubly robust — it's correct if either the reward model OR the logging propensities are correct.

Practical requirement: This only works if you log the propensity scores of the logging policy for every served item. If you don't log π(a|x), you cannot compute IPS or DR. This is why mature recommendation teams log propensities alongside every impression: it enables offline evaluation of new policies without running an A/B test.

Interleaving — a faster alternative to full A/B tests for ranking: Instead of routing users to either model A or model B, interleave the results from both models into a single ranked list for each user. Track which model's items get more clicks. Statistically much more efficient than A/B — requires 10–100× fewer users to reach the same confidence. Used heavily at Netflix and LinkedIn for ranking evaluation.