Imbalanced Classification: Metrics, Class Weights, SMOTE, and Threshold Tuning

Most interview prep treats class imbalance as a single problem with a single fix ('use SMOTE'). This is wrong on both counts. Imbalance is not inherently a problem — it becomes a problem only when three specific conditions are present, and the right fix depends on which one dominates.

Condition 1 — You chose a bad metric. Accuracy on a 99:1 imbalanced dataset is meaningless because the majority-class baseline already achieves 99%. The fix here isn't resampling — it's picking a metric sensitive to the minority class (PR-AUC, recall at fixed precision, F-beta).

Condition 2 — Your loss function assumes balanced priors. Standard cross-entropy treats every example equally, so with 1,000 negatives per positive, 99.9% of the gradient signal comes from negatives. The model drifts toward 'predict negative always' as a local optimum. The fix is algorithm-level: class weights, focal loss, or a Bayes-optimal threshold shift.

Condition 3 — Business costs are asymmetric. Missing a $50K fraud is not the same mistake as blocking a $12 coffee purchase. Resampling can't encode this — only a cost matrix or a business-cost-aware threshold can.

Real-world imbalance regimes:

• Ad CTR: 1–5% positive rate. Mild imbalance. Class weights usually sufficient.
• Rare disease screening: 0.1–1% positive. Moderate. Need focal loss or cost-sensitive learning.
• Fraud detection: 0.01–0.1% positive (Kaggle Credit Card Fraud benchmark: 0.172%). Severe. Cost-sensitive threshold + calibration are mandatory.
• Beyond 1:10,000: stop framing as classification. Switch to anomaly detection (Isolation Forest, one-class SVM, autoencoder reconstruction error).

The interview mistake most candidates make: jumping to SMOTE before diagnosing which condition actually applies.

Before any algorithm change, fix the metric. A model optimized for the wrong metric will look great and perform terribly, regardless of how you handle imbalance.

Accuracy — useless under imbalance. 99% accuracy on a 99:1 dataset is achieved by predicting the majority class for everything. Report it only as a sanity check, never as the primary metric.

ROC-AUC — silently optimistic under imbalance. Davis & Goadrich (2006) showed that ROC curves are deceptive when negatives vastly outnumber positives because FPR = FP / (FP + TN) has a huge TN in the denominator. A model can achieve ROC-AUC = 0.99 while its precision at any useful threshold is 10%. Use ROC-AUC for balanced tasks only.

PR-AUC (average precision) — the correct primary metric for imbalanced classification. Precision and recall both operate on the positive class, so the metric isn't dominated by the easy-to-classify majority negatives. Under severe imbalance, PR-AUC is typically far lower than ROC-AUC — that gap is honest; don't smooth it over.

Precision@k / Recall@k — when operational capacity is fixed (e.g., fraud analysts can review 1,000 alerts/day), report precision at the top-k scored examples. This maps directly to business impact.

Recall at fixed precision (e.g., recall@precision=0.9) — the metric that decouples model quality from threshold choice and directly answers 'how much fraud can I catch without blocking more than 10% legitimate transactions?'

F-beta — F1 weights precision and recall equally; F-beta with β = sqrt(C_FN / C_FP) weights them by their business costs. For fraud with 100:1 cost ratio, β = 10.

Expected cost — the only metric with units of dollars: EC = FP × C_FP + FN × C_FN. Impossible to game. Report this as the headline metric when cost data is available.

Algorithm-level solutions modify the loss function rather than the data. They are almost always the right first move because they don't distort the data distribution and don't require post-hoc calibration correction.

Class weights (simplest, most underused): scikit-learn's class_weight='balanced' sets weight for class c to N / (K × N_c). For a 99:1 dataset this is {0: 0.505, 1: 50.0} — positive examples contribute 100× more to the loss. In XGBoost and LightGBM, use scale_pos_weight = N_neg / N_pos (≈ 99 for 99:1 imbalance). In PyTorch, nn.BCEWithLogitsLoss(pos_weight=torch.tensor([99.0])). Simple, no hyperparameter to tune (the ratio is determined by the data), works for most mild-to-moderate imbalance.

Cost-sensitive learning: extends class weights to a full cost matrix when C_FP ≠ C_FN. The weight ratio becomes (C_FN / C_FP) × (N_neg / N_pos). For 100:1 cost ratio and 1% base rate: scale_pos_weight = 100 × 99 = 9,900. This directly encodes business cost into the loss function.

Focal Loss (Lin et al., 2017 — RetinaNet): down-weights easy examples, not just minority ones. The key distinction from class weights:

• Class weights scale all positives uniformly — a positive that the model already predicts correctly at p=0.99 still contributes fully weighted gradient.
• Focal loss FL = -(1-p_t)^γ · log(p_t) down-weights both easy positives and easy negatives. The remaining gradient comes from hard, ambiguous examples.

With γ=2, an easy negative at p=0.99 contributes (0.01)² = 10^-4 — essentially zero. A hard example at p=0.5 contributes 0.25 — nearly full weight. The net effect is implicit hard example mining without an explicit mining pipeline. Standard in RetinaNet, YOLO v5+, EfficientDet.

When focal loss beats class weights: one-stage object detection (99,990 easy background anchors dominate CE), severe imbalance beyond 100:1, when you can't afford offline hard negative mining.

When focal loss loses to class weights: tabular gradient boosting (trees already do implicit selection), balanced data (focal over-weights the few hard examples), early training (the modulating factor only activates once easy examples become easy — typically after 1K–5K iterations).

Threshold tuning (underused, massive impact): regardless of what loss you use, the default threshold of 0.5 is almost always wrong. For asymmetric costs, the Bayes-optimal threshold is τ* = C_FP / (C_FP + C_FN). This single change often delivers 80% of the total gain available from imbalance handling.

Data-level solutions modify the training set rather than the loss. They are popular because they're intuitive, but they carry risks that most tutorials skip.

Random oversampling (duplicate minority): trivially implemented, but duplicates cause overfitting — the model memorizes minority points verbatim. Rarely outperforms class weights.

SMOTE (Chawla et al., 2002): Synthetic Minority Oversampling Technique. For each minority point, pick a random neighbor among its k nearest minority neighbors and generate a synthetic point on the line segment between them. Intuition: fill in the minority manifold with interpolated examples.

The trap: SMOTE assumes the minority class lies on a convex manifold where linear interpolation produces plausible examples. This is often false on tabular data — the line between two fraud transactions may land in a region of feature space that is categorically legitimate (a fraudulent transaction of $500 on a high-limit card and a fraudulent transaction of $5 on a prepaid card, interpolated, yields a $252 transaction with half-high / half-prepaid card attributes — physically impossible). Empirically, Elor & Averbuch-Elor (2022) benchmarked SMOTE variants against class weights across 73 tabular datasets and found that SMOTE rarely beats properly tuned class weights on gradient boosting. For tabular + XGBoost/LightGBM: skip SMOTE.

Variants (Borderline-SMOTE, ADASYN) focus synthesis on the decision boundary but don't fix the manifold assumption. Tomek links and ENN remove overlapping points — these are cleaning techniques, not balancing; apply cautiously.

Random undersampling: drop majority examples. Information loss is severe for data-scarce problems but can help when computational budget is the constraint. EasyEnsemble (Liu et al., 2009) undersamples the majority multiple times and trains an ensemble on each subsample — recovers the lost information through bagging.

The single most important SMOTE trap: apply SMOTE BEFORE the train/val split and you leak test information into training. SMOTE synthesizes new points from neighbors; if a validation point ends up as a neighbor used to generate a training synthetic, information has flowed across the split. Your CV score becomes optimistic by 5–15 points in PR-AUC. Fix: apply SMOTE inside each CV fold using imblearn.pipeline.Pipeline, not sklearn.pipeline.Pipeline.

Resampling changes the class prior in the training data, which means the model's predicted probabilities no longer reflect the true population prior. This is catastrophic for any downstream decision that depends on calibrated probabilities (cost-sensitive thresholds, credit pricing, risk ranking).

The intuition: if you undersample negatives at rate β (keeping β × N_neg examples), your training data has an artificially boosted minority rate. A model trained on this data outputs P(fraud | x, resampled data), not P(fraud | x, true data). If your true fraud rate is 0.1% and you undersampled to a 50/50 training set, your model's predicted 0.5 probability actually corresponds to a true probability of ~0.1%.

Pozzolo et al. (2015) derived the exact correction for undersampling. If p_s is the probability predicted by the model trained on undersampled data and β is the subsampling rate applied to the majority class, the true probability is:

p = β · p_s / (β · p_s - p_s + 1)

Or equivalently: p_s = p / (p + (1-p)/β).

Apply this as a post-processing step on every prediction. Without it, your Bayes-optimal threshold calculation is wrong, your probability-weighted expected value is wrong, and any downstream system (pricing, ranking by expected fraud loss) is systematically biased.

Practical workflow (production-grade):

1. Train on undersampled balanced data (fast, simple loss landscape).
2. Apply Pozzolo correction on inference-time probabilities.
3. Validate with a reliability diagram on held-out, un-resampled data. Group predictions into 10 equal-frequency bins; mean predicted probability in each bin should match the observed positive rate.
4. If the reliability diagram still deviates, fit Platt scaling or isotonic regression on top of the Pozzolo-corrected probabilities.

For SMOTE (oversampling), the correction is different because SMOTE inflates minority without removing majority information. Platt scaling or isotonic regression on held-out untouched data is the standard approach — no closed-form correction.

For class weights, calibration is usually preserved because the data distribution is untouched; only the gradient weighting changes. Still, tree-based models (XGBoost, LightGBM) produce uncalibrated scores regardless — always run a reliability diagram.

Beyond 1:10,000 imbalance, the classification framing starts to break down. You don't have enough positives to learn a class boundary, and the 'positive class' is often heterogeneous — many distinct rare phenomena lumped together (different fraud schemes, different disease subtypes, different system failures).

Anomaly detection reframes the problem: instead of 'learn what fraud looks like', learn 'what normal looks like' and flag deviations. This requires only negative examples (abundant) and implicitly handles novel attack patterns that pure classifiers miss.

Isolation Forest (Liu et al., 2008): builds random trees where paths to leaves are shorter for outliers (anomalies are easier to isolate with random splits). Scales well, no tuning beyond contamination rate. Good baseline for tabular anomaly detection.

One-Class SVM: learns a decision boundary around the normal data. Sensitive to kernel choice and the nu parameter; kernel computation is O(N²) — doesn't scale past ~100K points.

Autoencoder reconstruction error: train an autoencoder on normal data only; anomalies reconstruct poorly because the network never learned to represent them. Works well on high-dimensional data (images, network traffic). Reconstruction error itself is the anomaly score.

Hybrid approach (used at Stripe Radar, per their engineering blog): anomaly detection as a first-stage filter that surfaces candidates to a supervised classifier. The anomaly detector handles novel patterns; the supervised model refines known fraud types.

When NOT to reframe as anomaly detection: when you have abundant labeled minority data (10K+ positives), when the minority class is homogeneous (same fraud scheme), when calibrated probabilities are required (anomaly scores are not probabilities). In these cases, supervised classification with focal loss + threshold tuning outperforms anomaly detection.

A non-obvious staff-level insight: production ranking models at FAANG almost never rebalance their training data. This is counterintuitive — YouTube's recommendation system sees 1:millions imbalance (user watches one of millions of candidate videos), and yet their training pipelines don't use SMOTE or focal loss.

The reason: ranking problems don't use pointwise classification loss. They use pairwise or listwise losses that are inherently imbalance-robust.

Pairwise loss (RankNet, LambdaRank): for each positive example, sample K negatives and minimize -log(sigmoid(score_pos - score_neg)). The loss operates on score differences, so the absolute class ratio doesn't matter — what matters is that the positive outranks the negatives. YouTube's two-tower recommender uses this pattern with importance sampling: negatives are drawn weighted by popularity to simulate in-batch negatives from the full catalog without materializing it.

Listwise loss (ListMLE, Softmax over in-batch): for each positive, compute softmax over all negatives in the batch. With batch size 8,192, each positive sees 8,191 implicit negatives. Used by two-tower models at Google, LinkedIn, Pinterest.

Hard negative mining: at FAANG scale, uniformly random negatives are trivially easy — any random video is clearly not what the user wanted. Instead, hard negatives are mined from (1) in-batch top-scored non-positives, (2) semantically similar but not-clicked items, (3) the model's own high-confidence false positives.

Why this matters for interviews: if you're asked about class imbalance in a recommendation or search context, the answer is not 'use SMOTE'. It's 'reframe as a ranking problem with pairwise/listwise loss and cost-aware negative sampling'. This is the depth that separates staff-level answers from senior-level answers on imbalance questions in an MLSD context.

See YouTube's 'Sampling-Bias-Corrected Neural Modeling for Large Corpus Item Recommendations' (Yi et al., 2019) for the canonical reference on importance-sampled softmax for extreme-scale ranking.