Preview — Pro guide
You are seeing a portion of this guide. Sign in and upgrade to unlock the full article, quizzes, and interview answers.
Sections
Related Guides
Linear & Logistic Regression: From OLS to FTRL
Machine Learning
XGBoost: Gradient Boosting Deep Dive
Machine Learning
Bias-Variance Tradeoff & ML Debugging
Machine Learning
ML Evaluation Metrics: The Complete Guide
Machine Learning
Feature Engineering: Leakage-Safe Encoding, Interactions, Temporal, and Production Parity
Machine Learning
SVM & Kernel Methods: Maximum Margin, Duals, and the Kernel Trick
Master Support Vector Machines from first principles: hard/soft margin primals, the Lagrangian dual, KKT conditions, Mercer's theorem, the kernel trick, RBF/polynomial kernels, SMO training, Platt calibration, and when SVMs still beat XGBoost and deep nets in 2026.
Why SVMs Still Matter in 2026
Support Vector Machines were the dominant classifier from roughly 1995 to 2012, displaced in most benchmarks by deep nets (vision, speech) and gradient-boosted trees (tabular). But SVMs are not a historical curiosity — they still win in three regimes that matter: (1) small-data problems (n < 10K) where tree ensembles overfit, (2) high-dimensional sparse inputs like TF-IDF text where linear SVM matches or beats XGBoost for 10× less training cost, and (3) problems with a known geometric or structural kernel — bioinformatics sequences (spectrum kernel), graphs (Weisfeiler-Leman), medical imaging. More importantly, SVMs are the cleanest vehicle for teaching three ideas that show up everywhere in modern ML: convex duality, the kernel trick, and the representer theorem. Gaussian processes, neural tangent kernels, and contrastive learning all inherit this machinery. An interviewer asking about SVMs is almost always testing whether you understand kernels, duals, and margins — not whether you can call sklearn.svm.SVC.
What 9/10 Answers Do That 6/10 Answers Don't
6/10: Recites that SVMs 'find the maximum margin hyperplane' and mentions the kernel trick as 'mapping to higher dimensions'.
9/10: Derives the dual formulation and points out that it depends only on inner products ⟨x_i, x_j⟩ — which is why the kernel trick works, not a separate feature. Names specific kernels for specific data (linear for text, RBF for numeric with unknown structure, spectrum kernel for bio), explains why the Gram matrix being O(n²) prevents kernel SVMs from scaling past ~100K samples, and reaches for Nyström approximation or random Fourier features (Rahimi & Recht, 2007) when asked about scaling. Mentions that predict_proba in sklearn is Platt-calibrated and is often poorly calibrated vs. CalibratedClassifierCV.
Clarifying Questions Before Reaching for an SVM
How large is n (samples)?
If n > 100K, kernel SVMs are effectively ruled out — the Gram matrix is n×n and SMO is ~O(n²) typical, O(n³) worst case. Use linear SVM (LIBLINEAR), or switch to XGBoost/LightGBM. If n < 10K, SVM is competitive with XGBoost and may win if the kernel matches the data geometry.
How high is d (features) and how sparse?
High-dim sparse (text, TF-IDF, n-grams, d > 10K, 99% zeros): linear SVM with L2 regularization is the gold standard — Joachims showed SVMs beat every other method on Reuters and similar text benchmarks for years. Dense low-dim (d < 100): RBF kernel is the default; try polynomial if you suspect feature interactions.
Do you need probability outputs or just predictions?
SVMs output signed distance to the margin, not probabilities. If downstream uses require P(y=1|x) — credit scoring, ad bidding, medical — either wrap in Platt scaling (sigmoid fit on held-out) or use logistic regression / calibrated trees instead. Do NOT trust sklearn's default predict_proba without checking calibration on a held-out set.
Is the problem linearly separable after feature engineering?
If yes, linear SVM is faster, more interpretable, and scales to n = 10M+ with LIBLINEAR. If no, you're choosing a kernel — and the kernel choice matters far more than the C hyperparameter. Wrong kernel (RBF on text, linear on MNIST pixels) is unrecoverable.
Are classes clean or noisy / overlapping?
SVMs are sensitive to noise because support vectors determine the decision boundary — a few mislabeled points near the margin can swing the boundary. Either clean labels aggressively, or lower C (more regularization), or switch to a model with softer decisions (logistic regression, gradient boosting).