Preview — Pro guide
You are seeing a portion of this guide. Sign in and upgrade to unlock the full article, quizzes, and interview answers.
Random Forest & Ensemble Methods
Random Forest from first principles — bootstrap aggregating, the bias-variance decomposition of ensembles, feature importance via Gini/permutation, out-of-bag error, and when to choose Random Forest vs XGBoost vs GBM. 7 hard interview questions with detailed answers.
Why Ensemble Methods Work — The Wisdom of Crowds
A single decision tree is a high-variance, low-bias model: it perfectly fits training data (zero training error if deep enough) but generalizes poorly. The key insight: if you train many trees on different data samples and average their predictions, the variance decreases while bias stays roughly the same. This is the Bagging principle.
For this to work, the errors of individual models must be uncorrelated. If all models make the same mistakes (correlated errors), averaging doesn't help. Random Forest introduces randomness in two ways: (1) different bootstrap samples for each tree and (2) random feature subset at each split. Both reduce correlation between trees. The result: an ensemble with lower variance than any individual component, at the cost of interpretability and some bias increase from the feature subsetting.