Feature Engineering: Leakage-Safe Encoding, Interactions, Temporal, and Production Parity

"Feature engineering is dead" is true for computer vision and natural language — CNNs and transformers learn their own features end-to-end. For tabular data and recommendation systems, the opposite is true: a tuned LightGBM with thoughtful features routinely beats a TabNet or MLP on datasets below ~1M rows (Shwartz-Ziv & Armon, 2022). At the staff level, the interview signal is knowing which regime you are in and why.

The actual job of feature engineering has three parts, and most candidates only know the first:

1. Representation — how you encode a column so the model can use it (one-hot vs target encoding vs embedding).
2. Leakage control — making sure no future-or-out-of-fold information slips into training features. This is the silent killer: offline AUC is great, production is terrible.
3. Training-serving parity — making sure the feature you computed at training time is bit-for-bit identical to the one served at inference. A "30-day click count" computed over full history for training but only the last 30 days for serving is a different feature with a different distribution — the model decays without an error log.

The single key insight: a feature is not a transformation of a column. A feature is a contract that must hold identically at training time, serving time, and at every historical point-in-time you join on. Everything in this topic — K-fold target encoding, ColumnTransformer inside a pipeline, feature stores — exists to enforce that contract.

Categorical encoding is the most failure-prone part of feature engineering. The decision is not "one-hot vs target encoding" — it is a four-axis decision: cardinality, model family, leakage tolerance, and serving cost.

One-hot encoding (K < 15): creates K binary columns. Safe for linear models, neural nets, k-NN. Hurts trees above ~K=100 — sparse one-hot columns create splits with ~1/K of the data on one side, which is rarely the optimal split, so boosted trees waste depth finding them. LightGBM's categorical_feature parameter uses a native Fisher-optimal split that is ~1.5–2× more accurate than one-hot on K=50+ columns.

Label / ordinal encoding: map categories to integers 0..K-1. Only valid when (a) the model is a tree (which treats integers as ordered, so it can split x ≤ 3.5) AND (b) the categories are actually ordinal (size: S=0, M=1, L=2, XL=3) OR you don't care about the imposed order (trees can still split on any threshold). Never use label encoding for linear/NN models — you are forcing a linear ordering on nominal categories.

Frequency encoding: replace each category with its count. Surprisingly effective on trees because frequency is often correlated with target (rare merchants are more suspicious). Zero leakage risk. Good first-pass for K > 100 on tree models.

Feature hashing (K unbounded): hash(category) % 2^18. Fixed memory regardless of cardinality, works for streaming / online learning where new categories appear forever (URLs, user agents, device fingerprints). Downside: collisions. Use with linear models (Vowpal Wabbit built its reputation on this) or as input to an embedding.

Embedding lookup: a K × d matrix learned end-to-end. Standard for deep recommenders (YouTube, DLRM). Dimension heuristic: d ≈ min(50, K^0.25) (fast.ai) or d ≈ 6·K^0.25 (TensorFlow).

Target (mean) encoding + K-fold + smoothing (K in 50–1000): the most powerful and most dangerous encoding. Replace each category with the mean target of training rows in that category. Adds signal density in one column where one-hot would need 500. But computing this on the full training set leaks every row's label into its own feature — training AUC inflates 5–10 points, production collapses. Always use out-of-fold K-fold + Bayesian smoothing toward the global mean.

The first question to ask: do I need to transform numerics at all? For trees (XGBoost, LightGBM, Random Forest), scaling is a no-op — tree splits are threshold-based and scale-invariant. Standardizing age before LightGBM is wasted engineering effort. For everything else, transforms matter.

StandardScaler (subtract mean, divide by std): default for linear models, SVM, k-NN, PCA, neural networks. Assumes roughly Gaussian distributions. Breaks down with heavy outliers — one outlier inflates σ and crushes the signal for the other 99% of rows.

RobustScaler (subtract median, divide by IQR): use when outliers are real and you want to keep them. Credit-card transaction amounts, income, page-view counts — all heavy-tailed.

MinMaxScaler (scale to [0, 1]): use when the algorithm requires bounded inputs (image pixels for a CNN, variational autoencoders). Do not use as a default — a single new outlier in production crushes the whole range.

Log / Box-Cox / Yeo-Johnson for right-skewed positives (income, price, latency, counts). Log is simplest and invertible. Box-Cox requires strictly positive data; Yeo-Johnson is the generalization that handles zeros and negatives. A log-transform on a right-skewed feature often turns a mediocre linear model into a strong one — the same feature in a tree changes nothing because trees can approximate monotonic transforms on their own.

Binning: discretize a numeric into bins. Three variants: equal-width (fast, bad for skewed data), equal-frequency / quantile-based (robust to skew — each bin has the same number of rows), and supervised / tree-based (use a single decision tree to find target-optimal cut points). Useful for linear models that need to capture non-linearities without degree-2 polynomials. Not useful for trees.

Rank / quantile transformation: map each value to its rank. Collapses all monotone transforms to the same representation, making the model strictly invariant to outliers at the cost of losing magnitude information. Common in LightGBM pipelines for very heavy-tailed features.

Trees capture interactions automatically. A decision tree split age > 40 followed by income > 100K inside that branch is exactly an (age, income) interaction — the model learned that high-income-and-over-40 behaves differently than either condition alone. This is why GBDTs on tabular data rarely need explicit interaction features.

Linear models and shallow NNs do not. A linear model y = w1·age + w2·income + b has no cross term. You must manufacture it. Two approaches:

Polynomial features (degree 2): compute all pairwise products x_i · x_j. With d features, you get O(d²) new features — explodes fast. Rarely use degree > 2. Useful when you have ~10–50 features and suspect non-linearity. For sparse one-hot inputs, use PolynomialFeatures(interaction_only=True) so you don't generate squares of binary indicators.

Feature crosses (categorical × categorical): concatenate two categorical values into a single new category. Example: city="SF" × day="Mon" → "SF_Mon". Facebook's 2014 ad-CTR paper (He et al., "Practical Lessons from Predicting Clicks on Ads at Facebook") showed that crossing categorical features and feeding them into a logistic regression on top of GBDT-derived leaf indicators lifted CTR significantly over either model alone — this "GBDT + LR" architecture became the default for ad-click prediction for half a decade.

When explicit crosses still matter for trees: when cardinality of either side is high, the tree may not find the combination inside its depth budget. A pre-computed cross exposes the combination at depth 1 instead of depth 6. For a depth-6 LightGBM model, only features in the top ~50 splits get used; a manual cross forces a promising combination to the front.

Temporal features are where leakage is most common and most catastrophic. A few patterns, each with a specific purpose:

Cyclical (sin / cos) encoding: hour of day, day of week, month, day of year are cyclical — hour 23 is close to hour 0, not 23× further from it. Encode as two features: sin(2π·hour/24) and cos(2π·hour/24). This lets any model (especially linear/NN) learn periodic patterns without treating hour=0 and hour=23 as far apart. Trees can learn thresholds on raw hour but benefit less from this transform.

Lag features: y_{t-1}, y_{t-7}, y_{t-30}. The single strongest predictor of tomorrow's sales is today's sales. For retail, finance, and demand forecasting, lagged features are the entire game. Check: every lag must be strictly in the past relative to the label timestamp — using y_{t-0} is a bug.

Rolling windows: mean(y, window=7d), std(y, window=30d). Captures trend and volatility. Use shift(1) before rolling to avoid the current day leaking in: df["rolling_7d"] = df["y"].shift(1).rolling(7).mean().

Exponentially-weighted moving average (EWMA): v_t = α·y_{t-1} + (1-α)·v_{t-1}. Smoother than rolling window, responds faster to recent changes. α ∈ [0.1, 0.5] typical.

Time-since-last-event: seconds since user's last purchase, login, click. Often one of the top-5 features in a churn or recsys model. Captures recency in a way that raw timestamps cannot.

Calendar features: day-of-week, is_holiday, is_weekend, is_month_end, is_payday. Especially strong for retail and financial data.

For tabular problems where a short text column (product title, search query, job description) is one of many features, you rarely need a BERT embedding. Start lightweight:

TF-IDF on unigrams and bigrams: sklearn.feature_extraction.text.TfidfVectorizer(ngram_range=(1, 2), max_features=50_000, min_df=5). For a LightGBM or linear model, this is often 80% of what a transformer gives at 1% of the cost. Output is sparse; use scipy.sparse throughout.

Character n-grams (3–5): robust to misspellings, abbreviations, and out-of-vocabulary words. Critical for user-generated text — "iphoen" and "iphone" share most character 3-grams. Standard in fraud detection on free-text fields.

Pre-trained sentence embeddings as features: SBERT / all-MiniLM-L6-v2 produces 384-d embeddings. Run offline, store as 384 numeric features, feed into LightGBM alongside the rest. Gets most of the semantic benefit without training a transformer end-to-end. Latency at serve time: batch inference on CPU ~10ms per sentence; on GPU ~1ms.

Simple length / punctuation features: character count, word count, uppercase ratio, exclamation-mark count. Strong signal for spam, sentiment, and review quality. Cost: near zero.

The rule: reach for a fine-tuned LLM only when lighter approaches plateau, not as the default. On Kaggle tabular competitions with text fields, TF-IDF + LightGBM is still the baseline to beat.

Leakage is information from outside the training-at-the-time-of-prediction distribution sneaking into features. It inflates offline metrics and silently destroys production models. There are four distinct types and every candidate should be able to name each with a fix:

Target leakage: a feature is directly or causally downstream of the label. Classic: using customer_complained to predict churn when complaints are caused by churn decisions. Less obvious: using policy_cancelled which is set after the fraud flag is raised. Fix: build a causal timeline — every feature must be computable strictly before the label event.

Train/test contamination (preprocessing leakage): fitting a scaler, imputer, or encoder on the full dataset before splitting. The test set's mean/std/category distribution shapes the transform; model learns a transform that would not exist in production. Fix: wrap every fitted step in sklearn.pipeline.Pipeline and pass the pipeline — not the preprocessed arrays — into cross_val_score. The pipeline re-fits every step inside each fold.

Group leakage: multiple rows share an entity (user_id, patient_id, session_id). Random split puts some of a user's rows in train and others in test — the model memorizes user identity. Classic in medical ML. Fix: GroupKFold(n_splits=5) with groups=user_ids.

Temporal leakage: for time-series, random K-fold puts future rows in training. The model sees events from next Tuesday while training on events through today. Fix: TimeSeriesSplit (expanding-window) or explicit chronological cutoffs. Also applies to target encoding — the encoding for category k at time t must only use rows with timestamp < t.

The senior-level move: when a colleague reports "CV AUC = 0.95," the first question is "what is your CV scheme and what does your pipeline object look like?" Not "what model did you use?" Leakage is 10× more likely than model choice to explain the number.

Rubin's 1976 taxonomy is still the correct framework. The right imputation depends on why the value is missing:

MCAR (Missing Completely At Random): the probability of missingness is independent of both observed and unobserved data. Sensor dropouts, random survey non-response. Safe to drop or mean/median impute. In practice, true MCAR is rare.

MAR (Missing At Random): missingness depends on observed features but not the missing value itself. Income missing more often for younger users — missingness is explained by age, which is observed. Model-based imputation works: regress the missing column on the other columns (MICE — Multivariate Imputation by Chained Equations, van Buuren 2011). Also: add a <col>_was_missing binary indicator — the fact that a value was missing is often itself predictive.

MNAR (Missing Not At Random): missingness depends on the unobserved value. Income missing because it is very high (users refuse to report). This is the hardest case — no unbiased imputation exists without domain modeling. Treat missingness as a feature: users who refuse to report income are systematically different from those who report; encode that.

Practical defaults:

• Simple: SimpleImputer(strategy="median") for numerics, "most_frequent" or constant="UNK" for categoricals.
• Better: KNNImputer(n_neighbors=5) — leverages similar rows. Expensive at scale.
• Best (MAR): IterativeImputer (scikit-learn's MICE implementation). Iterates a regressor per column until convergence.
• Always: add a missingness indicator column for any feature with >1% missing rate.
• Trees (XGBoost, LightGBM) handle missing values natively via an optimal-direction default for NaN — no imputation required. Linear models and NNs require imputation.

With 500 candidate features and 50K rows, the model overfits to spurious correlations. Four families of selection methods:

Filter methods (cheap, univariate): variance threshold, Pearson/Spearman correlation with target, mutual information (MI) for non-linear relationships, chi-squared for categorical features. Fast but miss interactions — a feature useless alone can be critical in combination.

Wrapper methods (expensive, model-aware): Recursive Feature Elimination (RFE) trains a model, removes the weakest feature by model importance, and repeats. Captures interactions but is O(d) model fits.

Embedded methods (built into model): L1 (Lasso) drives coefficients to zero → automatic selection for linear models. Tree-based feature_importances_ (split gain) comes free with GBDT. Critical caveat: default gain-based importance is biased toward high-cardinality features (Strobl et al., 2007) — a continuous variable has many possible split points and can accumulate gain spuriously. Never rank features by raw LightGBM feature_importances_ alone.

Permutation importance (unbiased, model-agnostic): randomly shuffle one feature's values in the validation set and measure the drop in score. Unbiased with respect to cardinality. Slower (one prediction per feature) but the standard for honest feature ranking. Use sklearn.inspection.permutation_importance with n_repeats=10.

Boruta (shadow features): add a shuffled "shadow" copy of every feature; a real feature is kept only if it outperforms its shadow across multiple runs. Strong but computationally expensive — good for a final feature set, not for exploration.

Staff-level caveat on SHAP: SHAP explains feature contributions to the model's predictions. It does not explain causal effects in the world. A SHAP plot showing age as the top feature means the model uses age heavily — if the model is miscalibrated on minority groups, SHAP will still rank age highly. Correlation-in-model ≠ real-world cause.

Every senior ML engineer has been burned by training-serving skew at least once. The pattern is always the same: offline evaluation looks great, deploy to production, AUC silently drops 5–15 points, on-call gets paged a week later when a product metric dips. The cause is almost never the model.

The top three skew sources:

1. Different code paths. Training uses a Pandas notebook; serving uses a Java microservice. Both claim to compute "30-day click count." One uses timezone-aware timestamps, the other UTC. One counts distinct events, the other total events. Distributions diverge. Fix: single feature definition in a registry (Feast, Tecton, Michelangelo, Zipline). Both paths call the same function.

2. Different time horizons. Training computes "user lifetime purchase count" over full history. Serving only has 90 days due to a data retention policy. The feature the model was trained on does not exist in production. Fix: enforce retention parity at the feature registry, not at the consumer.

3. Missing point-in-time correctness. Training joins labels from January with features as of today. The feature user_30d_fraud_rate for a January label row includes fraud that happened in February. The model learns to use future-leaking features; at serve time, those features are much weaker. Fix: entity_df point-in-time joins (native to Feast, Tecton, BigQuery AS OF).

The staff-level framing: a feature store is not a database. It is a regression test harness for feature contracts. Every production feature should have a daily job comparing the online distribution to the offline distribution (Population Stability Index, PSI > 0.25 = alert). This is ongoing, not a one-time setup. This is what separates "I use Feast" from "I operate a feature platform."