Hyperparameter Tuning: Search Strategy, Budgeting, and Production Discipline

Hyperparameter tuning is not trying values until accuracy improves. It is constrained experimental optimization under finite compute and time budgets. Every tuning decision involves an implicit tradeoff: search budget versus exploration coverage, evaluation fidelity versus run cost, and reproducibility versus speed.

Why random search beats grid search (Bergstra & Bengio, 2012): grid search evaluates all combinations of a discrete parameter grid. If a model has 6 hyperparameters but only 2 matter significantly, grid search wastes the majority of its budget evaluating unimportant dimensions. Random search independently samples each hyperparameter, so every trial covers a unique projection onto each dimension. With the same budget, random search typically evaluates more distinct values of important hyperparameters — this is the key insight.

Hyperparameter importance is empirically unequal. Across most ML tasks: learning rate is the dominant factor (often a 10× range matters), then batch size (affects generalization via noise scale), then regularization strength, then architecture choices (depth, width), and finally optimizer-specific parameters (β₁, β₂ in Adam). Knowing this ordering lets you allocate search budget efficiently: explore learning rate exhaustively before worrying about Adam's β₂.

Bayesian optimization builds a probabilistic surrogate model (typically a Gaussian Process) over the performance landscape. After each trial, it updates the posterior estimate of where good hyperparameters might be. The acquisition function (Expected Improvement, Upper Confidence Bound) balances exploration (trying uncertain regions) with exploitation (evaluating near known good regions). This is sample-efficient for expensive training runs — 20-50 trials of Bayesian optimization often outperforms 200+ random search trials when each trial costs hours.

Multi-fidelity methods (Hyperband, ASHA) exploit the observation that relative hyperparameter rankings often stabilize early in training. Run many configurations for a small number of steps (1% of full training), eliminate the bottom half, double resources for survivors, and repeat. ASHA (Asynchronous Successive Halving) adapts this for parallel settings. The result: the same compute budget evaluates 10–100× more configurations while ensuring the winner actually trains to convergence.

Strong process: establish a robust baseline → define search space with domain priors → use a budget-aware scheduler → track and reproduce all experiments → validate top candidates across multiple seeds and data slices.

Learning rate is consistently the highest-impact hyperparameter across model families. Getting it right deserves dedicated strategy, not just a single tuned value.

Cyclical learning rates (Smith, 2017): oscillate the learning rate between a lower and upper bound following a triangular or cosine schedule. The key insight: allowing the learning rate to temporarily increase causes the optimizer to escape sharp local minima and explore flatter regions, which generalize better. The 1-cycle policy trains for one cycle with the learning rate rising to a maximum then decaying to near-zero — in practice, this reaches within 1-2% of fully-tuned performance in roughly 1/5 of the epochs, making it a highly efficient tuning approach.

Warmup + cosine decay: the dominant schedule in large-scale deep learning (BERT, GPT, ViT). Warmup linearly increases the learning rate from near-zero to peak for the first 5-10% of training steps (avoids early instability when gradients are high-variance), then decays following a cosine curve. The cosine shape provides faster initial decay and slower final decay compared to linear, empirically outperforming both step-decay and linear schedules on large models.

Learning rate range test (LR finder): train for 100-200 steps with learning rate increasing exponentially from 1e-7 to 1. Plot loss vs. learning rate. The optimal learning rate is slightly below where loss starts diverging. This provides a principled search starting point rather than arbitrary grid values.

Batch size interaction: scaling laws suggest learning rate should scale proportionally with batch size (linear scaling rule, Goyal et al. 2017). Doubling batch size → double learning rate, with a warmup period. This allows larger batches without sacrificing training dynamics — critical for distributed training.