Hyperparameter Optimization: Grid Search, Bayesian and Optuna Compared

Hyperparameter optimization is where many machine learning projects quietly succeed or fail. I have seen teams spend weeks refining architectures and feature pipelines, only to leave 5-10% performance on the table because they relied on a default tuning strategy that did not fit the problem's structure. After tuning hundreds of models from gradient boosting classifiers on tabular data to transformer fine-tuning runs, my view is straightforward: grid search, random search, Bayesian optimization, and modern frameworks like Optuna are not interchangeable tools. They represent fundamentally different tradeoffs in sample efficiency, compute budget, parallelism, and operational complexity. This article compares them directly, with implementation details and lessons from production use.

The Exhaustion Problem: Why Grid Search Scales So Poorly

Grid search is the most intuitive approach to hyperparameter tuning. You define a discrete set of values for each hyperparameter, generate the Cartesian product of all combinations, and evaluate every point. For two or three low-cardinality categorical parameters, it works. For anything beyond that, it collapses under combinatorial explosion.

Consider a typical gradient boosting model where you want to tune learning_rate in {0.01, 0.05, 0.1}, max_depth in {4, 6, 8, 10}, n_estimators in {500, 1000, 2000}, subsample in {0.8, 1.0}, and colsample_bytree in {0.8, 1.0}. That is 3 * 4 * 3 * 2 * 2 = 144 configurations. If each training run with cross-validation takes 5 minutes, you are looking at 12 hours of sequential compute. Add one more continuous parameter like reg_lambda with 4 values and you jump to 576 trials. In my experience, this is where teams first realize that exhaustive search is not exhaustive in any meaningful sense - it is just a sparse, axis-aligned sampling of a high-dimensional space.

The Curse of Dimensionality and Wasted Budget

The deeper issue is not just cost, but efficiency. When you discretize continuous parameters for a grid, you impose an arbitrary resolution. If the optimum lies between 0.05 and 0.1 for learning rate, a grid will never find it. More importantly, Bergstra and Bengio's work on random search showed that for most models, only a small subset of hyperparameters actually matters for a given dataset. Grid search wastes the majority of its budget evaluating unimportant dimensions at many redundant levels. If max_depth is the dominant parameter and subsample is largely irrelevant for your data, grid search will still evaluate every subsample value for every max_depth value, multiplying evaluations without new information.

I still use grid search in two narrow cases: when the search space is genuinely small and discrete, such as choosing between {l1, l2, elasticnet} penalties with a solver constraint, and when I need a fully reproducible, embarrassingly parallel baseline for audit purposes. For everything else, it is a strawman. If you are working through Cross-Validation Strategies: Beyond K-Fold for Real-World Data, you will see the same pattern - coupling an expensive grid with nested cross-validation quickly becomes infeasible.

from sklearn.model_selection import GridSearchCV
from xgboost import XGBClassifier

param_grid = {
    'max_depth': [4, 6, 8],
    'learning_rate': [0.01, 0.05, 0.1],
    'subsample': [0.8, 1.0],
    'colsample_bytree': [0.8, 1.0]
}

# 48 fits * cv=5 = 240 model trainings
grid = GridSearchCV(
    estimator=XGBClassifier(n_estimators=1000, use_label_encoder=False, eval_metric='logloss'),
    param_grid=param_grid,
    cv=5,
    scoring='roc_auc',
    n_jobs=-1,
    verbose=1
)
grid.fit(X_train, y_train)
print(f"Best score: {grid.best_score_:.4f} with {grid.best_params_}")

This code is simple and reliable, but notice the tradeoff: you must predefine the grid. In practice, I have found teams re-run grids iteratively with finer resolution around the best point, which is essentially a manual, inefficient version of sequential optimization.

Bayesian Optimization Demystified: Gaussian Processes, TPE, and Acquisition Functions

Bayesian optimization reframes tuning as a sequential decision problem. Instead of evaluating points independently, it builds a probabilistic surrogate model of the objective function p(y|x) where x is the hyperparameter configuration and y is the validation score, and uses an acquisition function to decide where to sample next. The surrogate is cheap to evaluate; the true objective (training a model) is expensive.

Surrogate Models: Gaussian Processes vs Tree-Parzen Estimators

The classical surrogate is a Gaussian Process (GP). A GP defines a distribution over functions and provides both a mean prediction and uncertainty estimate for any candidate point. This is powerful for continuous, low-dimensional spaces (typically <15-20 dimensions) with smooth objectives. Libraries like scikit-optimize use GPs effectively. However, GPs scale poorly. The computational cost is O(n^3) in the number of observations due to matrix inversion, and they struggle with categorical variables, conditional hyperparameters, and high dimensionality.

Tree-structured Parzen Estimator (TPE), which powers Optuna by default, takes a different approach. Instead of modeling p(y|x), TPE models p(x|y) and p(y). It splits observations into two groups - good and bad - based on a quantile threshold (often the top 15% of trials), and fits kernel density estimators to each group: l(y) for good points and g(y) for bad points. The acquisition function then maximizes the ratio l(x) / g(x). This formulation handles categorical and conditional spaces naturally and scales linearly with trials. In my experience tuning XGBoost and LightGBM on tabular data with 20+ hyperparameters including conditional logic like "only tune min_child_weight if grow_policy is depthwise", TPE consistently outperforms GP-based methods in both final score and wall-clock time.

Acquisition Functions and the Exploration Tradeoff

The acquisition function balances exploration (sampling where uncertainty is high) and exploitation (sampling where the mean prediction is high). Expected Improvement (EI), Upper Confidence Bound (UCB), and Probability of Improvement (PI) are common choices. With TPE, EI is implicit in the l/g ratio. One nuance I have learned to watch: early in optimization, the surrogate is inaccurate and the acquisition function will behave almost randomly. Do not judge a Bayesian optimizer by its first 20 trials. After 40-50 trials on a moderately complex space, the model has enough signal to guide search meaningfully. This sample efficiency is the core advantage. Where grid search needs 200 trials to cover the space coarsely, a well-configured Bayesian optimizer often finds a comparable or better optimum in 60-80 trials.

from skopt import BayesSearchCV
from skopt.space import Real, Integer

search_space = {
    'max_depth': Integer(3, 10),
    'learning_rate': Real(0.01, 0.3, prior='log-uniform'),
    'reg_lambda': Real(1e-3, 10.0, prior='log-uniform'),
    'subsample': Real(0.6, 1.0),
    'colsample_bytree': Real(0.6, 1.0)
}

opt = BayesSearchCV(
    estimator=XGBClassifier(n_estimators=1000, use_label_encoder=False, eval_metric='logloss'),
    search_spaces=search_space,
    n_iter=64,  # number of Bayesian optimization steps
    cv=5,
    scoring='roc_auc',
    n_jobs=-1
)
opt.fit(X_train, y_train)

A practical tip: always specify the prior correctly. Using a uniform prior for learning_rate or reg_lambda is a common mistake. These parameters operate on a logarithmic scale; a log-uniform prior gives the optimizer appropriate resolution across orders of magnitude. When you are evaluating the Bias-Variance Tradeoff: A Practical Guide to Model Selection, getting regularization parameters like reg_lambda right with a proper prior often matters more than the choice of optimizer itself.

Optuna Under the Hood: How Define-by-Run and Pruning Change the Workflow

Optuna is not just "another Bayesian optimizer." Its design decisions solve specific pain points I encountered repeatedly with older frameworks like Hyperopt and Spearmint. The two most important are the define-by-run API and asynchronous pruning.

Define-by-Run vs Define-and-Run

Traditional frameworks use a define-and-run pattern where you declare the entire search space upfront as a static dictionary. This breaks down when hyperparameters are conditional. For example, in a CatBoost or LightGBM search, if boosting_type is 'dart', you should tune drop_rate; if it is 'gbdt', that parameter is irrelevant. With a static dictionary, you end up sampling irrelevant parameters or writing hacked workarounds.

Optuna's define-by-run lets you construct the search space dynamically in Python code. The objective function itself contains the sampling logic, with full access to control flow. This makes conditional spaces trivial and keeps the search space definition close to the model construction logic, which improves maintainability. I have found this especially valuable when tuning pipelines that include preprocessing choices, such as whether to tune PCA components or feature selection thresholds as covered in Feature Engineering for Tabular Data: Techniques That Actually Work.

Pruning: MedianPruner, Successive Halving, and ASHA

Pruning is Optuna's second major advantage. Unpromising trials can be terminated early based on intermediate validation scores. By default, MedianPruner compares a trial's intermediate value at a given step to the median of previous trials at the same step. If it is worse, the trial is pruned. More advanced pruners implement Asynchronous Successive Halving (ASHA), which aggressively allocates resources to promising trials. When tuning neural networks where one trial might run for 50 epochs, pruning can save 40-60% of total compute without degrading final performance, because many configurations reveal themselves as poor after only a few epochs.

To make pruning work, you must report intermediate values correctly. For gradient boosting, that means reporting after each boosting round; for PyTorch training, after each epoch. The PyTorch Documentation on training loops and the Hugging Face Documentation on Trainer callbacks both show patterns for emitting step-wise metrics that integrate cleanly with Optuna's report and should_prune API.

import optuna
from sklearn.model_selection import cross_val_score
import lightgbm as lgb

def objective(trial):
    param = {
        'objective': 'binary',
        'metric': 'auc',
        'verbosity': -1,
        'boosting_type': trial.suggest_categorical('boosting_type', ['gbdt', 'dart']),
        'num_leaves': trial.suggest_int('num_leaves', 20, 150),
        'learning_rate': trial.suggest_float('learning_rate', 1e-3, 0.3, log=True),
        'feature_fraction': trial.suggest_float('feature_fraction', 0.6, 1.0),
        'bagging_fraction': trial.suggest_float('bagging_fraction', 0.6, 1.0),
        'bagging_freq': trial.suggest_int('bagging_freq', 1, 7),
        'reg_lambda': trial.suggest_float('reg_lambda', 1e-3, 10.0, log=True),
    }
    # Conditional hyperparameter
    if param['boosting_type'] == 'dart':
        param['drop_rate'] = trial.suggest_float('drop_rate', 0.05, 0.3)

    model = lgb.LGBMClassifier(**param, n_estimators=2000)

    # Use pruning via early stopping + Optuna pruning callback
    pruning_callback = optuna.integration.LightGBMPruningCallback(trial, 'auc')
    scores = cross_val_score(model, X_train, y_train, cv=5, scoring='roc_auc',
                             fit_params={'callbacks': [pruning_callback]})
    return scores.mean()

study = optuna.create_study(direction='maximize',
                            sampler=optuna.samplers.TPESampler(multivariate=True),
                            pruner=optuna.pruners.MedianPruner(n_warmup_steps=20))
study.optimize(objective, n_trials=100, n_jobs=4)
print(f"Best AUC: {study.best_value:.4f}")
print(f"Best params: {study.best_params}")

One detail I have learned to tune is multivariate=True for the TPESampler. The default independent TPE models each parameter separately, while multivariate TPE captures correlations between parameters, such as the known interaction between learning_rate and n_estimators or between num_leaves and max_depth. On the Gradient Boosting Showdown: XGBoost, LightGBM and CatBoost Compared workloads I run, enabling multivariate TPE gives a modest but consistent lift in final performance for only a small overhead.

Empirical Tradeoffs: Cost, Sample Efficiency, and Wall-Clock Time Compared

Theoretical advantages mean little without empirical context. The table below summarizes how the three approaches behave on a typical mid-sized tabular problem: 100k rows, 40 features, LightGBM with 5-fold CV, 10-core machine, and a budget of 100 model trainings. Your numbers will differ, but the relative ordering is stable across many datasets I have tested.

What the table hides is variance. Grid search is deterministic - run it twice and you get the same result. Bayesian methods are stochastic and sensitive to initial random points. I always set n_startup_trials to 15-20 to force a random exploration phase before TPE takes over, and I run at least two independent studies with different seeds for critical models to check stability.

Another tradeoff is interpretability of the search itself. Optuna provides optuna.visualization.plot_param_importances and plot_parallel_coordinate which I use in every post-tuning review. In one recent project tuning a time-series forecaster, the importance plot revealed that seasonality_prior_scale dominated while changepoint_prior_scale had almost no effect, which changed our modeling direction entirely. Grid search offers no such diagnostic. Recent surveys on the arXiv Machine Learning list show growing work on multi-objective and cost-aware acquisition functions, which Optuna now supports natively via NSGAIISampler for cases where you care about AUC vs inference latency.

Operationalizing Tuning: Parallelism, Cross-Validation, and Avoiding Leakage

The best optimization algorithm will still produce misleading results if your evaluation protocol is flawed. The most common failure I see in production tuning is leakage through improper cross-validation or preprocessing.

Correct Parallel Execution and Storage

For Optuna, scaling beyond one machine requires a shared study storage. Use a relational backend like PostgreSQL, not SQLite, and set load_if_exists=True so workers coordinate. Each worker runs study.optimize independently. Without this, parallel Bayesian optimization degrades to independent random searches because trials do not share the surrogate model state. In my experience, even with 8 workers, the overhead of database synchronization is negligible compared to model training time if you set heartbeat_interval appropriately.

Nesting Preprocessing Inside the Objective

Any transformation that learns from data - imputation, scaling, target encoding, feature selection - must be fitted inside each CV fold inside the objective function, not once outside. If you fit a target encoder on the full training set and then tune hyperparameters via cross-validation, you leak information from validation folds into training and will overestimate performance. The objective function should create a full Pipeline and evaluate it with cross_val_score or manual folding. This increases cost per trial but is non-negotiable for reliable tuning. Use the same cv splitter across all trials for comparability, and for temporal or grouped data, ensure the splitter respects that structure.

Budget-Aware Tuning and Early Stopping

For gradient boosting, couple Optuna with early stopping via the pruning callback rather than fixing n_estimators as a hyperparameter. Let the model stop when validation AUC plateaus, and tune learning_rate, num_leaves, and regularization instead. For neural networks, treat number of epochs as a pruned resource rather than a tuned parameter. I also recommend defining a strict time budget rather than a fixed trial count. study.optimize(objective, timeout=3600*6) for a 6-hour budget often yields more practical results than n_trials=200, especially when trial runtimes vary due to pruning.

Finally, log everything. Optuna's study.trials_dataframe() gives you a full history with parameter values, scores, and durations. I archive this dataframe alongside the best parameters for every production tuning run. When a model degrades three months later, that history is invaluable for diagnosing whether the degradation is due to data drift or simply that the previous optimum was unstable. Tuning is not a one-off step; it is part of your model governance.

Frequently Asked Questions