What Is Hyperparameter Optimization?

Two types of parameters govern the operation of machine learning (ML) models, model parameters and hyperparameters. Model parameters are parameters the model has learned during training, such as weights and biases in a neural network. Hyperparameters are settings and regulations that are set by the developer before training to control how the algorithm applies the model to the data. When a model is first being created, the best combinations and settings for the hyperparameters are unknown. This creates a need for hyperparameter optimization.

Hyperparameter optimization is the process of finding the best set of hyperparameter values for a given data set and problem. It begins by identifying the search space, which is a theoretical space where each hyperparameter to be optimized acts as a dimension; each point in this space represents a possible configuration of the model. There are several methods developers use to search through these configurations and identify the best candidates. H2O.ai provides defaults for many data sets to act as a starting point for additional optimization. H2O.ai assists clients with this process by grouping the hyperparameters by their importance to the Flow UI. All users should look closely at those labeled as critical, while the others may only be necessary in special cases.

Hyperparameter Optimization Methods

Manual Search

Manual search is the traditional method of hyperparameter optimization. Users manually select several settings for each hyperparameter and then compare the performance of the model with each combination of those selected settings. For instance, users might select ntrees settings of 50, 100, and 200, and max_depth settings of 5, 10, 15, and 20. This would give users twelve combinations and twelve models to compare. However, this method becomes less practical when there are many hyperparameters and settings to compare.

Grid Search

A grid search is similar to a manual search. It involves determining a set of values for each hyperparameter and evaluating the model with each possible combination of these values. It is less tedious than manual search but still involves the developers guessing at which values may be optimal.

Random Search

The random search method involves the system randomly selecting the values for each hyperparameter and then running the model to see how it performs. H2O.ai specializes in random search optimization and has proven to yield models that are equal or better than those generated by grid and manual optimization in a shorter timespan. In addition, H2O.ai allows users to compare the various models generated and then narrow the search parameters to those areas which look promising allowing developers to refine their search and be more efficient.

Bayesian Optimization

Bayesian optimization involves running sets of hyperparameters then using the results to improve the hyperparameter sets for the next search. Bayesian optimization is more efficient than other methods and can reduce optimization time and improve results. 

Why Does Hyperparameter Optimization Matter?

Hyperparameter optimization is essential to machine learning and AI systems because it allows developers to identify the best configurations of a model for a given situation. It helps machine learning models generalize, which is the ability to operate just as effectively whether encountering training data or new data. This leads to better performance of the model and better results for users.