Bayesian Optimization (BO) is a technique for optimizing an objective function
BO can be expressed succinctly in 7 fundamental steps:
- Gather initial samples
- Initialize the surrogate model
- Construct the acquisition function
$\alpha(x)$ - Optimize the acquisition function
$x^* = \arg\max_x \alpha(x)$ - Sample new data at
$x^*$ and update surrogate - Repeat until the budget is exhausted
- Make final recommendation
$x^{final}$
The algorithm above naturally induces some dependencies. The code is seperated into distinct interellated chunks that enable Bayesian Optimization. We provide a high-level overview of each.
kernels.jl
Our kernel objects have several fields that allow us to query the hyperparamters, the evaluation
of the kernel and all of it's partials, including the partials with respect to our hyperparameters.
We leverage automatic differentiation to compute these partials and the KernelGeneric function
is responsible for facilitating the construction of a Kernel that abides by this specification.
Kernels have several properties that are worth writing support for, but we've only implemented 3 of the many, i.e. kernel addition, kernel scaling, and kernel multiplication. In order to construct arbitrary kernels from arbitrary operations, we express our kernel objects as nodes in an expression tree.
surrogates.jl
- Implement logic for computing the gradient of the log likelihood
- Write support for updating/conditioning the Gaussian Process on new observations
- Update abstract type dependencies in
surrogates.jl