Abstract:

This project aims to understand the theoretical aspects of the Gaussian Process as a non- parametric statistical learning tool as well as its role as a surrogate function in Bayesian optimization. This is achieved through a comprehensive review of the literature associ- ated with both the Gaussian Process and Bayesian optimization along with introductory examples on how they are used to conduct inference over functions. We then motivate the use of Bayesian optimization in real world applications where the collection of data is either computationally or commercially expensive through a series of diverse examples. This includes a discussion around the trade-off between exploitation and exploration of the solution space in search of the global optimum. After this we assess the performance of the Bayesian optimization algorithm across three analytical test functions and against several popular optimization routines using a Monte Carlo simulation. This results in the conclusion that Bayesian optimization outperforms most methods on function surfaces which are non-convex and multi-modal but is largely unnecessary on surfaces which are not. We end with a discussion around the benefits of Bayesian optimization in situations where we seek to optimize a function which has no analytical form or is either expensive or impossible to evaluate.

Research conducted at the University of Cape Town under the supervision of Jake Stangroom and Emeritus Prof. Linda Haines