Optimisation

Terry Benzschawel

Although optimisation is typically considered separately from regularisation, both are used in combination to address undersampling and local minimum problems. The interest in optimisation techniques stems mainly from their role in deep learning neural networks: optimisation methods are used to choose parameters that are optimal for a given problem. Demonstrated successes of optimisation methods have inspired research designed to model more challenging machine learning problems, and to design new methods.

8.1 OPTIMISATION ALGORITHMS

This chapter describes the popular optimisation methods stochastic gradient descent (SGD), RMSProp, AdaGrad, AdaDelta, Adam and AdaMax.

Stochastic gradient descent

Although SGD was introduced earlier, in Chapter 6, for backpropagation, many optimisation methods are built on this method. First-order optimisation algorithms minimise or maximise a loss function E(x) using its gradient values with respect to the parameters. The most widely used first-order optimisation algorithm is gradient descent. The first-order derivative tells us whether the error function is decreasing or increasing at a particular point. The first-order derivative basically gives us a