C++ Neural Networks and Fuzzy Logic: Preface

Supervised Learning

Supervised neural network paradigms to be discussed include :

•  Perceptron

•  Adaline

•  Feedforward Backpropagation network

•  Statistical trained networks (Boltzmann/Cauchy machines)

•  Radial basis function networks

The Perceptron and the Adaline use the delta rule; the only difference is that the Perceptron has binary output,

while the Adaline has continuous valued output. The Feedforward Backpropagation network uses the

generalized delta rule, which is described next.

Generalized Delta Rule

While the delta rule uses local information on error, the generalized delta rule uses error information that is

not local. It is designed to minimize the total of the squared errors of the output neurons. In trying to achieve

this minimum, the steepest descent method, which uses the gradient of the weight surface, is used. (This is

also used in the delta rule.) For the next error calculation, the algorithm looks at the gradient of the error

C++ Neural Networks and Fuzzy Logic:Preface

Delta Rule

103

---

surface, which gives the direction of the largest slope on the error surface. This is used to determine the

direction to go to try to minimize the error. The algorithm chooses the negative of this gradient, which is the

direction of steepest descent. Imagine a very hilly error surface, with peaks and valleys that have a wide range

of magnitude. Imagine starting your search for minimum error at an arbitrary point. By choosing the negative

gradient on all iterations, you eventually end up at a valley. You cannot know, however, if this valley is the

global minimum or a local minimum. Getting stuck in a local minimum is one well−known potential problem

of the steepest descent method. You will see more on the generalized delta rule in the chapter on

backpropagation (Chapter 7).