C++ Neural Networks and Fuzzy Logic: Preface

C++ Neural Networks and Fuzzy Logic

by Valluru B. Rao

MTBooks, IDG Books Worldwide, Inc.

ISBN: 1558515526   Pub Date: 06/01/95

Previous Table of Contents Next

Delta Rule

The delta rule is also known as the least mean squared error rule (LMS). You first calculate the square of the

errors between the target or desired values and computed values, and then take the average to get the mean

squared error. This quantity is to be minimized. For this, realize that it is a function of the weights themselves,

since the computation of output uses them. The set of values of weights that minimizes the mean squared error

is what is needed for the next cycle of operation of the neural network. Having worked this out

mathematically, and having compared the weights thus found with the weights actually used, one determines

their difference and gives it in the delta rule, each time weights are to be updated. So the delta rule, which is

also the rule used first by Widrow and Hoff, in the context of learning in neural networks, is stated as an

equation defining the change in the weights to be affected.

Suppose you fix your attention to the weight on the connection between the ith neuron in one layer and the jth

neuron in the next layer. At time t, this weight is w

ij

(t) . After one cycle of operation, this weight becomes

w

ij

(t + 1). The difference between the two is w

ij

(t + 1) − w

ij

(t), and is denoted by [Delta]w

ij

 . The delta rule

then gives [Delta]w

ij

 as :

[Delta]w

ij

 = 2[mu]x

i

(desired output value – computed output value)

j

Here, [mu] is the learning rate, which is positive and much smaller than 1, and x

i

 is the ith component of the

input vector.