C++ Neural Networks and Fuzzy Logic: Preface

Adding the Momentum Term

A simple change to the training law that sometimes results in much faster training is the addition of a

momentum term. The training law for backpropagation as implemented in the simulator is:

Weight change = Beta * output_error * input

Now we add a term to the weight change equation as follows:

Weight change = Beta * output_error * input +

                Alpha*previous_weight_change

The second term in this equation is the momentum term. The weight change, in the absence of error, would be

a constant multiple by the previous weight change. In other words, the weight change continues in the

direction it was heading. The momentum term is an attempt to try to keep the weight change process moving,

and thereby not get stuck in local minimas.

Code Changes

The effected files to implement this change are the layer.cpp file, to modify the update_weights() member

function of the output_layer class, and the main backprop.cpp file to read in the value for alpha and pass it to

the member function. There is some additional storage needed for storing previous weight changes, and this

affects the layer.h file. The momentum term could be implemented in two ways: