C++ Neural Networks and Fuzzy Logic: Preface

Statistical Training and Simulated Annealing

The Boltzmann machine (and Cauchy machine) uses probabilities and statistical theory, along with an energy

function representing temperature. The learning is probabilistic and is called simulated annealing. At different

temperature levels, a different number of iterations in processing are used, and this constitutes an annealing

schedule. Use of probability distributions is for the goal of reaching a state of global minimum of energy.

Boltzmann distribution and Cauchy distribution are probability distributions used in this process. It is

obviously desirable to reach a global minimum, rather than settling down at a local minimum.

Figure 6.1 clarifies the distinction between a local minimum and a global minimum. In this figure you find the

graph of an energy function and points A and B. These points show that the energy levels there are smaller

than the energy levels at any point in their vicinity, so you can say they represent points of minimum energy.

The overall or global minimum, as you can see, is at point B, where the energy level is smaller than that even

at point A, so A corresponds only to a local minimum. It is desirable to get to B and not get stopped at A

itself, in the pursuit of a minimum for the energy function. If point C is reached, one would like the further

movement to be toward B and not A. Similarly, if a point near A is reached, the subsequent movement should

avoid reaching or settling at A but carry on to B. Perturbation techniques are useful for these considerations.

Figure 6.1

  Local and global minima.