C++ Neural Networks and Fuzzy Logic: Preface

Eliminate Correlated Inputs Where Possible

You have seen that getting to the minimum number of inputs for a given problem is important in terms of

minimizing DOF and simplifying your model. Another way to reduce dimensionality is to look for correlated

inputs and to carefully eliminate redundancy. For example, you may find that the Swiss franc and German

mark are highly correlated over a certain time period of interest. You may wish to eliminate one of these

inputs to reduce dimensionality. You have to be careful in this process though. You may find that a seemingly

redundant piece of information is actually very important. Mark Jurik, of Jurik Consulting, in his paper on

data preprocessing, suggests that one of the best ways to determine if an input is really needed is to construct

neural network models with and without the input and choose the model with the best error on training and

test data. Although very iterative, you can try eliminating as many inputs as possible this way and be assured

that you haven’t eliminated a variable that really made a difference.

Another approach is sensitivity analysis, where you vary one input a little, while holding all others constant

and note the effect on the output. If the effect is small you eliminate that input. This approach is flawed

C++ Neural Networks and Fuzzy Logic:Preface

Eliminate Correlated Inputs Where Possible

301

---

because in the real world, all the inputs are not constant. Jurik’s approach is more time consuming but will

lead to a better model.

The process of decorrelation, or eliminating correlated inputs, can also utilize a linear algebra technique

called principal component analysis. The result of principal component analysis is a minimum set of variables

that contain the maximum information. For further information on principal component analysis, you should

consult a statistics reference or research two methods of principal component analysis: the Karhunen−Loev

transform and the Hotelling transform.