Beginning Anomaly Detection Using

Figure 5-49.  Standardizing every value except for the columns the label encoder 

transformed

Figure 5-50.  The code in a Jupyter cell

Figure 5-51.  The first part of the output showing that most of the values have been 

transformed

Figure 5-52.  The same output but scrolled right to show that more of the values 

have been transformed

Chapter 5   Boltzmann maChines

---

205

As you can see, most of the zero value entries have been standardized in accordance 

with all of the values in their respective columns. The few nonzero entries in these 

columns will help the scaler to standardize the rest of the values in that column.

Just as you want to avoid massive values in the training set, you also seek to avoid 

large amounts of zero value entries in the data. In both such cases, the calculations 

for the gradient will be thrown off, resulting in cases such as the “exploding gradient” 

(gradients so big that the model can never converge on the local minimum) or the 

“vanishing gradient” (gradients so small that they are practically nonexistent, and 

the model never converges on the local minimum). An abundance of values that are 

too large or too small can negatively affect the training process, so it’s a good idea to 

preprocess the data set before training the model on it.

Now you can move on to defining your training and testing sets (see Figure 

5-53

).

Figure 5-53.  Defining the training and testing sets and printing out the shapes of each

The corresponding output is shown in Figure 

5-54

.

Chapter 5   Boltzmann maChines

---

206

The 43,000 entries indicate a roughly 80-20 split between the training and testing 

data sets.

Again, you drop the last column, since this is 

unsupervised training (although it is 

true that both the anomalies and the normal entries are labeled, the model only sees 

unlabeled data during the training and prediction processes).

With your data sets created, you can define and train the model (see Figure 

5-55

, 

Figure 

5-56

, and Figure 

5-57

).