Beginning Anomaly Detection Using

Figure 5-44.  Code to define an anomaly data set and a normal data set. Then, the 

normal data set is shuffled to ensure random selections, and a new data set named 

novelties is formed

Figure 5-45.  The output of the code in Figure 

5-44

One thing about the KDDCUP data set is that there are a massive amount of entries 

with data values as either miniscule values or as 0. You’ve dealt with massive values 

with the credit card data set, and you know that those values can throw off the training 

process entirely. Likewise, massive amounts of zero values or really tiny data values can 

also hamper the training process.

Chapter 5   Boltzmann maChines

---

202

Since novelties.head() only displays some of the columns, you’ll have to use 

something else to check every column, so look at the code in Figure 

5-46

.

Figure 5-46.  Code to print all the columns and five rows in the data frame

The parameters are self-explanatory. In the example, all 41 columns are displayed for 

the first 5 rows (Figure 

5-47

 and Figure 

5-48

).

Figure 5-47.  The output from the code in Figure 

5-46

. Notice the massive amount 

of zero values in the columns of the data entries

Chapter 5   Boltzmann maChines

---

203

While the large amount of zero-value entries might not have affected the isolation 

forest, they will certainly mess with the training process of the RBM, leading to terrible 

AUC scores. Therefore, standardizing all of the values will help the RBM during the 

training process and help it attain proper AUC scores.

You don’t want to standardize the data values for the columns protocol_type, flag, or 

label, so exclude them specifically (see Figure 

5-49

).

Figure 5-48.  The rest of the output continued from Figure 

5-46

. There are still 

many zero values or really small values in each entry

Chapter 5   Boltzmann maChines

---

204

The output showing the standardized data is shown in Figure 

5-50

, Figure 

5-51

, and 

Figure 

5-52

.