Beginning Anomaly Detection Using

Figure 5-42.  Using the label encoder on the categorical values in your data frame

Figure 5-43.  The output showing the new data frame with the categorical values 

converted to integer label equivalents

In this data set, the normal data entries comprise an overwhelmingly large 

proportion of the data entries, pretty much drowning out the anomalous data. Not only 

that, but you don’t want to pass in all of the data values into the RBM, so you will create 

a new data frame that contains a portion of normal data entries and all of the anomalous 

data entries. Run the code in Figure 

5-44

.

Chapter 5   Boltzmann maChines

---

201

As in Chapter 

2

, the normal labels are encoded as 4 so you can use them as the basis 

to separate the normal entries from the anomalies.

Since the data set is so large, the entries are shuffled randomly ten times before a sample of  

50,000 is selected from them. This is to ensure a random selection of values from the entire 

data set instead of having the entries just in the top 50,000. The output is shown in Figure 

5-45

.