Beginning Anomaly Detection Using

Figure 5-30.  Code to check the five-number summary of the normal data

The output should look somewhat like Figure 

5-31

.

Chapter 5   Boltzmann maChines

---

194

Now let’s check the five-number summary of the anomalies (see Figure 

5-32

).

Figure 5-31.  The five-number summary shows that the normal data is right 

skewed, since the values for each quartile are in the negative, while the outlier 

values in the tail bring the mean up into the positives

Figure 5-32.  The code to check the five-number summary for the anomalies

The output should look somewhat like Figure 

5-33

.

Figure 5-33.  Looking at the data, it seems that all of the anomalies are below 250. 

Knowing this, you can now pick a threshold value so only the relevant data is displayed 

on the graph

Chapter 5   Boltzmann maChines

---

195

Knowing the general distribution of the data, you can pick a threshold value so that 

only relevant data is shown on the graph. You know the majority of the normal data is 

situated around the value zero, so the outliers are irrelevant to you since they won’t show 

up on the graph anyways (a few values for 20,000 won’t show up when compared to tens 

of thousands of values around zero).

And so let’s choose a cutoff point of 250, since the maximum free energy for an 

 anomaly is at around 232. Figure 

5-34

 shows a graph of the free energy vs. the probabilities 

for the test set.

Figure 5-34.  Code to plot the free energies associated with x_test and the 

respective probabilities

Figure 

5-35

 shows the code.

Figure 5-35.  The code to graph the free energies of the data points and their 

probabilities

Chapter 5   Boltzmann maChines

---

196

The output graph is shown in Figure 

5-36

.

Figure 5-36.  The graph of the free energies vs. the probability of the normal and 

anomaly data points in the test set with costs less than 500

The graph automatically graphs the probabilities of the data points based on 

their free energies, but this isn’t exactly made very clear for you to see. The way the 

probabilities are computed correspond with this line of code:

probs = costs / np.sum(costs)

This essentially takes the individual free energy and divides it by the total free energy 

associated with the whole set.

The RBM seems to have learned the distribution well enough that you can see a 

pretty clear separation between the normal values and the anomalies, although there is 

a bit of an overlap. In any case, the RBM performed pretty well on the credit card dataset 

with an AUC of 95.84%.

Chapter 5   Boltzmann maChines

---

197