Beginning Anomaly Detection Using

rectified linear unit (ReLU) in the first 

dense layer, to 

softmax in the second.

Mathematically, the 

ReLU function is defined as y = max(0, x), so when the node 

calculates the dot products between the input and the weights and adds the bias, it 

simply outputs whatever is bigger between 0 or the calculation.

The graph for ReLU looks like Figure 

3-42

.

Figure 3-41.  Showing what a flatten layer does to an input 3x3 image

Chapter 3   IntroduCtIon to deep LearnIng

---

103

The general formula for softmax is shown in Figure 

3-43

.

As for the 

optimizer, it is set to the Adam optimizer, a type of gradient-based 

optimizer. By default, the parameter known as the 

learning rate is set to 0.001. Recall 

that the learning rate helps determine the step size taken by the optimization algorithm 

to see how much to adjust the weights by.

After executing the code in Figure 

3-43

, you get the output in Figure 

3-44

.