Beginning Anomaly Detection Using

Input is the previous layer, and training is a parameter that determines whether or not 

you want this dropout layer to function outside of training (such as during evaluation).

Figure 

B-18

 shows an example of how you can use this layer in the forward function.

Figure B-18.  The dropout layer in the forward function of a model

appendix B   intro to pytorch

---

383

Figure B-19.  The general formula that ReLU follows

So with dropout, you have two ways of applying it, both producing similar outputs. In 

fact, the layer itself is an extension of the functional version of dropout, which itself is an 

interface. This is really up to personal preference, since both are still dropout layers and 

there’s no real difference in behavior.

 ReLU

torch.nn.ReLU()

ReLU, or “Rectified Linear Unit”, performs a simple activation based on the function, 

as shown in Figure 

B-19

.

Here is the parameter:

• 

inplace: If set to True, it will perform the operation in place.  

Default = False.

For ReLU, the graph can look like Figure 

B-20

.

Figure B-20.  The general graph of a ReLU function

appendix B   intro to pytorch

---

384

Similarly to dropout, you can define this as a layer within the model itself, or apply 

ReLU in the forward function like so:

torch.nn.functional.relu(input, inplace=False)

Input is the previous layer.

Figure 

B-21

 shows an example of how you can use this layer in the forward function.

Just like with dropout, you have two ways of applying ReLU, but it all boils down to 

personal preference.

Figure B-22.  The general formula for softmax. The parameter i goes up until the 

total number of samples, which is K