Beginning Anomaly Detection Using

eps: (Optional). Some float value where epsilon e >= 0. Epsilon is 

some small number, described as 10E-8 in the paper, to help prevent 

division by 0. Default is 1e-8.

• 

weight_decay: A l2_penalty for weights that are too high, helping 

incentivize smaller model weights. Default = 0.

• 

amsgrad: A Boolean on whether or not to apply the AMSGrad 

version of this algorithm. For more details on the implementation 

of this algorithm, check out “On the Convergence of Adam and 

Beyond.” Default=False.

 RMSProp

torch.optim.RMSprop()

RMSprop is a good algorithm for recurrent neural networks. RMSprop is a gradient-

based optimization technique developed to help address the problem of gradients 

becoming too large or too small. RMSprop helps combat this problem by normalizing 

the gradient itself using the average of the squared gradients. In Chapter 

7

, it’s explained 

that one of the problems with RNNs is the vanishing/exploding gradient problem, 

appendix B   intro to pytorch

---

392

leading to the development of LSTMs and GRU networks. And so it’s of no surprise that 

RMSprop pairs well with recurrent neural networks.

This function has several parameters:

• 

params: Some iterable of parameters to optimize, or dictionaries 

with parameter groups. This can be something like model.

parameters().

•