Beginning Anomaly Detection Using

Figure 6-9.  Linear and nonlinear data plots

Figure 6-10.  tanh activation

Chapter 6   Long Short-term memory modeLS 

---

221

We also need sigmoid (another activation function) as a way to either remember or 

forget the information. A sigmoid activation function is shown in Figure 

6-11

.

Now, conventional RNNs have a tendency to remember everything including 

unnecessary inputs which results in an inability to learn from long sequences. By 

contrast, LSTMs selectively remember important inputs and this allows them to handle 

both short-term and long-term dependencies.

So how does LSTM do this? It does this by releasing information between the hidden 

state and the cell state using three important gates: the forget gate, the input gate, and 

the output gate. A common LSTM unit is composed of a cell, an input gate, an output 

gate, and a forget gate. The cell remembers values over arbitrary time intervals and 

the three gates regulate the flow of information into and out of the cell.

A more detailed LSTM architecture is shown in Figure 

6-12

. There are a couple of key 

functions used, the tanh and the sigmoid, which are activation functions. F

t

 is the forget 

gate, I

t

 is the input gate, and O

t

 is the output gate.