Beginning Anomaly Detection Using

Figure 6-7.  A RNN with tanh activation

Figure 6-8.  Common activation functions

The key idea behind activation functions is to add non-linearity to the data to align 

better with real-world problems and real-world data. In Figure 

6-9

, the top graph shows 

linearity and the bottom graph shows nonlinearity.

Chapter 6   Long Short-term memory modeLS 

---

220

Clearly, there is no linear equation to handle the nonlinearity so we need an 

activation function to deal with this property. The different activation functions are listed 

at 

https://keras.io/activations/

.

In time series data, the data is spread over a period of time, not some instantaneous 

set such as seen in Chapter 

4

 autoencoders, for example. So not only it is important to look 

at the instantaneous data at some time T, it is also important for older historical data to 

the left of this point to be propagated through the steps in time. Since we need the signals 

from historical data points to survive for a long period of time, we need an activation 

function that can sustain information for a longer range before going to zero. tanh is the 

ideal activation function for the purpose and is graphed as shown in Figure 

6- 10

.