Development of a fast relu activation function algorithm for deep learning problems

How to solve the dying ReLU problem?

A solution for that problem is the modification in the ReLU activation function resulted in variants of the ReLU like Noisy ReLU, Leaky ReLU, ELU mentioned in fig2.
LReLU: The derivative of the LReLU is 1 in the positive part and small fraction in the negative part. Instead of being 0 when z<0, a leaky ReLU allows a small, non-zero, constant gradient α (Normally, α=0.1).

An advantage of using LReLU is thus that we can worry less about the initialization of your neural network. but in the ReLU case, the neural network that never learns if the neurons are not activated at the start. we may have lots of dead ReLU without even knowing.
Example: ReLU layer in a Layer array
layers = [
#28x28x1 images with 'zerocenter' normalization
imageInputLayer([28 28 1])
#20 5x5 convolutions with stride &padding
convolution2dLayer(5,20)
#ReLU Layer
reluLayer
#2x2 maxpooling with stride 2x2 and padding [0 0 0 0]
maxPooling2dLayer(2,'Stride',2)
#10 fully connected layer
fullyConnectedLayer(10)
softmaxLayer
classificationLayer
]

Conclusion

The activation function is an important part of the convolution neural network, which can map the nonlinear features of the data, so that the convolution neural network has enough ability to capture the complex pattern. On the basis of the traditional convolution neural network, this paper enhances data, adds the local response normalization layer, and using the maximum pooling and so on. Besides the problem of insufficient expression of the Relu function, And the softsign activation function is nonlinear and the improved fault tolerance, an improved ReLu segmentation correction activation function is proposed. Based on the Google deep learning platform TensorFlow, this paper uses the activation function to construct the modified convolution neural network structure model.