Activation Functions in Neural Networks
An activation function in a neural network defines how the weighted sum of the input is transformed into an output from a node or nodes in a layer of the network. Sometimes it is called transfer function or squashing function.
Activation functions allow Deep Learning models to capture nonlinearity. If ANNs (artificial neural networks) are a symbolic representation of biological neural networks, then activation functions mirror the ability of neurons being able to fire with different levels of intensity based on the rapidity of how often they fire. A model with no activation functions would just be a linear model. In the output layer, activation functions make the results of the neural network’s forward propagation step interpretable.
The choice of activation function in the hidden layer will control how well the network model learns the training dataset. The choice of activation function in the output layer will define the type of predictions the model can make. All hidden layers typically use the same activation function. The output layer will typically use a different activation function from the hidden layers and is dependent upon the type of prediction required by the model. Activation functions are also typically differentiable, meaning the first-order derivative can be calculated for a given input value. This is required given that neural networks are typically trained using the backpropagation of error algorithm that requires the derivative of prediction error in order to update the weights of the model.
A neural network may have zero or more hidden layers. Typically, a differentiable nonlinear activation function is used in the hidden layers of a neural network. This allows the model to learn more complex functions than a network trained using a linear activation function. There are three most commonly used activation functions in hidden layers: Rectified Linear Activation (ReLU), Logistic (Sigmoid) and Hyperbolic Tangent (Tanh).
