ReLU Activation Function VariantsThe ReLU activation function suffers from a problem known as the dying ReLUs: during training, some neurons effectively die, meaning they…Dec 29, 2021Dec 29, 2021
The Vanishing/Exploding Gradients ProblemsGradients often get smaller and smaller as the algorithm progresses down to the lower layers. As a result, the Gradient Descent update…Nov 4, 2021Nov 4, 2021
Loss Function in Deep LearningIn the context of an optimization algorithm, the function used to evaluate a candidate solution (i.e. a set of weights) is referred to as…Oct 11, 2021Oct 11, 2021
Hidden Layer Activation FunctionsThis blog introduces three most commonly used activation functions in hidden layers: Rectified Linear Activation (ReLU), Logistic (Sigmoid)…Oct 10, 2021Oct 10, 2021
Activation Functions in Neural NetworksAn activation function in a neural network defines how the weighted sum of the input is transformed into an output from a node or nodes in…Oct 9, 2021Oct 9, 2021
Forward & Backward PropagationNeural Networks have two major processes: Forward Propagation and Back Propagation. During Forward Propagation, we start at the input layer…Oct 8, 2021Oct 8, 2021
Multilayer PerceptronAn MLP (Multilayer Perceptron) is composed of one passthrough input layer, one or more layers of TLUs (threshold logic units), called…Oct 8, 2021Oct 8, 2021
PerceptronAn ANN (artificial neural network) is a Machine Learning model inspired by the networks of biological neurons found in the brains. An…Oct 8, 2021Oct 8, 2021
AIC and BICThis blog introduces to two measures: AIC (Akaike information criterion) and BIC (Bayesian information criterion), which give a…Oct 6, 2021Oct 6, 2021
Gaussian Mixture ModelA Gaussian mixture model (GMM) is a probabilistic model that assumes that the instances were generated from a mixture of several Gaussian…Oct 5, 2021Oct 5, 2021
