Kinder ChenReLU Activation Function VariantsThe ReLU activation function suffers from a problem known as the dying ReLUs: during training, some neurons effectively die, meaning they…2 min read·Dec 29, 2021----
Kinder ChenThe Vanishing/Exploding Gradients ProblemsGradients often get smaller and smaller as the algorithm progresses down to the lower layers. As a result, the Gradient Descent update…2 min read·Nov 4, 2021----
Kinder ChenLoss Function in Deep LearningIn the context of an optimization algorithm, the function used to evaluate a candidate solution (i.e. a set of weights) is referred to as…2 min read·Oct 11, 2021----
Kinder ChenHidden Layer Activation FunctionsThis blog introduces three most commonly used activation functions in hidden layers: Rectified Linear Activation (ReLU), Logistic (Sigmoid)…2 min read·Oct 10, 2021----
Kinder ChenActivation Functions in Neural NetworksAn activation function in a neural network defines how the weighted sum of the input is transformed into an output from a node or nodes in…2 min read·Oct 9, 2021----
Kinder ChenForward & Backward PropagationNeural Networks have two major processes: Forward Propagation and Back Propagation. During Forward Propagation, we start at the input layer…2 min read·Oct 8, 2021----
Kinder ChenMultilayer PerceptronAn MLP (Multilayer Perceptron) is composed of one passthrough input layer, one or more layers of TLUs (threshold logic units), called…2 min read·Oct 8, 2021----
Kinder ChenPerceptronAn ANN (artificial neural network) is a Machine Learning model inspired by the networks of biological neurons found in the brains. An…2 min read·Oct 8, 2021----
Kinder ChenAIC and BICThis blog introduces to two measures: AIC (Akaike information criterion) and BIC (Bayesian information criterion), which give a…2 min read·Oct 6, 2021----
Kinder ChenGaussian Mixture ModelA Gaussian mixture model (GMM) is a probabilistic model that assumes that the instances were generated from a mixture of several Gaussian…2 min read·Oct 5, 2021----
