Logistic Regression

Logistic Regression (i.e., Logit Regression) is commonly used binary classifier to estimate the probability that an instance belongs to a particular class. Logistic Regression model computes a weighted sum of the input features with a bias term, and outputs the logistic of the result. The logistic is a sigmoid function (i.e., S-shaped) that outputs a number between 0 and 1. Logistic function is

The objective of Logistic Regression model training is to set the parameter vector θ so that the model estimates high probabilities for positive instances (y = 1) and low probabilities for negative instances (y = 0).

The cost function over the whole training set is the average cost over all training instances. Logistic Regression cost function (also called log loss) is

There is no known closed-form equation to compute the value of θ that minimizes this cost function. However, the cost function is convex, so Gradient Descent is guaranteed to find the global minimum. Logistic Regression models can be regularized using l1 or l2 penalties. Scikit-Learn actually adds an l2 penalty by default. The hyperparameter controlling the regularization strength of a Scikit-Learn LogisticRegression model is not alpha, but its inverse: C. The higher the value of C, the less the model is regularized.