Types of Multi Classification

This blog introduces different types of multi classification systems.

Multiclass Classification

Multiclass classifiers can distinguish between more than two classes other than binary classifiers. Stochastic gradient descent (SGD) classifiers, Random Forest classifiers, and naive Bayes classifiers etc. are capable of handling multiple classes natively. On the other hand, Logistic Regression or Support Vector Machine classifiers are strictly binary classifiers.

There are various strategies that you can use to perform multiclass classification with multiple binary classifiers. One way to create a system that can classify the instances into N classes is to train N binary classifiers. Then when we want to classify an instance, we get the decision score from each classifier for that instance and we select the class whose classifier outputs the highest score. This is called the one-versus-the-rest (OvR) strategy (also called one-versus-all). Another strategy is to train a binary classifier for every pair of instances. This is called the one-versus-one (OvO) strategy. If there are N classes, you need to train N×(N-1)/2 classifiers. The main advantage of OvO is that each classifier only needs to be trained on the part of the training set for the two classes that it must distinguish. Support Vector Machine classifiers scale poorly with the size of the training set. Therefore, OvO is preferred because it is faster to train many classifiers on small training sets than to train few classifiers on large training sets. For most binary classification algorithms, however, OvR is preferred.

Multilabel Classification

In some cases, we may want the classifier to output multiple classes for each instance. A classification system that outputs multiple binary tags is called a multilabel classification system.

One approach to evaluate a multilabel classifier is to measure the F1 score for each individual label (or any other binary classifier metric), then simply compute the average score.

Multioutput Classification

For multioutput classification, it is a generalization of multilabel classification where each label can be multiclass (i.e., it can have more than two possible values). Multioutput systems are not limited to classification tasks, we could even have a system that outputs multiple labels per instance, including both class labels and value labels.