Bagging and Pasting

A group of predictors is called an ensemble. The technique aggregating the predictions of a group of predictors is called Ensemble Learning, and an Ensemble Learning algorithm is called an Ensemble method. In this blog, one of the most popular Ensemble methods, bagging is introduced.

Bagging and Pasting

One approach to get a diverse set of classifiers is to use the same training algorithm for every predictor and train them on different random subsets of the training set. When sampling is performed with replacement, this method is called bagging (or bootstrap aggregating). When sampling is performed without replacement, it is called pasting. Both bagging and pasting allow training instances to be sampled several times across multiple predictors, but only bagging allows training instances to be sampled several times for the same predictor.

The aggregation function is typically the statistical mode, i.e., the most frequent prediction for classification, or the average for regression. Each individual predictor has a higher bias than if it were trained on the original training set, but aggregation reduces both bias and variance. Generally, the net result is that the ensemble has a similar bias but a lower variance than a single predictor trained on the original training set.

Bootstrapping introduces a bit more diversity in the subsets that each predictor is trained on, so bagging ends up with a slightly higher bias than pasting; but the extra diversity also means that the predictors end up being less correlated, so the ensemble’s variance is reduced. Overall, bagging often results in better models, which explains why it is generally preferred. However, if we have spare time and CPU power, we can use cross-validation to evaluate both bagging and pasting and select the one that works best.