Hands-On Machine Learning with Scikit-Learn and TensorFlow

198 | Chapter 7: Ensemble Learning and Random Forests

---

6
As 
m
grows, this ratio approaches 1 – exp(–1) ≈ 63.212%.
Figure 7-5. A single Decision Tree versus a bagging ensemble of 500 trees
Bootstrapping introduces a bit more diversity in the subsets that each predictor is
trained on, so bagging ends up with a slightly higher bias than pasting, but this also
means that predictors end up being less correlated so the ensemble’s variance is
reduced. Overall, bagging often results in better models, which explains why it is gen‐
erally preferred. However, if you have spare time and CPU power you can use cross-
validation to evaluate both bagging and pasting and select the one that works best.
Out-of-Bag Evaluation
With bagging, some instances may be sampled several times for any given predictor,
while others may not be sampled at all. By default a 
BaggingClassifier
samples 
m
training instances with replacement (
bootstrap=True
), where 
m
is the size of the
training set. This means that only about 63% of the training instances are sampled on
average for each predictor.
The remaining 37% of the training instances that are not
sampled are called 
out-of-bag
(oob) instances. Note that they are not the same 37%
for all predictors.
Since a predictor never sees the oob instances during training, it can be evaluated on
these instances, without the need for a separate validation set. You can evaluate the
ensemble itself by averaging out the oob evaluations of each predictor.
In Scikit-Learn, you can set 
oob_score=True
when creating a 
BaggingClassifier
to
request an automatic oob evaluation after training. The following code demonstrates
this. The resulting evaluation score is available through the 
oob_score_
variable: