Using our wildest imagination, we can picture a dataset consisting
of features X and labels Y, as on the left. Also imagine that we’d
like to generalize this relationship to additional values of X -
that we’d like to predict future values based on what we’ve
- already seen before.
-
+ already seen before.
With our imagination now undoubtedly spent, we can take a very
simple approach to modeling the relationship between X and Y by
just drawing a line to the general trend of the data.
@@ -138,12 +118,10 @@
A Simple Model
-
Our simple model isn’t the best at modeling the relationship -
clearly there's information in the data that it's failing to
- capture.
-
+ capture.
We'll measure the performance of our model by looking at the
mean-squared error
of its output and the true values (displayed in the bottom
barchart). Our model is close to some of the training points, but
overall there's definitely room for improvement.
-
The error on the training data is important for model tuning, but
what we really care about is how it performs on data we haven't
seen before, called test data. So let's check that out as well.
@@ -159,12 +136,10 @@
Low Complexity & Underfitting
-
Uh-oh, it looks like our earlier suspicions were correct - our
model is garbage. The test error is even higher than the train
- error!
-
+ error!
In this case, we say that our model is
underfitting the data: our
model is so simple that it fails to adequately capture the
@@ -177,16 +152,13 @@
>
A Complex Model
-
Our previous model performed poorly because it was too simple.
Let's try our luck with something more complex. In fact, let's get
as complex as we can - let's train a model that predicts every
- point in our training data perfectly.
-
+ point in our training data perfectly.
Great! Now our training error is zero. As the old saying goes in
Tennessee: Fool me once - shame on you. Fool me twice - er... you
can't get fooled again ;).
@@ -194,7 +166,6 @@
High Complexity & Overfitting
-Wait a second... Even though our training error from our model was effectively zero, the error on our test data is high. What gives? @@ -226,12 +197,9 @@
noise.
-
-
- Or, mathematically:
-
-
-
+
+ Or, mathematically:
+
We can’t do much about the irreducible
@@ -246,9 +214,7 @@
>Bias represents the difference between the average prediction
and the true value:
-
-
-
+
The term is a tricky one.
It refers to the average prediction after the model has been
trained over several independent datasets. We can think of the
@@ -260,7 +226,6 @@
comes from bias.
- -
- +
The term is a tricky one. It refers to the average prediction after the model has been trained over several independent datasets. We can think of the @@ -260,7 +226,6 @@
comes from bias.
Variance
@@ -270,8 +235,7 @@
>variance measures how much, on average, predictions vary for a
given data point:
-
-
+
As you can see in the bottom plot, predictions from overfit
(high-complexity) models show a lot more error from variance than
from bias. It’s easy to imagine that any unseen data points will
@@ -284,13 +248,11 @@
To obtain our best results, we should work to find a happy medium
between a model that is so basic it fails to learn meaningful
patterns in our data, and one that is so complex it fails to
- generalize to unseen data .
-
+ generalize to unseen data .
In other words, we don’t want an underfit model, but we don’t want
an overfit model either. We want something in between - something
with enough complexity to learn learn the generalizable patterns
- in our data.
-
+ in our data.
By trading some bias for variance (i.e. increasing the
complexity of our model), and without going overboard, we can
@@ -303,35 +265,29 @@
We just showed, at different levels of complexity, a sample of
model realizations alongside their corresponding prediction error
- decompositions.
-
+ decompositions.
Let’s direct our focus to the error decompositions across model
- complexities.
-
+ complexities.
For each level of complexity, we’ll aggregate the error
decomposition across all data-points, and plot the aggregate
- errors at their level of complexity.
-
+ errors at their level of complexity.
This aggregation applied to our balanced model (i.e. the middle
level of complexity) is shown to the left.
+ generalize to unseen data .
In other words, we don’t want an underfit model, but we don’t want an overfit model either. We want something in between - something with enough complexity to learn learn the generalizable patterns - in our data. -
+ in our data.
By trading some bias for variance (i.e. increasing the complexity of our model), and without going overboard, we can @@ -303,35 +265,29 @@
We just showed, at different levels of complexity, a sample of
model realizations alongside their corresponding prediction error
- decompositions.
-
+ decompositions.
Let’s direct our focus to the error decompositions across model
- complexities.
-
+ complexities.
For each level of complexity, we’ll aggregate the error
decomposition across all data-points, and plot the aggregate
- errors at their level of complexity.
-
+ errors at their level of complexity.
This aggregation applied to our balanced model (i.e. the middle
level of complexity) is shown to the left.
The Bias Variance Trade-off
+The Bias Variance Tradeoff
Repeating this aggregation across our range of model complexities,
we can see the relationship between bias and variance in
prediction errors manifests itself as a U-shaped curve detailing
- the trade off between bias and variance.
-
+ the tradeoff between bias and variance.
When a model is too simple (i.e. small values along the x-axis),
it ignores useful information, and the error is composed mostly of
- that from bias.
-
+ that from bias.
When a model is too complex (i.e. large values along the x-axis),
it memorizes non-general patterns, and the error is composed
- mostly of that from variance.
-
+ mostly of that from variance.
The ideal model aims to minimize both bias and variance. It
lays in the sweet spot - not too simple, nor too complex.
@@ -341,21 +297,18 @@