Bias–Variance Tradeoff

455. Bias–Variance Tradeoff

We have:

Bias (systematic error)

Bias (𝑥) = 𝔼 [\hat{𝑓} (𝑥)] - 𝑓 (𝑥)

The difference is the bias

Consistently across datasets

Variance (random error)

Variance (𝑥) = 𝔼 [{(\hat{𝑓} (𝑥) - 𝔼 [\hat{𝑓} (𝑥)])}^{2}]

The spread of those curves around their average prediction at each $𝑥$ is the variance

Inconsistent across different data

Example

True Function:

𝑥^{2}

Observations:

Observations differ due to noise.

Average across datasets

{\hat{𝑓}}_{avg} (- 1) = \frac{1.1 + 0.5 + 1.2}{3} = \frac{2.8}{3} \approx 0.933

{\hat{𝑓}}_{avg} (0) = \frac{- 0.2 + 0.3 - 0.1}{3} = 0

{\hat{𝑓}}_{avg} (1) = \frac{1.05 + 1.2 + 0.7}{3} = \frac{2.95}{3} \approx 0.983

Bias

Bias at each point

Bias (- 1) = 0.933 - 1 = - 0.067

Bias (- 0) = 0 - 0 = 0

Bias (1) = 0.983 - 1 = - 0.017

Overall Bias

Mean Square Bias

{Bias}^{2} = \frac{1}{𝑛} \sum_{𝑖 = 1}^{𝑛} {(𝔼 [\hat{𝑓} (𝑥_{𝑖})] - 𝑓 (𝑥_{𝑖}))}^{2}

{Bias}^{2} \approx \frac{{(- 0.067)}^{2} + 0^{2} + {(- 0.017)}^{2}}{3} = 0.0016

Variance

Deviations

\begin{aligned} 𝑑_{1} & = 1.1 - 0.933 = 0.166 \\ 𝑑_{2} & = 0.5 - 0.933 = - 0.433 \\ 𝑑_{3} & = 1.2 - 0.933 = 0.266 \end{aligned}

\begin{aligned} 𝑑_{1}^{2} & = 0.028 \\ 𝑑_{2}^{2} & = 0.188 \\ 𝑑_{3}^{2} & = 0.071 \end{aligned}

Var (- 1) = \frac{0.028 + 0.188 + 0.071}{3} = 0.096

\begin{aligned} 𝑑_{1} & = - 0.2 - 0 = - 0.2 \\ 𝑑_{2} & = 0.3 - 0 = 0.3 \\ 𝑑_{3} & = 0.1 - 0 = - 0.1 \end{aligned}

\begin{aligned} 𝑑_{1}^{2} & = 0.04 \\ 𝑑_{2}^{2} & = 0.09 \\ 𝑑_{3}^{2} & = 0.01 \end{aligned}

Var (- 1) = \frac{0.04 + 0.09 + 0.01}{3} = 0.047

\begin{aligned} 𝑑_{1} & = 1.05 - 0.983 = 0.067 \\ 𝑑_{2} & = 1.2 - 0.983 = 0.217 \\ 𝑑_{3} & = 0.7 - 0.983 = - 0.283 \end{aligned}

\begin{aligned} 𝑑_{1}^{2} & = 0.004 \\ 𝑑_{2}^{2} & = 0.047 \\ 𝑑_{3}^{2} & = 0.08 \end{aligned}

Var (1) = \frac{0.004 + 0.047 + 0.08}{3} = 0.044

Overall Variance

Average Variance

\frac{0.096 + 0.047 + 0.044}{3} \approx 0.062

Overfitting

Bias-Variance Tradoff