AIC & BIC

223. AIC & BIC

Picking between competing models

Maximized likelihood

223.1. AIC

AIC = \underset{penalty term}{\underset{⏟}{2 𝑘}} - \underset{fit term}{\underset{⏟}{2 \cdot ln (\hat{𝐿})}}

$𝑘$ : number of parameters in the model
$𝑛$ : number of data points

223.2. BIC

BIC = \underset{penalty term}{\underset{⏟}{𝑘 \cdot ln (𝑛)}} - \underset{fit term}{\underset{⏟}{2 \cdot ln (\hat{𝐿})}}

$𝑘$ : number of parameters in the model
$𝑛$ : number of data points

)

Explanation:

The fit term, $- 2 \cdot ln (𝐿)$ , is identical in both (Better fit $\to$ bigger L̂ $\to$ bigger ln(L̂) $\to$ this term goes down $\to$ criterion goes down
The penalty term is the only difference:
- AIC charges 2 per parameter
- BIC charges ln(n) per parameter

Example

You have n = 100 data points and two candidate models for them:

Model A (simpler): k = 3 parameters, maximized log-likelihood ln(L̂) = −120
Model B (fancier): k = 6 parameters, maximized log-likelihood ln(L̂) = −116

Model B fits better — its log-likelihood is higher (−116 > −120), which it should be, since it has more parameters to play with. The whole question is whether those 3 extra parameters are worth it.

AIC $2 𝑘 - 2 \cdot ln (\hat{𝐿})$

Model A: 2·3 − 2·(−120) = 6 + 240 = 246
Model B: 2·6 − 2·(−116) = 12 + 232 = 244

AIC picks Model B (244 < 246)

BIC $𝑘 \cdot ln (𝑛) - 2 \cdot ln (\hat{𝐿})$

Model A: 3·4.6 − 2·(−120) = 13.8 + 240 = 253.8
Model B: 6·4.6 − 2·(−116) = 27.6 + 232 = 259.6

BIC picks Model A (253.8 < 259.6)

Same data, same two models, opposite verdicts. Here’s the why, and it’s exactly that per-parameter cost from before: Going from A to B, the fit term drops by 8 (from 240 to 232) — that’s the reward for the better fit. The 3 extra parameters cost you:

BIC charges more per parameter (because ln(100) > 2), so it demands a bigger fit improvement before it’ll accept extra complexity. That’s why it leans toward simpler models.