222. Principal Component Analysis

222.1. PCA as application

Covariance Matrix

Principle components are always perpendicular

Let 𝑋𝑗 denote the 𝑗-th feature, and 𝑋1,𝑋2,,𝑋𝑑 are your 𝑑 features:

𝐶=[var(𝑋1)cov(𝑋1,𝑋2)cov(𝑋1,𝑋𝑑)cov(𝑋2,𝑋1)var(𝑋2)cov(𝑋2,𝑋𝑑)cov(𝑋𝑑,𝑋1)cov(𝑋𝑑,𝑋2)var(𝑋𝑑)]
Example
𝑋={(0,2),(2,0),(4,3),(6,4),(8,6)}

Step 1 Center

𝑥=0+2+4+6+85=5𝑦=2+0+3+4+65=3

Subtract (5, 4) from each point

𝑋̄={(4,1),(2,3),(0,0),(2,1),(4,3)}

Step 2 Covariance matrix

var(𝑥)=16+4+0+4+164var(𝑦)=1+9+0+1+94cov(𝑥,𝑦)=(4)(1)+(2)(3)+0+(2)(1)+(4)(3)4=6𝐶=[10665]
  • top-left = 10 → how much &x& spreads on its own
  • bottom-right = 5 → how much 𝑦 spreads on its own (𝑥 spreads twice as much)
  • off-diagonal = 6 → how 𝑥 and 𝑦 move together (positive, so they rise together)

Step 3 — Eigenvalues and eigenvectors

Each principal component is an eigenvector 𝑣 of 𝐶: a direction that 𝐶 only stretches, never rotates.

𝐶𝑣=𝜆𝑣(𝐶𝜆𝐼)𝑣=0

Eigenvalues — force a non-trivial solution with det(𝐶𝜆𝐼)=0:

det(𝐶𝜆𝐼)=(10𝜆)(5𝜆)36=𝜆215𝜆+14=(𝜆14)(𝜆1)=0𝜆1=14or𝜆2=1

Eigenvectors — substitute each 𝜆 back into (𝐶𝜆𝐼)𝑣=0.

For 𝜆1=14, with 𝐶14𝐼=[4669]:

4𝑣1+6𝑣2=02𝑣1=3𝑣2𝑣[32]

For 𝜆2=1, with 𝐶1𝐼=[9664]:

9𝑣1+6𝑣2=03𝑣1=2𝑣2𝑣[23]

Normalise to unit length:

PC1=113[32]PC2=113[23]