222. Principal Component Analysis
222.1. PCA as application
Covariance Matrix
Principle components are always perpendicular
Let denote the -th feature, and are your features:
Example
Step 1 Center
Subtract (5, 4) from each point
Step 2 Covariance matrix
- top-left = 10 → how much &x& spreads on its own
- bottom-right = 5 → how much spreads on its own ( spreads twice as much)
- off-diagonal = 6 → how and move together (positive, so they rise together)
Step 3 — Eigenvalues and eigenvectors
Each principal component is an eigenvector of : a direction that only stretches, never rotates.
Eigenvalues — force a non-trivial solution with :
Eigenvectors — substitute each back into .
For , with :
For , with :
Normalise to unit length: