SVD

89. SVD

Every real matrix $𝐴$ — square or rectangular, full-rank or rank-deficient — has a Singular Value Decomposition:

𝐴 = 𝑈 Σ 𝑉^{𝑇}

where, for an $𝑚 \times 𝑛$ matrix $𝐴$ :

$𝑈$ is $𝑚 \times 𝑚$ orthogonal — columns are the left singular vectors
$Σ$ is $𝑚 \times 𝑛$ diagonal with non-negative entries $𝜎_{1} \geq 𝜎_{2} \geq \dots \geq 0$ — the singular values
$𝑉$ is $𝑛 \times 𝑛$ orthogonal — columns are the right singular vectors

The singular values are the square roots of the eigenvalues of $𝐴^{𝑇} 𝐴$ (which is symmetric positive semi-definite):

𝜎_{𝑖} = \sqrt{𝜆_{𝑖} (𝐴^{𝑇} 𝐴)}

Number of non-zero singular values = rank of $𝐴$ .

Every linear transformation $𝐴 : ℝ^{𝑛} \to ℝ^{𝑚}$ decomposes into:

Rotate / reflect in the input space (multiply by $𝑉^{𝑇}$ )
Stretch along the new axes by factors $𝜎_{1}, 𝜎_{2}, \dots$ (multiply by $Σ$ )
Rotate / reflect in the output space (multiply by $𝑈$ )

So every matrix is a rotation–stretch–rotation. The singular values measure how much $𝐴$ stretches each direction.

Example

𝐴 = [\begin{matrix} 3 & 0 \\ 0 & 1 \\ 0 & 0 \end{matrix}]

Already in SVD form: $𝑈 = 𝐼_{3}$ , $Σ = 𝐴$ , $𝑉 = 𝐼_{2}$ . Singular values: $𝜎_{1} = 3, 𝜎_{2} = 1$ .

$𝐴$ maps the unit circle in $ℝ^{2}$ to an ellipse in $ℝ^{3}$ with semi-axes $3$ and $1$ .

For rank- $𝑟$ matrix $𝐴$ , drop the zero singular values:

𝐴 = 𝑈_{𝑟} Σ_{𝑟} 𝑉_{𝑟}^{𝑇}

where $𝑈_{𝑟}$ is $𝑚 \times 𝑟$ , $Σ_{𝑟}$ is $𝑟 \times 𝑟$ diagonal, $𝑉_{𝑟}$ is $𝑛 \times 𝑟$ . Same product, less storage.

The truncated SVD (keeping the top $𝑘$ singular values) is the best rank- $𝑘$ approximation of $𝐴$ in both Frobenius and spectral norms:

𝐴_{𝑘} = \sum_{𝑖 = 1}^{𝑘} 𝜎_{𝑖} 𝑢_{𝑖} 𝑣_{𝑖}^{𝑇}

This is the engine behind:

If $𝐴$ is symmetric positive semi-definite, the SVD coincides with the eigendecomposition (Spectral Theorem): $𝑈 = 𝑉$ and $Σ$ = eigenvalues.

For general $𝐴$ , $𝐴^{𝑇} 𝐴$ and $𝐴 𝐴^{𝑇}$ are both symmetric:

Right singular vectors of $𝐴$ = eigenvectors of $𝐴^{𝑇} 𝐴$ (with eigenvalues $𝜎_{𝑖}^{2}$ )
Left singular vectors of $𝐴$ = eigenvectors of $𝐴 𝐴^{𝑇}$ (with same eigenvalues $𝜎_{𝑖}^{2}$ )

When $𝐴$ is not square or not invertible, the Moore–Penrose pseudoinverse uses the SVD:

𝐴^{+} = 𝑉 Σ^{+} 𝑈^{𝑇}

where $Σ^{+}$ inverts the non-zero singular values and transposes the shape.