Quadratic Form

96. Quadratic Form

A quadratic form is a polynomial expression in $𝑛$ variables where every term has total degree exactly $2$ :

𝑄 (\vec{𝑥}) = \sum_{𝑖, 𝑗} 𝑎_{𝑖 𝑗} 𝑥_{𝑖} 𝑥_{𝑗}

It can always be written compactly using a symmetric matrix $𝐴$ :

𝑄 (\vec{𝑥}) = {\vec{𝑥}}^{𝑇} 𝐴 \vec{𝑥}

The matrix $𝐴$ is assumed symmetric without loss of generality — if $𝐵$ isn’t symmetric, replacing it with $\frac{𝐵 + 𝐵^{𝑇}}{2}$ gives the same quadratic form.

Example

𝑄 (𝑥, 𝑦) = 3 𝑥^{2} + 5 𝑥 𝑦 + 2 𝑦^{2}

As a matrix product (split the $5 𝑥 𝑦$ symmetrically as $\frac{5}{2} 𝑥 𝑦 + \frac{5}{2} 𝑦 𝑥$ ):

𝐴 = [\begin{matrix} 3 & \frac{5}{2} \\ \frac{5}{2} & 2 \end{matrix}], 𝑄 (\vec{𝑥}) = {\vec{𝑥}}^{𝑇} 𝐴 \vec{𝑥}

96.1. Definiteness

Quadratic forms are classified by the sign of $𝑄 (\vec{𝑥})$ over all non-zero $\vec{𝑥}$ :

Type	Sign of $𝑄$	All eigenvalues of $𝐴$
Positive definite	$> 0 always$	$> 0$
Positive semi-definite	$\geq 0 always$	$\geq 0$
Negative definite	$< 0 always$	$< 0$
Negative semi-definite	$\leq 0 always$	$\leq 0$
Indefinite	takes both signs	mixed

The eigenvalue characterization comes from the Spectral Theorem: diagonalizing $𝐴 = 𝑄 𝐷 𝑄^{𝑇}$ and changing variables $\vec{𝑦} = 𝑄^{𝑇} \vec{𝑥}$ gives

𝑄 (\vec{𝑥}) = \sum_{𝑖 = 1}^{𝑛} 𝜆_{𝑖} 𝑦_{𝑖}^{2}

— a sum of squares weighted by eigenvalues. The signs of the eigenvalues determine the signs of $𝑄$ .

96.2. Tests for positive definiteness

For a symmetric $𝐴$ , equivalent conditions:

All eigenvalues $> 0$
All leading principal minors $> 0$ (Sylvester’s criterion)
$𝐴 = 𝐿 𝐿^{𝑇}$ admits a Cholesky decomposition
${\vec{𝑥}}^{𝑇} 𝐴 \vec{𝑥} > 0$ for all $\vec{𝑥} \neq 𝟎$

96.3. Geometric pictures (2D)

For $𝑄 (𝑥, 𝑦) = {\vec{𝑥}}^{𝑇} 𝐴 \vec{𝑥}$ , the level sets $𝑄 (\vec{𝑥}) = 𝑐$ are conic sections aligned with the eigenvectors of $𝐴$ :

Positive definite: ellipse — closed, bounded, axes are eigenvectors with lengths $\frac{1}{\sqrt{𝜆_{𝑖}}}$
Negative definite: ellipse (with negative $𝑐$ )
Indefinite: hyperbola — open curves with asymptotes
Positive semi-definite (rank 1): parallel lines

96.4. Where they show up

Semi-definiteness / Hessian: the second-order Taylor approximation of a function near a critical point is a quadratic form. Definiteness classifies the critical point as min / max / saddle.
Norms: $‖ \vec{𝑥} ‖_{2}^{2} = {\vec{𝑥}}^{𝑇} \vec{𝑥} = {\vec{𝑥}}^{𝑇} 𝐼 \vec{𝑥}$ — a quadratic form with $𝐴 = 𝐼$
Mahalanobis distance: ${(\vec{𝑥} - \vec{𝜇})}^{𝑇} Σ^{- 1} (\vec{𝑥} - \vec{𝜇})$ — a quadratic form using the inverse covariance
Multivariate Gaussian density: $exp (- \frac{1}{2} {\vec{𝑥}}^{𝑇} Σ^{- 1} \vec{𝑥})$
Quadratic programming: optimize $\frac{1}{2} {\vec{𝑥}}^{𝑇} 𝑄 \vec{𝑥} + 𝑐^{𝑇} \vec{𝑥}$ subject to linear constraints
Kinetic energy / potential energy in mechanics
Variance of a linear combination: $Var (𝑎^{𝑇} 𝑋) = 𝑎^{𝑇} Cov (𝑋) 𝑎$

96.5. Diagonalization (principal axis theorem)

Every quadratic form can be brought to diagonal form by an orthogonal change of variables — equivalently, by rotating the coordinate axes to align with the eigenvectors of $𝐴$ :

\vec{𝑦} = 𝑄^{𝑇} \vec{𝑥} \Rightarrow 𝑄 (\vec{𝑥}) = 𝜆_{1} 𝑦_{1}^{2} + 𝜆_{2} 𝑦_{2}^{2} + \dots + 𝜆_{𝑛} 𝑦_{𝑛}^{2}

This is the “principal axes” of the conic / quadric.

96.6. See also

Symmetric Matrix
Spectral Theorem
Leading Principal Minor — Sylvester’s criterion
Cholesky Decomposition
Semi-Definiteness in Calculus II