Orthogonality

13. Orthogonality

Two vectors $\vec{𝑢}, \vec{𝑣}$ are orthogonal (written $\vec{𝑢} ⟂ \vec{𝑣}$ ) if their dot product is zero:

\vec{𝑢} ⟂ \vec{𝑣} \Leftrightarrow \vec{𝑢} \cdot \vec{𝑣} = 0

In $ℝ^{2}$ and $ℝ^{3}$ , this is the same as perpendicular. In higher dimensions or abstract inner product spaces, orthogonality generalizes the notion.

13.1. Orthogonal sets and orthonormal sets

A set ${{\vec{𝑣}}_{1}, \dots, {\vec{𝑣}}_{𝑘}}$ is:

Orthogonal if every pair is orthogonal: ${\vec{𝑣}}_{𝑖} \cdot {\vec{𝑣}}_{𝑗} = 0$ for $𝑖 \neq 𝑗$
Orthonormal if also each is a unit vector: $‖ {\vec{𝑣}}_{𝑖} ‖ = 1$

A useful concise formula uses the Kronecker delta:

{\vec{𝑣}}_{𝑖} \cdot {\vec{𝑣}}_{𝑗} = 𝛿_{𝑖 𝑗} = {\begin{cases} 1 if 𝑖 = 𝑗 \\ 0 if 𝑖 \neq 𝑗 \end{cases}

13.2. Key facts

Linear independence: any orthogonal set of non-zero vectors is automatically linearly independent
Pythagoras: if $\vec{𝑢} ⟂ \vec{𝑣}$ , then $‖ \vec{𝑢} + \vec{𝑣} ‖^{2} = ‖ \vec{𝑢} ‖^{2} + ‖ \vec{𝑣} ‖^{2}$
Easy coordinates: in an orthonormal basis ${{\vec{𝑞}}_{1}, \dots, {\vec{𝑞}}_{𝑛}}$ , the coordinates of $\vec{𝑥}$ are just dot products:

\vec{𝑥} = (\vec{𝑥} \cdot {\vec{𝑞}}_{1}) {\vec{𝑞}}_{1} + \dots + (\vec{𝑥} \cdot {\vec{𝑞}}_{𝑛}) {\vec{𝑞}}_{𝑛}

13.3. Orthogonal complement

For a subspace $𝑊 \subseteq ℝ^{𝑛}$ , its orthogonal complement is:

𝑊^{⟂} = {\vec{𝑥} \in ℝ^{𝑛} : \vec{𝑥} \cdot \vec{𝑤} = 0 for all \vec{𝑤} \in 𝑊}

— the set of vectors orthogonal to every vector in $𝑊$ .

Properties:

$𝑊^{⟂}$ is itself a subspace
$𝑊 \cap 𝑊^{⟂} = {𝟎}$
$dim (𝑊) + dim (𝑊^{⟂}) = 𝑛$
${(𝑊^{⟂})}^{⟂} = 𝑊$
$ℝ^{𝑛} = 𝑊 \oplus 𝑊^{⟂}$ (every vector decomposes uniquely as $\vec{𝑤} + {\vec{𝑤}}^{⟂}$ )

13.4. Four fundamental subspaces (and their complements)

For an $𝑚 \times 𝑛$ matrix $𝐴$ , four subspaces and two pairings:

\underset{\subset ℝ^{𝑚}}{\underset{⏟}{Col (𝐴)}} ⟂ \underset{\subset ℝ^{𝑚}}{\underset{⏟}{Null (𝐴^{𝑇})}}

\underset{\subset ℝ^{𝑛}}{\underset{⏟}{Row (𝐴)}} ⟂ \underset{\subset ℝ^{𝑛}}{\underset{⏟}{Null (𝐴)}}

The column space is orthogonal to the left null space; the row space (= column space of $𝐴^{𝑇}$ ) is orthogonal to the null space.

13.5. Orthogonal projection

For a subspace $𝑊$ with orthonormal basis ${{\vec{𝑞}}_{1}, \dots, {\vec{𝑞}}_{𝑘}}$ , the projection of $\vec{𝑥}$ onto $𝑊$ is:

{proj}_{𝑊} (\vec{𝑥}) = \sum_{𝑖 = 1}^{𝑘} (\vec{𝑥} \cdot {\vec{𝑞}}_{𝑖}) {\vec{𝑞}}_{𝑖}

In matrix form with $𝑄 = [\begin{matrix} {\vec{𝑞}}_{1} & \dots & {\vec{𝑞}}_{𝑘} \end{matrix}]$ : ${proj}_{𝑊} (\vec{𝑥}) = 𝑄 𝑄^{𝑇} \vec{𝑥}$ . The matrix $𝑃 = 𝑄 𝑄^{𝑇}$ is the projection matrix — see Projection.

13.6. Why orthogonality matters

Best approximation: projecting onto a subspace gives the closest point — basis of least-squares fitting
Decoupling: in an orthonormal basis, coordinates are independent — no cross-terms, simpler computations
Orthogonal matrices preserve distances and angles — rigid transformations
Spectral theorem: real symmetric matrices have orthogonal eigenvectors
Gram–Schmidt builds orthonormal bases from any basis