Cheatsheet

3. Cheatsheet

Vector

[\begin{matrix} 𝑥_{1} \\ 𝑥_{2} \\ ⋮ \\ 𝑥_{𝑛} \end{matrix}]

Scalar Vector Multiplication

\begin{matrix} 𝑐 \in ℝ \vec{𝑥} = [\begin{matrix} 𝑥_{1} \\ 𝑥_{2} \\ ⋮ \\ 𝑥_{𝑛} \end{matrix}] \\ 𝑐 \vec{𝑥} = [\begin{matrix} 𝑐 𝑥_{1} \\ 𝑐 𝑥_{2} \\ ⋮ \\ 𝑐 𝑥_{𝑛} \end{matrix}] \end{matrix}

Dot Product

\vec{𝑥} = [\begin{matrix} 𝑥_{1} \\ 𝑥_{2} \\ ⋮ \\ 𝑥_{𝑛} \end{matrix}] \vec{𝑦} = [\begin{matrix} 𝑦_{1} \\ 𝑦_{2} \\ ⋮ \\ 𝑦_{𝑛} \end{matrix}]

\begin{aligned} \vec{𝑥} \cdot \vec{𝑦} & = \sum_{𝑖 = 1}^{𝑛} 𝑥_{𝑖} 𝑦_{𝑖} \\ = 𝑥_{1} 𝑦_{1} + 𝑥_{2} 𝑦_{2} + \dots + 𝑥_{𝑛} 𝑦_{𝑛} \\ = {\vec{𝑥}}^{𝑇} \vec{𝑦} \end{aligned}

Cross Product (

ℝ^{3}

)

\vec{𝑎} = [\begin{matrix} 𝑎_{1} \\ 𝑎_{2} \\ 𝑎_{3} \end{matrix}] \vec{𝑏} = [\begin{matrix} 𝑏_{1} \\ 𝑏_{2} \\ 𝑏_{3} \end{matrix}]

\vec{𝑐} = \vec{𝑎} \times \vec{𝑏}

\vec{𝑐} = [\begin{matrix} 𝑎_{2} 𝑏_{3} - 𝑎_{3} 𝑏_{2} \\ 𝑎_{3} 𝑏_{1} - 𝑎_{1} 𝑏_{3} \\ 𝑎_{1} 𝑏_{2} - 𝑎_{2} 𝑏_{1} \end{matrix}]

Returns a vector orthogonal to the two vectors

Vector Space

Closure under addition

\vec{𝑢}, \vec{𝑣} \in 𝑉 \Rightarrow \vec{𝑢} + \vec{𝑣} \in 𝑉

Closure under scalar multiplication

\vec{𝑣} \in 𝑉 \land 𝑐 \in ℝ \Rightarrow 𝑐 \vec{𝑣} \in 𝑉

Commutativity of addition

\vec{𝑢} + \vec{𝑣} = \vec{𝑣} + \vec{𝑢}

Associativity of addition

(\vec{𝑢} + \vec{𝑣}) + \vec{𝑤} = \vec{𝑢} + (\vec{𝑣} + \vec{𝑤})

Additive identity

\exists 𝟎 \in 𝑉 | \vec{𝑣} + 𝟎 = \vec{𝑣}

Additive inverse

\forall \vec{𝑣} \in 𝑉 \exists - \vec{𝑣} \in 𝑉 | \vec{𝑣} + (- \vec{𝑣}) = 0

Scalar multiplication (compatibility)

𝑎 (𝑏 \vec{𝑣}) = (𝑎 𝑏) \vec{𝑣}

Distributivity over vector addition

𝑎 (\vec{𝑢} + \vec{𝑣}) = 𝑎 \vec{𝑢} + 𝑎 \vec{𝑣}

Distributivity over scalar addition

(𝑎 + 𝑏) \vec{𝑣} = 𝑎 \vec{𝑣} + 𝑏 \vec{𝑣}

Multiplicative identity

1 \vec{𝑣} = \vec{𝑣}

Subspace

Non-emptiness

$𝟎 \in 𝑉$

Closure under addition

$If \vec{𝑢}, \vec{𝑣} \in 𝑉, then \vec{𝑢} + \vec{𝑣} \in 𝑉$

Closure under scalar multiplication

$If \vec{𝑣} \in 𝑉, 𝑐 \in ℝ, then 𝑐 \vec{𝑣} \in 𝑉$

A subspace is a subset of a vector space that is itself a vector space, satisfying the same axioms as the original. If

𝑉

is a vector space in

ℝ^{𝑛}

, then the subspace

𝑈

is always contained in

ℝ^{𝑛}

, meaning

𝑈 \subseteq ℝ^{𝑛}

Vector Addition

\begin{matrix} \vec{𝑢} = (𝑢_{1}, 𝑢_{2}, \dots, 𝑢_{𝑛}) \\ \vec{𝑣} = (𝑣_{1}, 𝑣_{2}, \dots, 𝑣_{𝑛}) \\ \vec{𝑢} + \vec{𝑣} = (𝑢_{1} + 𝑣_{1}, 𝑢_{2} + 𝑣_{2}, \dots, 𝑢_{𝑛} + 𝑣_{𝑛}) \end{matrix}

Dot Product

\vec{𝑢} \cdot \vec{𝑣} = \sum_{𝑖 = 1}^{𝑛} 𝑢_{𝑖} 𝑣_{𝑖}

Orthogonality

\vec{𝑢} \cdot \vec{𝑣} = 𝟎

Angle between the two vectors is

90 °

Angle between vectors

Θ = arccos (\frac{\vec{𝑢} \cdot \vec{𝑣}}{‖ \vec{𝑢} ‖_{2} \cdot ‖ \vec{𝑣} ‖_{2}})

𝐿_{1}

Norm (Manhattan)

‖ \vec{𝑢} ‖_{1} = \sum_{𝑖 = 1}^{𝑛} | 𝑢_{𝑖} |

𝐿_{2}

Norm (Euclidean)

‖ \vec{𝑢} ‖_{2} = \sqrt{\sum_{𝑖 = 1}^{𝑛} 𝑢_{𝑖}^{2}}

𝐿_{1}

Distance (Manhattan)

𝑑 (\vec{𝑢}, \vec{𝑣}) = \sum_{𝑖 = 1}^{𝑛} | 𝑢_{𝑖} - 𝑣_{𝑖} |

𝐿_{2}

Distance (Euclidean)

𝑑 (\vec{𝑢}, \vec{𝑣}) = \sqrt{\sum_{𝑖 = 1}^{𝑛} {(𝑢_{𝑖} - 𝑣_{𝑖})}^{2}}

Projection

{proj}_{𝑤} (\vec{𝑣}) = \frac{\vec{𝑣} \cdot \vec{𝑤}}{\vec{𝑤} \cdot \vec{𝑤}} \vec{𝑤}

Linear Independence

A set of vectors is linearly independent if no vector in the set can be written as a linear combination of the others

A set of vectors ${{\vec{𝑣}}_{1}, {\vec{𝑣}}_{2}, \dots, {\vec{𝑣}}_{𝑛}}$ is linearly independent if the only solution to the the equation

𝑐_{1} {\vec{𝑣}}_{1} + 𝑐_{2} {\vec{𝑣}}_{2} + \dots + 𝑐_{𝑛} {\vec{𝑣}}_{𝑛} = 𝟎

is $𝑐_{1} = 𝑐_{2} = \dots = 𝑐_{𝑛} = 0$

Transformation

𝑇 : ℝ^{𝑛} \to ℝ^{𝑚}

𝑇 (\vec{𝑣}) = 𝐴 \vec{𝑣}

Additivity

$𝑇 (\vec{𝑢} + \vec{𝑣}) = 𝑇 (\vec{𝑢}) + 𝑇 (\vec{𝑣})$

Homogeneity

$𝑇 (𝑐 \vec{𝑢}) = 𝑐 𝑇 (\vec{𝑢})$

Surjective (onto)

Every element in $𝐵$ is the image of at least one element in $𝐴$ . The transformation covers the entire codomain.

$Range (𝑇) = 𝐵$

Injective (one-to-one)

Different inputs in $𝐴$ map to different outputs in $𝐵$ .The transformation is information-preserving — doesn’t collapse distinct vectors together

$𝑇 ({\vec{𝑥}}_{1}) = 𝑇 ({\vec{𝑥}}_{2}) \Rightarrow 𝑥_{1} = 𝑥_{2}$

Or equivalently:

$ker (𝑇) = {𝟎}$

Domain

\begin{matrix} 𝑇 : 𝑉 \to 𝑊 \\ Domain (𝑇) = 𝑉 \end{matrix}

Set of all input vectors

𝑉

that the transformation acts on

Codomain

\begin{matrix} 𝑇 : 𝑉 \to 𝑊 \\ Codomain (𝑇) = 𝑊 \end{matrix}

Set of all possible output vectors

𝑊

to which elements of the domain

𝑉

are mapped under the transformation

Transpose

\begin{matrix} det (𝐴) = det (𝐴^{𝑇}) \\ {(𝐴 𝐵)}^{𝑇} = 𝐵^{𝑇} 𝐴^{𝑇} \\ {(𝐴^{𝑇})}^{- 1} = {(𝐴^{- 1})}^{𝑇} \end{matrix}

Image

\begin{matrix} 𝑇 : 𝑉 \to 𝑊 \\ Im (𝑇) = {\vec{𝑤} \in 𝑊 | \vec{𝑤} = 𝑇 (\vec{𝑣}) for some \vec{𝑣} \in 𝑉} \end{matrix}

The image of a transformation $𝑇 : 𝑉 \to 𝑊$ is the set of all possible outputs $𝑇 (𝑣)$ for $𝑣 \in 𝑉$ :

It is a subspace of the codomain $𝑊$
If $𝑇$ is represented by a matrix $𝐴$ , the image of $𝑇$ is the column space of $𝐴$

Preimage

\begin{matrix} 𝑇 : 𝑉 \to 𝑊 \\ 𝑇^{- 1} (\vec{𝑤}) = {\vec{𝑣} \in 𝑉 | 𝑇 (\vec{𝑣}) = \vec{𝑤}} \end{matrix}

The preimage of a transformation refers to the set of all elements in the domain that map to a particular element or subset in the codomain

Span

Span ({𝑣_{1}, 𝑣_{2}, \dots, 𝑣_{𝑘}}) = {\sum_{𝑖 = 1}^{𝑛} 𝑐_{𝑖} {\vec{𝑣}}_{𝑖} | 𝑐_{𝑖} \in ℝ}

The span of a set of vectors is the collection of all possible linear combinations of those vectors

Composition

\begin{matrix} 𝑇_{1} : ℝ^{𝑛} \to ℝ^{𝑚} 𝑇_{2} : ℝ^{𝑚} \to ℝ^{𝑝} \\ 𝑇_{1} (\vec{𝑣}) = 𝐴 \vec{𝑣} 𝑇_{2} (\vec{𝑣}) = 𝐵 \vec{𝑣} \end{matrix}

𝑇 \circ 𝑆 (\vec{𝑣}) = 𝑇_{2} (𝑇_{1} (\vec{𝑣}))

𝑇_{2} \circ 𝑇_{1} (\vec{𝑣}) = 𝐵 (𝐴 \vec{𝑣}) = (𝐵 𝐴) \vec{𝑣}

Additivity

$(𝑇_{3} \circ 𝑇_{2}) \circ 𝑇_{1} = 𝑇_{3} \circ (𝑇_{2} \circ 𝑇_{1})$

Homogeneity

$𝑇_{2} \circ 𝑇_{1} (𝑐 \vec{𝑢}) = 𝑐 (𝑇_{2} \circ 𝑇_{1}) (\vec{𝑢})$

Identity Transformation

$𝐼 \circ 𝑇 = 𝑇 𝑇 \circ 𝐼 = 𝑇$

Column Space (Range)

$Col (𝐴) = {𝐴 𝑥 | \vec{𝑥} \in ℝ^{𝑛}}$

Or equivalently

\begin{matrix} 𝐴 = [\begin{matrix} {\vec{𝑐}}_{1} & {\vec{𝑐}}_{2} & \dots & {\vec{𝑐}}_{3} \end{matrix}] \\ Col (𝐴) = span ({\vec{𝑐}}_{1}, {\vec{𝑐}}_{2}, \dots, {\vec{𝑐}}_{3}) \end{matrix}

The column space (or range) of a matrix

𝐴

is the set of all linear combinations of its columns

Determinant

det (𝐴)

The determinant of a square matrix A measure of the "scale factor" by which the matrix A transforms a space

$det (𝐴) \neq 0$
- $𝐴$ does not collapse the space
- $𝐴$ has full rank
- $𝐴$ ’s columns are linearly independent
- $𝐴$ is invertable
$det (𝐴) = 0$
- $𝐴$ collapses the space into lower dimension
- $𝐴$ does not have full rank
- $𝐴$ ’s columns are linearly dependent
- $𝐴$ is non-invertable (singular)

Invertibility

$det (𝐴) \neq 0 ⟹ Invertible$

$det (𝐴) = 0 ⟹ Non-Invertible$

𝐴 𝐴^{- 1} = 𝐴^{- 1} 𝐴 = 𝐼_{𝑛}

{(𝐴 𝐵)}^{- 1} = 𝐵^{- 1} 𝐴^{- 1}

{(𝐴^{𝑇})}^{- 1} = {(𝐴^{- 1})}^{𝑇}

Basis

Linear Independence

\begin{matrix} 𝑐_{1} \vec{𝑣_{1}} + 𝑐_{2} \vec{𝑣_{2}} + \dots + 𝑐_{𝑘} \vec{𝑣_{𝑘}} = 𝟎 \\ \Rightarrow 𝑐_{1} = 𝑐_{2} = \dots = 𝑐_{𝑘} = 0 \end{matrix}

Spanning

\begin{matrix} \forall \vec{𝑣} \in 𝑉, \exists 𝑐_{1}, \dots, 𝑐_{𝑘} \in ℝ 𝑠 . 𝑡 . \\ \vec{𝑣} = 𝑐_{1} \vec{𝑣_{1}} + \dots + 𝑐_{𝑘} \vec{𝑣_{𝑘}} \end{matrix}

A basis of a vector space $𝑉$ is a set of linearly independent vectors that span the space
Every vector in $𝑉$ can be uniquely written as a linear combination of the basis vectors

E.g.:

Dimension

dim (𝑉)

Number of linearly independent vectors (basis) in a vector space $𝑉$

$𝑉 \subseteq ℝ^{𝑛}$

$dim (𝑉) = 0$	$𝑉 = {𝟎}$
$dim (𝑉) = 1$	$𝑉$ is a line through the origin in $ℝ^{𝑛}$
$dim (𝑉) = 2$	$𝑉$ is a plane through the origin in $ℝ^{𝑛}$
$dim (𝑉) = 𝑘$	$𝑉$ is a $𝑘$ -dimensional flat subspace of $ℝ^{𝑛}$
$dim (𝑉) = 𝑛$	$𝑉 = ℝ^{𝑛}$

$ℝ^{2}$ has dimension 2:
- A basis: ${[\begin{matrix} 1 \\ 0 \end{matrix}], [\begin{matrix} 0 \\ 1 \end{matrix}]}$
$ℝ^{3}$ has dimension 3:
- A basis: ${[\begin{matrix} 1 \\ 0 \\ 0 \end{matrix}], [\begin{matrix} 0 \\ 1 \\ 0 \end{matrix}], [\begin{matrix} 0 \\ 0 \\ 1 \end{matrix}]}$

Rank

Rank (𝐴) = dim (Col (𝐴)) = dim (Row (𝐴))

The rank of a matrix $𝐴$ is the dimension of its column space (or row space)
Number of linearly independent columns (or rows)

Eigen

𝐴 𝑥 = 𝜆 𝑥, 𝑥 \neq 𝟎

Set of all nonzero vectors $\vec{𝑥}$ such that when the transformation represented by matrix $𝐴$ is applied to $\vec{𝑥}$ , the result is a scaled version of $\vec{𝑥}$ itself

These vectors lie along directions that are preserved by the transformation:

$| 𝜆 | > 1$ : stretched
$0 < | 𝜆 | < 1$ : shrunk
$𝜆 < 0$ : flipped
$𝜆 = 1$ : stay the same

Null Space (kernel)

Null (𝐴) = {\vec{𝑥} \in ℝ^{𝑛} | 𝐴 \vec{𝑥} = 𝟎}

The null space of a matrix

𝐴

is the set of all input vectors that get mapped to the zero vector when you multiply them by

𝐴

Identity Matrix

𝐼_{𝑛} = [\begin{matrix} 1 & 0 & 0 & \dots & 0 \\ 0 & 1 & 0 & \dots & 0 \\ 0 & 0 & 1 & \dots & 0 \\ ⋮ & ⋮ & ⋮ & ⋱ & ⋮ \\ 0 & 0 & 0 & \dots & 1 \end{matrix}]

Matrix Inverse

𝐴 \cdot 𝐴^{- 1} = 𝐼

RREF

Row Swapping (Interchange)

𝑅_{1} \leftrightarrow︎ 𝑅_{2}

Row Scaling (Multiplication)

𝑅_{1} \to \frac{1}{3} 𝑅_{1}

Row Addition (Replacement)

𝑅_{1} \to 𝑅_{1} - 2 𝑅_{2}

3.1. Matrix

	$𝑛$
$𝑚$	$[\begin{matrix} 𝑎_{11} & 𝑎_{12} & \dots & 𝑎_{1 𝑛} \\ 𝑎_{21} & 𝑎_{22} & \dots & 𝑎_{2 𝑛} \\ ⋮ & ⋮ & ⋱ & ⋮ \\ 𝑎_{𝑚 1} & 𝑎_{𝑚 2} & \dots & 𝑎_{𝑚 𝑛} \end{matrix}])$

3.1.1. Matrix Vector Product

	$𝑛$
$𝑚$	$[\begin{matrix} 𝑎_{11} & 𝑎_{12} & \dots & 𝑎_{1 𝑛} \\ 𝑎_{21} & 𝑎_{22} & \dots & 𝑎_{2 𝑛} \\ ⋮ & ⋮ & ⋱ & ⋮ \\ 𝑎_{𝑚 1} & 𝑎_{𝑚 2} & \dots & 𝑎_{𝑚 𝑛} \end{matrix}]$


$𝑛$	$[\begin{matrix} 𝑥_{1} \\ 𝑥_{2} \\ ⋮ \\ 𝑥_{𝑛} \end{matrix}]$

[\begin{matrix} 𝑎_{11} 𝑥_{1} + 𝑎_{12} 𝑥_{2} + \dots + 𝑎_{1 𝑛} 𝑥_{𝑛} \\ 𝑎_{21} 𝑥_{1} + 𝑎_{22} 𝑥_{2} + \dots + 𝑎_{2 𝑛} 𝑥_{𝑛} \\ ⋮ \\ 𝑎_{𝑚 1} 𝑥_{1} + 𝑎_{𝑚 2} 𝑥_{2} + \dots + 𝑎_{𝑚 𝑛} 𝑥_{𝑛} \end{matrix}]

3.1.2. Matrix Multiplication

		$𝑛$
$𝐴 =$	$𝑚$	$[\begin{matrix} 𝑎_{11} & 𝑎_{12} & \dots & 𝑎_{1 𝑛} \\ 𝑎_{21} & 𝑎_{22} & \dots & 𝑎_{2 𝑛} \\ ⋮ & ⋮ & ⋱ & ⋮ \\ 𝑎_{𝑚 1} & 𝑎_{𝑚 2} & \dots & 𝑎_{𝑚 𝑛} \end{matrix}]$

		$𝑝$
$𝐵 =$	$𝑛$	$[\begin{matrix} 𝑏_{11} & 𝑏_{12} & \dots & 𝑏_{1 𝑝} \\ 𝑏_{21} & 𝑏_{22} & \dots & 𝑏_{2 𝑝} \\ ⋮ & ⋮ & ⋱ & ⋮ \\ 𝑏_{𝑛 1} & 𝑏_{𝑛 2} & \dots & 𝑏_{𝑛 𝑝} \end{matrix}]$

		$𝑛$
$𝐴 =$	$𝑚$	$[\begin{matrix} [\begin{matrix} 𝑎_{11} & 𝑎_{12} & \dots & 𝑎_{1 𝑛} \end{matrix}] \\ [\begin{matrix} 𝑎_{21} & 𝑎_{22} & \dots & 𝑎_{2 𝑛} \end{matrix}] \\ ⋮ \\ [\begin{matrix} 𝑎_{𝑚 1} & 𝑎_{𝑚 2} & \dots & 𝑎_{𝑚 𝑛} \end{matrix}] \end{matrix}]$

		$𝑝$
$𝐵 =$	$𝑛$	$[\begin{matrix} [\begin{matrix} 𝑏_{11} \\ 𝑏_{21} \\ ⋮ \\ 𝑏_{𝑛 1} \end{matrix}] & [\begin{matrix} 𝑏_{12} \\ 𝑏_{22} \\ ⋮ \\ 𝑏_{𝑛 2} \end{matrix}] & \dots & [\begin{matrix} 𝑏_{1 𝑝} \\ 𝑏_{2 𝑝} \\ ⋮ \\ 𝑏_{𝑛 𝑝} \end{matrix}] \end{matrix}]$

$𝐴$ : Row Representation
$\begin{matrix} 𝐴 = [\begin{matrix} 𝑟_{1} \\ 𝑟_{2} \\ ⋮ \\ 𝑟_{𝑚} \end{matrix}] \\ 𝑟_{𝑖} = [\begin{matrix} 𝑎_{𝑖 1} & 𝑎_{𝑖 2} & \dots & 𝑎_{𝑖 𝑛} \end{matrix}], for 𝑖 = 1, 2, \dots, 𝑚 \end{matrix}$

$𝐵$ : Column Representation
$\begin{matrix} 𝐵 = [\begin{matrix} 𝑐_{1} & 𝑐_{2} & \dots & 𝑐_{𝑝} \end{matrix}] \\ 𝑐_{𝑗} = [\begin{matrix} 𝑏_{1} 𝑗 \\ 𝑏_{2} 𝑗 \\ ⋮ \\ 𝑏_{𝑛 𝑗} \end{matrix}], for 𝑗 = 1, 2, \dots, 𝑝 \end{matrix}$

		$𝑝$
$𝐶 =$	$𝑚$	$[\begin{matrix} 𝑟_{1} \cdot 𝑐_{1} & 𝑟_{1} \cdot 𝑐_{2} & \dots & 𝑟_{1} \cdot 𝑐_{𝑝} \\ 𝑟_{2} \cdot 𝑐_{1} & 𝑟_{2} \cdot 𝑐_{2} & \dots & 𝑟_{2} \cdot 𝑐_{𝑝} \\ ⋮ & ⋮ & ⋱ & ⋮ \\ 𝑟_{𝑚} \cdot 𝑐_{1} & 𝑟_{𝑚} \cdot 𝑐_{2} & \dots & 𝑟_{𝑚} \cdot 𝑐_{𝑝} \end{matrix}]$

3.1.3. Transpose

𝐴^{𝑇}

		$𝑛$
$𝐴 =$	$𝑚$	$[\begin{matrix} 𝑎_{11} & 𝑎_{12} & \dots & 𝑎_{1 𝑛} \\ 𝑎_{21} & 𝑎_{22} & \dots & 𝑎_{2 𝑛} \\ ⋮ & ⋮ & ⋱ & ⋮ \\ 𝑎_{𝑚 1} & 𝑎_{𝑚 2} & \dots & 𝑎_{𝑚 𝑛} \end{matrix}]$

		$𝑚$
$𝐴^{𝑇} =$	$𝑛$	$[\begin{matrix} 𝑎_{11} & 𝑎_{21} & \dots & 𝑎_{𝑚 1} \\ 𝑎_{12} & 𝑎_{22} & \dots & 𝑎_{𝑚 2} \\ ⋮ & ⋮ & ⋱ & ⋮ \\ 𝑎_{1 𝑛} & 𝑎_{2 𝑛} & \dots & 𝑎_{𝑚 𝑛} \end{matrix}]$

		$𝑛$
$𝐴 =$	$𝑚$	$[\begin{matrix} 𝑎_{11} & 𝑎_{12} & \dots & 𝑎_{1 𝑗} & \dots & 𝑎_{1 𝑛} \\ 𝑎_{21} & 𝑎_{22} & \dots & 𝑎_{2 𝑗} & \dots & 𝑎_{2 𝑛} \\ ⋮ & ⋮ & ⋮ & ⋮ \\ 𝑎_{𝑖 1} & 𝑎_{𝑖 2} & \dots & 𝑎_{𝑖 𝑗} & \dots & 𝑎_{𝑖 𝑛} \\ ⋮ & ⋮ & ⋮ & ⋮ \\ 𝑎_{𝑚 1} & 𝑎_{𝑚 2} & \dots & 𝑎_{𝑖 𝑗} & \dots & 𝑎_{𝑚 𝑛} \end{matrix}]$

		$𝑚$
$𝐴^{𝑇} =$	$𝑛$	$[\begin{matrix} 𝑎_{11} & 𝑎_{21} & \dots & 𝑎_{𝑖 1} & \dots & 𝑎_{𝑚 1} \\ 𝑎_{12} & 𝑎_{22} & \dots & 𝑎_{𝑖 2} & \dots & 𝑎_{𝑚 2} \\ ⋮ & ⋮ & ⋮ & ⋮ \\ 𝑎_{1 𝑗} & 𝑎_{2 𝑗} & \dots & 𝑎_{𝑖 𝑗} & \dots & 𝑎_{𝑚 𝑗} \\ ⋮ & ⋮ & ⋮ & ⋮ \\ 𝑎_{1 𝑛} & 𝑎_{2 𝑛} & \dots & 𝑎_{𝑖 𝑛} & \dots & 𝑎_{𝑚 𝑛} \end{matrix}]$

3.2. Matrix Factorization

3.2.1. LU Decomposition

𝐴 = 𝐿 𝑈

where:

$𝐿$ : lower triangular matrix (entries above the diagonal are zero)
$𝑈$ : upper triangular matrix (entries below the diagonal are zero)

$𝑚 \times 𝑛$ Matrix (with $𝑚 \geq 𝑛$ )

		$𝑛$
$𝐿 =$	$𝑚$	$[\begin{matrix} 𝑙_{11} & 𝑙_{12} & \dots & 𝑙_{1 𝑛} \\ 𝑙_{21} & 𝑙_{22} & \dots & 𝑙_{2 𝑛} \\ ⋮ & ⋮ & ⋱ & ⋮ \\ 𝑙_{𝑚 1} & 𝑙_{𝑚 2} & \dots & 𝑙_{𝑚 𝑛} \end{matrix}]$

		$𝑛$
$𝑈 =$	$𝑛$	$[\begin{matrix} 𝑢_{11} & 𝑢_{12} & \dots & 𝑢_{1 𝑛} \\ 𝑢_{21} & 𝑢_{22} & \dots & 𝑢_{2 𝑛} \\ ⋮ & ⋮ & ⋱ & ⋮ \\ 𝑢_{𝑛 1} & 𝑢_{𝑛 2} & \dots & 𝑢_{𝑛 𝑛} \end{matrix}]$

$𝑚 \times 𝑛$ Matrix (with $𝑚 < 𝑛$ )

see QR Decomposition

3.3. Singular Value Decomposition (SVD)

𝐴 = 𝑈 Σ 𝑉^{𝑇}

		$𝑚$
$𝑈 =$	$𝑚$	$[\begin{matrix} 𝑢_{11} & 𝑢_{12} & \dots & 𝑢_{1 𝑚} \\ 𝑢_{21} & 𝑢_{22} & \dots & 𝑢_{2 𝑚} \\ ⋮ & ⋮ & ⋱ & ⋮ \\ 𝑢_{𝑚 1} & 𝑢_{𝑚 2} & \dots & 𝑢_{𝑚 𝑚} \end{matrix}]$

		$𝑛$
$Σ =$	$𝑚$	$[\begin{matrix} 𝜀_{11} & 𝜀_{12} & \dots & 𝜀_{1 𝑚} \\ 𝜀_{21} & 𝜀_{22} & \dots & 𝜀_{2 𝑚} \\ ⋮ & ⋮ & ⋱ & ⋮ \\ 𝜀_{𝑛 1} & 𝜀_{𝑛 2} & \dots & 𝜀_{𝑛 𝑚} \end{matrix}]$

		$𝑛$
$𝑉^{𝑇} =$	$𝑛$	$[\begin{matrix} 𝑣_{11} & 𝑣_{12} & \dots & 𝑣_{1 𝑛} \\ 𝑣_{21} & 𝑣_{22} & \dots & 𝑣_{2 𝑛} \\ ⋮ & ⋮ & ⋱ & ⋮ \\ 𝑣_{𝑛 1} & 𝑣_{𝑛 2} & \dots & 𝑣_{𝑛 𝑛} \end{matrix}]$

	$[\begin{matrix} 1 & 2 & - 1 & 2 \\ 2 & 3 & 1 & 5 \\ 3 & 4 & - 2 & 4 \end{matrix}]$
$\begin{matrix} 𝑅_{2} \to 𝑅_{2} - 2 𝑅_{1} \\ 𝑅_{3} \to 𝑅_{3} - 3 𝑅_{1} \end{matrix}$	$[\begin{matrix} 1 & 2 & - 1 & 2 \\ 0 & - 1 & 3 & 1 \\ 0 & - 2 & 1 & - 2 \end{matrix}]$
$𝑅_{2} \to - 𝑅_{2}$	$[\begin{matrix} 1 & 2 & - 1 & 2 \\ 0 & 1 & - 3 & - 1 \\ 0 & - 2 & 1 & - 2 \end{matrix}]$
$\begin{matrix} 𝑅_{1} \to 𝑅_{1} - 2 𝑅_{2} \\ 𝑅_{3} \to 𝑅_{3} + 2 𝑅_{2} \end{matrix}$	$[\begin{matrix} 1 & 0 & 5 & 4 \\ 0 & 1 & - 3 & - 1 \\ 0 & 0 & - 5 & - 4 \end{matrix}]$
$𝑅_{3} \to \frac{1}{- 5} 𝑅_{3}$	$[\begin{matrix} 1 & 0 & 5 & 4 \\ 0 & 1 & - 3 & - 1 \\ 0 & 0 & 1 & \frac{4}{5} \end{matrix}]$
$\begin{matrix} 𝑅_{1} \to 𝑅_{1} - 5 𝑅_{3} \\ 𝑅_{2} \to 𝑅_{2} + 3 𝑅_{3} \end{matrix}$	$[\begin{matrix} 1 & 0 & 0 & 0 \\ 0 & 1 & 0 & \frac{7}{5} \\ 0 & 0 & 1 & \frac{4}{5} \end{matrix}]$

Example

𝐴 = [\begin{matrix} 2 & - 1 & - 3 \\ - 4 & 2 & 6 \end{matrix}]

Null Space

The null-space of $𝐴$ , denoted $𝑁 (𝐴)$ , consists of all vectors $\vec{𝑥} \in ℝ^{3}$ such that $𝐴 𝑥 = 0$ . The set of all such vectors is the pre-image of the zero vector under the transformation defined by $𝐴$ . In other words, $𝑁 (𝐴) = {𝑥 \in ℝ^{3} | 𝐴 \vec{𝑥} = 0}$ , which represents the set of vectors that $𝐴$ maps to zero.

𝑁 (𝐴) = {\vec{𝑥} \in ℝ^{3} | 𝐴 \vec{𝑥} = 𝟎}

To find the null space $𝑁 (𝐴)$ of the matrix $𝐴$ , we can use the row-reduced echelon form (RREF). By augmenting the matrix $𝐴$ with a zero column and performing row operations, we reduce it to the form:

[\begin{matrix} 2 & - 1 & - 3 \\ - 4 & 2 & 6 \end{matrix}] [\begin{matrix} 𝑥_{1} \\ 𝑥_{2} \\ 𝑥_{3} \end{matrix}] = [\begin{matrix} 0 \\ 0 \end{matrix}]

	$[\begin{matrix} 2 & - 1 & - 3 & 0 \\ - 4 & 2 & 6 & 0 \end{matrix}]$
$\begin{matrix} 𝑅_{1} \to \frac{𝑅_{1}}{2} \\ 𝑅_{2} \to \frac{𝑅_{2}}{4} \end{matrix}$	$[\begin{matrix} 1 & - \frac{1}{2} & - \frac{3}{2} & 0 \\ - 1 & \frac{1}{2} & \frac{3}{2} & 0 \end{matrix}]$
$𝑅_{2} \to 𝑅_{2} - 𝑅_{1}$	$[\begin{matrix} 1 & - \frac{1}{2} & - \frac{3}{2} & 0 \\ 0 & 0 & 0 & 0 \end{matrix}]$

[\begin{matrix} 1 & - \frac{1}{2} & - \frac{3}{2} \\ 0 & 0 & 0 \end{matrix}] [\begin{matrix} 𝑥_{1} \\ 𝑥_{2} \\ 𝑥_{3} \end{matrix}] = [\begin{matrix} 0 \\ 0 \end{matrix}]

\begin{matrix} 𝑥_{1} - \frac{1}{2} 𝑥_{2} - \frac{3}{2} 𝑥_{2} = 0 \\ 𝑥_{1} = \frac{1}{2} 𝑥_{2} + \frac{3}{2} 𝑥_{2} \\ [\begin{matrix} 𝑥_{1} \\ 𝑥_{2} \\ 𝑥_{3} \end{matrix}] = 𝑥_{2} [\begin{matrix} \frac{1}{2} \\ 1 \\ 0 \end{matrix}] + 𝑥_{3} [\begin{matrix} \frac{3}{2} \\ 0 \\ 1 \end{matrix}] \\ 𝑁 (𝑎) = span ({[\begin{matrix} \frac{1}{2} \\ 1 \\ 0 \end{matrix}] [\begin{matrix} \frac{3}{2} \\ 0 \\ 1 \end{matrix}]}) \end{matrix}

The dimension of the null-space is the number of vectors in this basis, which is 2. This is important because the dimension of the null space gives us insight into how many degrees of freedom exist in the system of equations $𝐴 𝑥 = 0$

Column Space

\begin{aligned} 𝐶 (𝐴) & = span ({[\begin{matrix} 2 \\ - 4 \end{matrix}] [\begin{matrix} - 1 \\ 2 \end{matrix}] [\begin{matrix} - 3 \\ 6 \end{matrix}]}) \\ = span ({[\begin{matrix} 2 \\ - 4 \end{matrix}]}) \end{aligned}

Basis

[\begin{matrix} 2 \\ - 4 \end{matrix}]

Rank

Number of vector in the basis of our column space

Rank (𝐴) = 1