Cheatsheet

117. Cheatsheet

117.1. Derivative Objects

Gradient	$𝑓 : ℝ^{𝑛} \to ℝ$	$𝑛 \times 1$	Vector of first derivatives; special case of Jacobian (scalar output)
Jacobian	$𝐹 : ℝ^{𝑛} \to ℝ^{𝑚}$	$𝑚 \times 𝑛$	Matrix of first derivatives; general case for vector-valued functions
Hessian	$𝑓 : ℝ^{𝑛} \to ℝ$	$𝑛 \times 𝑛$	Matrix of second derivatives; Jacobian of the gradient

	Input/Output	Derivatives	Notation
Gradient (Derivative) ( $1 \times 1$ )	$𝑓 : ℝ \to ℝ$ One Input One Output Scalar Function	First Order	$\nabla 𝑓 (𝑥) = 𝑓^{'} (𝑥) = [\begin{matrix} \frac{𝜕 𝑓}{𝜕 𝑥} \end{matrix}]$
Jacobian ( $𝑚 \times 1$ )	$𝑓 : ℝ \to ℝ^{𝑚}$ One Input Multiple Output Vector Function	First Order	$𝐽_{𝑓} (𝑥) = [\begin{matrix} \frac{𝜕 𝑓_{1}}{𝜕 𝑥} \\ \frac{𝜕 𝑓_{2}}{𝜕 𝑥} \\ ⋮ \\ \frac{𝜕 𝑓_{𝑚}}{𝜕 𝑥} \end{matrix}]$
Gradient ( $𝑛 \times 1$ )	$𝑓 : ℝ^{𝑛} \to ℝ$ Multiple Input One Output Scalar Function	First Order	$\nabla 𝑓 (𝐱) = [\begin{matrix} \frac{𝜕 𝑓}{𝜕 𝑥_{1}} \\ \frac{𝜕 𝑓}{𝜕 𝑥_{2}} \\ ⋮ \\ \frac{𝜕 𝑓}{𝜕 𝑥_{𝑛}} \end{matrix}]$
Jacobian ( $𝑚 \times 𝑛$ )	$𝑓 : ℝ^{𝑛} \to ℝ^{𝑚}$ Multiple Input Multiple Output Vector Function	First Order	$\begin{aligned} 𝐽_{𝑓} (𝐱) & = [\begin{matrix} \nabla 𝑔_{1} {(𝑥)}^{𝑇} \\ \nabla 𝑔_{2} {(𝑥)}^{𝑇} \\ ⋮ \\ \nabla 𝑔_{𝑚} {(𝑥)}^{𝑇} \end{matrix}] \\ = [\begin{matrix} \frac{𝜕 𝑓_{1}}{𝜕 𝑥_{1}} & \dots & \frac{𝜕 𝑓_{1}}{𝜕 𝑥_{𝑛}} \\ ⋮ & ⋱ & ⋮ \\ \frac{𝜕 𝑓_{𝑚}}{𝜕 𝑥_{1}} & \dots & \frac{𝜕 𝑓_{𝑚}}{𝜕 𝑥_{𝑛}} \end{matrix}] \end{aligned}$
Hessian ( $𝑛 \times 𝑛$ )	$𝑓 : ℝ^{𝑛} \to ℝ$ Multiple Input One Output Scalar Function	Second Order	$\nabla^{2} 𝑓 (𝐱) = 𝐻_{𝑓} (𝐱) = [\begin{matrix} \frac{𝜕^{2} 𝑓}{𝜕 𝑥_{1}^{2}} & \dots & \frac{𝜕^{2} 𝑓}{𝜕 𝑥_{1} 𝜕 𝑥_{𝑛}} \\ ⋮ & ⋱ & ⋮ \\ \frac{𝜕^{2} 𝑓}{𝜕 𝑥_{𝑛} 𝜕 𝑥_{1}} & \dots & \frac{𝜕^{2} 𝑓}{𝜕 𝑥_{𝑛}^{2}} \end{matrix}]$
Laplacian	Trace of the Hessian (sum of diagonal entries)		$tr (𝐻_{𝑓} (𝐱)) = \sum_{𝑖 = 1}^{𝑛} \frac{𝜕^{2} 𝑓}{𝜕 𝑥_{𝑖}^{2}} (𝑥)$ $𝐻_{𝑓} (𝐱) = [\begin{matrix} \frac{𝜕^{2} 𝑓}{𝜕 𝑥_{1}^{2}} & \dots & \frac{𝜕^{2} 𝑓}{𝜕 𝑥_{1} 𝜕 𝑥_{𝑛}} \\ ⋮ & ⋱ & ⋮ \\ \frac{𝜕^{2} 𝑓}{𝜕 𝑥_{𝑛} 𝜕 𝑥_{1}} & \dots & \frac{𝜕^{2} 𝑓}{𝜕 𝑥_{𝑛}^{2}} \end{matrix}]$

Gradient	$1^{st}$ derivative	$𝑛 \times 1$
Hessian	$2^{nd}$ derivative	$𝑛 \times 𝑛$
$3^{rd}$ Order Tensor	$3^{rd}$ derivative	$𝑛 \times 𝑛 \times 𝑛$
$⋮$
$𝑘$ -th Order Tensor	$𝑘^{th}$ derivative	$𝑛 \times 𝑛 \times \dots \times 𝑛 (𝑘 times)$