Hessian Matrix

121. Hessian Matrix

Square matrix of second-order partial derivatives

Describes the local curvature of a multivariable function

𝑓 : ℝ^{𝑛} \to ℝ

Input: $𝐱 = [\begin{matrix} 𝑥_{1} \\ 𝑥_{2} \\ ⋮ \\ 𝑥_{𝑛} \end{matrix}]$

Output: Scalar

𝐻_{𝑓} (𝐱) = [\begin{matrix} \frac{𝜕^{2} 𝑓}{𝜕 𝑥_{1}^{2}} & \frac{𝜕^{2} 𝑓}{𝜕 𝑥_{1} 𝜕 𝑥_{2}} & \dots & \frac{𝜕^{2} 𝑓}{𝜕 𝑥_{1} 𝜕 𝑥_{𝑛}} \\ \frac{𝜕^{2} 𝑓}{𝜕 𝑥_{2} 𝜕 𝑥_{1}} & \frac{𝜕^{2} 𝑓}{𝜕 𝑥_{2}^{2}} & \dots & \frac{𝜕^{2} 𝑓}{𝜕 𝑥_{2} 𝜕 𝑥_{𝑛}} \\ ⋮ & ⋮ & ⋱ & ⋮ \\ \frac{𝜕^{2} 𝑓}{𝜕 𝑥_{𝑛} 𝜕 𝑥_{1}} & \frac{𝜕^{2} 𝑓}{𝜕 𝑥_{𝑛}^{2} 𝜕 𝑥_{2}} & \dots & \frac{𝜕^{2} 𝑓}{𝜕 𝑥_{𝑛}^{2}} \end{matrix}]

Diagonal entries: $\frac{𝜕^{2} 𝑓}{𝜕 𝑥_{𝑖}^{2}}$

Second derivative of $𝑓$ with respect to a single variable twice

How the slope of $𝑓$ changes as you move along the $𝑥_{1}$ direction

Off-diagonal entries: $\frac{𝜕^{2} 𝑓}{𝜕 𝑥_{𝑖} 𝜕 𝑥_{𝑗}}$ where $𝑖 \neq 𝑗$

How the slope in one direction changes when you move in a different direction

$\frac{𝜕^{2} 𝑓}{𝜕 𝑥_{1} 𝜕 𝑥_{2}}$ : How the rate of change of $𝑓$ along $𝑥_{1}$ changes as you move in the $𝑥_{2}$ direction

Symmetric (Clairaut-Schwarz theorem):

\frac{𝜕^{2} 𝑓}{𝜕 𝑥_{𝑖} 𝜕 𝑥_{𝑗}} = \frac{𝜕^{2} 𝑓}{𝜕 𝑥_{𝑗} 𝜕 𝑥_{𝑖}}

𝐻_{𝑓} (𝑥) = 𝐻_{𝑓} {(𝑥)}^{𝑇}

Example

𝑓 (𝑥, 𝑦) = 𝑥^{3} 𝑦 + 𝑦^{2}

\nabla^{2} 𝑓 (𝑥, 𝑦) = [\begin{matrix} \frac{𝜕^{2} 𝑓}{𝜕 𝑥^{2}} & \frac{𝜕^{2} 𝑓}{𝜕 𝑥 𝜕 𝑦} \\ \frac{𝜕^{2} 𝑓}{𝜕 𝑦 𝜕 𝑥} & \frac{𝜕^{2} 𝑓}{𝜕 𝑦^{2}} \end{matrix}] = [\begin{matrix} 6 𝑥 𝑦 & 3 𝑥^{2} \\ 3 𝑥^{2} & 2 \end{matrix}]

\nabla^{2} 𝑓 (1, 2) = [\begin{matrix} 6 \cdot 1 \cdot 2 & 3 \cdot 1^{2} \\ 3 \cdot 1^{2} & 2 \end{matrix}] = [\begin{matrix} 12 & 3 \\ 3 & 2 \end{matrix}]

Each entry is now a scalar, and the Hessian is numeric