Singular Value Decomposition (SVD)

Factor any matrix as A = U V^ via the eigenvalues of A^ A.

The idea

Singular value decomposition. Every real $m \times n$ matrix $A$ can be written as $A = U\,\Sigma\,V^{\top},$ where $U$ is an $m \times m$ orthogonal matrix, $V$ is an $n \times n$ orthogonal matrix, and $\Sigma$ is an $m \times n$ diagonal matrix whose diagonal entries $\sigma_1 \ge \sigma_2 \ge \cdots \ge 0$ are the singular values of $A$.

Read right to left, the factorization says every matrix performs three steps: $V^{\top}$ rotates or reflects the input, $\Sigma$ stretches each resulting axis direction by its own factor $\sigma_i$, and $U$ rotates or reflects the result. In the plane, $A$ therefore carries the unit circle to an ellipse, as in the figure: the perpendicular columns $v_1, v_2$ of $V$ are the input directions $A$ stretches cleanly, and they land along $\sigma_1 u_1$ and $\sigma_2 u_2$, the semi-axes of the ellipse, where $u_1, u_2$ are the columns of $U$.

The factorization exists for every matrix, unlike the diagonalization $X \Lambda X^{-1}$, which requires $A$ to be square with a full set of independent eigenvectors. The reason is that the SVD is built from $A^{\top}A$: that matrix is symmetric, and it satisfies $x^{\top}A^{\top}Ax = \lVert Ax \rVert^{2} \ge 0$, so it always has an orthonormal eigenbasis — the columns of $V$ — and nonnegative eigenvalues, the $\sigma_i^{2}$.

Ways to work on it

• Walkthrough. Why A = U V^ exists, and singular values from A^ A.
• Practice. Find a singular value from the eigenvalues of A^ A.
• Hardest. Singular values of a non-diagonal matrix via its characteristic equation.

Not sure where to start? Take the ten-question placement test.