Singular Value Decomposition (SVD)

Introduction to SVD

Singular Value Decomposition (SVD) is a fundamental matrix factorization technique in linear algebra. It expresses any real or complex matrix as a product of three matrices, revealing important geometric and algebraic properties. SVD is widely used in data analysis, signal processing, and numerical methods.

• Matrix Factorization: Any m × n matrix A can be written as , where P and Q are orthogonal matrices and D is a diagonal matrix.

• Generalization: SVD generalizes diagonalization to non-square or non-symmetric matrices.

Singular Values

Definition and Properties

Singular values are the square roots of the eigenvalues of (or ). They measure the 'stretching' effect of the matrix on vectors.

• Definition: For a matrix , the singular values are the non-negative square roots of the eigenvalues of .

• Notation: If is , the singular values are , where .

• Multiplicity: The number of nonzero singular values equals the rank of .

Example: If , then , so the singular values are $1.

Definition of Singular Value Decomposition

Formal Statement

A singular value decomposition of an matrix is a factorization:

• U: orthogonal matrix (columns are left singular vectors)

• \Sigma: diagonal matrix with singular values on the diagonal

• V: orthogonal matrix (columns are right singular vectors)

Example: For , and can be identity matrices, and is .

How to Find a SVD

Step-by-Step Procedure

1. Compute and find its eigenvalues and eigenvectors.

2. The singular values are the square roots of the eigenvalues of .

3. The right singular vectors (columns of ) are the normalized eigenvectors of .

4. The left singular vectors (columns of ) are computed as for nonzero .

Example: For , follow the steps above to compute the SVD.

Applications of SVD

Importance and Uses

• Data Analysis: SVD is used in principal component analysis (PCA) to reduce dimensionality.

• Low-Rank Approximation: SVD helps approximate a matrix by one of lower rank, useful in image compression and noise reduction.

• Solving Linear Systems: SVD provides stable solutions to least squares problems, even when is not invertible.

Example: Approximating a matrix by keeping only the largest singular values and corresponding vectors.

Exercises

Practice Problems

1. Find a singular value decomposition of the matrix .

2. Find a unit vector in the nullspace of if it exists.

3. Show that if is an matrix, then the singular values of are the square roots of the eigenvalues of .

4. Let be an orthogonal matrix. Show that all singular values of are $1$.