Orthogonal Vectors, Subspaces, Matrices, and Projections
Orthogonal Vectors and Their Length
The length (or norm) of a vector $x = (x_1, x_2, \dots, x_n)^T \in \mathbb{R}^n$ is \[ \|x\| = \sqrt{x_1^2 + x_2^2 + \dots + x_n^2} = \sqrt{x^T x}. \]
Length of $x = (1,2,3)$: \[ \|x\| = \sqrt{1^2 + 2^2 + 3^2} = \sqrt{14}. \]
Vector length in 2D:
Two vectors $x, y \in \mathbb{R}^n$ are orthogonal if \[ x^T y = 0. \]
Check orthogonality: $x = (2,2,-1)$ and $y = (-1,2,2)$: \[ x^T y = 2(-1) + 2(2) + (-1)(2) = 0 \] Hence, $x \perp y$.
A set of nonzero vectors $\{v_1, \dots, v_k\}$ that are mutually orthogonal is automatically linearly independent.
Orthogonal Subspaces and Complements
Two subspaces $V, W \subseteq \mathbb{R}^n$ are orthogonal if every vector $v \in V$ is orthogonal to every vector $w \in W$: \[ v^T w = 0 \quad \forall v \in V, \forall w \in W. \]
$V$ spanned by $v_1=(1,0,0,0), v_2=(1,1,0,0)$, $W$ spanned by $w=(0,0,4,5)$. Then $W \perp V$ because $v_i^T w = 0$.
The orthogonal complement of $V$ is \[ V^\perp = \{x \in \mathbb{R}^n \mid x^T v = 0, \forall v \in V\}. \]
Fundamental Theorem of Linear Algebra (Orthogonality Version): For $A \in \mathbb{R}^{m \times n}$,
\begin{align*}
N(A) &= (C(A^T))^\perp,
N(A^T) &= (C(A))^\perp.
\end{align*}
$A = \begin{bmatrix}1 & 3\\ 2 & 6\\ 3 & 9\end{bmatrix}$, rank 1. Row space multiples of $(1,3)$, nullspace $x=(-3,1)$. \[ A x = 0 \Rightarrow N(A) \perp \text{Row space}. \] Column space: line through $(1,2,3)$, left nullspace: plane $y_1+2y_2+3y_3=0$.
Orthogonal subspaces in 3D:
Orthogonal Matrices
$Q \in \mathbb{R}^{n \times n}$ is orthogonal if $Q^T Q = QQ^T = I_n$.
Properties of orthogonal matrices:
$Q^T = Q^{-1}$
$\det(Q) = \pm 1$
Columns (and rows) form an orthonormal set
$\|Qx\| = \|x\|$ for all $x$
Product of orthogonal matrices is orthogonal
$Q = \begin{bmatrix}0 & 1 \\ -1 & 0\end{bmatrix}$, $Q^T Q = I_2$, so $Q$ is orthogonal.
$\det \begin{bmatrix}1/2 & \sqrt{3}/2 \\ -\sqrt{3}/2 & 1/2\end{bmatrix} = 1$.
Projection Matrices
Projection matrix $P$ maps $x \in \mathbb{R}^n$ onto a subspace $W$: \[ Px = \text{proj}_W(x) \]
If $W$ spanned by $V=[v_1\dots v_k]$, then \[ P = V(V^T V)^{-1} V^T \]
Properties of projection matrices:
$P^2 = P$ (idempotent)
$P^T = P$ (symmetric)
Eigenvalues $0$ or $1$
Rank$(P) = \dim(W)$
$I-P$ projects onto $W^\perp$
Projection onto line spanned by $v = \begin{bmatrix}2\\1\\2\end{bmatrix}$: \[ P = \frac{vv^T}{v^T v} = \frac{1}{9} \begin{bmatrix}4 & 2 & 4 \\ 2 & 1 & 2 \\ 4 & 2 & 4\end{bmatrix}. \]
Project $x = \begin{bmatrix}1\\2\\3\end{bmatrix}$ onto line spanned by $v = \begin{bmatrix}1\\1\\1\end{bmatrix}$: \[ P = \frac{vv^T}{v^T v} = \frac{1}{3} \begin{bmatrix}1 &1 &1\\1 &1 &1\\1 &1 &1\end{bmatrix},\quad Px = \begin{bmatrix}2\\2\\2\end{bmatrix} \]
Projection onto a plane in 3D:
Orthogonal and Symmetric Matrices
A square matrix $A \in \mathbb{R}^{n \times n}$ is symmetric if \[ A^T = A. \]
A matrix $Q$ is both orthogonal and symmetric if \[ Q^T = Q \quad \text{and} \quad Q^T Q = I. \]
Properties of matrices that are both orthogonal and symmetric:
$Q^2 = I$ (since $Q^T Q = I$ and $Q^T = Q$)
Eigenvalues are either $+1$ or $-1$
$Q^{-1} = Q$
Check if \[ Q = \begin{bmatrix}1 & 0 \\ 0 & -1\end{bmatrix} \] is orthogonal and symmetric.
Solution: \[ Q^T = Q \quad \text{(symmetric)}, \quad Q^T Q = \begin{bmatrix}1 & 0 \\ 0 & 1\end{bmatrix} = I_2 \quad \text{(orthogonal)}. \] Hence $Q$ is both orthogonal and symmetric. Its eigenvalues are $1$ and $-1$.
If \[ Q = \begin{bmatrix}0 & 1 \\ 1 & 0\end{bmatrix}, \] then $Q^T = Q$ (symmetric) and $Q^T Q = I_2$ (orthogonal). Eigenvalues are $1$ and $-1$.
Observation: Any orthogonal and symmetric matrix represents a reflection across a subspace.
Applications of Orthogonality and Projections
Linear Independence and Orthogonal Vectors: A set of nonzero vectors $\{v_1, v_2, \dots, v_k\}$ that are mutually orthogonal is always linearly independent. This simplifies checking independence: if $v_i^T v_j = 0$ for $i \neq j$, the vectors are independent.
Check if the vectors \[ v_1 = \begin{bmatrix}1\\0\\1\end{bmatrix}, \quad v_2 = \begin{bmatrix}0\\1\\1\end{bmatrix}, \quad v_3 = \begin{bmatrix}1\\-1\\0\end{bmatrix} \] are linearly independent.
Solution: Compute inner products: \[ v_1^T v_2 = 0, \quad v_1^T v_3 = 1, \quad v_2^T v_3 = -1 \] Vectors are not mutually orthogonal, so we cannot immediately conclude independence. However, Gram-Schmidt can create an orthogonal basis to check independence.
Solving Systems of Linear Equations: For $Ax = b$, if columns of $A$ are orthogonal or orthonormal, the solution is straightforward: \[ x_i = \frac{a_i^T b}{a_i^T a_i} \quad \text{for each column } a_i \] where $A = [a_1 \dots a_n]$.
Solve \[ A x = b, \quad A = \begin{bmatrix}1 & 0 \\ 0 & 2\end{bmatrix}, \quad b = \begin{bmatrix}3\\4\end{bmatrix}. \]
Solution: Columns of $A$ are orthogonal. \[ x_1 = \frac{1 \cdot 3 + 0 \cdot 4}{1^2 + 0^2} = 3, \quad x_2 = \frac{0 \cdot 3 + 2 \cdot 4}{0^2 + 2^2} = 2 \] Hence, $x = \begin{bmatrix}3\\2\end{bmatrix}$.
Best Approximation using Projection: If $Ax = b$ is inconsistent, the orthogonal projection of $b$ onto the column space of $A$ gives the vector $\hat{b}$ closest to $b$ that is in the span of $A$. Then, solve $A x = \hat{b}$ for the least-squares solution: \[ \hat{x} = (A^T A)^{-1} A^T b \]
Find the projection of \[ b = \begin{bmatrix}3\\4\\5\end{bmatrix} \] onto the column space of \[ A = \begin{bmatrix}1 & 0 \\ 0 & 1 \\ 1 & 1\end{bmatrix}. \]
Solution: \[ \hat{x} = (A^T A)^{-1} A^T b, \quad A^T A = \begin{bmatrix}2 & 1 \\ 1 & 2\end{bmatrix}, \quad (A^T A)^{-1} = \frac{1}{3}\begin{bmatrix}2 & -1 \\ -1 & 2\end{bmatrix} \] \[ A^T b = \begin{bmatrix}8\\9\end{bmatrix}, \quad \hat{x} = \frac{1}{3}\begin{bmatrix}2 & -1 \\ -1 & 2\end{bmatrix} \begin{bmatrix}8\\9\end{bmatrix} = \begin{bmatrix}7/3\\10/3\end{bmatrix} \] \[ \text{proj}_{\text{Col}(A)}(b) = A \hat{x} = \begin{bmatrix}7/3\\10/3\\17/3\end{bmatrix} \]
Orthogonal Decomposition: Any vector $b$ in $\mathbb{R}^n$ can be uniquely decomposed into components along a subspace $W$ and its orthogonal complement $W^\perp$: \[ b = \text{proj}_W(b) + (b - \text{proj}_W(b)), \quad \text{with } (b - \text{proj}_W(b)) \in W^\perp \]
If \[ b = \begin{bmatrix}3\\4\end{bmatrix}, \quad W = \text{span}\{\begin{bmatrix}1\\1\end{bmatrix}\}, \] then \[ \text{proj}_W(b) = \frac{b^T v}{v^T v} v = \frac{3+4}{1+1} \begin{bmatrix}1\\1\end{bmatrix} = \frac{7}{2} \begin{bmatrix}1\\1\end{bmatrix} = \begin{bmatrix}3.5\\3.5\end{bmatrix} \] \[ b - \text{proj}_W(b) = \begin{bmatrix}-0.5\\0.5\end{bmatrix} \in W^\perp \]