Partitioned Least Squares Derivation

Setup

Given the linear model:

y=Xb+e\begin{align} \pmb y = X \pmb b + \pmb e \end{align}

We can partition the columns of XX into X1X_1 and X2X_2 to get:

y=X1b1+X2b2+e\begin{align} \pmb y = X_1 \pmb b_1 + X_2 \pmb b_2 + \pmb e \end{align}

Solving the least squares problem with the partitioned representation of the model, we find the normal equations:

[X1′X1X1′X2X2′X1X2′X2][b1b2]=[X1′yX2′y]\begin{align} \begin{bmatrix} X_1' X_1 & X_1'X_2 \\ X_2'X_1 & X_2'X_2\end{bmatrix}\begin{bmatrix}\pmb b_1 \\ \pmb b_2\end{bmatrix} = \begin{bmatrix}X_1'\pmb y \\ X_2' \pmb y\end{bmatrix} \end{align}

Solving

Our normal equations form a system of (two) equations. Therefore, solve for b1\pmb b_1 and substitute into b2\pmb b_2 to find the solution.

Finding b1\pmb b_1:

X1′X1b1+X1′X2b2=X1′y  ⟹  b1=(X1′X1)−1X1′(y−X2b2)\begin{align} X_1' X_1 \pmb b_1 + X_1' X_2 \pmb b_2 &= X_1'\pmb y\\ \implies \pmb b_1 &= (X_1'X_1)^{-1}X_1'(\pmb y - X_2 \pmb b_2)\\ \end{align}

Substituting into the second normal equation and expanding:

X2′y=(X2′X2)b2+X2′X1b1=(X2′X2)b2+X2′X1(X1′X1)−1X1′(y−X2b2)=(X2′X2)b2+X2′X1(X1′X1)−1X1′y−X2′X1(X1′X1)−1X1′X2b2\begin{align} X_2'\pmb y &= (X_2'X_2)\pmb b_2 + X_2'X_1 \pmb b_1 \\ &= (X_2'X_2)\pmb b_2 + X_2'X_1(X_1'X_1)^{-1}X_1'(\pmb y - X_2 \pmb b_2)\\ &= (X_2'X_2)\pmb b_2 + X_2'X_1(X_1'X_1)^{-1}X_1'\pmb y - X_2'X_1(X_1'X_1)^{-1}X_1'X_2 \pmb b_2\\ \end{align}

Continuing with algebra to isolate b2\pmb b_2:

X2′y+X2′X1(X1′X1)−1X1′y=(X2′X2)b2−X2′X1(X1′X1)−1X1′X2b2X2′(I+X1(X1′X1)−1X1′)y=X2′(I−X1(X1′X1)−1X1′)X2b2\begin{align} X_2'\pmb y + X_2'X_1(X_1'X_1)^{-1}X_1'\pmb y &= (X_2'X_2)\pmb b_2 - X_2'X_1(X_1'X_1)^{-1}X_1'X_2 \pmb b_2 \\ X_2'(I + X_1(X_1'X_1)^{-1}X_1')\pmb y&= X_2'(I - X_1(X_1'X_1)^{-1}X_1')X_2\pmb b_2 \end{align}

Substitute in the residual maker matrices:

X2′M1y=X2′M1X2b2\begin{align} X_2'M_1\pmb y &= X_2' M_1 X_2\pmb b_2 \end{align}

Recalling that these residual makers are idempotent and symmetric:

(X2′M1′)(M1y)=(X2′M1′)(M1X2)b2(M1X2)′(M1y)=(M1X2)′(M1X2)b2X~2′y~=X~2′X~2b2\begin{align} (X_2'M_1')(M_1\pmb y) &= (X_2' M_1') (M_1 X_2)\pmb b_2 \\ (M_1 X_2)'(M_1\pmb y) &= (M_1 X_2)' (M_1 X_2)\pmb b_2 \\ \tilde X_2' \tilde{\pmb y} &= \tilde X_2'\tilde X_2 \pmb b_2 \end{align}

Finally,

b2=(X~2′X~2)−1X~2′y~\begin{align} \pmb b_2 &= (\tilde X_2' \tilde X_2)^{-1}\tilde X_2'\pmb{\tilde y} \end{align}