Partitioned Least Squares

Understanding what it means to "partial out" variable effects.

Prerequisites: In order to understand the math in this article you should understand OLS projection matrices from this article.

Setup

Given the linear model:

y=Xb+e\begin{align} \pmb y = X \pmb b + \pmb e \end{align}

We can partition the columns of XX into X1X_1 and X2X_2 to get:

y=X1b1+X2b2+e\begin{align} \pmb y = X_1 \pmb b_1 + X_2 \pmb b_2 + \pmb e \end{align}

For clarity, think of X1X_1 as the control variables and X2X_2 as the variables of interest.

Solving the least squares problem with the partitioned representation of the model, we find the normal equations:

[X1′X1X1′X2X2′X1X2′X2][b1b2]=[X1′yX2′y]\begin{align} \begin{bmatrix} X_1' X_1 & X_1'X_2 \\ X_2'X_1 & X_2'X_2\end{bmatrix}\begin{bmatrix}\pmb b_1 \\ \pmb b_2\end{bmatrix} = \begin{bmatrix}X_1'\pmb y \\ X_2' \pmb y\end{bmatrix} \end{align}

From here we find 2 important results from two cases.

Case 1: Regressors of X1X_1 are uncorrelated with the regressors of X2X_2

Having uncorrelated explanatory variables is equivalent to:

Col(X1)⊥Col(X2)  ⟺  X1′X2=X2′X1=0\begin{align} \mathcal{Col}(X_1) \perp \mathcal{Col}(X_2) \iff X_1'X_2 = X_2'X_1 = 0 \end{align}

Given this condition, our normal equations are symmetric and independent.

Without loss of generality:

X1′X1b1=X1′y  ⟹  b1=(X1′X1)−1X1′y\begin{align} X_1' X_1 \pmb b_1 &= X_1'\pmb y \implies \pmb b_1 = (X_1'X_1)^{-1}X_1'\pmb y \end{align}

Thus, recovered coefficients are unchanged when uncorrelated regressors are included as controls.

Case 2: Regressors of X1X_1 and X2X_2 are correlated

If our variables of interest and controls are correlated, this is equivalent to:

X1′X2≠0\begin{align} X_1' X_2 \neq 0 \end{align}

Our normal equations form a system of (two) equations. After some algebra, we get the result

b2=(X~2′X~2)−1X~2′y~\begin{align} \pmb b_2 &= (\tilde X_2' \tilde X_2)^{-1}\tilde X_2'\pmb{\tilde y} \end{align}

Where the variables with a tilde are partialled out by the control variables.

Therefore, controlling for variables in OLS is equivalent to regressing the residuals of the outcome variable on the residuals of the explanatory variables of interest.

Geometrically, this means that controlling for variables leaves us with a collection of the orthogonal component of the variables of interest as our new basis for the variables of interest. Instead of finding the combination of these vectors that will be in the shadow of y\pmb y, we find the combination of these vectors that will be in the shadow of y~\tilde{\pmb y}.

By shadow, I mean the reachable (determined by X~2\tilde X_2) point that minimizes the length of the residual vector that connects y~^\pmb{\hat{\tilde y}} to y~\pmb{{\tilde y}} by the relationship y~=y~^+e~ \pmb{{\tilde y}} = \pmb{\hat{\tilde y}} + \pmb{\tilde e}.