Solved – Why ANOVA/Regression results change when controlling for another variable

Linear regression can be illustrated geometrically in terms of an orthogonal projection of the predicted variable vector $boldsymbol{y}$ onto the space defined by the predictor vectors $boldsymbol{x}_{i}$. This approach is nicely explained in Wicken's book "The Geometry of Multivariate Statistics" (1994). Without loss of generality, assume centered variables. In the following diagrams, the length of a vector equals its standard deviation, and the cosine of the angle between two vectors equals their correlation (see here). The simple linear regression from $boldsymbol{y}$ onto $boldsymbol{x}$ then looks like this:

$hat{boldsymbol{y}} = b cdot boldsymbol{x}$ is the prediction that results from the orthogonal projection of $boldsymbol{y}$ onto the subspace defined by $boldsymbol{x}$. $b$ is the projection of $boldsymbol{y}$ in subspace coordinates (basis vector $boldsymbol{x}$). This prediction minimizes the error $boldsymbol{e} = boldsymbol{y} – hat{boldsymbol{y}}$, i.e., it finds the closest point to $boldsymbol{y}$ in the subspace defined by $boldsymbol{x}$ (recall that minimizing the error sum of squares means minimizing the variance of the error, i.e., its squared length). With two correlated predictors $boldsymbol{x}_{1}$ and $boldsymbol{x}_{2}$, the situation looks like this:

$boldsymbol{y}$ is projected orthogonally onto $U$, the subspace (plane) spanned by $boldsymbol{x}_{1}$ and $boldsymbol{x}_{2}$. The prediction $hat{boldsymbol{y}} = b_{1} cdot boldsymbol{x}_{1} + b_{2} cdot boldsymbol{x}_{2}$ is this projection. $b_{1}$ and $b_{2}$ are thus the ends of the dotted lines, i.e. the coordinates of $hat{boldsymbol{y}}$ in subspace coordinates (basis vectors $boldsymbol{x}_{1}$ and $boldsymbol{x}_{2}$).

The next thing to realize is that the orthogonal projections of $hat{boldsymbol{y}}$ onto $boldsymbol{x}_{1}$ and $boldsymbol{x}_{2}$ are the same as the orthogonal projections of $boldsymbol{y}$ itself onto $boldsymbol{x}_{1}$ and $boldsymbol{x}_{2}$.

This allows us to directly compare the regression weights from each simple regression with the regression weights from the multiple regression:

$hat{boldsymbol{y}}_{1}$ and $hat{boldsymbol{y}}_{2}$ are the predictions from the simple regressions $boldsymbol{y}$ onto $boldsymbol{x}_{1}$, and $boldsymbol{y}$ onto $boldsymbol{x}_{2}$. Their endpoints give the individual regression weights $b^{1} = rho_{x_{1} y} cdot sigma_{y}$ and $b^{2} = rho_{x_{2} y} cdot sigma_{y}$, where $rho_{x_{1} y}$ is the correlation between $boldsymbol{x}_{1}$ and $boldsymbol{y}$, and $sigma_{y}$ is the standard deviation of $boldsymbol{y}$. In contrast, the endpoints of the dotted lines give the regression weights from the multiple regression of $boldsymbol{y}$ onto $boldsymbol{x}_{1}$ and $boldsymbol{x}_{2}$: $b_{1} = beta_{1} sigma_{y}$, where $beta_{1}$ is the standardized regression coefficient.

Now it is easy to see that $b^{1}$ and $b^{2}$ will coincide exactly with $b_{1}$ and $b_{2}$ only if $boldsymbol{x}_{1}$ and $boldsymbol{x}_{2}$ are orthogonal (or if $boldsymbol{y}$ is orthogonal to the plane spanned by $boldsymbol{x}_{1}$ and $boldsymbol{x}_{2}$). It is also easy to geometrically construct cases that sometimes seem puzzling, e.g., when the regression weight has the opposite sign as the bivariate correlation between a predictor and the predicted variable:

Here, $boldsymbol{x}_{1}$ and $boldsymbol{x}_{2}$ are highly correlated. Now the sign of the correlation between $boldsymbol{y}$ and $boldsymbol{x}_{1}$ is positive (red line: orthogonal projection of $boldsymbol{y}$ onto $boldsymbol{x}_{1}$), but the regression weight from the multiple regression is negative (end of green line onto subspace defined by $boldsymbol{x}_{1}$.