The formula for adjusted $R^2$ is:
$$
1 – frac{(n-1)}{(n-p-1)}(1-R^2)
$$
where $r^2$ is the coefficient of determination, $n$ is the number of points, and $p$ is the number of parameters the model has.
If I want the adjusted $R^2$ from a holdout set, is $n$ the number of points in the data the model was trained on, or the number of points in the holdout set?
Best Answer
There is no need to make the adjustment on a hold-out sample, or if you did, $p$ would equal 1. For evaluation in a hold-out sample, all coefficients estimated in the training sample must be frozen. But you really should not be using that formula for $R^2$ as it implies you are forcing $R^2$ to be positive. The correct formula allows $R^{2} < 0$ because predictions can be worse than chance. So compute $R^{2} = 1 – frac{SSE}{SST}$ where $SSE$ is sum of squared errors and $SST$ is sum of squares total ($(n-1) times$ the variance of $Y$).