Solved – the advantage of reducing dimensionality of predictors for the purposes of regression

What are the applications or advantages of dimension reduction regression (DRR) or supervised dimensionality reduction (SDR) techniques over traditional regression techniques (without any dimensionality reduction)? These class of techniques finds a low-dimensional representation of the feature set for the regression problem. Examples of such techniques include Sliced Inverse Regression, Principal Hessian Directions, Sliced Average Variance Estimation, Kernel Sliced Inverse Regression, Principal Components Regression, etc.

  1. In terms of cross-validated RMSE, if an algorithm performed better on a regression task without any dimensionality reduction, then what is the real use of the dimensionality reduction for regression? I don't get the point of these techniques.

  2. Are these techniques by any chance used to reduce space and time complexity for regression? If that is the primary advantage, some resources on complexity reduction for high-dimensional datasets when this techniques are put to use would be helpful. I debate this with the fact that running a DRR or SDR technique itself requires some time and space. Is this SDR/DRR + Regression on a low-dim dataset faster than only regression on a high-dim dataset?

  3. Has this setting been studied out of abstract interest only, and does not have a good practical application?

As a side thought: at times there are assumptions that the joint distribution of the features $X$ and the response $Y$ lies on a manifold. It makes sense to learn the manifold from the observed sample in this context for solving a regression problem.

According to the manifold hypothesis, the data is assumed to lie on a low-dimensional manifold, the implication being that the residual is noise, so if you do your dimensionality reduction correctly, you should improve performance by modeling the signal rather than the noise. It's not just a question of space and complexity.

Similar Posts:

Rate this post

Leave a Comment