Solved – LDA vs. SVM for Dimensionality Reduction

Whats the difference between LDA and Linear SVM for dimensionality reduction. I am little confuse as LDA also looks for projection that separates the classes of data and SVM we also look for hyperplane …

None of the algorithms are for dimensionality reduction.(but see last comment)

In terms of classification, LDA finds the hyperplane that "best separates" all the data points while linear SVM looks for the hyperplane that "best separates" only the points in the frontier between the two classes.

Thus LDA has to do with the mean of each of the two classes (since it deals with all the points and the mean is a "good summary" of all the points). If the two classes are basically spherical then the LDA hyperplane is perpendicular to the line linking the two means of the points.

What is confusing is that most of the explanations of LDA talks about "projection of the data that best separates the classes". What is confusing is that the projection is onto a line and not a hyperplane – the LDA projection line (or LDA direction) is the line orthogonal to the separating hyperplane.

Take the image below (I did not create the image – found through Google but the original site no longer exists. I will remove it if it violates any copyright).

The LDA is the pane in the right. The green line is the LDA direction . The LDA hyperplane is the line perpendicular to the green that separates the two regions (further confusing is that in a 2D data the separating hyperplane is also a line!!)

typical LDA example figure

The figure in sklearn example of LDA shows the separating hyperplane of the LDA (also notice that since the data are not spherically distributed in each class the LDA separating hyperplane is not perpendicular to the line joining the two centers/means)

Finally, LDA can be seen as an extreme dimensionality reduction (contrary to what I wrote first). It reduces the data to one dimension – if you have to remember only one dimension of the data and you still want to classify it, the best thing you can do is to project the data to the LDA direction!!!

Similar Posts:

Rate this post

Leave a Comment