Solved – Should I use the Kernel Trick whenever possible for non-linear data

For linear data this is of course not helpful, but for non-linear data, this seems always useful. Using linear classifiers is much easier than non-linear in terms of training time and scalability.

@BartoszKP already explained why is kernel trick useful. To fully address your question however I would like to point out, that kernelization is not the only option to deal with non linearly separable data.

There are at least three good, common alternatives for delinearization of the model:

• Neutal network based methods, where you add one (or more) layers of processing units, able to transform your data into the linearly separable case. In the simplest case it is a sigmoid-based layer, which adds non-linearity to the process. Once randomly initialized they are getting updates during the gradient-based optimization of the upper layer (which actualy solves the linear problem).
• In particular – deep learning techniques can be used here to prepare data for further linear classification. It is very similar idea to the previous one, but here you first train your processing layers in order to find a good starting point for further fine-tuning based on training of some linear model.
• Random projections – you can sample (non linear) projections from some predefined space and train linear classifier on top of them. This idea is heavily exploited in so called extreme machine learning, where very efficient linear solvers are used to train a simple classifier on random projections, and achieve very good performance (on non linear problems in both classification and regression, check out for example extreme learning machines).

To sum up – kernelization is a great delinearization technique, and you can use it, when the problem is not linear, but this should not be blind "if then" appraoch. This is just one of at least few interesting methods, which can lead to various results, depending on the problem and requirements. In particular, ELM tends to find very similar solutions to those given by kernelized SVM while in the same time can be trained rows of magnitude faster (so it scales up much better than kernelized SVMs).