I'm just learning about optimization, and the difference between an analytical solution, and a numerical one. Suppose there is a cost function f(x), and we want to find the value of x which minimizes this. In an analytical solution, we would differentiate with respect to x, i.e. find df(x)/dx. Whereas in a numerical solution, we would try values of x, and see in which direction we need to change x in order to move towards the solution (gradient descent).
It seems to me that the analytical solution is always preferred because it gives you an exact answer. You do not get stuck in local minima and you can be sure that your solution is correct. So why do we not always use analytical solutions?
I understand that numerical solutions make things easier because you don't have to work out by hand the analytical solution, but it seems to me that solving something by hand to get a precise answer is highly preferred to doing it computationally for an approximate answer.
Can somebody clear this up for me? Thanks!
Best Answer
Your question is interesting, because it is a starting point into optimization in general.
It is maybe better to start with a concrete example on the existence of analytical solution. If you look at your standard polynomial of 2nd degree, $f(x)=ax^2+bx+c$. You have a formula for the zeroes of the function, which is an analytical solution. If you have a polynomial of order 5 or higher, no such formula exists.
This is of course not an optimization problem as in minimizing or maximizing a function, but when you find where the derivative is zero, you are essentially looking for a zero of a function, namely the derivative.
So it seems that you cannot always solve a problem with pen and paper. This is also dependent on the scale of the problem. Sometimes you need to estimate billions of parameters, and it is just not feasible for a human to work out an analytical solution or even find out if it exists.
In the case of ordinary least squares, or fitting a linear model, an analytical solution exists. But it is very easy to change the model slightly, such that no such solution exists, (at least not one we know of). An example of this is the lasso.
So the use of numerical methods is completely justified, both by the fact that an analytical solution may not exist, or it is not feasible to work out such a solution.
Similar Posts:
- Solved – Why are numerical solutions preferred to analytical solutions
- Solved – Why are numerical solutions preferred to analytical solutions
- Solved – When is logistic regression solved in closed form
- Solved – Is EM feasible when there is no closed form maximization of the expectation of log likelihood
- Solved – Most suitable algorithm for optimizing Maximum likelihood function