I know the activation function (logistic) is linear around 0.
Does this then imply a neural net should be able to approximate a GLM given the right bias and weights?
I am currently playing in R with a simple model y~x1.
The GLM performs well out of sample, yet the nnet with 1 hidden variable does poorly.
What am I missing?
Best Answer
Using the same reasoning, you could say a 2nd degree polynomial function can approximate a linear function. It is true, but the polynomial model is more complex.
This means that you will need more data to fit it correctly and it will have a stronger tendency to overfit. So, if the true model is a line and you fit/evaluate a linear model versus a 2nd degree polynomial model, the linear model will do likely better for a given amount of data.
Similar Posts:
- Solved – Logistic Regression – Adding a polynomial basis to the input matrix make sense
- Solved – Polynomial approximations of nonlinearities in neural networks
- Solved – Polynomial approximations of nonlinearities in neural networks
- Solved – Rationale behind shrinking regression coefficients in Ridge or LASSO regression
- Solved – Difference between a non-linear neuron vs non-linear activation function