What is parameter instability and how can I measure it?
If my model is having a hard time to forecast out-of-time samples, could parameter instability or populational instability be the cause of it?
Best Answer
I agree with @gung there are some terminology or definition issues needs to be clarified.
What you described "hard time to forecast out-of-time samples" can be viewed as a "overfitting" problem in machine learning literature. For example, if we fit a very complex model on small amount of data, it is very likely the model "overfit" the data given, and fail to generalize to out of time samples.
Here are a widely used examples for overfitting (from wikipedia). You can see the model is trying too hard to fit the given data perfectly. Because the data contains noise, the model is not really generalizable to out of time sample.
To measure if the model is overfitting, bais and variance of the model can be calculated. For OLS, an example can find here. Intuitive explanation of the bias-variance tradeoff?
In fact, the first thing comes to my mind by reading the question title is numerical stability. This means because computer can only process finite digits numbers, there are algorithms that work theoretically but not working in real world.