I'm aware of the bias variance trade off.
Intuitively I understand how as the model becomes more complex the variance decreases and the bias increases, after a certain point.
But I don't really understand bias.
For example:
If we have a predictor variable x, and we want to estimate a y.
Bias = E[x] – y
residual = x*B – y <=> E[x] – y
Best Answer
A bias is a property of an estimator or a statistics, NOT of a stochastic realization. It means that an estimator or a statistics is calculated in a way that it is SYSTEMATICALLY different from the quantity that is supposed to summarize / estimate.
These things are NOT examples of bias:
- Residuals for a single experiment
- The difference of a parameter estimate or prediction from the truth for a single experiment (unless it is systematic)
- Anything else that is stochastic and not systematic
The bias variance trade-off is maybe not an ideal name, it should maybe have better been called interpolation/extrapolation trade-off. Anyway, the motivation for the name is that that when adding more parameters / complexity, you have
- Less systematic error (bias) in your model (supposedly, because it is more flexible, I would argue it depends on what you call error / bias)
- More variance in the estimation of the model parameters (because it is more flexible)
Similar Posts:
- Solved – Why must one trade off between bias and variance
- Solved – Variance/bias trade off regularisation penalty – why does it take this form
- Solved – Does bias in statistics and machine learning mean the same thing
- Solved – the relationship between estimation error, approximation error, bias, variance in machine learning
- Solved – the difference between an estimator and a statistic