Suppose I make a bunch of probabilistic forecasts like:
- 70% probability that sales growth will be 10-15% in Q1, 10% probability that sales growth will be > 15%, 20% probability that sales growth will be < 10%
Given the actual data, what's the best way to measure or track my accuracy? Brier score?
And can I average my Brier score for different types of forecasts? (e.g. Find the brier score for the prediction "there is 80% chance of rain" and average it with the sales growth forecast)
Best Answer
Your comment sounds as if you are really looking for a density forecast rather than a point forecast, i.e., you want to forecast the full probability distribution of the future outcome(s). This is a very good idea. Density forecasting is common in financial or econometric forecasting, but unfortunately it is rarely treated in other forecasting textbooks and courses. Tay & Wallis (2000, Journal of Forecasting) give a useful early survey.
The most common way of evaluating density forecasts uses the Probability Integral Transform (PIT). The canonical reference is Diebold, Gunther & Tay (1998, International Economic Review). Berkowitz (2001, Journal of Business & Economic Statistics) and Bao, Lee & Saltoglu (2007, Journal of Forecasting) give alternatives.
Recently, interest has risen in (proper) scoring rules, like the Brier score you mention. Literature includes Mitchell & Wallis (2011, Journal of Applied Econometrics) and Gneiting, Balabdaoui & Raftery (2007, JRSS-B).
Finally, Gneiting & Katzfuss (2014, Annual Review of Statistics and its Application) gives a more recent overview of density (or probabilistic) forecasting, focusing again on scoring rules.
Similar Posts:
- Solved – How to judge the accuracy of Nate Silver’s predictions
- Solved – How to judge the accuracy of Nate Silver’s predictions
- Solved – the difference between probabilistic forecasting and quantile forecasting
- Solved – Is 10 years data enough for forecasting
- Solved – Is 10 years data enough for forecasting