Solved – How to measure probabilistic forecast accuracy

Suppose I make a bunch of probabilistic forecasts like:

  • 70% probability that sales growth will be 10-15% in Q1, 10% probability that sales growth will be > 15%, 20% probability that sales growth will be < 10%

Given the actual data, what's the best way to measure or track my accuracy? Brier score?

And can I average my Brier score for different types of forecasts? (e.g. Find the brier score for the prediction "there is 80% chance of rain" and average it with the sales growth forecast)

Your comment sounds as if you are really looking for a density forecast rather than a point forecast, i.e., you want to forecast the full probability distribution of the future outcome(s). This is a very good idea. Density forecasting is common in financial or econometric forecasting, but unfortunately it is rarely treated in other forecasting textbooks and courses. Tay & Wallis (2000, Journal of Forecasting) give a useful early survey.

The most common way of evaluating density forecasts uses the Probability Integral Transform (PIT). The canonical reference is Diebold, Gunther & Tay (1998, International Economic Review). Berkowitz (2001, Journal of Business & Economic Statistics) and Bao, Lee & Saltoglu (2007, Journal of Forecasting) give alternatives.

Recently, interest has risen in (proper) scoring rules, like the Brier score you mention. Literature includes Mitchell & Wallis (2011, Journal of Applied Econometrics) and Gneiting, Balabdaoui & Raftery (2007, JRSS-B).

Finally, Gneiting & Katzfuss (2014, Annual Review of Statistics and its Application) gives a more recent overview of density (or probabilistic) forecasting, focusing again on scoring rules.

Similar Posts:

Rate this post

Leave a Comment