Suppose I make a bunch of probabilistic forecasts like:

- 70% probability that sales growth will be 10-15% in Q1, 10% probability that sales growth will be > 15%, 20% probability that sales growth will be < 10%

Given the actual data, what's the best way to measure or track my accuracy? Brier score?

And can I average my Brier score for different types of forecasts? (e.g. Find the brier score for the prediction "there is 80% chance of rain" and average it with the sales growth forecast)

**Contents**hide

#### Best Answer

Your comment sounds as if you are really looking for a density forecast rather than a point forecast, i.e., you want to forecast the full probability distribution of the future outcome(s). This is a very good idea. Density forecasting is common in financial or econometric forecasting, but unfortunately it is rarely treated in other forecasting textbooks and courses. Tay & Wallis (2000, *Journal of Forecasting*) give a useful early survey.

The most common way of evaluating density forecasts uses the Probability Integral Transform (PIT). The canonical reference is Diebold, Gunther & Tay (1998, *International Economic Review*). Berkowitz (2001, *Journal of Business & Economic Statistics*) and Bao, Lee & Saltoglu (2007, *Journal of Forecasting*) give alternatives.

Recently, interest has risen in (proper) scoring rules, like the Brier score you mention. Literature includes Mitchell & Wallis (2011, *Journal of Applied Econometrics*) and Gneiting, Balabdaoui & Raftery (2007, *JRSS-B*).

Finally, Gneiting & Katzfuss (2014, *Annual Review of Statistics and its Application*) gives a more recent overview of density (or probabilistic) forecasting, focusing again on scoring rules.

### Similar Posts:

- Solved – How to judge the accuracy of Nate Silver’s predictions
- Solved – How to judge the accuracy of Nate Silver’s predictions
- Solved – the difference between probabilistic forecasting and quantile forecasting
- Solved – Is 10 years data enough for forecasting
- Solved – Is 10 years data enough for forecasting