I am using ROC curves and full AUC values to compare different models, using simulated data. Now I think I am confused with the interpretations of ROC curves and AUC values. Please see the figure below (sorry it is partial from screen shots…)
There are three models compared, and I know that the model shown in green should preform best of all. However, as you can see, the green curve is superior to the other two before the FPR reaching around 0.2. This cut-off of 0.2 is quite interesting: it is the percentage of differentially expressed genes that I specify in my simulation (i.e. 20% of the observations are simulated to be positives).
My concern are:
given that people in reality will seldom choose a FPR cut-off of 0.5 or higher, why people would prefer a ROC curve with FPR ranging from 0 to 1 and use the full AUC value (i.e. calculate the entire area under the ROC curve) instead of just reporting the area made from, say, 0 to 0.25 or to 0.5? Is that called "partial AUC"?
in the figure below, what can we say about the performances of the three models? The AUC values are: green (0.805), red (0.815), blue (0.768). The red curve turns out to be superior, but as you see, the superiority is only reflected after FPR > 0.2. Thanks 🙂
Best Answer
I agree with your concerns.
given that people in reality will seldom choose a FPR cut-off of 0.5 or higher, why people would prefer a ROC curve with FPR ranging from 0 to 1 and use the full AUC value (i.e. calculate the entire area under the ROC curve) instead of just reporting the area made from, say, 0 to 0.25 or to 0.5? Is that called "partial AUC"?
- I'm a big fan of having the complete ROC, as it gives much more information that just the sensitivity/specificity pair of one working point of a classifier.
- For the same reason, I'm not a big fan of summarizing all that information even further into one single number. But if you have to do so, I agree that it is better to restrict the calculations to parts of the ROC that are relevant for the application.
in the figure below, what can we say about the performances of the three models? The AUC values are: green (0.805), red (0.815), blue (0.768). The red curve turns out to be superior, but as you see, the superiority is only reflected after FPR > 0.2. Thanks 🙂
- That depends entirely on your application. In your example, if high specificity is needed, then the green classifier would be best. If high sensitivity is needed, go for the red one.
As to the comparison of classifiers: there are lots of questions and answers here discussing this. Summary:
- classifier comparison is far more difficult than one would expect at first
- not all classifier performance measures are good for this task. Read @FrankHarrells answers, and go for so-called proper scoring rules (e.g. Brier's score/mean squared error).
Similar Posts:
- Solved – ROC AUC and PR AUC: Are the AUC values different for each class
- Solved – An intuitive meaning of the area under the PR curve
- Solved – what is the difference between area under roc and weighted area under roc
- Solved – Generating the ROC curve for ensemble Classifier
- Solved – What exactly the ROC curve can tell us or can be inferred