I was asked a relatively simple problem and was curious as to how to solve.
Say I had a bomb detector at the airport, and it is 99.99% correct. That being, when the detector goes off or does not go off, it is correct 99.99% of the time. And in the population, 1/10,000 people actually have a bomb.
What is the probability that when it went off, the person actually had a bomb?
EDIT**
This is a Bayes Theorem question. We want to know the probability of a person having a bomb given the false positive/ true positive probability as given here
Best Answer
As you claim that 1 in 10000 has a bomb, we can construct following confusion matrix (assuming our population has a size of 10000, WLOG):
Bomb NoBomb Detected TP FN Not Det. FP TN 1 9999 Your definition of "correctness" is called Accuracy
$ Accuracy = frac{TP + TN}{TP + TN + FP + FN}= frac{9999}{10000} = 0.9999 $
So mathematically we have 2 variants: either (TP=1, TN=9998), or (TP=0, TN=9999)
Your want to get the value of Precision, that is
$ Precision = frac{TP}{TP + FN} $
So in first case we have the Precision=1, that is the system always detects the bomber, and in the second case, Precision=0 that is system never detects a bomber. And you have no indication which is really your case.
The purpose of this exercise is not to find the value of Precision. It is to show, that in case of highly imbalanced class distribution, the value of Accuracy is not really indicative, and should not be used to measure system performance. In reality, for imbalanced classes, discriminators tend to detect always the major class. So your bomb detection system will never find a bomber, and still will have excellent Accuracy 😉