It's not a dumb question at all. The classification accuracy and "uncertainty" are different, but the explanation depends on what you mean by uncertainty.
Let me try and give one intuitive explanation; if others would like to chime in with something better, by all means.
Let's suppose that you are classifying objects in images - say bananas and oranges, but it could be tumors or anything that you like.
So we train a classifier to predict this, and we find that of 100 classifications on a hold-out set, we get 73 of them correct. You might, quite reasonably, interpret this to mean that if we randomly select a new image of either an orange or a banana, we will have a 0.73 probability of classifying it correctly. (There are actually some subtleties in this interpretation which I'm ignoring, but they aren't so important for the point I want to make.)
Suppose, however, that we draw out an image that we want to feed into our classifier, and we look at it for a moment. Suppose further that this image contains an object that is long, thing, curved and yellow. We'd expect our classifier to classify it as a banana, and sure enough, it does. Now we draw out another image, except this one has an object that is long, but bent almost completely in a circle, and is more orange than yellow. Now, we might still expect our classifier to classify this as a banana, but should the classifier really be as certain about this prediction as it was about the previous one? Intuitively, I would say not. However, the overall classification accuracy remains unchanged, and so we can't say anything in particular about the certainty of this prediction.
So uncertainty isn't just the proportion of your results that you classify correctly.
Furthermore, it also isn't exactly equivalent to the class probability produced by your classifier, though I don't think this is the best forum for me to get into the details on that.
laingc|9 years ago
Let me try and give one intuitive explanation; if others would like to chime in with something better, by all means.
Let's suppose that you are classifying objects in images - say bananas and oranges, but it could be tumors or anything that you like.
So we train a classifier to predict this, and we find that of 100 classifications on a hold-out set, we get 73 of them correct. You might, quite reasonably, interpret this to mean that if we randomly select a new image of either an orange or a banana, we will have a 0.73 probability of classifying it correctly. (There are actually some subtleties in this interpretation which I'm ignoring, but they aren't so important for the point I want to make.)
Suppose, however, that we draw out an image that we want to feed into our classifier, and we look at it for a moment. Suppose further that this image contains an object that is long, thing, curved and yellow. We'd expect our classifier to classify it as a banana, and sure enough, it does. Now we draw out another image, except this one has an object that is long, but bent almost completely in a circle, and is more orange than yellow. Now, we might still expect our classifier to classify this as a banana, but should the classifier really be as certain about this prediction as it was about the previous one? Intuitively, I would say not. However, the overall classification accuracy remains unchanged, and so we can't say anything in particular about the certainty of this prediction.
So uncertainty isn't just the proportion of your results that you classify correctly.
Furthermore, it also isn't exactly equivalent to the class probability produced by your classifier, though I don't think this is the best forum for me to get into the details on that.
nextos|9 years ago
But you want the uncertainty conditioned to the particular observations you have made.
A ton more here:
http://mlg.eng.cam.ac.uk/yarin/blog_2248.html
nerfhammer|9 years ago