[blackshaw]

So? 25% doesn't mean it can't happen. I wouldn't cross the road if there was a 25% of being hit by a car.

The real way to measure predictive accuracy is to find _all_ the times they gave something a 25% chance of happening. If the predictions are accurate then roughly 25% of those things should have happened.

[nojs]

If I am asked to predict whether it will rain every day for a year, and every day I blindly predict 30% because on average it rains 30% of days in this location, am I correct? Maybe, but I’m not a very useful forecaster.

What you should optimise instead is something like log loss between the given probability and the true outcome (0 or 1). That way you’re rewarded not only for being right, but for being confident and right.