[enjeyw]

It absolutely has been!

In general prediction markets can’t be “correct” or “incorrect” - for instance if a prediction market says there’s a 60% chance of an event occurring, and it doesn’t occur, was the market right or wrong? Well it’s hard to say - certainly the market said the event was more likely to occur than not, but only just, and who knows? Maybe the event _only just_ occurred, and very nearly didn’t!

So generally we say a prediction market is “correct” if it is “well calibrated”, which is to say that if we took all the events that the market said had a 60% chance of occurring, then approximately 60% percent of these events occurred (with the same holding true for all other percentages).

On this note, an interesting phenomenon that used to occur was “favorite-longshot bias”, where markets would consistently overestimate the likelihood of longshot events occurring - so events that the market predicted would occur 10% of the time would only occur 5% of the time. What’s fascinating is that once people realized that this bias exited, they began to exploit it by making bets against longshots, which had the effect of moving the market and removing the biases, making the markets well calibrated. It’s a pretty neat example of the efficient market hypothesis in action!

[BrenBarn]

Some of the longshot biases still exists and can't be removed due to technical constraints on the platforms. A lot of times there is a minimum contract price, which effectively means the probability of unlikely events cannot be modeled as lower than 1% or 0.1% or whatever. But there are contracts for events much less likely than that.

[pseudo0]

There are also issues with the time value of money for long-shot events. Someone has to be willing to buy a share of "No", and if that works out to a return lower than the risk-free rate (eg. buying t-bills) there will be no incentive to take the "No" position. That makes anything roughly under 3-4% per year pretty unreliable.

[thaumasiotes]

> for instance if a prediction market says there’s a 60% chance of an event occurring, and it doesn’t occur, was the market right or wrong? Well it’s hard to say - certainly the market said the event was more likely to occur than not, but only just, and who knows? Maybe the event _only just_ occurred, and very nearly didn’t!

For most events like this, you'd want to see the market spike to 0% or 100% as the deadline approached. And in particular for an event that happens, you want to see the spike to 100% before it happens. Remaining at 60% until after the fact is wrong because the occurrence of the event becomes more certain as it gets closer.

Being "well-calibrated" as you describe is a very bad quality metric in the sense that two sets of predictions can achieve the same calibration profile while differing markedly in quality. The farther the predictions are from 50%, the better they are, but your calibration metric doesn't take this into account.

[jjmarr]

Charlie Kirk has a 3% chance of winning a Nobel Peace Prize right now according to Polymarket. He's climbed from 1% since Maduro was arrested.

It seems unlikely since Nobels aren't awarded posthumously.

[pseudo0]

The issue there is time. The Nobel prizes will be announced in around 9 months. Buying a share of "No" would currently cost 98.2 cents, working out to a (basically) risk-free return of around 2.4%. Alternatively someone who wants a very low-risk investment product could just buy 1-year t-bills with a return of... ~3.5%. And that doesn't require messing around with buying crypto and the inherent risk of trusting Polymarket with your money.

Anything under 3%/year of time until decision is going to have pretty limited predictive value within that range. Anything starting above that range will end up hitting that floor rather than going to zero because of the difficulty of finding a counterparty.

[rich_sasha]

How much volume on this bet? Let's ignore black swan events and say it's a guaranteed 3% return. On how much? $1? $10? $1m?

I'd weigh the accuracy by how much money is at stake...

Even then, a "perfect" prediction market need not be accurate, if people use it for hedging. If some low probability event is really bad for me, I may pay over odds (pushing the implied probability up) to get paid if it happens. The equilibrium probability may be efficient, reasonable and biased.

[avadodin]

Nobel Peace Prizes are Peace Prizes as much as they are Nobel Prizes.

I'm not sure the same(any) rules apply.

[vintermann]

Well, Nobel peace prizes aren't usually awarded to people calling for invasions of their home country either, or cheering for the extrajudicial double-tap killing of smugglers/random fishermen.

Who's to say a dead person can't have done the most to "promote peace conferences" as mentioned in Nobel's will? These days, I'd say dead people make a larger net contribution to peace than most politicians.

[hobofan]

Normally they aren't, but maybe the US will take over Sweden and the Nobel Foundation and make it happen.

[pinkmuffinere]

To be fair you’re not really providing a hard stance against the estimate. You say it is unlikely, and indeed the prediction is a 3% chance. That’s unlikely.

[kurtis_reed]

No, markets are evaluated on accuracy, not calibration

[enjeyw]

Well markets are evaluated on a number of different metrics depending on what you’re trying to determine.

If you want to go be pedantic about it and select one metric, markets are evaluated on their Brier Score or some other Proper Scoring Rule, not accuracy.

However, I prefer calibration as a high level way to explain prediction market performance to people, as it’s more intuitive.

[stingraycharles]

Noob question from me: what’s the difference between accuracy and calibration? A well calibrated market would be more accurate and vice versa, not?

Edit: just found the answer myself: “accuracy measures the percentage of correct predictions out of total predictions, while calibration assesses whether a prediction market's assigned probabilities align with the actual observed frequency of those outcomes”