[kenjackson]

Everyone is already at 80% for that one. Crazy that we were just at 50% with GPT-4o not that long ago.

[XCSme]

But 80% sounds far from good enough, that's 20% error rate, unusable in autonomous tasks. Why stop at 80%? If we aim for AGI, it should 100% any benchmark we give.

[Davidzheng]

I'm not sure the benchmark is high enough quality that >80% of problems are well-specified & have correct labels tbh. (But I guess this question has been studied for these benchmarks)

[kenjackson]

Are humans 100%?