[fishpham]

Yes, but benchmarks like this are often flawed because leading model labs frequently participate in 'benchmarkmaxxing' - ie improvements on ARC-AGI2 don't necessarily indicate similar improvements in other areas (though it does seem like this is a step function increase in intelligence for the Gemini line of models)

[jstummbillig]

Could it also be that the models are just a lot better than a year ago?

[bigbadfeline]

> Could it also be that the models are just a lot better than a year ago?

No, the proof is in the pudding.

After AI we're having higher prices, higher deficits and lower standard of living. Electricity, computers and everything else costs more. "Doing better" can only be justified by that real benchmark.

If Gemini 3 DT was better we would have falling prices of electricity and everything else at least until they get to pre-2019 levels.

[layer8]

Isn’t the point of ARC that you can’t train against it? Or doesn’t it achieve that goal anymore somehow?

[egeozcan]

How can you make sure of that? AFAIK, these SOTA models run exclusively on their developers hardware. So any test, any benchmark, anything you do, does leak per definition. Considering the nature of us humans and the typical prisoners dilemma, I don't see how they wouldn't focus on improving benchmarks even when it gets a bit... shady?

I tell this as a person who really enjoys AI by the way.

[theywillnvrknw]

* that you weren't supposed to be able to

[XenophileJKO]

https://chatgpt.com/s/m_698e2077cfcc81919ffbbc3d7cccd7b3

[aleph_minus_one]

I don't understand what you want to tell us with this image.

[olalonde]

Would be cool to have a benchmark with actually unsolved math and science questions, although I suspect models are still quite a long way from that level.

[gowld]

Does folding a protein count? How about increasing performance at Go?