[fsh]

Companies are optimizing for all the big benchmarks. This is why there is so little correlation between benchmark performance and real world performance now.

[czhu12]

Isn’t there? I mean, Claude code has been my biggest usecase and it basically one shots everything now

[fsh]

Yes, LLMs have become extremely good at coding (not software engineer though). But try using them for anything original that cannot be adapted from GitHub and Stack Overflow. I haven't seen much improvement at all at such tasks.