[downWidOutaFite]

There was a step change over the last few years but that rate of improvement is not continuing. The currently known techniques seem to have hit a plateau. It's impossible to know when the next "Attention is All You Need" will materialize. It could be in 2025 or 2035.

[FeepingCreature]

o1 sort of techniques (tree search, test-time training type things) have not hit any recognizable plateau. There's still low hanging fruits all around.

[blharr]

This discussion includes o1. It's marginal improvement compared to, say, GPT-3 to GPT-4.

[sdenton4]

Or 2023... There's a lot of papers out there!