[andreybaskov]

I agree scaling alone is not enough, and transformers itself is a proof of that - it was an iteration on the attention mechanism and a few other changes.

But no matter what the next big thing is, I'm sure it would immediately fill all available compute to maximize its potential. It's not like intelligence has a ceiling beyond which you don't need more intelligence.