[cweld510]

Context windows are a natural improvement, but new architectures are completely speculative and it’s unclear we can make any sort of predictable progress with new, better architectures. Most progress has been made on essentially the same architecture paradigms, although we did move from dense models to MoE at some point.