[Kuxe]

A discovery by a giant is in some sense a new base vector in the space of discoveries. The interesting question is if a statistical machine can only perform a linear combination in the space of discoveries, or if a statistical machine can discover a new base vector in the space of discoveries.. whatever that is.

[D-Machine]

For sure we know modern LLMs and AIs are not constrained by anything particularly close to simple linear combinations, by virtue of their depth and non-linear activation functions.

But yes, it is not yet clear to what degree there can be (non-linear) extrapolation in the learned semantic spaces here.