[scottmas]

Not really IMO. At scale the model learns that the rainbow emoji is literally a synonym for “rainbow”. Computers are great at memorizing giant lists of facts such as this. It feels way more amazing for an LLM to guess the Carebear Movie from the rainbow emoji and bear emoji but it’s literally the same as asking the LLM to guess a movie that involves rainbows and bears with words.

It’s just an embedding space buried within the bowls of the LLM.