I was there. They had no idea. The purpose and expectation of the transformer architecture was to enable better translation at scale and without recurrence. This in itself would have been a big deal (and they probably expected some citations), but the actual impact of the architecture was a few orders of magnitude greater.
No comments yet.