[two_in_one]

As it's still work in progress may I suggest? It would be nice if you go beyond what others have already published and add more details. Like different position encodings, MoE, decoding methods, tokenization. As it's educational easy to use should be a priority, of course.

[rasbt]

Thanks, comparing positional encodings, MoEs, kv-caches etc are all good topics that I have in mind for either supplementary material and/or a follow-up book. The reason why it probably won't land in this current book is the length and time line. It's already going to be a big book as it is (400-500 pages). And I also want to be a bit mindful of the planned release date. However, these are indeed good suggestions.