The Structure of Neural Embeddings

[jmward01]

Current embeddings are badly trained and are massively holding back networks. A core issue is something I call 'token drag'. Low frequency tokens, when they finally come up, drag the model back towards an earlier state causing a lot of lost training. This leads to the first few layers of a model effectively being dedicated to just being a buffer to the bad embeddings feeding the model. Luckily fixing this is actually really easy. Creating a sacrificial two layer network to predict embeddings in training (and then just calculating the embeddings once for prod inference) gives a massive boost to training. To see this in action check out the unified embeddings in this project: https://github.com/jmward01/lmplay

[tomrod]

Do you have a peer reviewed source you could link on this approach, or is it something you thought of and are experimenting with yourself? I couldn't tell from the LMPlay repo in my skim, and the idea is intriguing

[tomrod]

Oh wow, great set of reads. Thanks to @sean_pedersen for posting, looking forward to reviewing this in my closeout this year.