[aljungberg]

THe RWKV model seems really cool. If you could get transformer-like performance with an RNN, the “hard coded” context length problem might go away. (That said, RNNs famously have infinite context in theory and very short context in reality.)

Is there a primer for what RWKV does differently? According to the Github page it seems the key is multiple channels of state with different decaying rates, giving I assume, a combination of short and long term memory. But isn’t that what LSTMs were supposed to do too?

[thegeomaster]

There's already research that tries to fix this problem with transformers in general, like Transformer-XL [1]. I'm a bit puzzled that I don't see much interest in getting a pre-trained model out that uses this architecture---it seems to give good results.

[1]: https://arxiv.org/abs/1901.02860

[gok]

T5 uses relative positional encoding

[solomatov]

My understanding is that RNNs aren't worse than Transformers per se, they are just slower to train, and use GPU much more efficiently, i.e. much more stuff could be run in parallel.

[Hendrikto]

Also slower to perform inference on. RNNs have to be much more sequential.

[euclaise]

We also don't have evidence that they scale the way transformers do

[eternalban]

They explains it here:

https://github.com/BlinkDL/RWKV-LM#the-rwkv-language-model-a...

[swyx]

> RNNs famously have infinite context in theory and very short context in reality.

any sources to read more about this please? its the first ive heard of it

[nl]

Read about "RNN Vanishing Gradients". LSTMs help here, but see eg https://medium.com/analytics-vidhya/why-are-lstms-struggling... for the problems there.

[georgehill]

I am not sure this article will answer your question, but Karpathy has an article about RNNs.

https://karpathy.github.io/2015/05/21/rnn-effectiveness

[solomatov]

Naive RNN have vanishing gradient, but LSTMs and GRUs are much better in this respect.