[sdrg822]

Cool. Important note:

""" One may ask whether the conditionality introduced by the use of CMM does not make FFFs incompatible with the processes and hardware already in place for dense matrix multiplication and deep learning more broadly. In short, the answer is “No, it does not, save for some increased caching complexity." """

It's hard to beat the hardware lottery!

[algo_trader]

Infact, as stated in the paper, this is bad news

> We therefore leave the attention layers untouched

Meaning, presumably, that the GPU memory remains the bottleneck

Flops really are quite cheap by now, e.g. vision inference chip ~$2/teraflop/s !!

[marcinzm]

Bottleneck for larger models however this would presumably allow for cheaper models at scale or on compute constrained devices (like phones).

[ashirviskas]

>Flops really are quite cheap by now, e.g. vision inference chip ~$2/teraflop/s !!

I'm really interested, can you share where you got these numbers?

[theGnuMe]

There's another paper replacing attention with FF networks so just combine the two and you've got something.

[YetAnotherNick]

> ~$2/teraflop/s

H100 is basically ~$2/(2000 tflops/s)/hour or $1 for 4*10^18 floating point operations.