[larntz]

Does that entire model fit in gpu memory? How's it run?

I tried running a model larger than ram size and it loads some layers into the gpu but offloads to the cpu also. It's faster than cpu alone for me, but not by a lot.

[shosca]

you're right, actually noticed gpu clocking up and down with 32b, 14b clocks up fully and actually runs faster