[bioemerl]

I'm spoiled by 4 bit and unfortunately it doesn't appear to be supposed here so this isn't of much use to me, but it's awesome to see people working on the inference speed side of things regardless.

[george_123]

this approach to managing KV cache can work with 4bit. imagine the speedup of pagedattention with quantization..

[zhisbug]

yep, it is agonistic to 4-bit. You can deploy a 4-bit model and still use vllm + pagedattention to double or even triple your serving throughput.