[cbo100]

I get the right answer on the 8B model too.

It could be the quantized version failing?

[ein0p]

My models are both 4 bit. But yeah, that could be - small models are much worse at tolerating quantization. That's why people use LoRA to recover the accuracy somewhat even if they don't need domain adaptation.