[junrushao1994]

yeah we tried out popular solutions like exllama and llama.cpp among others that support inference of 4bit quantized models