[fredliu]

I might be wrong, but looks like this could help with speculative decoding which can already vastly improves the inference speed?