[mufasachan]

Disclaimer: This is just my intuition, I do not have knowledge about LoRA on small models.

It's possible that does not work. LoRA (for Low Rank) benefits from the "small changes" introduced during finetuning of a model. The update of the weights has a low rank. If you take a smaller model, it might induce that the rank is not so low, resulting in degradation in metrics by LoRA compression. I would be interested to see if LoRA still has a benefit in this configuration.

[dragonwriter]

> LoRA (for Low Rank)

Pedantic, but it is actually for Low Rank Adaptation.