Fine tune LLAMA3 on million scale dataset in consumer GPU using QLora, DeepSpeed

[unraveller]

This is a thorough "how to" but it is missing a "why for" about any of the chosen starting elements.

I don't understand why you would use an old dataset that worked for llama2 and just fine-tune llama3 on it. Isn't it most likely that the new model has covered off everything it missed last time around and now the last dataset is only valuable for the last gen.

[factorymoo]

This might be an unfair statement but it really feels like all of these blogs don't know why. They copy/paste each other (you often seem the same errors in multiple notebooks/blogs) and I have a feeling no one really deeply understands what they're doing.

[sa-code]

Thank you for saying this! The number of people that would need to fine tune vs just using RAG is really small. People that are not familiar with the source often jump to fine tuning as an option

[blackoil]

Dataset may not be public. All large companies have millions of internal documents. Internal LLM can be trained on them.

[imjonse]

The "why for" is usually learning/gaining experience/FOMO.

[iAkashPaul]

With unsloth's optimizations you can do llama-3-8b's QLoRA fine-tuning on your 8GB card(mine's a 2070S) with 900MB to spare with BS of 4.

[SunlitCat]

Since the crypto (currency) craze of 2017, every time I hear "consumer GPU" somewhere in a story that has nothing to do with gaming, it sends a chill down my spine.

[j0hnyl]

RIP your spine for the foreseeable future.