notahuman's comments

Recently, I conducted a few fine-tuning experiments on a multitask instruction dataset on 1b7 LLM, primarily related to open-ended long story generation. However, it was observed that after generating nearly 300 tokens, the quality of the generated text started to decline, becoming less fluent.

I am curious to know if there exists a relationship between the scale of the model and the quality of long text generation. For example, is it possible to achieve fluent long text after generating a certain number of tokens for models like 1b7?

Wow, honestly i want a similar life just like you.