[zeruh]

Wouldn't it result in overfitting?

[triyambakam]

The details on how exactly they may have used it to train their model is vague. I believe transfer learning or knowledge distillation are valid techniques based on the inference from other models.

[kevsim]

I would also think it'd be an incredibly expensive way to train a model.

[ReptileMan]

Depends. I wonder what is the minimum reasonable amount of different tokens needed to lift up the weights.

[Jensson]

You store the output from ChatGPT, you don't run it again every time you do a training step. Generating millions of examples to add to your own training wont cost much at all relatively.