They discuss that if you train models with the output of other models you can get worse results. If we all start using code from those models how will that impact GPT-5?
This paper is a joke in the ML community. Model collapse is not real. This paper made some seriously faulty assumptions in their contrived lab experiment. State of the art models are trained entirely or largely on "synthetic" generated datasets to surpass models trained on purely human data in quality and capability.
Even training on uncurated synthetic generated data, from web scrapes after the advent of genersgofe AI, actually leads to more capabilities and quality in models in practice (the exact opposite of what is prediction by the failed model collapse paper).
almatabata|2 years ago
https://browse.arxiv.org/pdf/2305.17493v2.pdf?
They discuss that if you train models with the output of other models you can get worse results. If we all start using code from those models how will that impact GPT-5?
MacsHeadroom|2 years ago
Even training on uncurated synthetic generated data, from web scrapes after the advent of genersgofe AI, actually leads to more capabilities and quality in models in practice (the exact opposite of what is prediction by the failed model collapse paper).