[fhood]

I am in way over my head here, so I wasn't able to tell if the authors addressed this, but my intuition is that this should be somewhat mitigated so long as people are providing the filter between which results are discarded and which might end up back in the training pool.

I would think that the human selection process would help to head off this conversion, both by selecting against incorrect results, and also by introducing variance outside of the model. On the other hand since a person can only act as a filter, I can also see how that would be of limited value long term.

[gwern]

They don't address that. They just assume random sampling, so there's no equivalent to human curation or quality metrics, which would preserve tails or, by manual use, create tails. The contraction they observe is pretty much what you would expect in the random sampling setting, since you can only lose tails with a finite sample, and never gain them. (They also need to have a very large ratio of synthetic to real/original data.)

So, while interesting for nailing down that phenomenon, the broader implications everyone wants to draw from it are not very good - very few people are using random GPT-3/4 or Stable Diffusion samples!

[rossdavidh]

We have already heard reports of companies that get paid for human tagging, and similar services, using LLM's to automate their processes.