[stefanka]

Where does one find a good robots.txt? Are there any well maintained out there?

[dmit]

https://github.com/ai-robots-txt/ai.robots.txt

[skrtskrt]

Cloudflare actually has this as a free tier feature so even if you don't want to use it for your site you can just setup a throwaway domain on Cloudflare and periodically copy the robots.txt they generate from your scraper allow/block preferences, since they'll be keeping up to date with all the latest.