[SalmonSnarker]

Projects like this should respect that if a site's robots.txt contains a long list of Disallow entries for other AI scrapers that they are probably not welcome to scrape either.

(of course this project doesn't do that)

[sachou]

Good point! Thanks for the feedback.

Let me add that to my todos.

[keyle]

It boggles my mind that you would launch without that as a prime directive.