WingNews logo WingNews
top | new | best | ask | show | jobs
top | item 44896202

(no title)

Scrounger | 6 months ago

> My issue is that crawlers aren’t respecting robots.txt

Cloudflare has a toggle switch to automatically block LLM's + scrapers etc:

https://blog.cloudflare.com/declaring-your-aindependence-blo...

discuss

order

No comments yet.

powered by hn/api // news.ycombinator.com