[wtf242]

The amount of AI bots scraping/indexing content is just mind boggling. for my books site https://thegreatestbooks.org, without blocking any bots, I was probably getting 500,000~ requests a day from ONLY ai bots. Claudebot, amazon ai bot, bing ai bot, bytespider, openai. Endless ai bots just non-stop indexing/scraping my data.

Before i moved my dns to cloudflare and got on their pro plan, which offers robust bot blocking, they were severely hurting my performance to the point that I bought a new server to offload the traffic.