[gwittel]

I work a product that involves a security crawler (phish, malware detection, etc). It’s just a new arms race. Crawlers will adapt.

Cloudflare is already heavily abused by threat actors to host, and gate their malicious content. This means our crawler has to handle anti-bot and CAPTCHAs. It’s a pain. Cloudflare is no help.

They have a “verified bot” program but it’s a joke for security. You must register a unique, identifiable user agent, and come from a set of self declared IPs. Cloudflare users can check a box to filter these bots out. And now you're easily fingerprintable so the bad guys can just filter you even without Cloudflare’s help.

So now we have a choice. Operate above board and miss security threats. Or operate outside the rules (as opaquely defined by Cloudflare), and do right by our customers.

All of this on CFs side is to solve a real problem. Unfortunately by not working with the industry in a productive manner, Cloudflare is just creating new problems for everyone else.

[disqard]

"verified bot" really sounds like an oxymoron :)