[r1ch]

Residential proxy botnets have exploded since LLMs became a thing. The amount of DDoS-level scraping we receive from residential IPs has exploded over the last year, one of our sites that typically sees around 10k unique IPs per day jumped to over 2M before we were able to deploy appropriate mitigations. We originally started blocking the IPs, but then we ended up blocking legitimate users as they seem to specifically use ISPs that have very dynamic IPs (i.e. the customer's IP will change even if their router stays on 24/7).

[ATechGuy]

Mind sharing what kind of mitigations you put in place and how well they worked?

[r1ch]

At first they were easily detectable using HTTP header analysis - e.g. pretending to be Chrome but not sending the headers that Chrome always sends. Now it's a combination of TLS / HTTP protocol level analysis and application layer - e.g. we send a cookie on the user's "normal" page view and check it exists on the higher-resource usage pages they might later visit - the bots don't care about normal viewing patterns and try to hit the higher-resource pages on their first visit, so they get blocked.

[Citizen8396]

Can you give some examples of these ISPs?