[pocketheyman]

Kind of interesting, according to the case file, the PACER records were being pulled en masse during normal court hours (typically when courts are also accessing the PACER database). A user noticed that PACER was going slow and notified PACER of the apparent slowness. Looks like they investigated, shut the PACER system down and were able to detect the requests were coming from an Amazon Web Hosting account linked to Swartz.

I find this interesting because it wasn't some flag on the PACER system screaming "HEY SOMEONE IS DOWNLOADING THESE EVERY TWO SECONDS" but instead was noticed because some law clerk was irritated at how slow the server was at responding.

[meowface]

This is similar to how many breaches and DDoS attacks are discovered. Lots of companies have absolutely no controls to detect the most basic of flooding or spidering behavior.

[UnoriginalGuy]

First off: Totally true.

Secondly: Devil's advocate, but it is a "hard problem." It is easy to look for behaviour on the system, it is very hard to look for patterns of behaviour.

I mean let's say that some of your users are normal court clerks, it wouldn't be unusual to see them sit around and pull tons of records all day every day. So how do you pick up normal requests on-mass and unusual requests on-mass?

If I was in charge of protecting such a system I wouldn't even attempt to detect this (too hard). Instead what I would do is make it impossible to get records sequentially (e.g. 1, 2, 3...9999999) instead each record had a unique randomly generated token associated with it (a UUID/GUID).

So in order for someone to gain every single record they would either need to conduct a "real" break in and steal the files, or search for every possible criteria (which, for them, becomes a huge hassle/problem).

PS - Most DDoS are, these days, against layer 3 (network). Since it is far harder to defeat a layer 3 attack (as it can literally crash a lot of network hardware). While layer 7 (software) DDoS attacks still exist, they're often conducted by less formidable adversaries and they're much easier to stop (e.g. return a JavaScript redirect instead of the normal page, most browser-users won't notice, but it will defeat a targeted attack until they re-target (and you could rename it every 10 minutes)).

[Zigurd]

Do you find it credible that one request every 2-3 seconds could create a noticable load?

[readme]

He was only making 1 request per 3 seconds. Must have been a really slow system.

[Zigurd]

"A user" unnamed "noticed" and complained to another unnamed person, etc, etc.

Yeah. Sure.