As always it’s easy to overlook the insider threat. Grammatically dubious extortion emai? Bitcoin ransom? Did it not occur to you that the cat was the one behind the attack?
We don’t have very many earthquakes in Kansas… but I remember the first/only one I felt.
I was sound asleep when my Siamese woke me up by pawing my face… he then went and sat on the edge of the bed and growled aggressively (very out of character)… Not 30s later, things started shaking.
No idea how he knew, but it was pretty wild. He passed away in 2020, still miss him.
Dang, sorry for your loss. That’s a dope memory of the cat tho. As someone that happened find themselves in Taipei a couple weeks ago for the 7.4, my only thought was getting back to my dog, whom I promised I would get back to. She was sorta freaking out before I left; either could sense me leaving, or sense the earthquake I was heading to…
There are two types of sound in rock. P and S waves. P waves are pressure waves and go faster. S waves go side to side and are a bit slower. So you cat was woken by a hiss from the P waves, which arrive a bit before the earthquake that you can feel.
Maybe the phone was silent but still flashing a screen? Mine does that in that mode.
At my first job we had a guy who could spot incidents coming on the monitoring dashboard before they happened. He never managed to explain or even understand what he was looking for and no-one else picked it up, but he would just see something that made him say things were odd, and most of the time we'd get an alert shortly after.
Make or get a human to stare at streams long enough and they’ll attune to the patterns. We’re wired for patterns. It doesn’t even have to be conscious and explainable. The signals just suddenly aren’t right.
Maybe some signals just bypass the part of the brain which deals with well defined facts. I read somewhere about a construction foreman (HN comment maybe?) who gained the respect of the crew by having an unusually good hit rate in finding piping in the ground or walls. He started to believe in his superpowers but later came to the conclusion that he just subconsciously learned the typical patterns, plus an occasional non-obvious sign. Something like a vent pipe in the wall of the building telling you that sewer piping is probably below ground.
happened to me when playing cyberpunk 2077 on psf ataunch (shudder).
I got to a point where I could reliably tell "the game is about to crash, better save." I save, and 10 seconds after resuming the game, it crashes. I still don't know how I could tell.
The princely sum of $5,000. We got that at my employer back in 2016. We got hit by a ddos, and decided to ignore it, though we did dig up some BTC just in case. We enacted a bunch of DDoS protection as a result, costing way more than $5,000, but not paying money to extortionists is worth every penny.
About 20 years ago I was kinda accidentally the guy who dealt with the DDoS attacks in the sysadmin team. There was a sequence of extortion emails during about 2 week period:
1. $50k or we attack - didn’t register anything
2. $25k or else - a minor overload on the server but nothing serious.
3. $10k or else - a serious attack which affected the service in a major way.
4. $5k or we really pissed - this time they took down a whole Tier2 ISP and Datacenter in London for a day. Other carriers peering on London Internet Exchange had to blackhole traffic to our service provider and finally kept blackholing one of our IPs for a while. I had to scramble to find a DDoS mitigation service, new DC and servers.
We did not respond to any of the emails. The attackers were also quite dumb, they attacked the web servers which were located in a well connected place.
The money making service of the business was in the Caribbean with a 1,5Mbps T1 and a 0,5Mbps satellite backup. They could have saturated those much easier for much longer and the impact then would have been about $1M revenue loss per hour.
The problem with paying extortion or ransoms is that you incentivize the attacker to come back and do it again. It may have been $5k to pay off one attacker and more than that to build the defense, but now you have defenses and are less likely to suffer attackers in the future. And as you say, not paying money to criminals is inherently worthwhile.
Yep, I know it could only invite more trouble. It was fun to fantasize about though, especially after watching the video I linked to in the post: https://www.youtube.com/watch?v=dWzz3NeDz3E
Quite some time ago, someone from my family was alerted by their cat when the dishwasher was leaking. Their conclusion was that the cat was either trying to save them or the cat was trying to kill them.
One of my all-time favorite novels, Anansi Boys by Neil Gaiman, includes an anecdote: a crow's call wakes up someone who's sleeping outdoors, just as a large cat (a tiger, perhaps) is sneaking up on him.
One character suggests the crow was trying to warn the man. Another posits the bird was bringing the sleeper to the tiger's attention so it could enjoy the scraps after the meal.
On a more serious note, do you think there will ever be a way to stop ddos attacks once and for all?
While all threats are bad, ddos is the most lame type of attacks there is; no special skill or knowledge are needed, just load a script or, heck, pay someone who'll execute it for you as a service.
It's not as simple as "loading a script" - IP addresses (or in the case of IPv6, subnets) are (for the average person) a limited resource, as is bandwidth, and most amplification attacks require IP spoofing which is not possible from most connections.
If it's a volumetric attack, the side with more bandwidth wins (the attacker may be able to amplify here). If it's a load-based/application-level attack, blocking the attacker IPs at the firewall level solves it. This was application level, not (purely) volumetric, since they already had a WAF/Cloudfront.
Identifying attacker IPs to block is a matter of correctly attributing cost to a source IP, correctly attributing benefit (i.e. legit user activity) to a source IP, then blocking the IPs or ranges where the cost significantly exceeds the benefit you see from that IP or range.
That's easier said than done, since cost can come in many forms (e.g. open connections clogging up memory, TLS handshakes, requests that are expensive to parse for your web server, requests that trigger expensive database queries, in/out bandwidth, ...) which is why most just slap Cloudflare (or here, Cloudfront) in front of it and work around with manual rules like in this example.
There's application level DDoS, which you generally stop by not doing expensive work for clients that haven't done expensive work for you. Sometimes, easier said than done.
And then there's volumetric DDoS. You can stop this by having more bandwidth than everyone else... but that's pretty hard and it makes you a potential attacker.
Innovation here is in the form of using BGP to disseminate traffic filters. Null routing is the MVP here: this IP is being attacked, so drop traffic to it as soon as possible. But I've seen there's some systems with more precision, like drop udp, drop fragments, drop packets to/from udp/tcp port X.
Most of these systems are designed so that these specialized routes don't propagate beyond immediate peers, but potentially, it might be desirable if they did.
Maybe if the network was much more distributed and lower bandwidth?
If most of your customers are in Mexico, Canada is DDoSing you, and the pipes between you and Canada start filling up as a result that isn’t a big problem, right? As long as consumer routers on you/Mexico’s side of the Canadian clog don’t decide to help out.
Writing this post did make me think that if someone had a well-trained dog, they could hook up a monitoring service to something that makes a particular sound, which tells the dog to alert the person.
Her name was (I sadly lost her to cancer) Bamboo! Because one of the first things she did after I adopted her was to try to eat my bamboo plant.
Funny, that's how the very first customer realised that the Australian telco Optus was down. The wireless cat feeder relied on the Internet and when no food appeared, the cat decided to complain to management.
I think this would be like a firewall or ingress thing that would drop packets that resulted in excess load before they make it to the application server.
Tangential question that rose up regarding availability vs. quality of life.
For a small startup whose products are only available on the US, does it always make sense to do nightly oncall? This doesn't work for some products, but if, for example, you have a site that sells mattresses in the US, would you wake someone up to fix the site at 3AM?
I guess here the main $$ loss would come from accepting so much traffic. But I wonder if we can better differentiate what's worth waking up for.
> we didn’t have a formal on-call rotation yet. That was a deliberate decision, since being on-call is painful, and the team was good about just collectively keeping an eye out for urgent alerts.
That seems like a terrible solution. Yeah, being on-call is painful, but at least I know beforehand when I'll be on-call and get compensated for it. Always being expected to keep an eye out for urgent alerts just sucks all around.
I know it sounds bad, but in practice, it really did work fine for us for quite a while.
1. We didn't experience that many incidents that couldn't wait until working hours.
2. There was never an explicit expectation to keep an eye out. We did it anyway because we were at an early-stage startup, and we all deeply cared about making our products work for our customers.
2 European teams I worked on paid a bonus for on-call duty, and the systems were so stable that enough people volunteered for the few who didn't want it, weren't forced to.
It was pretty great, I took a week shift every month or so except when I was going on holiday, and aside from lugging a backpack with my laptop everywhere, didn't affect my life at all except 1 or 2 minor issues
You usually get some extra money for the duty. And if you get woken up, the hours you spend are counted towards your normal working hours - so you aren't expected to show up in the morning after putting out a fire. Or you get some more bonus (like 2x hourly pay for the night work). That's about the balance when people are ok doing it.
But it depends on stability of your service. If it is messed up and people are woken up often, then you won't find many volunteers if they have other choice.
Cats love to hang out in warm areas, even sunbathe, so I doubt they'd do anything to get attention in that situation! Their body temperature is a few degrees warmer than that of humans.
We once detected a DDOS because all our office phones went down. Silly attackers didn’t realise that our (money-making) APIs weren’t colocated with our public website and phone system.
Kinda reads like one, but if he was on GCP and used their ddos shield then it'd read like an ad for their service instead. Would be better if he'd been a bit more abstract and said cloud provider instead of naming AWS.
It's translated by duckduckgo.com's chatgpt interface. don't down vote plz :D
I suspect that I am somewhat sensitive to electromagnetic fields and magnetic fields. There have been times when I have not felt well the next day after sleeping on an electric heating pad, and I have experienced severe discomfort after sleeping on a mattress with magnets.
When I used a CRT monitor, I often had diarrhea if I spent a long time in front of the monitor.
Since using LCD monitors or laptops, those symptoms have disappeared.
When I sleep, there is a wireless router on the right side of my head, and I play youtube videos on my smartphone on the left side. I have strange dreams and wake up early from sleep. However, if I put the smartphone on the right side of my head while sleeping, those symptoms are lessened.
Thus,
Even though there was no sound, wouldn't your cat have sensed that as well?
jameshart|1 year ago
shawn_w|1 year ago
Mawr|1 year ago
nusl|1 year ago
ordu|1 year ago
exabrial|1 year ago
I was sound asleep when my Siamese woke me up by pawing my face… he then went and sat on the edge of the bed and growled aggressively (very out of character)… Not 30s later, things started shaking.
No idea how he knew, but it was pretty wild. He passed away in 2020, still miss him.
jmprspret|1 year ago
In the recent NYC ones there are videos of dogs howling before any of the tremors are noticeable by people. This is a common phenomena I believe.
nyjah|1 year ago
btilly|1 year ago
There are two types of sound in rock. P and S waves. P waves are pressure waves and go faster. S waves go side to side and are a bit slower. So you cat was woken by a hiss from the P waves, which arrive a bit before the earthquake that you can feel.
See https://manoa.hawaii.edu/exploringourfluidearth/physical/oce... to verify that there are two types of waves, and the P waves arrive first.
johnnyAghands|1 year ago
lmm|1 year ago
At my first job we had a guy who could spot incidents coming on the monitoring dashboard before they happened. He never managed to explain or even understand what he was looking for and no-one else picked it up, but he would just see something that made him say things were odd, and most of the time we'd get an alert shortly after.
Waterluvian|1 year ago
seanthemon|1 year ago
praptak|1 year ago
vrighter|1 year ago
I got to a point where I could reliably tell "the game is about to crash, better save." I save, and 10 seconds after resuming the game, it crashes. I still don't know how I could tell.
leo150|1 year ago
fragmede|1 year ago
jnsaff2|1 year ago
1. $50k or we attack - didn’t register anything
2. $25k or else - a minor overload on the server but nothing serious.
3. $10k or else - a serious attack which affected the service in a major way.
4. $5k or we really pissed - this time they took down a whole Tier2 ISP and Datacenter in London for a day. Other carriers peering on London Internet Exchange had to blackhole traffic to our service provider and finally kept blackholing one of our IPs for a while. I had to scramble to find a DDoS mitigation service, new DC and servers.
We did not respond to any of the emails. The attackers were also quite dumb, they attacked the web servers which were located in a well connected place.
The money making service of the business was in the Caribbean with a 1,5Mbps T1 and a 0,5Mbps satellite backup. They could have saturated those much easier for much longer and the impact then would have been about $1M revenue loss per hour.
vsnf|1 year ago
ed_mercer|1 year ago
Not replying is the only valid answer. Trolling them could potentially put you more on their radar and get targeted for other attacks. And for what?
dguo|1 year ago
benraz123|1 year ago
EveryPizza|1 year ago
macintux|1 year ago
One character suggests the crow was trying to warn the man. Another posits the bird was bringing the sleeper to the tiger's attention so it could enjoy the scraps after the meal.
readyplayernull|1 year ago
An inverted Schrödinger cat.
ahmedfromtunis|1 year ago
Ah, the days before ChatGPT!
On a more serious note, do you think there will ever be a way to stop ddos attacks once and for all?
While all threats are bad, ddos is the most lame type of attacks there is; no special skill or knowledge are needed, just load a script or, heck, pay someone who'll execute it for you as a service.
tgsovlerkhgsel|1 year ago
If it's a volumetric attack, the side with more bandwidth wins (the attacker may be able to amplify here). If it's a load-based/application-level attack, blocking the attacker IPs at the firewall level solves it. This was application level, not (purely) volumetric, since they already had a WAF/Cloudfront.
Identifying attacker IPs to block is a matter of correctly attributing cost to a source IP, correctly attributing benefit (i.e. legit user activity) to a source IP, then blocking the IPs or ranges where the cost significantly exceeds the benefit you see from that IP or range.
That's easier said than done, since cost can come in many forms (e.g. open connections clogging up memory, TLS handshakes, requests that are expensive to parse for your web server, requests that trigger expensive database queries, in/out bandwidth, ...) which is why most just slap Cloudflare (or here, Cloudfront) in front of it and work around with manual rules like in this example.
flafla2|1 year ago
It would be pretty cool if there was a way to DDOS-harden at the protocol layer. Not sure if that’s even possible though
toast0|1 year ago
And then there's volumetric DDoS. You can stop this by having more bandwidth than everyone else... but that's pretty hard and it makes you a potential attacker.
Innovation here is in the form of using BGP to disseminate traffic filters. Null routing is the MVP here: this IP is being attacked, so drop traffic to it as soon as possible. But I've seen there's some systems with more precision, like drop udp, drop fragments, drop packets to/from udp/tcp port X.
Most of these systems are designed so that these specialized routes don't propagate beyond immediate peers, but potentially, it might be desirable if they did.
squarefoot|1 year ago
> Ah, the days before ChatGPT!
The topic made me read that as CatGPT, and now I can't pull it out of my head.
bee_rider|1 year ago
If most of your customers are in Mexico, Canada is DDoSing you, and the pipes between you and Canada start filling up as a result that isn’t a big problem, right? As long as consumer routers on you/Mexico’s side of the Canadian clog don’t decide to help out.
swampthinker|1 year ago
Regardless, very cute - what’s your cat’s name?
dguo|1 year ago
Her name was (I sadly lost her to cancer) Bamboo! Because one of the first things she did after I adopted her was to try to eat my bamboo plant.
chris_wot|1 year ago
867-5309|1 year ago
retreatguru|1 year ago
jart|1 year ago
avg_dev|1 year ago
I think this would be like a firewall or ingress thing that would drop packets that resulted in excess load before they make it to the application server.
hnlmorg|1 year ago
You could still overload the service with a sufficiently large attack in either volume of connection requests or number of unique IP addresses.
Token buckets are usually part of an overall resilience strategy rather than a silver bullet to solve all denial of service concerns.
nullderef|1 year ago
For a small startup whose products are only available on the US, does it always make sense to do nightly oncall? This doesn't work for some products, but if, for example, you have a site that sells mattresses in the US, would you wake someone up to fix the site at 3AM?
I guess here the main $$ loss would come from accepting so much traffic. But I wonder if we can better differentiate what's worth waking up for.
euroderf|1 year ago
Denvercoder9|1 year ago
That seems like a terrible solution. Yeah, being on-call is painful, but at least I know beforehand when I'll be on-call and get compensated for it. Always being expected to keep an eye out for urgent alerts just sucks all around.
dguo|1 year ago
1. We didn't experience that many incidents that couldn't wait until working hours.
2. There was never an explicit expectation to keep an eye out. We did it anyway because we were at an early-stage startup, and we all deeply cared about making our products work for our customers.
bongodongobob|1 year ago
cocoa19|1 year ago
willsmith72|1 year ago
It was pretty great, I took a week shift every month or so except when I was going on holiday, and aside from lugging a backpack with my laptop everywhere, didn't affect my life at all except 1 or 2 minor issues
tossandthrow|1 year ago
Also, if you can get an equivalent role with less requirements such as being on call, then I guess it is just a question of grabbing it!
krab|1 year ago
But it depends on stability of your service. If it is messed up and people are woken up often, then you won't find many volunteers if they have other choice.
matricaria|1 year ago
mmahemoff|1 year ago
Anyway, better experience than being woken up by a dozen SMS alerts.
hunter2_|1 year ago
Operyl|1 year ago
com|1 year ago
ro_bit|1 year ago
peterburkimsher|1 year ago
avg_dev|1 year ago
... that you know of
csin|1 year ago
johnnyAghands|1 year ago
AtlasBarfed|1 year ago
fragmede|1 year ago
xeromal|1 year ago
Talks about useful tech
"iS tHiS An aStroTurFing Ad"
fuzztester|1 year ago
cattackstrophic!
dontdieych|1 year ago
I suspect that I am somewhat sensitive to electromagnetic fields and magnetic fields. There have been times when I have not felt well the next day after sleeping on an electric heating pad, and I have experienced severe discomfort after sleeping on a mattress with magnets.
When I used a CRT monitor, I often had diarrhea if I spent a long time in front of the monitor.
Since using LCD monitors or laptops, those symptoms have disappeared.
When I sleep, there is a wireless router on the right side of my head, and I play youtube videos on my smartphone on the left side. I have strange dreams and wake up early from sleep. However, if I put the smartphone on the right side of my head while sleeping, those symptoms are lessened.
Thus,
Even though there was no sound, wouldn't your cat have sensed that as well?