>'They' spin up spot instances which isn't subject to Billing Alerts. You'll need to cancel those spot instances, revoke your AWS credentials, and change your account password," he said.
This doesn't make sense at all. Amazon should let us if monthly bill > X send me a priority email and phone call. Why do they hide behind these dark patterns? I thought they were better than that.
The fact of the matter is amazon is a giant company and cannot thoroughly think through each piece of logic in their system.
The reward for focusing on this before-hand is much lower than just writing a check for $5k to this person and then fixing later (lot of $5k checks from amazon today. Wheres mine?)
Why the cliffhanger? Why not just finish your thought? As is, this story provides nothing other than showing the Amazon provided some helpful customer service.
We had an almost identical event last Sunday in one of our dev accounts: multiple high end spot instances in multiple regions with a newly created security group pointing to a suspect IP.
We caught and corrected it quickly, but we still don't know how the keys leaked out - we have chalked it up to lower security practices since it's not a production account and is shared by more people (e.g. no 2-factor on it). We started to investigate, but then Heartbleed happened.
I wish there were more mechanism in AWS to prevent bills from mounting up, but the basic billing alarms worked in this case. I can't imagine how or why spot instances would be excluded from alerts, their cost certainly is included in the estimates that alerts are based on.
Another reminder to be extra careful about checking in AWS credentials into version control. You never know when you might open source that repository and someone can easily yank it from Github.
Actually this is what happened to us... our AWS credentials had accidentally been put into Github for a hackathon project several months ago.
Coincidentally, the incident also occurred around the same time (April 1-2). We were hit with $13,000 worth of EC2 usage before we shut them down and changed our AWS key... We reported to Amazon, and they are working on a refund.
The instance was spun up on April 2, but Heartbleed wasn't disclosed for almost a week later. I highly doubt anybody used the Heartbleed 0-day to access your account.
A similar thing happened to me when I made public a previously private repo on Amazon and forgot to scrub it for AWS key. Luckily I got an email from Amazon within 24 hours of the instances being started, and as such my bill was just $360, which in any case they waived off.
You might want to check your Github repos.
The fact that the consultant - unprompted - knows exactly what is going on suggests to me that Amazon needs to fix this. If one customer gets burned then the customer is an idiot. If lots of customers get burned then maybe its not the customers that are the problem.
Is it too pedantic to want the tech support to just say they were probably mining cryptocurrency instead of bitcoin? It's most likely the intruder was mining either litecoin or whatever coin is most profitable for the month. I know it's all very much the same but they were almost certainly not mining bitcoin.
A $5,000 AWS instance would mine about $1 worth of bitcoin and would not be worth the time logging into someones account.
[+] [-] sergiotapia|12 years ago|reply
This doesn't make sense at all. Amazon should let us if monthly bill > X send me a priority email and phone call. Why do they hide behind these dark patterns? I thought they were better than that.
[+] [-] unknown|12 years ago|reply
[deleted]
[+] [-] theboss|12 years ago|reply
The reward for focusing on this before-hand is much lower than just writing a check for $5k to this person and then fixing later (lot of $5k checks from amazon today. Wheres mine?)
[+] [-] bequanna|12 years ago|reply
Spot Instances aren't subject to AWS Billing Alerts? Is this common knowledge?
[+] [-] ceejayoz|12 years ago|reply
[+] [-] matthuggins|12 years ago|reply
[+] [-] Jare|12 years ago|reply
We caught and corrected it quickly, but we still don't know how the keys leaked out - we have chalked it up to lower security practices since it's not a production account and is shared by more people (e.g. no 2-factor on it). We started to investigate, but then Heartbleed happened.
I wish there were more mechanism in AWS to prevent bills from mounting up, but the basic billing alarms worked in this case. I can't imagine how or why spot instances would be excluded from alerts, their cost certainly is included in the estimates that alerts are based on.
[+] [-] colinbartlett|12 years ago|reply
[+] [-] friendstock|12 years ago|reply
Coincidentally, the incident also occurred around the same time (April 1-2). We were hit with $13,000 worth of EC2 usage before we shut them down and changed our AWS key... We reported to Amazon, and they are working on a refund.
[+] [-] nilved|12 years ago|reply
[+] [-] viseztrance|12 years ago|reply
[+] [-] abhimir|12 years ago|reply
[+] [-] jes5199|12 years ago|reply
[+] [-] prattbhatt|12 years ago|reply
[+] [-] Havoc|12 years ago|reply
[+] [-] sandy23|12 years ago|reply
[+] [-] mek|12 years ago|reply
[deleted]
[+] [-] josephagoss|12 years ago|reply
A $5,000 AWS instance would mine about $1 worth of bitcoin and would not be worth the time logging into someones account.