Ask HN: My VPS got hacked and now I'm facing a massive bill. What can I do?

[Koekoeksklok]

I've got a VPS which I use for small programming projects and college assignments. Two weeks ago I received an e-mail from my provider, stating that "your VPS has been transmitting a lot of outgoing traffic which results in a very large traffic usage bill". In September on my 500 GB data-limit VPS, it had been transmitting 27 TB of data traffic. This resulted in a € 3300 extra charge on my € 15 VPS. I'm expecting a similar bill for this month.

Of course I immediately shut down my VPS after the notice two weeks ago, but by then it had been using these amounts of traffic for a month and a half.

What are my options here? I can't afford to pay > € 5000 unfortunately. Does anyone have similar experiences?

[patio11]

Incidentally, since many HNers probably come at this from a mental model of "Anything which appears on an invoice is non-negotiable and simply must be paid": a B2B service provider which collects payment after services are rendered is knowingly taking on credit risk and has already priced non-collectability of some accounts into their services. You may be overestimating how much drama is required for someone at their company to say "Wow, really? OK, sorry about that. I'll write it off."

This is one of many, many, many reasons why we don't generally do cost-based pricing and, when we do do cost-based pricing, the markup is absolutely phenomenal. It has to include risk premiums. As long as it do include risk premiums, you don't have to sweat the small stuff like e.g. an uncollectable $4k invoice. (n.b. Small stuff! $4k hiccups are utterly routine events and largely dealt with by processes rather than by treating them as sudden emergencies, even if they feel like that to natural humans.)

[readme]

Also OP, if they don't write it off make sure to tell us all who it was so we can avoid them.

The same thing happened to me with amazon. Amazon pid for it. It's highly unreasonable in my opinion to ask the customer of a VPS to pay for damages caused by a malicious attacker. It's tantamount to a landlord expecting you to pay after an arsonist comes along and burns down your apartment, just because you happened to be renting it at the time.

[13]

I have had this experience. In my case the bill was altered to $0 before I'd even got to the point in the conversation where I intended to ask for some relief. It most certainly doesn't cost them that much, and it's within their interests to keep customers coming back for repeat payments than soured by being forced to pay an exorbitant fee for something which was obviously not their fault.

[switch007]

I worked for a hosting company and we'd normally charge time spent handling the issue plus the upstream cost of bandwidth, depending on how nice the customer acted.

[Blahah]

1. Report the incident to the police. Right now.

2. Report it to the VPS provider. Explain that you've reported it to the police. Ask for their cooperation in investigating the problem.

You do not have to pay. If they try to force you to pay, depending on your country, you'll probably end up in small claims court where you'll find judges are very reasonable people who usually side with the little guy. (IANAL)

[monstermonster]

Yes. This happened to a company I worked for in the UK in the last 1990s. We had a half rack full of NT4 machines and someone got into our kit and used it to run a pr0n FTP. They took us to court to pay up and the magistrate said we had no bill to pay and that it was a waste of court time as it wasn't intentional.

You're right about reporting a crime though even if the police don't take it seriously. A crime ref number goes a long way on its own.

Edit: we had to move our kit sharpish though as the company exercised their right to throw it on the street within 24 hours.

[burtonator]

MAKE SURE you file a police report , and are diligent in recording and logging everything and have a long paper trail.

Including ALL customer support interaction , police logs, diary , journal, etc.

Judges will show favor if you have a paper trail.

[donniezazen]

I hope cyber crime police has improved their working in recent years. 5-6 years ago a Ebay seller defrauded me. I filed a complaint with all the information I had since everything is online and involved bank transactions there had to be a money trail. I never heard back from cyber crime cell. I don't know how many folks were defrauded by that person before and after the incident with me.

[minopret]

I wonder what experiences we have with police in similar situations in recent years. In the USA many years ago I brought a telephone bill dispute to police. They explained, not without kindness and patience, that the legal question involved intent. Consequently it wasn't a matter where they could easily provide assistance.

[BukhariH]

I've been in the exact same situation with AWS ( http://cl.ly/SHOu ).

It was a nerve recking couple of days but I contacted AWS support and they were extremely good. They helped me secure my machine and then cancelled the 1.4K payment they were going to take from my account.

In all the whole process took 2.5 weeks and I only had to pay $15 for the I/O requests.

The best thing I can recommend is to talk to your host and tell them honestly you can't pay that much and you weren't the cause of the charges either.

[UnoriginalGuy]

AWS also has billing alarms. If you're running an account which should cost $15/month then for the love of god set an alarms at e.g. $20/month.

I had a reserved instance for 12 months, forgot to renew it, and on the 13th month (when it was on-demand) the usage creeped over my cap and I started getting alarms allowing me to kill the instance, renew my reserved, and restart it. Saved me at least $10.

On a related topic, I wish VPS providers allowed you to pre-pay. With Microsoft's Azure I have an MSDN Ultimate account, which has $150 of pre-paid credit on Azure. When you go over the $150 they just shut your stuff down rather than charging you (in fact I don't have a CC on there at all). They don't even offer this kind of service to non-MSDN subscriptions which sucks, I'd love to just pre-pay $50/month to them and have everything shut off when I exceed it (so it becomes a "no risk" playground).

[e40]

I've seen AWS credit charges that people intended to make but didn't want. We have some public AMIs that include charges for our product. Twice AWS support contact us to ask if we would credit someone back on the order of $500 out of our pocket because their customer didn't realize running an AMI would incur charges. Yeah, I was pretty surprised by this, too. It seems AWS gives people one freebie, though.

[rmc]

In aws, you can set up billing alerts, so they will email you if you go over X per month. It's a good idea to set that up, so at least you'll be alerted as soon as possible if you get hacked.

[malditojavi]

Same here, not as big as 4k, but only $200 more. AWS took care in an awesome way.

[thewhk]

I work for a VPS provider in the US. These situations are common and we usually just issue a credit and give a reminder to the customer to please secure their server

That brings me to my point. How did the hack occur? When you get a VPS you are fully responsible for what goes on in there. It is your responsibility to secure it and keep it updated. It's not the provider's fault you did not apply the latest security updates. It's not the provider's fault your Java application was using outdated and vulnerable libraries nor is it their fault you didn't set a CAPTCHA in front of your submission forms. Either hire a competent sysadmin if you can't take care of that yourself or find a provider that offeres managed hosting instead of a VPS, as that's what you'd most likely need.

There are some cases where it's the provider's fault such as the Linode BitCoin hack a few years back but mostly it's just poor server maintenance

[waxjar]

People that rent a $15/year VPS use it to run an IRC bouncer or a small web log, something you don't need to know a whole lot of sysadmin stuff for. They just need a machine that's always on.

It's hardly worth hiring a sysadmin for (I find that suggestion laughable, to be frank). Managed hosting doesn't allow you to do much else besides hosting a website in PHP, which is not enough for plenty of use-cases, including OPs.

[theonemind]

I work for a company that provides VPSes. In a situation like this, they can see the usage is aberrant and they can see it's not normal based on past bills. They'd likely offer a large credit if you say you didn't intend to do this, and it doesn't look like a fraudulent account. That being said, they themselves probably have bandwidth costs, and are not at all likely to forget all of the charge, perhaps half at best.

[onestone]

Stop using providers which charge a ridiculous price for bandwidth (like AWS). There are many excellent alternatives where a TB costs only a few dollars/euros.

[njsubedi]

like?

[patio11]

I would begin by contacting your VPS provider, explaining the circumstances which caused the bill, and asking "What are our options?"

[ColinCera]

Have you talked to your VPS provider? They should be able to cut you a break; after all, that 40TB of traffic cost them only a small fraction of what they're charging you, so if they're reasonable you should at least be able to get them to reduce the charges to their actual cost.

You might also offer to suggest writing up a post mortem for them, that they can provide to their customers as a lesson/tutorial on how to protect a VPS.

Finally, you can suggest that they might want to implement (and perhaps help them implement it) some kind of warning system, i.e., if a VPS suddenly begins using exorbitant amounts of bandwidth, and far more bandwidth than it ever has before, they really should email/text the owner an alert within 24 hours — not let it go on for 6 weeks. I'm surprised that they don't cap/throttle the bandwidth once you go over your plan's limit, to go along with sending you alerts. It borders on negligence on their part that they don't already have such a system in place.

[MangoDiesel]

In my opinion, it is negligence to an extent that OP should not have to pay for this, and he should find a new VPS provider.

[Jare]

Depends on your provider. Amazon AWS is known to have waived such bills in the past, see for example http://readwrite.com/2014/04/15/amazon-web-services-hack-bit...

[zhovner]

To prevent such incidents Linode have alerts of traffic/cpu/disk thresholds. For example you can configure notification if your bandwidth utilization more than N Mbit/s in duration more than N minutes. Very useful for DDoS prevention.

[bluedino]

And look at your dashboards once in a while. I'd find it unusual if I saw my toy VPS cranking out 100mbs for a week straight!

[emeraldd]

Make sure you tune those alerts appropriately and know when your systems should be doing things that will trip them. We've had a couple of boxes that regularly trip cpu warnings during normal operation, but only inside specific time windows. Knowing what's not normal is vitally important with this stuff.

[dangoldin]

I had something similar happen with AWS but the bill wasn't as high since they ended up flagging my box as spam-producing and shut off all outbound traffic. I'd just ask them and see if they can remove the charges, it worked in my case.

[matthewarkin]

I had the same thing happen to me. I wrote about it on my blog http://mattarkin.com/protect-your-azure-linux-vm-aka-how-to-.... Basically I complained to Microsoft, they said they'd waive the charge but since it was for a linux vm they said they couldn't cover it. Then I complained to American Express claiming it was an unauthorized and fraudulent charge. Amex sent the dispute to Microsoft and they never responded so I wont the chargeback.

[minopret]

I can understand how that could happen and what a problem it would be. I had an experience with a telephone bill myself, but the story is not going to help you.

I would suppose your first and best resort is to consult your lawyer, advocate, solicitor, barrister, Anwalt. I wonder what your relevant legal jurisdiction is.

I wonder whether it would help if you can account for your own whereabouts and your own usage of endpoint data services. I wonder if your method of payment to your VPS provider is mediated by a financial service that can help you dispute the bill.

I am not a lawyer.

[jnardiello]

I assume you are in europe. I'd suggest simply talking with your provider, explaining the issue and asking them to investigate. I honestly expect them to cooperate and be understanding.

If they insist for you to pay: simply don't. State the truth: You can't afford it. Tell them the only way they will see this money is by taking legal action against you and even in that case you won't be able to comply - as you don't have the money.

Hope it helps :(

[freshflowers]

Just in addition to some other helpful comments: based on posting I assume that your are Dutch or Belgian, located in Europe and are buying this VPS as a private consumer, not a company.

Which means your case is probably covered by consumer protection rules when it comes to informing you about data usage, and I seriously doubt a VPS provider has covered their ass as well as mobile providers tend to do.

[tdicola]

Anyone have tips on how to secure their Linux VPS? I just set one up and disabled SSH password login, locked down all the ports with iptables (using ufw), and enabled fail2ban. Anything else I should install or configure to make myself a little more secure? Was considering tripwire but I dunno how much a headache it would be with false positives as I change things on the server.

[MayIHaveAnother]

A very common attack vector is through installed web applications. Especially if you run wordpress with a lot of plugins installed, be sure to enable correct read/write settings for /var/www, and update your application frequently.

Malicious entities runs 24/7 scans towards indexed URL's attempting to exploit various vulnerabilities, and many of the vulnerabilities allows remote code execution, upload of php files etc. This can be used to upload malicious code, simple php-webshells, and then your VPS is suddenly a part of a DDoS/Scanning network.

Exploited Wordpress sites are a problem, Zeus/Zbot-Trojan is often seen downloading updates/configs from these, and they are also often used to redirect users to Exploit Kits.

[Ded7xSEoPKYNsDd]

On my personal machines, I also setup a cronjob to automatically install updates. There's a small risk of breakage (I had one in 5 years), but I prefer that over a bill like op's.

[applecore]

PSA: Set up billing alerts! You should always have a notification sent to you when your monthly bill exceeds one or more dollar amounts. For example, if you're using AWS, Amazon CloudWatch lets you set an alarm on a billing metric to notify you automatically.

[joshmn]

Post on WebHostingTalk.com - just do it. You'll get attention from the host, other hosts who will sympathize, and you'll see that they'll just write it off.

Post the link when you do and I'll be sure to comment on it (I'm somewhat very-active at WHT)

[decisiveness]

I seem to be missing something. You knew it was happening when you got the first bill, but let it continue for another half month before shutting it down?

[logn]

In addition to the other comments, make it absolutely clear to them (with proof if needed) that you're a student.

[gregcmartin]

Make sure Elastic Search is not accessible from a public IP address (this is what likely got you in the mess to begin)

[general_failure]

Do you know how you got hacked?

[zack19]

cut your credit card report it as stolen tell the host that it wasn't u :p

[ishener]

I feel really sorry for you situation. I first suggest talking to the hosting provider and explain what happened. Any decent service will give some discount in this case.

Unfortunately, I can't think of anything else. I wish it was realistic to tell you to go to the police.

Also, if you would give your email, I would definitely consider sending a donation through paypal... Hopefully other readers here will do the same.

[patio11]

There is one great reason to go to the police: it establishes a paper trail documenting that a crime was committed. It isn't necessary that the police catch the bad guys for that paper trail to be advantageous.

(Examples: police reports make CC disputes and legal declarations much easier and more likely to be given weight as other than self-serving explanations of a deadbeat. It may also trigger insurance policies either for you or for the VPS company.)