Only 8 accounts were affected. Do not worry. Minor breach. Not much harm done.
It seems to me the truth is the attacker looked for bitcoin wallets and emptied them. The fact he could identify 8 accounts and access them suggests the attacker could have accessed far more accounts if they wished. I think this is the most worrying thing about the breach.
I don't really understand how bitcoin works but it seems that people with wallets need to set up multiple wallets on multiple providers and limit the amount of bit coins in each wallet to limit any losses from breaches like this.
If I was a linode customer I would be thinking about moving. This message, while fairly open, doesn't give me much confidence there aren't other security issues with the platform.
Just imagine that BitCoin is like having cash in your wallet, because that's more or less its intended model. There are a lot of 'anti-counterfeiting' measures because computers are very good at copying, and you don't want people to be able to copy BitCoins the way they can copy music -- and when you ask "what is BitCoin?" people basically start to tell you about the anticounterfeiting technology, and the limits on printing uncontrollable amounts of money. But it's essentially stamped paper in your wallet in any other sense, worth whatever people using it on the Internet will pay for it, not backed by anything in particular but its usefulness.
Basically a lot of people were renting storage rooms in an apartment complex run by Linode, you get your own key to enter the door and retrieve and store things -- whatever. Some people left their wallets inside these buildings, with cash therein. Someone else used some unidentified systematic security flaw, but we don't yet know what it was. Maybe there is a ventilation system which is easily navigable once you know how to get in; or maybe all of the rooms have unlocked windows for no good reason; we haven't been told yet. (There are some suggestions that they stole a key from one of the janitors who cleans these rooms up.)
What we have been told is that some burglar stole eight wallets, and that "All activity by the intruder was limited to a total of eight customers, all of which had references to 'bitcoin'." That suggests that the burglar did indeed peek in the windows beforehand somehow, to find out that these 8 rooms had wallets inside. Otherwise, presumably they would say something like, "The intruder broke into many of our customers' accounts but didn't actually do anything in 99% of cases." In that sense I think the scary bit isn't that he accessed the 8 accounts, it is the fact that he identified them in the first place.
Amortizing the loss across many points of failure may be a good idea, but it wouldn't seem to solve the central problem. Suppose I put $20 in two accounts with 5% chance of compromise, rather than $40 in one account with 5% chance of compromise -- either way, I should expect to lose $2. What I've changed is that I am more likely to lose some of my money (9.75%), but I am less likely to lose all of my money (0.25%). This may appeal more to risk-averse people but it is not fundamentally changing the situation.
Perhaps a better approach is to keep a BitCoin wallet encrypted, since that's pretty simple to do in day-to-day life. This is something that you can't do with your wallet -- you cannot turn your wallet into a steel vault with two-foot-thick walls.
Indeed, we are a Linode customer, and this message only helps a bit. Yes, I know now that we are not affected. But little other information is given: were the user accounts compromised by a vulnerability in Linode's VM management software? If so, was this vulnerability found and fixed? Or did the attacker compromise the account of one of Linode's employees?
It wasn't clear from the memo, but how did the attacker know which 8 specific accounts had 'references to bitcoin' if they didn't access other accounts too?
I'd like to understand the what actions will be taken to prevent similar attacks in the future. Also, what can I as a linode customer to prevent my host from being compromised in a similar fashion.
Implementation of two factor authentication for your customers and requiring it for a root password reset would go a ways to preventing similar attacks.
All this talk about banks being safe yada yada and cloud hosting not safe for US50k. Real banking companies (with billions of dollars on hand) do use commodity cloud hosting including Linode, for even sensitive parts. Take for example Natwest online banking login. On initial login page they load a cookie via an image from www.advanced-web-analytics.com and then once you enter a customer number the next page loads a ...drum roll... javascript file from www.omni-traffic.com. Now who can tell me what one can do when you have control over the Javascript on a banking login page?
Ah crap. It looks they have been moved to Amazon EC2, ~8 months ago they were hosted on conventional Linode VPSs. Points still stands though.
In my experience working on a US financial website, a bank would never consider using a VPS like Linode to store actual banking and customer data. It's not even close to Level 1 PCI compliant.
The user who was affected by the incident quoted an email from linode that stated "Our investigation has revealed a customer support interface was used to access your account.", based on that and all the information of that post you get the impression that through the 'interface' the attacker was able to change the vps root password.
Now a reply from linode comes and says "The portal does not have access to credit card information or Linode Manager user passwords". So if the portal doesn't have access to Linode Manager how the attacker gained ability to change the root passwords ?
Thy should give more details on the incident, i do have a certain trust in the ability of linode to have a secure environment & i can understand that things like that will happen at some point to everyone. However its one thing for someone to get access in your system because you had your roots password to 'password' and another if there was a bug that got exploited.(yea this is an extreme example)
> So if the portal doesn't have access to Linode Manager
They didn't say that, they said it doesn't have access to the passwords. They have an interface to change details, they just can't read them. So they can reset your password to "hunter2" but they can't see if it's "hunter2".
I run a web app. I built an administrative interface for managing it. This interface includes the ability to log me in to any individual user's account (by appropriately initializing a session with that user's ID, the same as logging in normally would do), and to reset any individual user's password. This interface does not let me view any user's passwords; it's both technically impossible, as I don't store them in plain text or encrypted form, and unnecessary.
Unless you believe they're lying, then Linode has the same thing. Some interface where they can access their users' Linode Manager accounts, but that interface does not show Linode Manager passwords or credit card information.
Suspicious events prompted an immediate investigation and the compromised credentials used by this intruder were then restricted
Does that mean the credentials were gained outside of Linode and used to change the root passwords of the accounts for the purposes of the theft, or were those credentials used as part of an exploit in Linodes systems?
It sounds like Linode's user system is different from their server management system. They probably have some administration tools for resetting VPS passwords, but don't have access to sensitive user details.
I've got to wonder too. If you rent out an apartment and someone breaks in with a key they stole from the landlord and takes your TV, can you sue the landlord? On the other hand, if you rent a security deposit box from a bank and it gets broken into, is the bank liable?
These bitcoin servers trusted their currency exchange to cloud servers--VPSes, really--and they want Linode to compensate them for the money lost? Insane. I don't claim to know anything about the bitcoin infrastructure hosted here but I'm going to go out on a limb here and say that there was no dedicated hardware firewall in front of it, no IDS, no WAF, nothing but some Linux instances running iptables.
The payment card industry wouldn't certify this hardware to process credit cards for a mom-and-pop online business, yet these guys use it like their bank? Come again?
If you put $12k of value on their service, why do they take on that risk? Why wouldn't the risk fall squarely on your shoulders?
Why would a hosting provider take on the liability of what you host on it? If the underlying filesystem had an error that Linode could avoid and you lost data, would you expect them to replace the value that was lost? Why is it not limited (at most) to the value of the VPS itself? (I can't imagine them compensating for hardware failure either).
One user (slush) lost $12k worth of bitcoins, but another (bitcoinica) lost upwards of [$40k-$50k - wrong].
EDIT: first report from bitcoinica was "over 10k btc". Most recent report is 43,554 BTC, which would be worth almost $200k if liquidated on MtGox at the moment.
Best thing to do would be for Linode to generously reimburse these people, but they have no obligation to do so.
As a policy, it would neither be good business nor appropriate for Linode to assume all of the risk in a situation like this.
Also, if Linode did put a policy in place to assume some of the risk (some sort of insurance policy) they open themselves up to scams (just get your friend to rob your bitcoins and cash in on Linode's good will insurance policy).
In Linode's FAQ, they mention that their virtualization management software was developed in-house. They seem quite proud of this, quipping: "The Linode Manager is custom software, written in house, and is not for sale (although others have tried to mimic it)."
It seems to me that this is a classic example of security failures following inevitably from a lack of peer review. Maybe Linode didn't consider its LM software to be peer-reviewable, but I bet the victims of the bitcoin thefts wish that someone else had tested the code (and human systems surrounding it) for vulnerabilities.
Is this not exactly what Bruce Schneier frequently points out? Anything that must withstand attacks to protect the valuables within should be tested by attacking it. A lot. My hunch is that the vulnerability exploited by this attacker would have been found and fixed already if the LM software were more open.
I think you may be a little off here. The statements in the thread seem to indicate that the compromise was not based on a vulnerability in custom software, but compromised credentials. You can certainly argue that the management console should be protected by two-factor (and it should be), but their software doesn't seem to be at fault here.
I would be willing to bet that they have had the system tested by external security contractors and scanned with automated scanning tools. This seems to be a people problem and features problem not a vulnerability problem.
I guess we just don't know at this point. You may very well be correct. I guess if you want to use an open source provider, just make sure they are running OpenStack (openstack.org).
I really hope the various bitcoin related incidents don't poison the well for online currencies in general. Yes, you need a greater level of security when dealing with transferrable, relatively anonymous or pseudonymous (and rapidly extractable) online assets that you don't need for book-entry accounting with an audit trail and reversible transactions (credit cards, ACH, etc.). Yes, this is beyond what even most banks currently use. No, it's not beyond current technological state of the art.
Gaming (i.e. casinos), at least some of them, do a reasonable job with some very similar security problems.
> I really hope the various bitcoin related incidents don't poison the well for online currencies in general.
It shouldn't do significantly. In the public's mind the well is some distance off, many not even knowing it exists, so there is plenty of time to get security better sorted before the average man on the street has his money invested.
Also, everyone knows that cash and other forms of investment are far from safe anyway. Hopefully people will eventually see online currency as being no less safe than other forms, and if the security is done right they'll see it as more safe (as it could be).
But this seems to me to be a general security issue with decentralised anything, not a bitcoin specific problem. If you remove central control, and take as full control yourself as possible, then you remove responsibility from other people and that is something you need to seriously think about. Providers like Linode will not be taking responsibility for financial loss in these cases and paying for hefty insurance policies to underwrite the possibility of said loss: they'll just add clauses to their contract disavowing themselves of responsibility if such clauses don't already exist.
One way to protect yourself is to spread the money around multiple places, then one hacked provider doesn't put all your resource at risk. Again this isn't bitcoin specific: if you have more then 50K to invest over here (in the UK) then you split it between multiple organisations as only 50K per registered organisation is guaranteed protected by national safety nets provided by government.
Back onto "poising the well" this could of course work the other way around: if the virtual currency is worth the effort of stealing then it might be seen as more valuable in the eyes of the public - much like a sign you have a good product is that knock-off copies start appearing.
While the incident is unfortunate, I do give them credit for being up-front, honest, and relatively speedy in their response. Sad to say, I'm not sure a lot of other hosting providers would be as quick to admit culpability.
That said, there's no way that resetting the root password should be something a customer service rep can do. Particularly when Linode are very explicit about not being a service for newbs - and are in general unwilling to provide help setting up a Linux system.
I wonder about that. It seems to me that the bitcoin wallet shouldn't have been accessible after reboot, at least until someone came by to give the agent managing it the passphrase that would allow it to decrypt it's state. From the sound of things the wallet was just laying around unencrypted?
Agreed. This is in sharp contrast to the Media Temple / Plesk debacle. Which affected how many thousands of sites? How long did it take them to admit it? Meanwhile, Linode reports full details of a hack affecting no more than 8 accounts the SAME day.
The breach originated on the Lionode customer service system. Which the attacker used to reset user passwords. Unlikely the incidents are directly related.
[+] [-] VonLipwig|14 years ago|reply
Only 8 accounts were affected. Do not worry. Minor breach. Not much harm done.
It seems to me the truth is the attacker looked for bitcoin wallets and emptied them. The fact he could identify 8 accounts and access them suggests the attacker could have accessed far more accounts if they wished. I think this is the most worrying thing about the breach.
I don't really understand how bitcoin works but it seems that people with wallets need to set up multiple wallets on multiple providers and limit the amount of bit coins in each wallet to limit any losses from breaches like this.
If I was a linode customer I would be thinking about moving. This message, while fairly open, doesn't give me much confidence there aren't other security issues with the platform.
[+] [-] drostie|14 years ago|reply
Basically a lot of people were renting storage rooms in an apartment complex run by Linode, you get your own key to enter the door and retrieve and store things -- whatever. Some people left their wallets inside these buildings, with cash therein. Someone else used some unidentified systematic security flaw, but we don't yet know what it was. Maybe there is a ventilation system which is easily navigable once you know how to get in; or maybe all of the rooms have unlocked windows for no good reason; we haven't been told yet. (There are some suggestions that they stole a key from one of the janitors who cleans these rooms up.)
What we have been told is that some burglar stole eight wallets, and that "All activity by the intruder was limited to a total of eight customers, all of which had references to 'bitcoin'." That suggests that the burglar did indeed peek in the windows beforehand somehow, to find out that these 8 rooms had wallets inside. Otherwise, presumably they would say something like, "The intruder broke into many of our customers' accounts but didn't actually do anything in 99% of cases." In that sense I think the scary bit isn't that he accessed the 8 accounts, it is the fact that he identified them in the first place.
Amortizing the loss across many points of failure may be a good idea, but it wouldn't seem to solve the central problem. Suppose I put $20 in two accounts with 5% chance of compromise, rather than $40 in one account with 5% chance of compromise -- either way, I should expect to lose $2. What I've changed is that I am more likely to lose some of my money (9.75%), but I am less likely to lose all of my money (0.25%). This may appeal more to risk-averse people but it is not fundamentally changing the situation.
Perhaps a better approach is to keep a BitCoin wallet encrypted, since that's pretty simple to do in day-to-day life. This is something that you can't do with your wallet -- you cannot turn your wallet into a steel vault with two-foot-thick walls.
[+] [-] microtonal|14 years ago|reply
[+] [-] mdda|14 years ago|reply
[+] [-] orofino|14 years ago|reply
Implementation of two factor authentication for your customers and requiring it for a root password reset would go a ways to preventing similar attacks.
[+] [-] eli|14 years ago|reply
[+] [-] videoappeal|14 years ago|reply
Ah crap. It looks they have been moved to Amazon EC2, ~8 months ago they were hosted on conventional Linode VPSs. Points still stands though.
[+] [-] tomg|14 years ago|reply
[+] [-] glfomfn|14 years ago|reply
The user who was affected by the incident quoted an email from linode that stated "Our investigation has revealed a customer support interface was used to access your account.", based on that and all the information of that post you get the impression that through the 'interface' the attacker was able to change the vps root password.
Now a reply from linode comes and says "The portal does not have access to credit card information or Linode Manager user passwords". So if the portal doesn't have access to Linode Manager how the attacker gained ability to change the root passwords ?
Thy should give more details on the incident, i do have a certain trust in the ability of linode to have a secure environment & i can understand that things like that will happen at some point to everyone. However its one thing for someone to get access in your system because you had your roots password to 'password' and another if there was a bug that got exploited.(yea this is an extreme example)
[+] [-] citricsquid|14 years ago|reply
They didn't say that, they said it doesn't have access to the passwords. They have an interface to change details, they just can't read them. So they can reset your password to "hunter2" but they can't see if it's "hunter2".
[+] [-] dangrossman|14 years ago|reply
Unless you believe they're lying, then Linode has the same thing. Some interface where they can access their users' Linode Manager accounts, but that interface does not show Linode Manager passwords or credit card information.
[+] [-] keeran|14 years ago|reply
[+] [-] dissident|14 years ago|reply
[+] [-] look_lookatme|14 years ago|reply
[+] [-] nwmcsween|14 years ago|reply
[+] [-] keypusher|14 years ago|reply
[+] [-] amalag|14 years ago|reply
[+] [-] chrissnell|14 years ago|reply
The payment card industry wouldn't certify this hardware to process credit cards for a mom-and-pop online business, yet these guys use it like their bank? Come again?
[+] [-] _Lemon_|14 years ago|reply
Why would a hosting provider take on the liability of what you host on it? If the underlying filesystem had an error that Linode could avoid and you lost data, would you expect them to replace the value that was lost? Why is it not limited (at most) to the value of the VPS itself? (I can't imagine them compensating for hardware failure either).
[+] [-] kylebrown|14 years ago|reply
EDIT: first report from bitcoinica was "over 10k btc". Most recent report is 43,554 BTC, which would be worth almost $200k if liquidated on MtGox at the moment.
[+] [-] techiferous|14 years ago|reply
As a policy, it would neither be good business nor appropriate for Linode to assume all of the risk in a situation like this.
Also, if Linode did put a policy in place to assume some of the risk (some sort of insurance policy) they open themselves up to scams (just get your friend to rob your bitcoins and cash in on Linode's good will insurance policy).
[+] [-] PsyVisions|14 years ago|reply
[deleted]
[+] [-] desade|14 years ago|reply
It seems to me that this is a classic example of security failures following inevitably from a lack of peer review. Maybe Linode didn't consider its LM software to be peer-reviewable, but I bet the victims of the bitcoin thefts wish that someone else had tested the code (and human systems surrounding it) for vulnerabilities.
Is this not exactly what Bruce Schneier frequently points out? Anything that must withstand attacks to protect the valuables within should be tested by attacking it. A lot. My hunch is that the vulnerability exploited by this attacker would have been found and fixed already if the LM software were more open.
[+] [-] jarito|14 years ago|reply
I would be willing to bet that they have had the system tested by external security contractors and scanned with automated scanning tools. This seems to be a people problem and features problem not a vulnerability problem.
I guess we just don't know at this point. You may very well be correct. I guess if you want to use an open source provider, just make sure they are running OpenStack (openstack.org).
[+] [-] rdl|14 years ago|reply
Gaming (i.e. casinos), at least some of them, do a reasonable job with some very similar security problems.
[+] [-] dspillett|14 years ago|reply
It shouldn't do significantly. In the public's mind the well is some distance off, many not even knowing it exists, so there is plenty of time to get security better sorted before the average man on the street has his money invested.
Also, everyone knows that cash and other forms of investment are far from safe anyway. Hopefully people will eventually see online currency as being no less safe than other forms, and if the security is done right they'll see it as more safe (as it could be).
But this seems to me to be a general security issue with decentralised anything, not a bitcoin specific problem. If you remove central control, and take as full control yourself as possible, then you remove responsibility from other people and that is something you need to seriously think about. Providers like Linode will not be taking responsibility for financial loss in these cases and paying for hefty insurance policies to underwrite the possibility of said loss: they'll just add clauses to their contract disavowing themselves of responsibility if such clauses don't already exist.
One way to protect yourself is to spread the money around multiple places, then one hacked provider doesn't put all your resource at risk. Again this isn't bitcoin specific: if you have more then 50K to invest over here (in the UK) then you split it between multiple organisations as only 50K per registered organisation is guaranteed protected by national safety nets provided by government.
Back onto "poising the well" this could of course work the other way around: if the virtual currency is worth the effort of stealing then it might be seen as more valuable in the eyes of the public - much like a sign you have a good product is that knock-off copies start appearing.
[+] [-] wmf|14 years ago|reply
[+] [-] strags|14 years ago|reply
That said, there's no way that resetting the root password should be something a customer service rep can do. Particularly when Linode are very explicit about not being a service for newbs - and are in general unwilling to provide help setting up a Linux system.
[+] [-] meow|14 years ago|reply
[+] [-] tptacek|14 years ago|reply
[+] [-] unknown|14 years ago|reply
[deleted]
[+] [-] jostmey|14 years ago|reply
[+] [-] olefoo|14 years ago|reply
[+] [-] Lazare|14 years ago|reply
[+] [-] frsandstone|14 years ago|reply
[+] [-] mrschwabe|14 years ago|reply
[+] [-] shazow|14 years ago|reply
[+] [-] frankydp|14 years ago|reply