This is a combination of posters not understanding what the technology is used for and the tech guy exaggerating the urgency of the situation.
Remote cardiac monitoring isn't for people who are imminently going to die of catastrophic heart attacks or who risk dropping into a fatal arrhythmia (wonky heart rhythm). In fact, there isn't even solid evidence that it saves lives. What it is used for is investigating patients who have vague symptoms which might be related to transient changes in their heart beat.
These are people who come to their doctor at the age of 54 and say "I felt my heart beating really quickly and felt kind of faint" or "I felt kind of dizzy and then passed out. It's happened twice in the last month." The more traditional way to investigate these patients are with what's called a Holter monitor, as alluded to by a previous poster. These are little belt packs you carry around while wired-up and that record your ECG for 24-48 hours at a time. The main weakness? You said it: the device only has a 24-48 hour windows to capture the weird, often rare, rhythm.
There are different ways remote cardiac monitoring systems report their results and I'm not sure which this particular company was using, but it doesn't really matter. Some of them only report weird stretches, others only report events when the patient says they're feeling symptom X, others are reporting continuously.
Take-home message: this is not life-or-death data. When doctors (at least those who are allowed to keep practicing) think a patient needs critical cardiac monitoring, they admit them to hospital.
This was a tech guy, looking to jump the queue by trying to raise a red flag because his servers were being used for--OMFG!--cardiac monitoring. A lot of my doctor buddies use similar strategies when they're caught speeding by the police. "I just got called in to the hospital!"
From the website I think this analysis is correct. Holters store the 24 hour data on the machine as well. The 12 leads are not continuous but set pieces which also store locally. If they were providing real-time monitoring with alerts then this would be more serious. This seems to be a backup system, ironically itself without backups.
b) Good to know from responses that amazon's paid support is crap. They'll officially tell you "were on it".
c) People are bashing this person for relying on amazon not failing all at once. I would be surprised if they hosted on amazon relying on each individual node being up 99.98% of the time. That is crazy. You build on cloud computing knowing that each one node can fail at any moment, but you build it to fall over to another node. From experience, sometimes nodes just... die... or get really unstable.
d) People bashing the dude vs offering future advice. I'm sure if his company is in deep shit, he's well aware of it.
This reminds me of doing tech support back during the dot bomb. Everyday we'd have day traders phone in complaining that their internet was down and they were losing thousands of dollars. Then I'd advise them to switch to their backup connection, then they'd say they didn't have one.
Finally, I'd ask why if they were running a business that could generate / lose thousands of dollars in a few minutes why they had chosen a residential internet service instead of opting for the business package and why they did not have a backup connection as neither the business packages or residential packages came with a SLA that was suitable for the type of risk inherent in their business.
If their data is so important why are they hosting it on the absolute cheapest and shittiest servers they can find without a backup solution in place? If they read the TOS for EC2 it probably says not to run critical infrastructure on it.
Yes, it costs more than EC2, but there's a reason for that. Wait until this company finds out that if their values exceed what MySQL expects it will silently truncate it.
We are a monitoring company and are monitoring hundreds of cardiac patients at home.
If this is a serious post and there really is a service that monitors cardiac patients from home, first you would have to deal with the unreliability of in-home broadband connections. What if a router needs to be reset, or the DSL goes out? Even if you ignore that, you would expect that any server setup will go down at some time. It has happened to Google, Facebook, Twitter, and now AWS and PlayStation Network. I remember when Rackspace went down a few years back (a truck crashed into the transformer that powered the datacenter - http://www.datacenterknowledge.com/archives/2007/11/13/truck...). All of Armenia was knocked offline when a woman cut a fiber optic line while digging for scrap metal. I'm not sure I'd trust the internet with any kind of life or death monitoring.
I could be way off base here since I only know what I saw on the site, but it doesn't look like this is life-or-death monitoring. It looks like its long term monitoring for later review by a doctor. It doesn't seem at all likely that they have someone monitoring the incoming data in realtime for heart attacks, that would be far better suited by hooking up the system to dial 911.
I think some IT person was freaking out that they had downtime and slightly exaggerated the life endangering part, they likely just lost information that may have been used in a future diagnostic manner.
I don't think it's so much about trust as alternatives and best practices. Ideally you should not trust any system when running mission critical apps and always have multiple levels of backups and monitoring including humans on call.
Interesting quote from a company providing such services:
"The Remote Health Monitoring System is the IT backbone that supports HealthFrontier’s entire portfolio of solutions, including the ecg@home™ and the microtel™/ecgAnywhere™. It captures data transmitted by the patient through USB, Bluetooth, or trans-telephonically. The RHMS™ then stores the information in a database, which can be accessed by the patient’s doctor through a web-based interface. Benefits include:
Cloud computing: The RHMS is hosted on a network of servers across the US, which increases reliability and eliminates the need for a large capital investment in on-site hardware and maintenance."
How many other companies provide home ECG monitoring which report over the web? I'm looking but having a hard time finding more. I will update this post as I do.
[UPDATE]
Other Home-based ECG monitoring services:
My uncle owns a few elder care franchises and had me look into monitoring services as a funemployment project. So I've spent a bit more time with this than I'd like to admit...
The company I was most impressed with was http://www.halomonitoring.com . The device seems to be designed well and the monitoring services (online portal for caregivers or family members) are impressive, comparatively.
There's kind of a sea change happening with home health monitoring -- from the "I've fallen and can't get up" devices to more robust and interactive solutions. GE recently bought a company that provides institutional monitoring equipment and services.
This is very popular. There are several products that take ECG samples at timed intervals and submit them in several different ways.
In addition these devices can be used to take 12-lead EKG measurements in transit (e.g. inside the ambulance) and send them to the doctor for analysis before the patient arrives.
I'm very surprised that they're hosting these services on EC2. The company I worked for recently finished building a huge highly-redundant data center specifically designed for hosting this kind of medically-sensitive information. I'm willing to bet it was a bit more fortified than even Amazon's.
Hmm... That would rather seem to be the monitoring company's own fault. Admittedly I haven't used ec2 for about 3 years, but I seem to recall there being an explicit disclaimer about how it wasn't supposed to be used for this kind of uptime-critical application. I want to say it even specifically mentioned medical stuff.
EC2 is fine for part of the servers. I'd honestly say you should have multiple locations of servers, managed via different means (so the single points of failure are not there).
Agree. I'd avoid cloud for half as critical apps where I cannot CLEARLY picture where my servers are in a physical sense and how I can physically get a hold of them if shit hits the fan.
Elastic Load Balancer, anyone? I have no sympathy for this company. They deserve to get sued for this.
Yes, for something life threatening like this, EC2 is a bad idea, but they didn't even bother to take advantage of the geographic redundancy, much less something so basic as having backup AMIs ready in another AZ in your current region.
Of course, a whole lot of companies are learning this now.
ELB doesn't work across regions, the failure at EC2 affected multiple Availability zones in the Virginia Region. ELB might have helped, but it's likely that some people had all their "availabiliy" zones knocked out.
In fact, it's looking like all the automatic-failovers from people hosting on those data centers compounded the problem when everyone's scripts tried to recover at the same time – what was referred to in a previous article as a "bank run".
But yeah, if you're doing heart monitoring, you need to have your servers replicated across a wide geographic area.
Unfortunately, this last AWS problem affected multiple AZs in the us-east region. The OP very well may have had an alternate AZ failover plan, but like Quora, Sencha, Reddit, FourSquare and Heroku, they probably kept it region specific.
As for backing up to multiple regions, I can imagine them thinking that sending everything over the public internet as being a bad idea. However, not having a multi-region/multi-provider failover plan was a worse one.
This is where you wish you had a data center address and a cabinet number where you could pop in, grab your servers and move the heck out to another DC.
I wonder if this will get more companies to maintain a non-cloud, vanilla setup on some dedicated or collocated boxes.
May be we should have an annual day to bring down our primary servers and see how the backups do. The idea shouldn't be to confirm if everything works or not. It should be to determine what did work and did not work.
You know, that gets me thinking- wouldn't it be pretty damn slick if Amazon offered a way to plug your own hardware into the greater EC2 cluster remotely, so that if Big EC2 goes down, you still have copies of your machines running locally?
Maybe you could even timeshare your machines out for some other instances for a discount on your service, similar to how you can send solar power back up the power grid for an electricity credit. Then ec2 really WOULD be a cloud!
I'd like to see more back story first. Medicine is about risk management. There are places where you need 1/100,000 reliability, and there are places where you need 1/1000, and there are even places where 1/10 is good enough. There are primary systems, and there are auxiliary systems.
Without any background, it is really tough to tell where this fits. You can't blame the people for making the wrong decision until you know what the risk profile for the failure is.
Later in the thread they say two instances are up but the third has most of the patients. If any of their patients die, they should be convicted of negligent manslaughter, not Amazon. Amazon explicitly warns that single instances can be lost. Any EC2 system needs redundancy, let alone a patient critical one!
I recently had the "good" fortune of being at the receiving end of a sales pitch from one cloud hosting service (not Amazon). The pitch centered more around flexibility (ability to add CPUs, RAM, storage, network nodes) rather than reliability issues as the one EC2 is facing right now. All questions about reliability were brushed aside with blanket statements like "it's in the cloud" as if that was an end-all-be-all. After much prodding the sales rep admitted that the hosting was all in one location (low lying hurricane prone area) and if we desired redundancy we would have to mirror our apps at a different geographical location. As with any other product, don't believe the sales pitch. If it sounds too good to be true, it probably is. Always use redundancy specially for critical apps.
I wish they offered some more detail on how important/life-threatening this is. Are we talking monitoring as in "detect signs of impending/occurring cardiac event and notify emergency services"? Or is this monitoring in a longitudinal sense, assisting physicians to look at trends in their patients? If it is the later case, is data stored on the device until it's uploaded, or is it simply lost if the connection fails?
Still, not really forgivable to be 100% AWS, but it's a very different story if this is primarily a recording device vs. a monitoring device.
This is too incredible to be true. But if it is true then I would expect the company in question to be out of business quite soon. A company that put's a life critical system on AWS shows a total disregard for their patients well being.
I've seen things much worse than this with medical records and data.
Up to a few months ago we had a ASP.NET shared hosting customer that was doing some kind of data relay web service for medical imaging. No encryption. Patient data in full view on the server. No redundancy.
Apparently it was used for outsourcing imagery review or something. If it didn't work doctors would have to drive in from home which slowed down the diagnostic process.
"Mission critical" on a $30 a month shared hosting plan. Very much not HIPPA compliant.
Just wow! How difficult can it be to have a backup dedi server with the DB mirrored running at some other provider to quickly switch to, just in case? Even many simple web sites have a config like that. Clouds have been around for decades, they are still new and may fail.
This is just supplementary monitoring, not a true life-or-death situation. Everyone (the company involved included) is welcome to stop blowing it out of proportion any time.
[+] [-] stick|15 years ago|reply
Remote cardiac monitoring isn't for people who are imminently going to die of catastrophic heart attacks or who risk dropping into a fatal arrhythmia (wonky heart rhythm). In fact, there isn't even solid evidence that it saves lives. What it is used for is investigating patients who have vague symptoms which might be related to transient changes in their heart beat.
These are people who come to their doctor at the age of 54 and say "I felt my heart beating really quickly and felt kind of faint" or "I felt kind of dizzy and then passed out. It's happened twice in the last month." The more traditional way to investigate these patients are with what's called a Holter monitor, as alluded to by a previous poster. These are little belt packs you carry around while wired-up and that record your ECG for 24-48 hours at a time. The main weakness? You said it: the device only has a 24-48 hour windows to capture the weird, often rare, rhythm.
There are different ways remote cardiac monitoring systems report their results and I'm not sure which this particular company was using, but it doesn't really matter. Some of them only report weird stretches, others only report events when the patient says they're feeling symptom X, others are reporting continuously.
Take-home message: this is not life-or-death data. When doctors (at least those who are allowed to keep practicing) think a patient needs critical cardiac monitoring, they admit them to hospital.
This was a tech guy, looking to jump the queue by trying to raise a red flag because his servers were being used for--OMFG!--cardiac monitoring. A lot of my doctor buddies use similar strategies when they're caught speeding by the police. "I just got called in to the hospital!"
…to see a patient with a really nasty rash.
[+] [-] mamp|15 years ago|reply
[+] [-] GrandMasterBirt|15 years ago|reply
b) Good to know from responses that amazon's paid support is crap. They'll officially tell you "were on it".
c) People are bashing this person for relying on amazon not failing all at once. I would be surprised if they hosted on amazon relying on each individual node being up 99.98% of the time. That is crazy. You build on cloud computing knowing that each one node can fail at any moment, but you build it to fall over to another node. From experience, sometimes nodes just... die... or get really unstable.
d) People bashing the dude vs offering future advice. I'm sure if his company is in deep shit, he's well aware of it.
[+] [-] _3u10|15 years ago|reply
Finally, I'd ask why if they were running a business that could generate / lose thousands of dollars in a few minutes why they had chosen a residential internet service instead of opting for the business package and why they did not have a backup connection as neither the business packages or residential packages came with a SLA that was suitable for the type of risk inherent in their business.
If their data is so important why are they hosting it on the absolute cheapest and shittiest servers they can find without a backup solution in place? If they read the TOS for EC2 it probably says not to run critical infrastructure on it.
Perhaps for sensitive data upon which people's lives depend they might want to look into something like:http://h20223.www2.hp.com/nonstopcomputing/cache/76385-0-0-0...
Yes, it costs more than EC2, but there's a reason for that. Wait until this company finds out that if their values exceed what MySQL expects it will silently truncate it.
[+] [-] byoung2|15 years ago|reply
If this is a serious post and there really is a service that monitors cardiac patients from home, first you would have to deal with the unreliability of in-home broadband connections. What if a router needs to be reset, or the DSL goes out? Even if you ignore that, you would expect that any server setup will go down at some time. It has happened to Google, Facebook, Twitter, and now AWS and PlayStation Network. I remember when Rackspace went down a few years back (a truck crashed into the transformer that powered the datacenter - http://www.datacenterknowledge.com/archives/2007/11/13/truck...). All of Armenia was knocked offline when a woman cut a fiber optic line while digging for scrap metal. I'm not sure I'd trust the internet with any kind of life or death monitoring.
[+] [-] esrauch|15 years ago|reply
I think some IT person was freaking out that they had downtime and slightly exaggerated the life endangering part, they likely just lost information that may have been used in a future diagnostic manner.
[+] [-] zaidf|15 years ago|reply
[+] [-] recampbell|15 years ago|reply
"The Remote Health Monitoring System is the IT backbone that supports HealthFrontier’s entire portfolio of solutions, including the ecg@home™ and the microtel™/ecgAnywhere™. It captures data transmitted by the patient through USB, Bluetooth, or trans-telephonically. The RHMS™ then stores the information in a database, which can be accessed by the patient’s doctor through a web-based interface. Benefits include:
Cloud computing: The RHMS is hosted on a network of servers across the US, which increases reliability and eliminates the need for a large capital investment in on-site hardware and maintenance."
http://www.healthfrontier.com/rhms.php
How many other companies provide home ECG monitoring which report over the web? I'm looking but having a hard time finding more. I will update this post as I do.
[UPDATE] Other Home-based ECG monitoring services:
Medtronic CareLink: http://www.medtronic.com/your-health/heart-failure/device/ca...
[UPDATE] A more complete listing: http://www.medcompare.com/matrix/1475/Outpatient-Cardiac-Mon...
[+] [-] jmm|15 years ago|reply
The company I was most impressed with was http://www.halomonitoring.com . The device seems to be designed well and the monitoring services (online portal for caregivers or family members) are impressive, comparatively.
There's kind of a sea change happening with home health monitoring -- from the "I've fallen and can't get up" devices to more robust and interactive solutions. GE recently bought a company that provides institutional monitoring equipment and services.
Lemme know if you want the full report :)
[+] [-] yellowbkpk|15 years ago|reply
In addition these devices can be used to take 12-lead EKG measurements in transit (e.g. inside the ambulance) and send them to the doctor for analysis before the patient arrives.
I'm very surprised that they're hosting these services on EC2. The company I worked for recently finished building a huge highly-redundant data center specifically designed for hosting this kind of medically-sensitive information. I'm willing to bet it was a bit more fortified than even Amazon's.
[+] [-] sliverstorm|15 years ago|reply
[+] [-] gte910h|15 years ago|reply
EC2 just shouldn't be ALL of the servers.
[+] [-] zaidf|15 years ago|reply
[+] [-] madaxe|15 years ago|reply
[+] [-] teilo|15 years ago|reply
Yes, for something life threatening like this, EC2 is a bad idea, but they didn't even bother to take advantage of the geographic redundancy, much less something so basic as having backup AMIs ready in another AZ in your current region.
Of course, a whole lot of companies are learning this now.
[+] [-] kinofcain|15 years ago|reply
In fact, it's looking like all the automatic-failovers from people hosting on those data centers compounded the problem when everyone's scripts tried to recover at the same time – what was referred to in a previous article as a "bank run".
But yeah, if you're doing heart monitoring, you need to have your servers replicated across a wide geographic area.
[+] [-] gregburek|15 years ago|reply
As for backing up to multiple regions, I can imagine them thinking that sending everything over the public internet as being a bad idea. However, not having a multi-region/multi-provider failover plan was a worse one.
[+] [-] zaidf|15 years ago|reply
I wonder if this will get more companies to maintain a non-cloud, vanilla setup on some dedicated or collocated boxes.
May be we should have an annual day to bring down our primary servers and see how the backups do. The idea shouldn't be to confirm if everything works or not. It should be to determine what did work and did not work.
[+] [-] sliverstorm|15 years ago|reply
Maybe you could even timeshare your machines out for some other instances for a discount on your service, similar to how you can send solar power back up the power grid for an electricity credit. Then ec2 really WOULD be a cloud!
[+] [-] cypherpunks|15 years ago|reply
Without any background, it is really tough to tell where this fits. You can't blame the people for making the wrong decision until you know what the risk profile for the failure is.
[+] [-] blantonl|15 years ago|reply
"This not just some social network website issue, but a serious threat to peoples lives!"
[+] [-] kierank|15 years ago|reply
[+] [-] unknown|15 years ago|reply
[deleted]
[+] [-] ceejayoz|15 years ago|reply
[+] [-] renegadedev|15 years ago|reply
[+] [-] protomyth|15 years ago|reply
Sometime during the last couple of days, a lawyer was getting a lecture from IT on how the system works and is going to have a very, very bad year.
[+] [-] jamesbkel|15 years ago|reply
Still, not really forgivable to be 100% AWS, but it's a very different story if this is primarily a recording device vs. a monitoring device.
[+] [-] st0p|15 years ago|reply
[+] [-] bigsassy|15 years ago|reply
Seriously? I know there's HIPAA standards for securing sensitive health records, but aren't there standards for system up-times?
[+] [-] lawnchair_larry|15 years ago|reply
Given the information we had a week ago, I can see how Amazon may have looked like the best solution for HA. Their site never goes down, after all.
[+] [-] forgotAgain|15 years ago|reply
[+] [-] jauer|15 years ago|reply
Up to a few months ago we had a ASP.NET shared hosting customer that was doing some kind of data relay web service for medical imaging. No encryption. Patient data in full view on the server. No redundancy. Apparently it was used for outsourcing imagery review or something. If it didn't work doctors would have to drive in from home which slowed down the diagnostic process.
"Mission critical" on a $30 a month shared hosting plan. Very much not HIPPA compliant.
[+] [-] csomar|15 years ago|reply
[+] [-] troels|15 years ago|reply
[+] [-] yaix|15 years ago|reply
[+] [-] arkitaip|15 years ago|reply
[+] [-] dasil003|15 years ago|reply
[+] [-] code_duck|15 years ago|reply