I was "ringleader" (many diverse groups have been involved over the years) of the OpenEMR project in 2003-2005 and went on to create the open source ClearHealth and HealthCloud EMR (electronic medical record) systems. OpenEMR has a lot of dedicated folks in it and has been a project of some sort of another for ~25 years at this point.
You can read quite a bit about open source in healthcare in my book, Hacking Healthcare. A bit dated but still in print.
Unfortunately there are massive headwinds against open source in US healthcare settings. Regulation requires certifications that cost upwards of $100K first time, $10K with each release, just in fees. Licensed data sets also make for real difficulties in licensing. Required 3rd parties like SureScripts are openly hostile to open source. Most largest buyers of systems are institutional and most current interpretations of law make it so that open source systems cannot be sold as "sole source" which makes life very hard to close and keep those deals. Finally, until a business model emerges that favors open source and patient health, everyone makes more money with lock-in and so that perpetuates.
Ask me anything.
Can confirm, a little concerningly, that code I wrote 17 years ago is still widely present in the OpenEMR codebase including my old office number for test patients, lol.
I work for a large hosp. operations company and serve as the Dir. Engineering for our clinical operations group. Hacking Healthcare is required reading for new members of my team. It serves as an excellent introduction (with a healthy amount of critique) to the dynamics in the hc technology ecosystem. Thank you for providing this perspective on the industry and its challenges with tech.
We've been successful developing using open source technology internally. In fact, I take a fairly hard stance on disallowing proprietary healthcare specific "solutions" from working their way into our stack (aside from the EHR itself, it has staying power). We're lucky in that we are positioned as somewhat of a startup within a larger org, and are able to take that approach.
To avoid some of the issues you raise, we generally are working to reduce the surface area of the EHR to become simply the transactional backend which is then mirrored to a larger ecosystem of custom apps. This has the effect of boxing in the regulated entity. We focus on data integration (by spending $$$$ on custom HL7 interfaces, unfortunately not everyone can afford) to get outside of the walled garden. This means we can use the information/data for new and interesting purposes without worrying about the EHR vendor's roadblocks/tolls. More importantly to some people, we don't disrupt the billing cycle that originates from the EHR.
Do you notice any trends where healthcare operations/providers are starting to develop internal technology that integrates with the EHR to compliment vs. replace the core transactional system?
Howdy there! I was one of the original authors of OpenEMR back in high school. I'm still good friends with at least one of the other authors. We're always stunned to see OpenEMR in the news, and watching it creep up on HackerNew today has been fun.
I've always been curious why OpenEMR seemed to dominate in the OSS space after we walked away from it. I can only theorize that the code was more approachable than other projects (PHP), and that the GPL kept the work from being captured by any one business. I can't imagine that the code was the best, I'm painfully aware of how poor the security practices must have been in hindsight.
You've given me the chance to ask a question I never knew who to ask: Why, back in 2003 (just after we stopped giving the project attention), was OpenEMR the project you decided to spend time on? What made it the attractive thing to invest in?
If you can tell me I'll bottle that elixir and pour it into every OSS effort I work on today.
> Unfortunately there are massive headwinds against open source in US healthcare settings.
This reminds me of one of the first articles I've read about Linux and open source in general. It was about a CEO (and largest shareholder) of Medsphere Systems Corp, who open sourced their tech stack (I believe called OpenVista) and was promptly sued by his own company (!)
Unfortunately it seems that the sands of time have eroded the original content (which was apparently hosted on linux-watch.com, which now redirects to a VPS provider), but I've still managed to find something [0] [1] [2]
I recently worked on a very small, very custom open source 'EMR' system for a friend involved in prenatal care in a developing country, and so what I wonder is are there attempts to get OpenEMR/Clear Health etc into countries and settings where there is not the same huge regulatory barriers? To me it seems like the U.S. is largely lost for a generation for open source/patient-centric EMR, but maybe there is hope for other countries?
I've noticed a lot of practitioners use PhraseExpander or some other shortcut writing tool to write their patient notes - I'm curious to know how they get around the HIPAA certifications, especially since they are a dedicated key logger on top of any OS.
Do you have any insight in this arena? I wonder if it is because they are not 'the record,' but instead are 'tools to create' the record that is eventually uploaded and stored in another platform?
Might be a tangential question, thanks for the patience
Unfortunately there are massive headwinds against open source in US healthcare settings. Regulation requires certifications that cost upwards of $100K first time, $10K with each release, just in fees.
Seems to me like this could be overcome by licencing but not in the general sense but more in a you need accreditation, join this other pool of people utilizing the system and buy an accreditation licence. Seems like there would still be a value proposition there.
As for the 3rd party seems that could be hit or miss again if it is money, build those out as modules that cover the licence fees the third party is looking to recoup. Maybe with a little bone in it for the open source developers as well.
Thanks for AMA. My previous startup was in self-insured employer space so I only saw the issues you're describing from a distance.
> "Finally, until a business model emerges that favors open source and patient health, everyone makes more money with lock-in and so that perpetuates"
I'm curious to know if you've thought about what such a business model might look like. The closest to a viable business model I've seen gives patients control of their data & allows them to monetize it. But that feels like a pipe dream at the moment because a) EHR vendors don't have incentives to share data, and b) there is no marketplace of buyers for said data.
As with anything in the b2b healthcare space, most of these systems suffer from quite a bit of legacy and at-best-average code quality. Despite that, many doctors, clinics and even small hospitals use them because the private solutions (think Epic [1], but smaller) aren't necessarily better code-wise (don't ask me how I know). I wish more FAANG-calibre devs would look into contributing to and evangelizing these platforms rather than writing yet another note-taking/"productivity management" app. It has a direct impact on the quality of care delivery in certain parts of the world and a positive impact on tool-related clinician burnout [2].
I've worked in the space for a few years and I'd discourage anyone from trying to build a career as a software dev in the Healthcare IT / EMR space. It's extremely sales driven (devs aren't valued at all), code quality is terrible and the systems you write are mostly for the benefit of insurance companies / compliance than doctors or patients.
I think there was a YC funded iPad EMR startup that tried to be cool / hip / provider first until they got smacked in the face by reality.
> I wish more FAANG-calibre devs would look into contributing to and evangelizing these platforms rather than writing yet another note-taking/"productivity management" app.
I would like to see the US Digital Service continue to task technologists with improving EMR systems at CMS (Centers for Medicare and
Medicaid Services), but made free to use by all practitioners and citizens (and of course, open sourcing the resulting codebase). It seems sort of inefficient we keep reinventing the wheel (Epic and the like, which are crazy expensive, or self hosted solutions, when practitioners should not be spending time maintaining EMRs), when your records should be stored for your benefit by your government over the course of your life. This is where, imho, high calibre engineers provide the most leverage (one way ratchets on public goods at scale).
One of the hurdles of EMR systems is that there is a pretty significant minimum viable product due to various standards that can't be ignored. Thankfully, this space is far more open now than it used to be; HL7 for example used to require payment just to see the standards. Pretty much everything you would need access to (HL7, CCDA, SNOMED, LOINC, ICD-10, etc) is freely available now!
Generally just having an EMR system is not enough; you also need practice management, scheduling, billing, insurance claims, etc. Interoperability between separate software for these things is... tenuous at best, though some practices do manage to handle it, it can be very fragile. Hence integrated solutions are pretty much the best way to go, and also prevent disruption from competitors which may be better in one space but not another, since it's so hard to get them to talk to each other well.
> Pretty much everything you would need access to (HL7, CCDA, SNOMED, LOINC, ICD-10, etc) is freely available now!
The AMA’s CPT-4, incorporated as a component of HCPCS, is not free, and is the mandated code set for most professional procedure coding.
And while otherwise that may be true for most of what you need for core EMR functionality, everyone wants EMR and billing/insurance transaction handling to be modules of the same core system (because you are going to need both, and they need to interface smoothly to avoid a whole lot of operational friction), and most of the mandatory billing/insurance standards are decidedly not gratis; older versions of at least the X-12 standards in this space were subsidized by CMS and available for free, but that hasn't been the case for the versions required since 2010. And that's just basic transaction standards, a lot of the code set standards are also proprietary.
(In addition to not being free, the standards in this space are exceptionally poorly written, ambiguous, self-contradictory, and incorporate vast quantities of external material, often also not free, by reference—and often not hyperlinks, but “here is the name of the document and the postal address from which you can contact the entity from which you can order it.”)
I think what would help us the most is not writing software, but instead explaining the requirements in detail (like a specification). There are many people looking for a nice self-contained FOSS project to work on, but many don't know where to look and joining an existing codebase might be too daunting.
There will be an Uber of this space, someone who says "I understand the problems these laws are trying to solve, unfortunately they are a kludge and we can solve them much better with better technology". So complying with all these standards will be a second priority done for backwards compat for the person who comes in and disrupts this space.
Yeah its a total mess, most health systems I work with have 30-50 different vendors all with some various forms of integration with the EMR... It's always a mess, with no end in sights.
I was release coordinator 5-7 years ago for an EMR software (DIPS) in Norway, to one of the bigger hospitals (Kalnes).
It was said that there was a unwritten policy that they never use open source software, the only exception was for the ERP database running on Linux. There was more than 2000 systems in that portfolio to various hospitals under the same policy.
After checking with other employees if found the reason; they had to make sure the supplier did not have to many levels of sub contractors, and had to be close to the core development.
So not open source in it self, but open source was seen as a big red flag.
In this space it's worth mention OpenMRS https://openmrs.org/ which also aims to do the same thing. Most of its deployments are in developing countries (I was part of a team driving adoption of it in Kenya) but I'm yet to see a successful large scale deployment of the platform.
Experimented with it and seemed to scale well.
3x load balancers (nginx or httpd or haproxy or iis) https only
3x apache tomcats to deploy openmrs war ... https only. azul jre with -Xmx=2g. Tomcats were also as close to default configuration as possible.
3x mysql clustered ... https only
No issues on any of these OS's:
Windows Server 2008r2
Windows Server 2016
CentOS 7
CentOS 7.5
Raspian Lite
MACOS
OSes ran on virtualized (hyper-v, virtualbox, xen) and baremetal. Commodity hardware (old laptops, desktops with at least 4G ram. and PIs) and real server hardware. Purposely used mixed computing resources to simulate what a team say in Ndola, Zambia might have at their disposal.
I clicked on "features", trying to find out what EMR means, and was greeted by a banner that really clarified things:
"ONC Certified HIT! 2014 Complete Edition EHR!"
I've been coming back to the project intermittently over the last few years and have been pleased to see the progress. The reason I came back this time was that our managing partner just told us we'll be paying next year in our small outpatient clinic--truly unreal. I dream of a day when we could just use our own system, maybe something that I could even manage myself (maybe not realistic?). Bravo to the contributors of this project!
1. Certification, in the USA if you want to get paid from the government you must be certified. This isn't a small undertaking at all and requires quite a bit of development and then the cost of actually doing the third party testing. It has driven companies out of business and forced consolidation even among the larger players.
2. Configurability. EMR's are crazy configurable to meet any hospitals requirements. This means lots of consultant hours to get things setup and running. Take a look at how much money Epic and Cerner make just from "consulting".
3. Interoperability. Again, there are standards like HL7 and FHIR are widely used but the data isn't always great. We are seeing more and more API endpoints all of this requires a level of customization.
All of this adds up to a ton of cost for a small-ish market with a large pool of no or low profit buyers and pretty much a replacement market.
Oh, and you are building software that could cause harm or death. I can't imagine why people don't want to come into this industry and really push the state of the art.
Open EMR was the first choice for our project, but it did fit our workflow needs and the UI/UX is also quite outdated. It's FHIR api also had minimal support. So we ended up building our own EMR on top of HAPI FHIR repository, (https://hapifhir.io).
That's still better than a lot of healthcare software. I have no issues with it running PHP, CVS is more of a hassle for devs. My current EMR only recently came out with a UI that works outside of IE and their proprietary ActiveX controls.
Kaiser spent $4 billion on implementing an EMR with Epic, after abandoning their own project developed with IBM. Imagine if that kind of investment was directed towards a open source consortium. I realize these kinds of projects are incredibly complicated to do correctly even for a single customer, but the benefit of a shared medical record nationwide would be enormous.
I would not accept EMR unless my records are encrypted and can only be unlocked with a smartcard that remains in my posession. Or something close to that.
So (hope it never happens) if you have met with a bad accident, the first responders must break in to your house, call lockpickinglawyer (youtube lock picking guy) break your safe open, retrieve your smart card, then bring it to hospital and start your treatment? And I thought time is very critical. Privacy is important, but its implementation must be reasonable, come on.
[+] [-] duffpkg|5 years ago|reply
You can read quite a bit about open source in healthcare in my book, Hacking Healthcare. A bit dated but still in print.
Unfortunately there are massive headwinds against open source in US healthcare settings. Regulation requires certifications that cost upwards of $100K first time, $10K with each release, just in fees. Licensed data sets also make for real difficulties in licensing. Required 3rd parties like SureScripts are openly hostile to open source. Most largest buyers of systems are institutional and most current interpretations of law make it so that open source systems cannot be sold as "sole source" which makes life very hard to close and keep those deals. Finally, until a business model emerges that favors open source and patient health, everyone makes more money with lock-in and so that perpetuates.
Ask me anything.
Can confirm, a little concerningly, that code I wrote 17 years ago is still widely present in the OpenEMR codebase including my old office number for test patients, lol.
[+] [-] datahead|5 years ago|reply
I work for a large hosp. operations company and serve as the Dir. Engineering for our clinical operations group. Hacking Healthcare is required reading for new members of my team. It serves as an excellent introduction (with a healthy amount of critique) to the dynamics in the hc technology ecosystem. Thank you for providing this perspective on the industry and its challenges with tech.
We've been successful developing using open source technology internally. In fact, I take a fairly hard stance on disallowing proprietary healthcare specific "solutions" from working their way into our stack (aside from the EHR itself, it has staying power). We're lucky in that we are positioned as somewhat of a startup within a larger org, and are able to take that approach.
To avoid some of the issues you raise, we generally are working to reduce the surface area of the EHR to become simply the transactional backend which is then mirrored to a larger ecosystem of custom apps. This has the effect of boxing in the regulated entity. We focus on data integration (by spending $$$$ on custom HL7 interfaces, unfortunately not everyone can afford) to get outside of the walled garden. This means we can use the information/data for new and interesting purposes without worrying about the EHR vendor's roadblocks/tolls. More importantly to some people, we don't disrupt the billing cycle that originates from the EHR.
Do you notice any trends where healthcare operations/providers are starting to develop internal technology that integrates with the EHR to compliment vs. replace the core transactional system?
[+] [-] mixonic|5 years ago|reply
I've always been curious why OpenEMR seemed to dominate in the OSS space after we walked away from it. I can only theorize that the code was more approachable than other projects (PHP), and that the GPL kept the work from being captured by any one business. I can't imagine that the code was the best, I'm painfully aware of how poor the security practices must have been in hindsight.
You've given me the chance to ask a question I never knew who to ask: Why, back in 2003 (just after we stopped giving the project attention), was OpenEMR the project you decided to spend time on? What made it the attractive thing to invest in?
If you can tell me I'll bottle that elixir and pour it into every OSS effort I work on today.
[+] [-] absorber|5 years ago|reply
This reminds me of one of the first articles I've read about Linux and open source in general. It was about a CEO (and largest shareholder) of Medsphere Systems Corp, who open sourced their tech stack (I believe called OpenVista) and was promptly sued by his own company (!)
Unfortunately it seems that the sands of time have eroded the original content (which was apparently hosted on linux-watch.com, which now redirects to a VPS provider), but I've still managed to find something [0] [1] [2]
0: https://70.42.23.9/servers/a-medical-open-source-legal-hell-...
1: https://medicalconnectivity.com/2007/10/25/medsphere-settles...
2: https://www.informationweek.com/medsphere-settles-lawsuit-wi...
[+] [-] russnewcomer|5 years ago|reply
[+] [-] ethbr0|5 years ago|reply
Bespoke front-ends and UX have never been open source's forte, but shared serving technology running behind the scenes has been wildly successful.
Health care seems like a good fit for that.
(Said as someone with clients in insurance, and well aware of how quickly data interchange can embrittle an architecture)
[+] [-] anonymouse008|5 years ago|reply
Do you have any insight in this arena? I wonder if it is because they are not 'the record,' but instead are 'tools to create' the record that is eventually uploaded and stored in another platform?
Might be a tangential question, thanks for the patience
[+] [-] kls|5 years ago|reply
Seems to me like this could be overcome by licencing but not in the general sense but more in a you need accreditation, join this other pool of people utilizing the system and buy an accreditation licence. Seems like there would still be a value proposition there.
As for the 3rd party seems that could be hit or miss again if it is money, build those out as modules that cover the licence fees the third party is looking to recoup. Maybe with a little bone in it for the open source developers as well.
[+] [-] dman7|5 years ago|reply
> "Finally, until a business model emerges that favors open source and patient health, everyone makes more money with lock-in and so that perpetuates"
I'm curious to know if you've thought about what such a business model might look like. The closest to a viable business model I've seen gives patients control of their data & allows them to monetize it. But that feels like a pipe dream at the moment because a) EHR vendors don't have incentives to share data, and b) there is no marketplace of buyers for said data.
[+] [-] schoolornot|5 years ago|reply
[+] [-] BadInformatics|5 years ago|reply
As with anything in the b2b healthcare space, most of these systems suffer from quite a bit of legacy and at-best-average code quality. Despite that, many doctors, clinics and even small hospitals use them because the private solutions (think Epic [1], but smaller) aren't necessarily better code-wise (don't ask me how I know). I wish more FAANG-calibre devs would look into contributing to and evangelizing these platforms rather than writing yet another note-taking/"productivity management" app. It has a direct impact on the quality of care delivery in certain parts of the world and a positive impact on tool-related clinician burnout [2].
[1] https://news.ycombinator.com/item?id=18735023 [2] https://news.ycombinator.com/item?id=24336039
[+] [-] safog|5 years ago|reply
I think there was a YC funded iPad EMR startup that tried to be cool / hip / provider first until they got smacked in the face by reality.
[+] [-] toomuchtodo|5 years ago|reply
I would like to see the US Digital Service continue to task technologists with improving EMR systems at CMS (Centers for Medicare and Medicaid Services), but made free to use by all practitioners and citizens (and of course, open sourcing the resulting codebase). It seems sort of inefficient we keep reinventing the wheel (Epic and the like, which are crazy expensive, or self hosted solutions, when practitioners should not be spending time maintaining EMRs), when your records should be stored for your benefit by your government over the course of your life. This is where, imho, high calibre engineers provide the most leverage (one way ratchets on public goods at scale).
[1] https://www.usds.gov/resources/USDS-Impact-Report-2020.pdf
[2] https://www.va.gov/health-care/get-medical-records/
[+] [-] sidlls|5 years ago|reply
There are just some things that “throw devs (of any quality) at it” just doesn’t work. The health care industry is one of them.
[+] [-] breck|5 years ago|reply
[+] [-] adzm|5 years ago|reply
Generally just having an EMR system is not enough; you also need practice management, scheduling, billing, insurance claims, etc. Interoperability between separate software for these things is... tenuous at best, though some practices do manage to handle it, it can be very fragile. Hence integrated solutions are pretty much the best way to go, and also prevent disruption from competitors which may be better in one space but not another, since it's so hard to get them to talk to each other well.
[+] [-] dragonwriter|5 years ago|reply
The AMA’s CPT-4, incorporated as a component of HCPCS, is not free, and is the mandated code set for most professional procedure coding.
And while otherwise that may be true for most of what you need for core EMR functionality, everyone wants EMR and billing/insurance transaction handling to be modules of the same core system (because you are going to need both, and they need to interface smoothly to avoid a whole lot of operational friction), and most of the mandatory billing/insurance standards are decidedly not gratis; older versions of at least the X-12 standards in this space were subsidized by CMS and available for free, but that hasn't been the case for the versions required since 2010. And that's just basic transaction standards, a lot of the code set standards are also proprietary.
(In addition to not being free, the standards in this space are exceptionally poorly written, ambiguous, self-contradictory, and incorporate vast quantities of external material, often also not free, by reference—and often not hyperlinks, but “here is the name of the document and the postal address from which you can contact the entity from which you can order it.”)
[+] [-] amelius|5 years ago|reply
[+] [-] breck|5 years ago|reply
[+] [-] bearjaws|5 years ago|reply
[+] [-] punnerud|5 years ago|reply
After checking with other employees if found the reason; they had to make sure the supplier did not have to many levels of sub contractors, and had to be close to the core development. So not open source in it self, but open source was seen as a big red flag.
[+] [-] subsaharancoder|5 years ago|reply
[+] [-] desktopninja|5 years ago|reply
No issues on any of these OS's: Windows Server 2008r2 Windows Server 2016 CentOS 7 CentOS 7.5 Raspian Lite MACOS
OSes ran on virtualized (hyper-v, virtualbox, xen) and baremetal. Commodity hardware (old laptops, desktops with at least 4G ram. and PIs) and real server hardware. Purposely used mixed computing resources to simulate what a team say in Ndola, Zambia might have at their disposal.
[+] [-] thinkmassive|5 years ago|reply
[+] [-] simonebrunozzi|5 years ago|reply
My AWS-bias made me initially think of an open version of EMR (Elastic Map Reduce).
[+] [-] ptx|5 years ago|reply
[+] [-] hanklazard|5 years ago|reply
[+] [-] WesBrownSQL|5 years ago|reply
2. Configurability. EMR's are crazy configurable to meet any hospitals requirements. This means lots of consultant hours to get things setup and running. Take a look at how much money Epic and Cerner make just from "consulting".
3. Interoperability. Again, there are standards like HL7 and FHIR are widely used but the data isn't always great. We are seeing more and more API endpoints all of this requires a level of customization.
All of this adds up to a ton of cost for a small-ish market with a large pool of no or low profit buyers and pretty much a replacement market.
Oh, and you are building software that could cause harm or death. I can't imagine why people don't want to come into this industry and really push the state of the art.
[+] [-] sreeni_ananth|5 years ago|reply
[+] [-] vishnuyar|5 years ago|reply
[+] [-] ramimac|5 years ago|reply
[+] [-] csense|5 years ago|reply
[+] [-] burnte|5 years ago|reply
[+] [-] roywiggins|5 years ago|reply
[+] [-] mixonic|5 years ago|reply
Yeah it started in CVS in 1998ish :-p
[+] [-] SftwreEngnr|5 years ago|reply
[+] [-] Forge36|5 years ago|reply
[+] [-] recursive|5 years ago|reply
Lacking a single, blessed, vendor that can do this seems like it might be an obstacle for adoption.
[+] [-] ahupp|5 years ago|reply
[+] [-] oogetyboogety|5 years ago|reply
[+] [-] ElijahLynn|5 years ago|reply
[+] [-] slater|5 years ago|reply
[+] [-] throwaway201103|5 years ago|reply
[+] [-] anaganisk|5 years ago|reply
[+] [-] JshWright|5 years ago|reply