I don’t like relative risk and relative risk reduction because it tends to overestimate the effectiveness of the intervention.
In this case, the absolute risk when measuring for death in the GIM pre-intervention and GIM post-intervention are 0.0215 (2.15%) and 0.0146 (1.46%) with an absolute risk reduction of 0.0069 (.69%).
While the relative risk is 26% across the pre- and post-intervention, the absolute risk reduction is only 0.69% with a NNT (number needed to treat) of 1/156. Which means that 1 patient in 156 was helped by this intervention.
In addition, they had 2 false alarms for each true alarm and could suggest that interventions were performed in patients who did not require it — more tests, medications and possibly increased risk from said interventions.
This shows that the CHARTwatch ML/AI is not helping at all that much clinically.
The question is will this lead to better care or a reduction in resources? Technology allows companies to become 'just good enough'. Any better than 'just good enough' and resources are withdrawn. If there is a 26% improvement in x and x was 'just good enough' before then the only 'rational' move by administration is to reduce other resources until x hits 'just good enough' again. That being said, I think the improvements are coming so rapidly in healthcare that we have a real chance of causing the entire system to shift into a new dynamic so maybe we will actually capture some of these gains for patients.
This takes place in Canada. There are no for-profit hospital complexes like the USA. All of our major hospitals are non-profit, reimbursed by the single-payer healthcare system and philanthropists getting stuff named after them. The profit-motive isn't as significant of a factor here.
That being said, I'm fine with a reduction of resources if additional resources don't increase the quality of my care. In Canada, doctors don't really like to prescribe antibiotics for minor infections.
Americans find this bizarre, but for a minor infection antibiotics are going to screw up your stomach bacteria and long-term health to maybe treat a disease that your body can easily handle on its own.
There's no magic value that comes from allocating resources to a problem. Oftentimes spending money has zero or negative impact beyond virtue-signalling that you care about the problem.
I think you're missing an important part of the equation: it's outcome quality per amount paid. If you could have gotten 20% better results but it would mean tripling the costs of healthcare because we'd need to hire a lot more staff, perhaps we felt that was a bad deal.
If you can get 20% by paying... what, presumably <5% more for some ML tool that double-checks stuff and flags risky stuff... perhaps it's something we want to do.
I think in this case it's unlikely because I don't think the problems the tool solves correspond 1-1 with reduced staffing or other resources. The tool mostly seems to provide ongoing diagnosis at a level of detail the clinical team doesn't have regular bandwidth for (they might make one diagnosis of the patient per time they visit the patient rather than on an ongoing basis). It doesn't really reduce the amount of time staff can spend with patients. They can't get rid of doctor diagnosis entirely so they can't really reduce time per patient in any effective way.
Starving the beast is an ongoing program, the budget will be cut (or fixed, hence silently cut through inflation) either way. My hope is that improvements like this will stave off the harmful effects of the budget cuts.
This question has an implicit assumption that you're talking about a US-style health system and the incentives that exist in a system of that structure.
This is exactly why a structure like the UK NHS which is going for "what's the most healthcare I can get for the country with a fixed pot of money" is a better setup.
For instance, in the UK the female contraceptive pill is free to whoever wants it. Because that is a whole lot cheaper than extra (unwanted) pregnancies. Similarly the NHS has spent money on reducing smoking because that's cheaper than dealing with the health effects.
This scenario exists only in progressive and HNers heads.
Companies make money and Capitalism works by offering more services not fewer. Are there companies that does short-term thinking? Yes.
But overall, our standard of living and quality of services has always improved
Unclear what "AI" brings to the table here. Sounds like traditional automation & monitoring could do the job here. No mention of how the model works, or what kind of training is involved.
> white blood cell count was "really, really high"
You don't need AI for this.
I wish they would provide a more compelling example.
It’s a regression model. You don’t “need” AI for anything. But using ML to identify thresholds for decision making is extremely useful.
I don’t like calling everything AI, but I’m even more irritated by people that don’t understand the value of simple ML models for low hanging fruit decisions like the one shown here
It's the difference between "give the programmer this medical report and have them parse out the white blood cell count" versus s/progammer/AI. And the same every time that the report changes in any way.
I've been that programmer more times than I can count. I'm much happier about being able to work on better problems instead than I am worried about AI taking away my rice bowl.
I think there is some element of "technology laundering" here that I saw during the blockchain hype. Even if plain ol' monitoring and automation could solve your problem no executives want to back that. If you say it's adding AI, blockchain, etc. they get to feel like a visionary so they'll fund your project
Machine learning is extremely good at recognising patterns and I'd much rather trust an LLM's spotting accuracy for an early warning system than the regex code of hospital IT workers
Knowing what I know about workplace dynamics in hospitals I'm gonna go out on a limb and say that the "new hotness" factor of the term "AI" probably does a lot of heavy lifting here when it comes to getting buy in from management and users.
Forgoing a decade of income to get some letters beside your name selects for people who don't take orders from Clippy unless you market it well.
This is a great example of "classic AI" being more than good enough.
Using AI to find patterns in patients and intervene was something I worked on in my last job in Specialty Pharma. Theres many red flags on patients long before they even start treatment, sadly income is one of the largest red flags here in the States.
We were able to perform interventions earlier and improve outcomes with a simple regression model that tried to determined the number of missed doses.
Medical professionals, mostly nurses, are spread extremely thin. They are so busy
and/or jaded that they often neglect to show any compassion or empathy until they see somebody else doing it. Having a family member nearby also keeps them accountable.
We have the term “GOFAI” to distinguish “modern” AI from the older stuff (big bag of if statements, behavior trees, etc., but do we need a new term now to distinguish pre-LLM / Diffusion models (neural networks and tree based models)? Everyone thinks “ChatGPT” when they hear AI now but surely this is something more like XGBoost or a neural network under the hood.
No, we don't use GOFAI, we call it machine learning. LLMs are a subset of the field, and if you want to refer to them just use the term LLM. We don't need new terms when we already have easy to use precise language.
Marketing will abuse any term they get their hands on, and certainly "AI" has been abused, but in the field it usually the umbrella term for all areas of research into making "intelligent" behaviour. Be it expert systems, logic systems, machine learning, statistical machine learning, or otherwise.
Important to note that the timing of this means that it's dedicated, specific AI, not "throw a wrapper and a specific prompt in front of ChatGPT" AI. Of course it's all muddied now.
Test results are already reported from testing equipment a value and expected range (to account for a specific machine/reagant's calibration). Notifying when out of range hardly seems like a AI, but it certainly might be marketed as such.
Maybe there is some nuance for things like a patient in for liver issues where their liver enzymes are expected to be abnormal, but identifying when it is abnormal for them.
There's a thriller plot hidden in here where the medicos ask an AI to reduce unexpected deaths so it manipulates both predictions and deaths to optimize the statistic. When they can manipulate the world we'll have to treat prompts as if they were wish fulfillment demands to a hostile djinn.
> That warning showed the patient's white blood cell count was "really, really high," recalled Bell, the clinical nurse educator for the hospital's general medicine program.
I’m not sure how an alarm for “high white cell count” should have had so much impact. Here in China once the doctor prescribes a finger blood test, we sample finger blood after lining up for 15 minutes, and the result is available within 30 minutes. The patient prints the results from a kiosk and any patient who cares enough about their own health will see the exceptionally high white cell count and request an urgent appointment with the doctor for diagnosis right away. Even in normal cases we usually have the doctor see the report within two hours. Why wait several hours?
> While the nursing team usually checked blood work around noon, the technology flagged incoming results several hours beforehand.
> But in health care, he stressed, these tools have immense potential to combat the staff shortages plaguing Canada's health-care system by supplementing traditional bedside care.
This sounds like the deaths prevented by this tech are caused by delays and staff shortage and what this tech does is to prioritize patients with serious issues? While I appreciate using new tools to cut deaths, it looks like the elephant in the room is staff shortage?
Don't judge me. I am not an ML expert. I am just wondering how this is an AI or ML thing. Is it not match the threshold of WBC in the body, if it is above or below the range, generate an alert. Can any ML guy tell me how this system is actually working?
I don't know the details, but I suspect its a bit more. Probably takes as input all of the factors over a time series and then determines based on these inputs over time there is a higher likelihood of Y. When that likelihood reaches some threshold it sends an alert to the nurse. It's almost certainly not as simple as temperature at 105 -> alert (although a temp of 105, would certainly signal a problem).
These studies are the only way AI will be implemented in Medical.
This stuff will not happen because its good technology that can save lives. Rather, the public pressure from AI performing better at saving lives that humans.
The anecdotes of 'oh it was wrong that one time', will pale in comparison to success. Maybe Insurance companies will be the winners and be our advocate. I've already seen medical professionals use 'that one time it was wrong' as a way to ignore technology.
I am thankful that we can expect fewer unexpected events now. Today I was saved from death at least five times because an automated traffic signal alerted me to machines hurtling dangerously in the wrong direction. I was able to push commands that halted my conveyance until the risk of death had plummeted.
I am also reminded of Dilbert's PHB decreeing that all future unplanned outages must be announced at least 48 hours in advance.
“While the nursing team usually checked blood work around noon, the technology flagged incoming results several hours beforehand”
So the blood was collected and labs done but it wasn’t scheduled to be reviewed until later?
Seems like a win-win. For those saying you don’t need AI, the alternative would be either across-the-board thresholds for flags for each line item (too many false positives) or manually setting it for each patient (too intensive).
Stuff like this is not exactly new but it’s great it’s receiving desired outcomes. The company I work for developed a sepsis alert back in 2010 that helped inform clinicians to possible sepsis in patients by analyzing lab results. Lot of success stories but of course false positives. Tools like this are very useful when they are one of many factors driving a clinician’s decision and not the only reason.
This is really awesome. As someone that has entered an emergency room in severe pain and is shocked at how long it takes to see a physician. I hope this system can monitor people waiting to be admitted as well.
The real weasel word here is "unexpected". If the AI is going around terminating patients and this counts as expected behaviour... technically correct!
[+] [-] esoleyman|1 year ago|reply
In this case, the absolute risk when measuring for death in the GIM pre-intervention and GIM post-intervention are 0.0215 (2.15%) and 0.0146 (1.46%) with an absolute risk reduction of 0.0069 (.69%).
While the relative risk is 26% across the pre- and post-intervention, the absolute risk reduction is only 0.69% with a NNT (number needed to treat) of 1/156. Which means that 1 patient in 156 was helped by this intervention.
In addition, they had 2 false alarms for each true alarm and could suggest that interventions were performed in patients who did not require it — more tests, medications and possibly increased risk from said interventions.
This shows that the CHARTwatch ML/AI is not helping at all that much clinically.
[+] [-] qsort|1 year ago|reply
https://doi.org/10.1503/cmaj.240132
[+] [-] tantalor|1 year ago|reply
I found this interesting:
> 1 truly alerted patient for every 2 falsely alerted patients was deemed an acceptable number of false alarms
[+] [-] jampekka|1 year ago|reply
https://en.m.wikipedia.org/wiki/Multivariate_adaptive_regres...
[+] [-] pj808|1 year ago|reply
[+] [-] meindnoch|1 year ago|reply
[+] [-] jmward01|1 year ago|reply
[+] [-] jjmarr|1 year ago|reply
That being said, I'm fine with a reduction of resources if additional resources don't increase the quality of my care. In Canada, doctors don't really like to prescribe antibiotics for minor infections.
Americans find this bizarre, but for a minor infection antibiotics are going to screw up your stomach bacteria and long-term health to maybe treat a disease that your body can easily handle on its own.
There's no magic value that comes from allocating resources to a problem. Oftentimes spending money has zero or negative impact beyond virtue-signalling that you care about the problem.
[+] [-] doe_eyes|1 year ago|reply
If you can get 20% by paying... what, presumably <5% more for some ML tool that double-checks stuff and flags risky stuff... perhaps it's something we want to do.
[+] [-] TimPC|1 year ago|reply
[+] [-] wesselbindt|1 year ago|reply
[+] [-] rkangel|1 year ago|reply
This is exactly why a structure like the UK NHS which is going for "what's the most healthcare I can get for the country with a fixed pot of money" is a better setup.
For instance, in the UK the female contraceptive pill is free to whoever wants it. Because that is a whole lot cheaper than extra (unwanted) pregnancies. Similarly the NHS has spent money on reducing smoking because that's cheaper than dealing with the health effects.
[+] [-] MichaelZuo|1 year ago|reply
Even declaring that is the case doesn’t change that it’s still clearly a personal judgement depending on the individual.
[+] [-] zooq_ai|1 year ago|reply
[+] [-] tantalor|1 year ago|reply
> white blood cell count was "really, really high"
You don't need AI for this.
I wish they would provide a more compelling example.
[+] [-] jncfhnb|1 year ago|reply
I don’t like calling everything AI, but I’m even more irritated by people that don’t understand the value of simple ML models for low hanging fruit decisions like the one shown here
[+] [-] jampekka|1 year ago|reply
[+] [-] wesselbindt|1 year ago|reply
A 26% reduction in unexpected deaths, apparently.
[+] [-] hiddencost|1 year ago|reply
[+] [-] delichon|1 year ago|reply
I've been that programmer more times than I can count. I'm much happier about being able to work on better problems instead than I am worried about AI taking away my rice bowl.
[+] [-] snapcaster|1 year ago|reply
[+] [-] byteknight|1 year ago|reply
[+] [-] JumpCrisscross|1 year ago|reply
And you don’t need Dropbox for file sync. Machine learning makes integrating automation easier.
[+] [-] charlie0|1 year ago|reply
[+] [-] nisten|1 year ago|reply
[+] [-] potato3732842|1 year ago|reply
Forgoing a decade of income to get some letters beside your name selects for people who don't take orders from Clippy unless you market it well.
[+] [-] bearjaws|1 year ago|reply
Using AI to find patterns in patients and intervene was something I worked on in my last job in Specialty Pharma. Theres many red flags on patients long before they even start treatment, sadly income is one of the largest red flags here in the States.
We were able to perform interventions earlier and improve outcomes with a simple regression model that tried to determined the number of missed doses.
[+] [-] gyutff|1 year ago|reply
If a loved one is in the hospital, stay with them as long as the hospital will allow you to.
[+] [-] geocrasher|1 year ago|reply
Medical professionals, mostly nurses, are spread extremely thin. They are so busy and/or jaded that they often neglect to show any compassion or empathy until they see somebody else doing it. Having a family member nearby also keeps them accountable.
I have seen it personally too many times.
[+] [-] resource_waste|1 year ago|reply
Medical isnt science, and its frightening.
The weirdest thing I've experienced as a patient is that Physicians will urge you against second opinions or having multiple doctors.
Hope telemedicine becomes more mainstream, I'd like to avoid US physicians as much as possible.
[+] [-] larsiusprime|1 year ago|reply
[+] [-] tensor|1 year ago|reply
Marketing will abuse any term they get their hands on, and certainly "AI" has been abused, but in the field it usually the umbrella term for all areas of research into making "intelligent" behaviour. Be it expert systems, logic systems, machine learning, statistical machine learning, or otherwise.
[+] [-] SketchySeaBeast|1 year ago|reply
[+] [-] Tobani|1 year ago|reply
Maybe there is some nuance for things like a patient in for liver issues where their liver enzymes are expected to be abnormal, but identifying when it is abnormal for them.
[+] [-] delichon|1 year ago|reply
[+] [-] renonce|1 year ago|reply
I’m not sure how an alarm for “high white cell count” should have had so much impact. Here in China once the doctor prescribes a finger blood test, we sample finger blood after lining up for 15 minutes, and the result is available within 30 minutes. The patient prints the results from a kiosk and any patient who cares enough about their own health will see the exceptionally high white cell count and request an urgent appointment with the doctor for diagnosis right away. Even in normal cases we usually have the doctor see the report within two hours. Why wait several hours?
> While the nursing team usually checked blood work around noon, the technology flagged incoming results several hours beforehand.
> But in health care, he stressed, these tools have immense potential to combat the staff shortages plaguing Canada's health-care system by supplementing traditional bedside care.
This sounds like the deaths prevented by this tech are caused by delays and staff shortage and what this tech does is to prioritize patients with serious issues? While I appreciate using new tools to cut deaths, it looks like the elephant in the room is staff shortage?
[+] [-] pknerd|1 year ago|reply
[+] [-] kenjackson|1 year ago|reply
[+] [-] fabiospampinato|1 year ago|reply
Like I'm not sure what this measure means, it's not like 26% of people that would die in the hospital would be made immortal or something.
[+] [-] resource_waste|1 year ago|reply
This stuff will not happen because its good technology that can save lives. Rather, the public pressure from AI performing better at saving lives that humans.
The anecdotes of 'oh it was wrong that one time', will pale in comparison to success. Maybe Insurance companies will be the winners and be our advocate. I've already seen medical professionals use 'that one time it was wrong' as a way to ignore technology.
[+] [-] unknown|1 year ago|reply
[deleted]
[+] [-] AStonesThrow|1 year ago|reply
I am also reminded of Dilbert's PHB decreeing that all future unplanned outages must be announced at least 48 hours in advance.
[+] [-] JumpCrisscross|1 year ago|reply
So the blood was collected and labs done but it wasn’t scheduled to be reviewed until later?
Seems like a win-win. For those saying you don’t need AI, the alternative would be either across-the-board thresholds for flags for each line item (too many false positives) or manually setting it for each patient (too intensive).
[+] [-] magicmicah85|1 year ago|reply
[+] [-] daft_pink|1 year ago|reply
[+] [-] Kalanos|1 year ago|reply
that's just an alert, not ai
[+] [-] Oarch|1 year ago|reply
[+] [-] loeg|1 year ago|reply