Some radiologists think that AI will be really good for filtering out normal images, so that they only have to review anomalies. But I don't think that a model that detects diseases will be too successful, even if they manage to make it really work.
For one, it's actually difficult to interpret and find signs in radiologic images. Obvious signs are obvious, but there are others that could be image artifacts, or just indolent variations, or point to something serious. Even with a generally good accuracy, it'll be hard that a general model performs well on those anomalies with low prevalence.
Second, radiologic signs are just signs. Most diseases are diagnosed with more than just radiologic signs. Most signs are compatible with a lot of diseases. If you see a model that pretends to diagnose a certain disease, well, they're looking at it wrong.
Third, you need a way to have responsibility for diagnoses, and a way to find and correct errors. I don't think that's possible with an unsupervised AI, you'll always need a doctor there to check the image and verify the output. There won't be much savings there. Whenever someone says "AI is going to revolutionize medicine", it's really hard to believe them. I mean, you just have to look at EKGs, modern machines can detect anomalies but doctors still learn how to interpret EKGs and double check what the machine says. It's a help, but not a replacement.
>Some radiologists think that AI will be really good for filtering out normal images, so that they only have to review anomalies. But I don't think that a model that detects diseases will be too successful, even if they manage to make it really work.
Radiologists say this until they get malpractice lawsuits.
Seriously, I once injured my wrist and the first radiologist said it was just a sprain, but no surprise the hospital was using a outsourced...overseas provider for that quick analysis. The hospital had a inhouse radiologist review it as a course of business later on in a few days and the hospital panicked with phone calls to get me back in because there were numerous wrist bone fractures and they took the first step towards liability issues
Could an AI do it? Sure. But you sure as hell need a second opinion
If we can have cheap scanners or other instruments to go with the AI, I think it could easily appear as a screening tool that less qualified medical professionals (midwives, nurses, GPs) or even the patients themselves can use. Those people already do various rough screening tests and refer the patient up the chain of expertise according to the outcome.
Your argument would be better if you stated that most diseases are diagnosed at an advanced stage when it's obvious. The rest are incidental and then we sometimes occasionally we get lucky. We don't routinely CT scan people because it's harmful to do so.
No disagreement here from my part, but I wouldn't be surprised if imaging artifacts could be detected/fixed by ML. E.g., there has been quite a bit of research in MRI boundary effects, although I wouldn't know if that turned out any practical solutions.
The place where I feel ML can help is finding out new disease markers or help upscale the quality of a scan (something I believe Facebook is looking at)
I've been at a few startups and ML has typically meant shoddy rules-based logic or some out of the box model. Only one company applied it with rigor and even then it was a slog - lackluster results, tinkering with different models, poring through research papers to figure out where the cognitive gap came from. The rest of the company thought we were brilliant as did prospective clients. Funny thing is it's possible (though less common) to get paid just as much if not more doing data analysis / engineering than vaunted ML/AI work. Businesses are swimming in data but it's siloed or dirty. I don't really see that and the actual analysis being automated away - too much messiness (human error inputting data into systems like Salesforce, ETL breaks in prod leading to gaps, etc).
> Funny thing is it's possible (though less common) to get paid just as much if not more doing data analysis / engineering than vaunted ML/AI work
Which makes me wonder how much of the supposed magical 10x value of AI actually just comes from getting your shit together in terms of ETL and data pipelines.
Shoddy rules-based logic is ok, as long as it created 10x value. When you masquerade it as AI/ML, you are doing disservice to your own company and clients.
My question is very simple, at what point Shoddy rules-based engine becomes "AI-driven approach"?
I was a moderately successful scientist who reorganized his career around applying machine learning to medicine. After working in the field for a few years, I noticed that all the folks getting attention for their papers were basically publishing bullcrap. I have a high bar for publishing, I don't publish stuff that is bullcrap. But my competitors were, and they had a serious advantage over me (because most people couldn't recognize the bullcrap).
ML folks today are under immense pressure to show small-digit improvements over existing methods and they're using all the wrong techniques (p-hacking, massive hyperparamter runs, press releases that overstate the impact). It's really sad to see all the snake oil.
This is true for every other domain. When every company calls itself a AI company these days. Therefore value of AI/ML and credibility gets diluted.
It becomes very difficult to classify real orgs using hardcode stats from fake ones. When I hear "we are solving x with AI" or "AI driven", it gives me jitters.
Most of the "AI" I see is just linear regressions run on meager datasets which lets them stick on that AI label but could have been written by a person with a little effort.
> By far the biggest problem — and the trickiest to solve — points to machine learning’s Catch-22: There are few large, diverse data sets to train and validate a new tool on, and many of those that do exist are kept confidential for legal or business reasons.
Pretty doubtful. The top innovating nations are such because they attract the top talent from around the world, and almost no talent at all moves to China. There's a lot of breakthroughs in AI left and whoever gets the most talent is going to have the advantage.
So China's (1.4 bil and dropping) smart people vs the rest of the world's smart people (6-7 bil). And China doesn't get easy stepping stones when it wants by stealing secrets from western companies, uni's, and the us military.
> By far the biggest problem — and the trickiest to solve — points to machine learning’s Catch-22: There are few large, diverse data sets to train and validate a new tool on, and many of those that do exist are kept confidential for legal or business reasons.
This is exactly my sense of the problem. Not saying that this is the only problem, just that this seems like the biggest immediate problem.
There are a bunch of questionable ML startups that try to do something with, say, a model trained on ImageNet. You can get pretty far starting with ImageNet. ImageNet was made with Mechanical Turk, but there is no Mechanical Turk for radiologists. If you work really hard you might get patient & treatment notes but interpreting those notes presents its own problem.
I came away from this convinced that somebody needs to build a data sharing system for medical images.
If there was a system for sharing images that had existed before Covid and enough data had been contributed that reviewers could demand evaluation on some predefined test sets, then a lot of these inconsistent evaluation issues could be weeded out.
The article mentions federated learning, but I feel like that's solving a different problem than reliable evaluation.
A lot of these issues also have to do with the publishing system.
There is no incentive in publishing stuff that don't work even though they were reasonable things to try. This indirectly pushes a lot of bad/flawed/incomplete papers and results to be published. Even if your methodology is flawed but you try hard enough, you have a >0 probability of getting your paper published somewhere.
As part of a solution, I think we would benefit from the existence of prestigious journals for negative results, which would incentivise also publishing what doesn't work. This way researchers wouldn't have to try massaging their data and experiments until it looks like it works. They could just publish that it doesn't work, and it would be good for them too.
AI has been breaking its pick at this coalface for at least 50 years, with negligible success. I don’t know why such a difficult field has been so popular.
It seems like the only result of consequence has been Zork, developed on the DM machine at the MIT AI lab in the 70s. No, it was not developed as a medical application but I believe that machine was owned by the medical decision making group.
Stanford’s Knowledge Systems Lab (Ed Feigenbaum’s lab) was next to and affiliated with the Medical School too.
No wonder it’s facing a credibility crisis. Almost every single Machine Learning for Neuroimaging paper I see is reporting 0.9+ AUC whilst using <50 subjects.
With the rampant lack of statistical rigour it blows my mind they get published. This great paper shows that reported accuracy plummets as dataset size increases: https://hal.inria.fr/hal-01545002/document
Unless the AI can talk tot the patient to get some context, it's going to be taking decisions with only very partial information, no matter how good it is.
Still, I'm thinking that as it improves, it's going to show that doctors are not that good at their job on average, and that's going to be fun to watch.
> Still, I'm thinking that as it improves, it's going to show that doctors are not that good at their job on average, and that's going to be fun to watch.
Medical AI is trained on labels generated by doctors. Can you explain how it will exceed the performance of doctors on average? Are you assuming that the labels will be generated by the "top x%" of doctors? If so, how will you identify those individuals? Or is there some other mechanism you're expecting to improve the performance?
Radiologists aren't talking to the patient either and don't have earlier scans available, they "just" comment on the image; so the doctor who does have the patient, their history and the radiology report(s) might as well receive the report from a ML system instead of a human radiologist.
I would really like to get a job in medical ML, but without a related degree it seems impossible. I'm "only" a Veterinarian and I don't even get interviews...
Reading some of your replies, you are good at coming up with reasons you cannot be successful.
A friendly suggestion: this attitude may be holding you back more than you realized before now.
What to do instead, is repeatedly ask yourself: "how can I do this?"
Apply this to many specific areas:
- How can I change my CV so it comes across differently?
- How can I begin a small startup without money or other resources?
- How can I move towards my goal without any new degree?
- How is my training in veterinary medicine an asset?
- How can I do medical ML for veterinary medicine?
- Et cetera...
You have a choice now to dig into your heels and say to yourself reasons why none of what I say above will work, and why your situation is truly and utterly hopeless.
OR you can say different things to yourself, maybe get a different result.
A friend of mine once got rejected from a big name company after a few months of interviews. Reason was that he did not have a PhD. It's hard to get into the business.
You could try to start-up and then plan to get acquihired. Or go and do a PhD. It's never too late for more schooling.
So much FUD on this thread. If a doctor can identify something fishy on a x-ray OF COURSE an appropriate machine learning algorithm can do it as well. It's just a question of gathering the right dataset and experimenting with different architectures.
[+] [-] gjulianm|4 years ago|reply
For one, it's actually difficult to interpret and find signs in radiologic images. Obvious signs are obvious, but there are others that could be image artifacts, or just indolent variations, or point to something serious. Even with a generally good accuracy, it'll be hard that a general model performs well on those anomalies with low prevalence.
Second, radiologic signs are just signs. Most diseases are diagnosed with more than just radiologic signs. Most signs are compatible with a lot of diseases. If you see a model that pretends to diagnose a certain disease, well, they're looking at it wrong.
Third, you need a way to have responsibility for diagnoses, and a way to find and correct errors. I don't think that's possible with an unsupervised AI, you'll always need a doctor there to check the image and verify the output. There won't be much savings there. Whenever someone says "AI is going to revolutionize medicine", it's really hard to believe them. I mean, you just have to look at EKGs, modern machines can detect anomalies but doctors still learn how to interpret EKGs and double check what the machine says. It's a help, but not a replacement.
[+] [-] delfinom|4 years ago|reply
Radiologists say this until they get malpractice lawsuits. Seriously, I once injured my wrist and the first radiologist said it was just a sprain, but no surprise the hospital was using a outsourced...overseas provider for that quick analysis. The hospital had a inhouse radiologist review it as a course of business later on in a few days and the hospital panicked with phone calls to get me back in because there were numerous wrist bone fractures and they took the first step towards liability issues
Could an AI do it? Sure. But you sure as hell need a second opinion
[+] [-] rscho|4 years ago|reply
Uh, TBH no professional so much as glances through the automated EKG summary. It's utterly useless and could be deleted with zero consequences.
[+] [-] exporectomy|4 years ago|reply
[+] [-] sjg007|4 years ago|reply
[+] [-] tgv|4 years ago|reply
[+] [-] rdedev|4 years ago|reply
[+] [-] dkrysl|4 years ago|reply
[+] [-] qsort|4 years ago|reply
Which makes me wonder how much of the supposed magical 10x value of AI actually just comes from getting your shit together in terms of ETL and data pipelines.
[+] [-] debarshri|4 years ago|reply
My question is very simple, at what point Shoddy rules-based engine becomes "AI-driven approach"?
[+] [-] dekhn|4 years ago|reply
ML folks today are under immense pressure to show small-digit improvements over existing methods and they're using all the wrong techniques (p-hacking, massive hyperparamter runs, press releases that overstate the impact). It's really sad to see all the snake oil.
[+] [-] debarshri|4 years ago|reply
It becomes very difficult to classify real orgs using hardcode stats from fake ones. When I hear "we are solving x with AI" or "AI driven", it gives me jitters.
[+] [-] f6v|4 years ago|reply
[+] [-] delfinom|4 years ago|reply
[+] [-] beforeolives|4 years ago|reply
[+] [-] spoonjim|4 years ago|reply
This is why China will win the AI Age.
[+] [-] axelroze|4 years ago|reply
Data is the new Oil and USA is still clinging to the old oil while China has AI as number 1 priority.
[+] [-] chakkepolja|4 years ago|reply
[+] [-] buildingmateri|4 years ago|reply
[+] [-] randomopining|4 years ago|reply
Lets go.
[+] [-] klodolph|4 years ago|reply
This is exactly my sense of the problem. Not saying that this is the only problem, just that this seems like the biggest immediate problem.
There are a bunch of questionable ML startups that try to do something with, say, a model trained on ImageNet. You can get pretty far starting with ImageNet. ImageNet was made with Mechanical Turk, but there is no Mechanical Turk for radiologists. If you work really hard you might get patient & treatment notes but interpreting those notes presents its own problem.
[+] [-] Eridrus|4 years ago|reply
If there was a system for sharing images that had existed before Covid and enough data had been contributed that reviewers could demand evaluation on some predefined test sets, then a lot of these inconsistent evaluation issues could be weeded out.
The article mentions federated learning, but I feel like that's solving a different problem than reliable evaluation.
[+] [-] hrzn|4 years ago|reply
There is no incentive in publishing stuff that don't work even though they were reasonable things to try. This indirectly pushes a lot of bad/flawed/incomplete papers and results to be published. Even if your methodology is flawed but you try hard enough, you have a >0 probability of getting your paper published somewhere.
As part of a solution, I think we would benefit from the existence of prestigious journals for negative results, which would incentivise also publishing what doesn't work. This way researchers wouldn't have to try massaging their data and experiments until it looks like it works. They could just publish that it doesn't work, and it would be good for them too.
[+] [-] dilawar|4 years ago|reply
[+] [-] gumby|4 years ago|reply
It seems like the only result of consequence has been Zork, developed on the DM machine at the MIT AI lab in the 70s. No, it was not developed as a medical application but I believe that machine was owned by the medical decision making group.
Stanford’s Knowledge Systems Lab (Ed Feigenbaum’s lab) was next to and affiliated with the Medical School too.
[+] [-] FL33TW00D|4 years ago|reply
With the rampant lack of statistical rigour it blows my mind they get published. This great paper shows that reported accuracy plummets as dataset size increases: https://hal.inria.fr/hal-01545002/document
[+] [-] BiteCode_dev|4 years ago|reply
Still, I'm thinking that as it improves, it's going to show that doctors are not that good at their job on average, and that's going to be fun to watch.
[+] [-] unsrsly|4 years ago|reply
Medical AI is trained on labels generated by doctors. Can you explain how it will exceed the performance of doctors on average? Are you assuming that the labels will be generated by the "top x%" of doctors? If so, how will you identify those individuals? Or is there some other mechanism you're expecting to improve the performance?
[+] [-] bayesian_horse|4 years ago|reply
[+] [-] PeterisP|4 years ago|reply
[+] [-] bayesian_horse|4 years ago|reply
[+] [-] redsymbol|4 years ago|reply
A friendly suggestion: this attitude may be holding you back more than you realized before now.
What to do instead, is repeatedly ask yourself: "how can I do this?"
Apply this to many specific areas:
- How can I change my CV so it comes across differently?
- How can I begin a small startup without money or other resources?
- How can I move towards my goal without any new degree?
- How is my training in veterinary medicine an asset?
- How can I do medical ML for veterinary medicine?
- Et cetera...
You have a choice now to dig into your heels and say to yourself reasons why none of what I say above will work, and why your situation is truly and utterly hopeless.
OR you can say different things to yourself, maybe get a different result.
[+] [-] axelroze|4 years ago|reply
You could try to start-up and then plan to get acquihired. Or go and do a PhD. It's never too late for more schooling.
[+] [-] ecmascript|4 years ago|reply
[+] [-] rafaelero|4 years ago|reply