top | item 10391313

Theranos Has Struggled with Blood Tests

137 points| OopsCriticality | 10 years ago |wsj.com | reply

86 comments

order
[+] danso|10 years ago|reply
Here's a nice profile from the New Yorker in late 2014:

http://www.newyorker.com/magazine/2014/12/15/blood-simpler

I mean "nice" as in, a good read...it's a flattering profile of Holmes -- she comes off well, as a genuinely passionate person, and even if she had stayed in school, it sounds like she would've still made a great impact.

But the article also raises concerns that seem to be corroborated by the OP:

> Clarke argues that finger-stick blood tests aren’t reliable for clinical diagnostic tests; because the blood isn’t drawn from a vein, the sample can be contaminated by lanced capillaries or damaged tissue. Holmes strongly disagrees: “We have data that show you can get a perfect correlation between a finger stick and a venipuncture for every test that we run.” When I asked for evidence, I was sent a document by Daniel P. Edlin, Theranos’s senior product manager, titled “Select Data.” It purported to show favorable results from numerous comparison tests. I asked Edlin if the tests had been conducted by an independent third party. He replied by e-mail: “The clinical tests were conducted by a combination of Theranos and external labs,” but he wouldn’t say which ones.

Um, OK. I don't know much about this testing process...but...what trade secrets are being protected by hiding the methodology and source of the comparative test results?

[+] dbcooper|10 years ago|reply
I was helping a friend design a microfluidic device for platelet characterisation, and the first thing he told me was that only a vein was suitable for drawing blood for platelets. He had to take a phlebotomy course just for the blood for testing prototypes.
[+] downandout|10 years ago|reply
Like many of us, Elizabeth Holmes is smart and had a good idea to change a large industry. Unlike many of us, she has managed to convince several billionaires to make her one too, based almost solely on the strength of her idea. I don't know how much money she has personally taken off the table during her funding rounds, but for her sake I hope it's enough to last her for the rest of her life.

I don't doubt that she is sincere in wanting to build this business and take over the blood testing industry. But Theranos is looking more and more like a cautionary tale about why VC's and angels shouldn't give any Stanford dropout with little more than an idea and a dream a multi-billion dollar fictional valuation and tens or hundreds of millions of dollars in real cash to burn in the streets.

[+] NN88|10 years ago|reply
is she smart?
[+] Pyxl101|10 years ago|reply
> Mr. Balwani replied the next day, copying in Ms. Holmes. “I am extremely irritated and frustrated by folks with no legal background taking legal positions and interpretations on these matters,” he wrote. “This must stop.”

Wow. What a terrible response to get from company leadership. It should be completely appropriate for an employee to bring a concern to the attention of leadership based on a plain-English reading of a statute or regulation. Obviously, I don't know what kind of email the person sent, but if it was something like, "A plain English interpretation suggests that we are required to ... " then the right response is to thank them for bringing it to attention, and then engage people for an appropriate review. Every time I've dealt with a legal situation in a corporate environment, the plain English reading was ballpark a correct way to understand it, with occasionally some detailed nuances for which it was important to have legal input, but which did not radically change the situation. I've never run into a case where the required action was opposite than it seemed, or anything like that, although I'm sure surprising things come up from time to time.

I understand why they might ask people to stop discussing it, on a big company email list or something like that, but bringing it to the attention of the appropriate company leader, is exactly the right thing to do. Thank them and look into it.

[+] danso|10 years ago|reply
Without seeing the full emails, I'm hesitant to judge. The excerpt sounds reasonable, but for all we know, the subject line could've read, "WHAT WE ARE DOING IS WRONG!"...I thought company lawyers generally frown upon speculation of legal matters via email. Not necessarily to stamp out internal and genuine complaints, but because they end up being a rich source of material to be taken out of context should litigation come up...that said, Mr. Balwani probably should've kept that in mind before writing something that appears quite dickish when excerpted in an investigative news report.

The other parts of Theranos's media strategy, according to the WSJ, doesn't seem particularly friendly:

- Threatening to sue a widow whose Theranos-employed husband expressed doubts to her before killing himself.

- Going around to people who had been interviewed by the WSJ and asking them to sign statements (before the story was even published) that the WSJ had misquoted them.

- Holmes ignoring WSJ for _five months_. Then standing them up last week. The CEO is famous for being an inspirational, thoughtful speaker but can't even make time in _five months_ to give a bland statement of defense or explanation to the Wall Street Journal, on a story that strongly hints at violations of federal health regulations? It's one thing to refuse comment when a news organization ambushes you the night of the deadline -- it's inexcusable to hide for five months..._especially_ if the company isn't actually doing anything wrong.

[+] dbcooper|10 years ago|reply
A critical opinion piece on Theranos from the journal Clinical Chemistry and Laboratory Medicine:

http://www.degruyter.com/view/j/cclm.2015.53.issue-7/cclm-20...

[+] sremani|10 years ago|reply
Thanks for the article, this quote hit me >> Details of the Theranos technology have not been disclosed to scientific journals and for this reason it is not possible to comment. << So they have handful of tests that are FDA approved and then their protocols or technology never went through scientific scrutiny, but the whole company is valued $9billion? This is disconcerting.
[+] nrao123|10 years ago|reply
I am genuinely confused by the overall thrust of the article.

1st Theme says: Theranos doesn't use Edison (thier in house testing device) & instead use regular equipment from companies like Siemens. This is a marketing problem because the company is saying its using one device but is actually using another one.

2nd theme: lab tests from Theranos differ from generally accepted standards. How do they differ if in fact they are using the same tests as everybody else? Is it just the general variability of lab results and similar variability could be find in quest diagnostics as well?

3rd Theme: almost all people say Theranos is dramatically cheaper than competitors. How is that possible when they are using the same equipment as everybody else for most of the tests? Is it a process innovation in operations rather than from Edison/better equipment tech? Or are they just subsidizing these costs and being cheaper and possible have bad unit economics?

The are two plausible storylines that can seem to reconcile these three themes is:

Storyline 1 (Negative)- Theranos seem to be doing a process innovation rather than an underlying equipment innovation. That process innovation perhaps includes diluting blood samples 1) to meet thier marketing promise of taking less blood 2) somehow taking less blood and diluting the samples to meet the standard for traditional equipments AND still lead to cheaper operational costs that lead to lower prices. But - somehow these diluted blood samples show more than normal variability.

Storyline 2 (not so negative) Traditional lab companies are ridiculously inefficient from an operations perspective. Theranos is able to take the same equipment as everybody else but because of thier operational efficiency make the end service dramatically cheaper. The variability in tests results is kind of standard in the lab testing market.

Am I thinking about this the right way or missing any big parts?

[+] msravi|10 years ago|reply
It's really pretty simple, and rather clear from the article, that Theranos has been cheating, VW-style. They use regular equipment to get through "proficiency testing", but use the Edison machine for regular patient specimens:

From the article:

> The two types of equipment gave different results when testing for vitamin D, two thyroid hormones and prostate cancer. The gap suggested to some employees that the Edison results were off, according to the internal emails and people familiar with the findings.

> Former employees say Mr. Balwani ordered lab personnel to stop using Edison machines on any of the proficiency-testing samples and report only the results from instruments bought from other companies.

> The former employees say they did what they were told but were concerned that the instructions violated federal rules, which state that a lab must handle “proficiency testing samples…in the same manner as it tests patient specimens” and by “using the laboratory’s routine methods.”

[+] danso|10 years ago|reply
You seem to be missing the part where former employees claim that the company skirted federal regulations by using other machines for the proficiency testing, while using the Edison machines for patient testing. The accuracy of those claims have some relevance to the kind of speculation you hope to make.
[+] bhouston|10 years ago|reply
> 2nd theme: lab tests from Theranos differ from generally accepted standards. How do they differ if in fact they are using the same tests as everybody else? Is it just the general variability of lab results and similar variability could be find in quest diagnostics as well?

Because Edison and the dilution procedure produce low quality results. Thus for a large number of the tests Theranos is less accurate than traditional methods.

> 3rd Theme: almost all people say Theranos is dramatically cheaper than competitors. How is that possible when they are using the same equipment as everybody else for most of the tests? Is it a process innovation in operations rather than from Edison/better equipment tech? Or are they just subsidizing these costs and being cheaper and possible have bad unit economics?

The above is possible because they are not currently focused on profit, rather market share. All that money that they have raised has had to go somewhere.

[+] whitehat2k9|10 years ago|reply
Well if they're not using their own technology for most of their tests it isn't just a marketing problem, it's an image problem that would lead investors to believe they don't have faith in their own product. Secondly, according to the article, their protocols even when using conventional machines differ from the norm in that they are forced to dilute samples to increase sample volume because they take less blood to begin with. This is generally accepted as being a less accurate practice.
[+] frozenport|10 years ago|reply
>>Theranos claims that the usual delay of testing in centralized laboratories is approximately 3 days and that they will generate and deliver their data much faster (e.g., within 4 h). The 3-day delay claim is not accurate. The bulk of laboratory testing in centralized laboratories is completed within an hour or two (calculated from time of sample collection to time of results posting for physician review). For example, in our laboratory, more than 90% of creatinine and troponin requests from all wards are completed in <1 h and more than 97% in <2 h. It is thus questionable that Theranos’s technology will be able to deliver faster results than the ones mentioned... Consequently, faster analysis will not have a major impact on patient outcomes.

http://www.degruyter.com/view/j/cclm.2015.53.issue-7/cclm-20...

This is the same kind of bullshit I see in academia, where a new technology is promised to replace an old technology that works fine, but now they have a 9 billion dollar valuation!

[+] asgard1024|10 years ago|reply
> This is the same kind of bullshit I see in academia, where a new technology is promised to replace an old technology that works fine, but now they have a 9 billion dollar valuation!

I am not sure it is bullshit, maybe it's just competition?

Over time, companies within an industry become complacent, cut costs, and decrease quality of service. Somebody (a young idealist perhaps?) comes along and says, hey, we can do better! And because she believes in free markets, it must be that old technology (and indeed newer technology may have marginal benefit) is the culprit, not human greed or laziness or other deadly sin. So she goes, touts, implements and there is some success. However, then the giants will wake up to the threat. They will magically find the way to be more effective, as they always could have been, and they may even grudgingly implement some parts of new technology. And so the margin for the disruptor quickly disappears, and things will return back to the normal state of affairs, where companies can lazily collect fat paychecks for doing the necessary minimum. It's actually win-win for everybody, customers get a small technology/process upgrade, and she is now in the club!

The only minor problem is that this economic model is far from being optimal, but no one really worries about that except academic economists, since no one really knows how to attain the optimal model in the human society anyway.

[+] rcurry|10 years ago|reply
The more I read about this company, Theranos, the more I get this feeling that one day I'll be reading an article in the WSJ or NYT titled "The Greatest Diagnostics Company That Never Was", or something along that line.
[+] hanklazard|10 years ago|reply
Seems to me like this quote from the paper is not making a fair comparison. As far as I know, the main market that Theranos is targeting is outpatient / home testing. Yes, of course central labs in a hospital setting get simple and/or common tests back within a few hours--inpatient medicine involves a higher level of illness severity and thus a higher priority on lab values. I work with one of the major outpatient laboratory companies on a daily basis and I'll tell you that their basic results are almost always back within 24 hrs and stat labs are back within three hours of the runner picking them up. In fact, probably half of our wait time is from the pick-up schedule (late in the day) and had nothing to do with these "older" technologies taking too long.

That being said, this particular lab company provides horrible customer service and our nurses probably spend more than an hour on the phone with them every week trying to troubleshoot very simple situations. For this reason, I've been hopeful about Theranos' potential to bring some meaningful competition to the market. We'll see.

[+] spacecowboy_lon|10 years ago|reply
I would agree from an end user POV (I have to do a set of bloods every 6-8 weeks) the delay is getting test results from lab to specialist.

Though in the UK the fractured nature of the NHS means I can't get my tests done at a hospital near work - I have to take a days leave.

I wonder if There a market for mobile blood testers who come to your work place to take the sample.

[+] erroneousfunk|10 years ago|reply
When I get blood tests done at my GP's office, they sit around until the next morning when the guy from the lab picks them up. Then they go to the lab, where they might wait in a queue until the next day. Then the tests are run, then they send the results electronically to my GP's office. It takes about three days from the time blood is drawn from me, until the time my GP makes a recommendation based on the test results, and a big part of that is because of the the remote lab testing.
[+] robbiep|10 years ago|reply
I agree with your turnarounds (I mean, of course, if you don't turn around a trop in under an hour it's pretty much useless) not to mention things like the iStat machines which are used in tonnes of rural Australian hospitals for all range of blood tests, and ABG machines in every emergency department and Icu, all of which require tiny quantities of venous or even capillary blood to work
[+] Gatsky|10 years ago|reply
Aside from whether their technology actually works or not, the central premise that cheaper, easier and more frequent testing will improve outcomes is unproven. There are many potential downsides. It might increase health spending through extra doctor visits, the need for repeat testing and more invasive and expensive investigations. But will it actually make anyone's life better? Bear in mind that even a single, obvious test performed in a high risk population to screen for say breast cancer or ovarian cancer, either barely moves the needle in saving lives or ends up being useless. What effect does Theranos think indiscriminate widespread testing will have?

With all these competing risks, this frequent testing strategy needs to be evaluated in randomised double blind clinical trials. Before that though, Theranos needs to submit their tests for independent validation in many thousands of patients, and publish the results.

I note this single study which compared Theranos' CRP to a conventional method [1]. The Pearson correlation was 0.85, which I don't think is really good enough.

[1] http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4091290/

[+] gcanyon|10 years ago|reply
On the other hand you have things like blood glucose testing, which mid-twentieth century happened only in a doctor's office, when today diabetics do it several times daily. Obviously not all tests are going to be like that, and I haven't looked at the list of tests Theranos is performing, so all, some, or none of their tests might be similar.
[+] OopsCriticality|10 years ago|reply
> Aside from whether their technology actually works or not, the central premise that cheaper, easier and more frequent testing will improve outcomes is unproven. There are many potential downsides.

This is an excellent point. Nevertheless, it's very hard to convince patients, and the public, that more tests do not lead to better outcomes. Think on every time there are revisions to the recommended screening criteria for breast or prostate cancer---it's good evidence-based medicine, but there's an outcry from the public every time.

In addition to your considerations of more testing being invasive and expensive, I'd also add that overtesting can lead to interventions that present a non-minimal risk to the patient and ultimately increase M&M.

[+] qq66|10 years ago|reply
It's not necessarily bad business or bad practice to launch a company before the technology is ready, and to use a more conventional, cheaper substitute in the interim. I remember reading about a company that offered automated translation, but started out with human translators to validate the increased market size at the lowered price point. They used the validation to raise enough money to continue growing a user base at a per-unit-loss as well as develop the machine translation software to the point of marketability.
[+] ogezi|10 years ago|reply
Getting blood from someone's finger will lead to incorrect results in many tests because other things diffuse into the blood. The things can lead to very inaccurate results. This is a BIG problem for Theranos.
[+] gmarx|10 years ago|reply
It is conceivable that they could use software to correct for this. Is there is something they can measure that indicates the amount of dilution with interstitial fluid,there might be a way to take account of it. Similarly you'd want to correct for hemolysis
[+] cjbenedikt|10 years ago|reply
[+] thanatosmin|10 years ago|reply
"The sources relied on in the article today were never in a position to understand Theranos’ technology and know nothing about the processes currently employed by the company."

This is exactly the problem! Nobody has sufficient information to judge Theranos's technology outside the company, and they're asking the scientific community and public to accept their claims at face value because they've submitted some documents to the FDA.

[+] msravi|10 years ago|reply
> The sources relied on in the article today were never in a position to understand Theranos’ technology and know nothing about the processes currently employed by the company.

Note the use of the word "current". Implicitly, they're admitting that the process has changed since the time of the former employees, and they're not disputing the process described by those former employees. Which means, that at some point they indulged in rigging proficiency testing.

[+] danso|10 years ago|reply
> The Journal even declined an opportunity to experience the technology themselves by turning down our offer to send proprietary Theranos devices to their offices so they could have a demonstration of tests conducted themselves, and compare the results to those of other testing providers.

I can't believe the reporter didn't take Theranos up on its free offer to collect his blood and analyze it for personal health information that might be useful to Theranos...er, I mean, to the reporter.

[+] Pyxl101|10 years ago|reply
Sounds like the accuracy testing should be blind but isn't. Why not?

It seems like disclosing that a sample is a test is more likely to negatively affect the accuracy of the test than improve it. The need for double blind trials is normally well understood. This is not an experiment, but as a test it seems like it would benefit from at least single blind: lab is unaware - accuracy tests are mixed randomly into the genuine population of testing requests. Kind of like how the TSA is occasionally tested by inspectors who bring all sorts of weapons through.

[+] zaroth|10 years ago|reply
How are you going to get a vial of blood into their system with precisely known characteristics, but blindly? I mean, you can draw blood just before and hope it's "basically" the same? That's not really satisfying when you're going to pass/fail the lab based on a discrepancy.
[+] OopsCriticality|10 years ago|reply
Seems only sporting to also include a link to their PR response to the WSJ article: https://www.theranos.com/news/posts/statement-from-theranos

Typical PR drivel. It's also a bit rich to complain about scientific inaccuracy, and boast of opening things up to the FDA, when the Theranos testing platform remains to be thoroughly investigated in the peer-reviewed literature.

[+] webmasterraj|10 years ago|reply
> Theranos presented the facts to this reporter to prove the accuracy and reliability of its tests and to directly refute these false allegations, including more than 1,000 pages of statements and documents. Disappointingly, the Journal chose to publish this article without even mentioning the facts Theranos shared that disproved the many falsehoods in the article.

Then why doesn't the statement say what these facts are?

This reads more like an attack on the former employees who spoke up than a scientific rebuttal of the article. Which leads me to think even more that the article is right.

[+] dannylandau|10 years ago|reply
Interesting article. While the article paints a bleak picture of company operations, one item that sticks out in favor of the Company is that "Walgreens says its partnership with Theranos has gone smoothly overall."

Seems like there would a lot more complaints considering the number of tests conducted, and that the relationship would have been soured/severed if the results were not living up to Walgreen's satisfaction.

[+] sjg007|10 years ago|reply
Meh..

You can resell a test so long as you follow the rules of the contract of the manufacturer and applicable CLIA and other FDA regulations. These are subject to audit etc... You should not report clinical results from tests not run in a CLIA or FDA approved way. So if a test is run on Siemens for reporting clinical results then the audit goes on that pathway. Diluting the pinprick blood to run on a test that requires larger volumes probably won't work either unless they've shown it does and that of course depends on the CLIA and/or FDA regulations around the test to prove equivalence. It seems like it won't work since the tests have specific input requirements (e.g. 100mL plasma vs 1mL). You might be able to go the other way.. but you miss the cellular and interstitial components that may add noise (e.g. potassium which cells bioaccumulate etc)..

Which other rules apply probably depends on interpretation of whether these are lab developed tests or FDA regulated and this area may require clarification etc... which will depend test to test etc...

Otherwise it sounds like they may have calibration issues which are normal. Outliers happen all the time. If the technology doesn't have some fundamental limit of detection based on the sample volume then they just need to keep iterating. Or the components of blood from the pin prick differ significantly from blood taken from the vein. The data may just be noisy. What that noise level is matters in terms of the detection limits. It is probably too early to tell. Internally they will be tracking this and trying to figure it out.

So this article seems like click bait, but what do I know.

[+] kitkatz|10 years ago|reply
I believe parts are a scam and parts are good The pathology diagnostics field is ripe for disruption with easy access and visible pricing and even if they do this well they will probably disrupt the big players. In terms of new technology I very much doubt they have anything substantial.
[+] wehadfun|10 years ago|reply
Sounds like this is a problem with the testing authorities as well. Instead of sending them full blood samples taken from veins by a trained nurse, they should have sent them finger pricks done at Walgreens.