Katherine Hobson has a new article about IBM Watson in the US News and World Report Best Hospitals issue:
The IBM Wiz Kid’s Work on Cancer
Watson’s foray into oncology is only the first baby step toward applying “big data” to thorny medical problems.
As this giant Jeopardy-winning supercomputer continues its venture into medicine, it’s fascinating to watch, but I continue to scream “WAKE UP!!” to anyone at IBM who’ll listen, because I think there’s a vast opportunity that’s not being pursued. And we need it! (I’m quoted at the end of the article, opining on this.)
What they’re doing
I love that they’re exposing Watson to gobs and gobs of previous cases, as the article describes, so it can sniff for patterns. If there’s one thing a well programmed computer can do, it’s sniff for familiar patterns.
And I love that they’re using it to suggest diagnoses and treatments to the doctor, not make diagnoses or recommendations. (I’ve long said that I doubted IBM’s lawyers would want to be anywhere near liability for a wrong diagnosis and its consequences!) An “information-suck-and-sort” machine should do just that.
(At least for today. Who knows what we’ll think in a few years; I’m not in a position to predict.)
What they’re not doing (yet)
But I have a problem with how they’re limiting Watson’s info-sucking: they’re only letting it look at things that have been approved by the establishment. That means it will always fall short of what a smart patient community knows. Oops.
In my speeches I always talk various ways that a smart patient community knows things that haven’t filtered out to the establishment – and I want Watson to be as smart as those patients. For instance:
- As used today, Watson will always fail to know about new research that’s not out yet. e-Patients founder “Doc Tom” Ferguson called this “The lethal lag time.” Smart patient communities know these papers because they talk to the researchers; Watson doesn’t.
- As used today, Watson only knows about professional information.
- If the literature’s out of date, so is Watson. This is no small issue for me: the treatment I got for my kidney cancer is never mentioned to 3/4 of kidney cancer patients because the literature is out of date. The smart patient community knows this, but a robot that only sniffs where it’s told will get out-Jeopardy’d by patients who don’t have blinders.
- It’s blind to side effects that didn’t appear in the research. Hobson cites a good example from one of Watson’s own “family”:
Herbert Chase, professor of clinical medicine in biomedical informatics at Columbia University’s College of Physicians and Surgeons and a member of Watson’s advisory board.
Chase offers a personal example: He started taking Lipitor and experienced insomnia, but found no mention of that side effect in the literature. Patients on blogs were all over it.
- It’s blind to knowledge that doesn’t interest researchers. In my case, coping with side effects may have saved my life. (My oncologist says so.) Patient communities are all over this stuff – but even six years later there isn’t a shred of literature on the subject.
- Does Watson take into account the relative strength of the evidence in a paper? For instance, does it give more credence to a study that’s been repeated by an independent lab?
- Did you know most published studies have never been repeated by another lab? It should make you nervous, because it means you’re the second lab. Smart patient communities know this.
In short, I might say: It’s great that Watson serves the established way of doing medicine – but wouldn’t it be even better if Watson blew the lid off the establishment? If it went beyond what the best doctors know, and included what patients talk about, too?
So I was thrilled that the article ends thus:
For patients like [deBronkart], the lag between finishing a study and seeing it published can be deadly, he says, so what’s needed is to unleash Watson and its ilk to harvest this early, more tentative knowledge. He envisions a tool that reveals a potentially life-saving tidbit shared by a researcher “last Tuesday in Budapest, on slide 27 of his presentation.”
Why not?? If the slides are already posted on Slideshare, Watson can easily find them, right??
Hey Watson! Let patients help!
Although the version of Watson that won Jeopardy! grew out of a basement skunkworks project, it’s hardly surprising that the mature version is more conservative. I suspect part of this is due to its target customer base (large medical organizations) and part of it is due to liability issues (to which large medical organizations are keenly sensitive). I’ve never read of a doctor being sued for malpractice for not following a course of treatment recommended in non-peer-reviewed literature.
I agree that this omission represents an opportunity, but suspect IBM will not be the organization that seizes it.
Joe, I imagine you’re right – if I were IBM I’d certainly want to turn it into something hospitals WANT to buy. Rocking the boat isn’t usually part of such a scheme.
But yeah, how long will it be before a disruptor comes along and says “I wonder if I can do 90% as good a job at 1/10 the price”? Maybe they’d first offer it to severely disadvantaged areas – and of course they’d toss in “grey knowledge” from peer networks.
I completely agree with Dave. There’s a wealth of untapped knowledge in patient networks. By analyzing only published data it seems like Watson could be exacerbating publication bias among other issues.
So here are two examples of simple research on patient networks: Data Mining the Distribution of Drugs Mentioned by IBD Patients on crohnsforum AND A Brief Analysis on the Mentions of Side Effects on crohnsforum .
Certainly a super computer could do a better job. Right?
Dave,
The article is almost fun. It has a couple of interesting quotes, most notably from Herbert Chase, professor of clinical medicine in biomedical informatics at Columbia University’s College of Physicians and Surgeons and a member of Watson’s advisory board:
“Now that people are eagerly sharing their health experiences online, might that information be advantageously mined, too? Patients have such unparalleled access to medical information and to each other that their collective experience can be as valuable as, or even more valuable than, that of a single practitioner”
That sounds like a statement issued by a great potential member of SPM!
But I am also shocked at the poor journalism. The article doesn’t question the validity of the content fed to Watson. It doesn’t quite state that Watson oncology is not only used at MSKCC but is also a joint venture of some sort between IBM and MSKCC, adding a clear bias to what the MSKCC doctor says in the article. Once you add that information, the article first page appears more like an infomercial for MSKCC than an actual piece of journalism. The MSKCC doctor is advertising one of their products but not stating so.
It also states that Watson is using clinical guidelines and many other kinds of peer-reviewed content. Nowhere does the writer questions the validity of all this content. I am just finishing to read the amazing BMJ articles written by Jeanne Lenzer, “Why we can’t trust clinical guidelines” and “Ensuring the integrity of clinical practice guidelines: a tool for protecting patients” (written with Jerome Hoffman, Curt Furberg, and John Ioannidis). Let me quote some content from the first article:
“A recent survey found that 71% of chairs of clinical policy committees (preparing guidelines) and 90.5% of co-chairs had financial conflicts. Such conflicts can have a strong impact: FDA advisers reviewing the safety record of the progestogen drospirenone voted that the drug’s benefits outweighed any risks. However, a substantial number of the advisers had ties to the manufacturer and if their votes had been excluded the decision would have been reversed”
and
“Despite repeated calls to prohibit or limit conflicts of interests among guideline authors and their sponsors, most guideline panelists have conflicts making the guidelines they issue less than reliable.”
How do you think these guidelines influence what Watson produces? My uneducated guess is that they are weighed more than most other content in the Watson algorithms.
Giles —
I’m the author of the USNWR piece. Sorry you didn’t like it. I guess my question is that if you don’t want to include practice guidelines or other “official” content fed to Watson, what would you feed it? As imperfect as those are, and as subject to bias (http://www.ncbi.nlm.nih.gov/pubmed/23752105) as they are, are they the best guide we have to treating cancer patients with a specific diagnosis? If not, should they then be scrapped as the basis for treating patients? That, to me, is a different but very interesting and complex debate.
KH
Hi Katherine:
Nice to e-meet you! And thank you for responding so fast.
Did the MSKCC people tell you they were working together with IBM on Watson Oncology? There is a huge difference between using a tool provided by others and marketing a tool jointly developed. Just as I look carefully at any sign of bias in peer-reviewed publications (and guidelines) I do the same in articles about technology and health and the lack of mention of the commercial interest that MSKCC has in getting the word out about Watson oncology just caught my eyes.
The issue of guidelines bias is real and troublesome. I think it will be interesting to hear directly from the Watson team and their content partner (MSKCC) how they handle the issue. We, the advocates for patients taking charge of their care will keep on demanding more and more transparency and I guess we’ll have many things to say about Watson, as long as it is a black box spurting out advice without allowing the recipients of the advice to be able to assess the sources of the information. We haven’t worked so hard to gain access to knowledge and data to see advanced computer uses force us back into the realm of paternalistic medicine.
— Gilles Frydman
If it wasn’t clear that this is a commercial partnership, I apologize. I should have stated that explicitly rather than assuming it was apparent. Here’s the press release on the partnership: http://www.mskcc.org/pressroom/press/ibm-watson-hard-work-new-breakthroughs-transform-quality-care-patients
Thanks Dave, I agree with your post and also raised some additional concerns about Watson:
http://informaticsprofessor.blogspot.com/2013/06/what-is-thinking-informatician-to-think.html
Bill
I don’t particularly have concerns about Watson per se – I LOVE the idea of a smart “resource Hoover” that sucks up everything. But yeah, what RULES you give it – to me that ought to be totally configurable. “Show me everything,” “protect me,” etc – perhaps it should be like Google’s Safe Search feature??
Bill, I just clicked over and actually looked at your post. If I’m reading it correctly, you’re anticipating the use of Watson to make (or propose) diagnoses, yes? No?
I wouldn’t at all expect it to be used that way, except perhaps in the sense that the blood test machines in my wife’s veterinary experience spit out results with “this pattern is consistent with X.” Are they thinking of having it be an actual diagnoser?
My interest from the beginning has been the potential for it to be an “obscure facts aggregator,” as suggested by my quote in the article, “On Slideshare, see slide 27 in Dr. X’s talk last Tuesday in Budapest.” That’s my own fantasy – what do you think about that, compared to the possible diagnosis idea?
Dave, Watson had its origin as a question-answering system, and they still tout that as its goal, differing from information retrieval systems like Google and Pubmed that return pages, documents, etc..
It might be able to do something like what you envision in your fantasy, but that would be very, very hard. I think it needs to prove its mettle in more basic functions first, and show it is better than something like Google. It has not, to me anyways, proven that yet.
Interesting, Bill. I wonder what the IBM folks think.