Continuing the series started Friday – fourteen foundation truths to set the stage for the upcoming book Superpatients: Patients who extend science when medicine’s out of answers.
Principle #1 said that good science is measurable, verifiable, and reproducible. Here’s #2, which is a shocker to a lot of people. As the articles linked below say, it’s a big problem for doctors, not just for patients:
That’s right: good science is great, but –
- Some published studies are weak (read on for the grisly details)
- Some studies are corrupt (sorry, but it’s true)
- And believe it or not, most studies have never been verified by another lab.
The point here is not to indict doctors or researchers. It’s that superpatients are entirely justified in striking out on their own, because it’s demonstrably wrong to say “Just look in the literature.” For one thing, what you need might not be in the literature. For another, even if you find something, it might be bogus. Or simply undependable. And who needs undependable science? That’s why “verifiable and reproducible” was in #1.
In short: when medical science is out of answers, it’s perfectly legitimate to look further.
[You can stop reading here if you want. If you want details, the excerpt below is adapted from Superpatients.]
1. Some published studies are scientifically weak
A big part of the culture of medicine is to trust recognized authority. Well, look at what these editors of three top medical journals have said:
- In 2009, Richard Smith (25-year editor of the BMJ) said, “Most of what appears in peer-reviewed journals is scientifically weak.” (Read the rest of what he said at that link, too.) What??? Read on.
- That same year, Marcia Angell (long-time editor-in-chief of the New England Journal of Medicine) said, “It is simply no longer possible to believe much of the clinical research that is published …. I take no pleasure in this conclusion, which I reached slowly and reluctantly over my two decades as an editor …”
- In 2015, Lancet editor Richard Horton said, “The case against science is straightforward: Much of the scientific literature, perhaps half, may simply be untrue.”
That’s Lancet, BMJ, and the New England Journal. Holy crap.
2. Some studies are corrupt.
This is hard to swallow, but it’s true. A real problem is that it doesn’t need to be intentional – it can just be human nature.
Studies about a drug that are funded by the drug’s maker are more likely to come out favorable. And after Marcia Angell left her role at the New England Journal, in 2012 the Times wrote that her successor “and his colleagues reported that for some subjects, so few experts without financial ties could be found that the journal’s scope was becoming artificially curtailed.”
Yes, Angell’s successor said if we exclude tainted people, the NEJM won’t have enough authors. He said that. And a decade later the Washington Post reviewed a year of NEJM studies of new drugs (73 articles) and found that 60 had been funded by the drug company, and 53 were co-written by the companies’ employees!
Oh, and then there’s Ben Goldacre’s AllTrials.net project, which has discovered that many clinical trials with results unfavorable to the funder are covered up – they’re never published. If that’s not scientific corruption, I don’t know what is. (See Ben’s entertaining TEDMED talk, and maybe my noisy iPad interview with him that night.)
Note: this isn’t just a problem for patients – it’s a hell of a problem for doctors! They’re told to obey what’s in the literature, so what are they supposed to do??
3. Most clinical trials have never been replicated by another lab.
“But wait!” you cry – “That flies in the face of rule 1, ‘good science is reproducible!'” You’re absolutely right. But it turns out scientific studies are really expensive – and scientists generally don’t want to risk the chance that another lab would disagree with their results. (When a California lab started a service to replicate studies at 1/10 the cost, it got little uptake.)
There’s this New Yorker piece on “the decline effect” – things that were supposedly proven true start turning out to be less true. OMG.
The superpatient takeaway
The point isn’t that you can’t trust anyone about anything; the reality is that people are living longer than ever, and that’s no accident, which is proof positive that medicine works better than ever. The point is that there’s still a major fly in the ointment if you assume you can trust whatever’s in the literature, and if it’s not in the literature you can’t.
Again: when medical science is out of answers, there’s plenty of reason for superpatients to look further.
Further principles will build on this, later on.
Next in the series: #3: Don’t expect docs to know all and others to know nothing.
“…when medical science is out of answers”? This wording makes it sound like medical science has tried and tried, but can’t figure something out. Yet that’s not exactly what you’ve described, here, in this installment.
I would re-phrase it as “…when medical science has tried and not been able to deliver the goods, as well as when medical science has not (or will not) really try to deliver any bankable goods at all–which, as we have seen here, happens surprisingly often.”
[Typo edit: after posting this, I changed the phrase to “nothing more we can do.”]
Good thought, Brenda. I used the term “out of answers” in the sense of “has nothing more to suggest” or “to offer.” A familiar Hollywood scene might be worded “I’m sorry, but there’s nothing more we can do.”
Also, yes, re “in this installment” – correct, this installment is not an all-in-one case for action. This is one of (at least) fourteen principles that I think are necessary to fully understand why all these real-life cases exist where patients accomplished something their clinicians (supported by all of medical science) had not accomplished.
Make sense?
Yes, Dave. It makes sense. The phrase just seemed to me to imply effort on the part of medical science that may not, in fact, have been expended at all. As you pointed out, sometimes the reason medical science is “out of answers” is because they weren’t trying to find any, anyway!
Great stuff, Dave.
On your point: “And believe it or not, most studies have never been verified by another lab.”
Some issues/reasons.
1. Verification studies are hard to get published. Journals want to publish new and groundbreaking, not verification. Given that academics are rated on their publications, there is little incentive for them to replicate studies.
2. Because of (1) above, some studies HAVE been replicated, but, unless they add something new, the reviewers will trash the paper, and it will not get published.
3. When a negative-result study (i.e. a study that shows that something does NOT work) is done, it is also next to impossible to get this published – journals do not want to publish something that says “Good idea, but it does not work.” It’s called “Publication Bias”). (It was for this reason that the site Clinical Trialss.gov was established, forcing researchers to at least give information about studies not published. (https://clinicaltrials.gov/).
So, the end result is many once-off studies :-(.
Yes, I’m well aware of all that, and trimmed it because this isn’t the body of the book, it’s just the foundation principles!
Should I made it a series? (Only half kidding…)
Well, if not a series, then at least plan 2nd and 3rd edition, please – hopefully, the system will improve, and later editions of the book will be able to report on improvements (we live in hope :-)
Another great article Dave and good reminder that just because a paper is published doesn’t mean it is good! I think we all have a tendency to blindly believe ‘research’ if it is published but the truth is a lot of research is not much better than the content the general media feed us with everyday which is riddled with bias and misinformation. I was a little shocked to find out the statistic you posed: that 60 of 73 studies in NEJM were funded by drug companies! I guess it doesn’t surprise but it is disappointing that there isn’t a little more due diligence carried out on funding sources for these studies.
Great article, Dave. I hope everyone reads, believes, and uses their own critical thinking as you did in your own recovery. Quite an inspiration.