Does chocolate really help weight loss? Does aspartame cause seizures? Did an Italian doctor discover a simple operation to cure multiple sclerosis? Do dryer sheets cause cancer? The answer to all these claims is a resounding NO. So why do these, and hundreds of other phony health stories, continue to circulate? And how can you sort good health and science information from utter crap?
I spend lots of time reading news, and I care passionately about sorting truth from lies. So I’m going to write a series of blog posts to share what I’ve learned over a lifetime of working at this Sisyphean task. First: phony news sites. Second: Satire beyond The Onion. Third: Outright lies and hoaxes. Today: Not really science and not really health.
John Bohannon staged the phony chocolate story, and then explained how he did it. Two journalists approached him in December 2014:
“They wanted me to help demonstrate just how easy it is to turn bad science into the big headlines behind diet fads. And Onneken wanted to do it gonzo style: Reveal the corruption of the diet research-media complex by taking part.”
Bohannon is a PhD, so calling himself Dr. Johannes Bohannon only meant changing his first name, not falsifying his credentials. Of course, his PhD is in molecular biology, not medicine or human biology. But really, aren’t all doctorates interchangeable? Isn’t that why it’s legit for Florida anesthesiologist Dr. Ted Noel to diagnose Hillary Clinton with “advanced Parkinson’s disease,” based on photographs and prejudices?
But back to Dr. Bohannon, whose chocolate research was based on a “100 percent authentic” study.
“My colleagues and I recruited actual human subjects in Germany. We ran an actual clinical trial, with subjects randomly assigned to different diet regimes. And the statistically significant benefits of chocolate that we reported are based on the actual data. It was, in fact, a fairly typical study for the field of diet research. Which is to say: It was terrible science. The results are meaningless, and the health claims that the media blasted out to millions of people around the world are utterly unfounded.”
The results were first published in one of hundreds of pay-to-publish, utterly unreliable “journals.” (In 2013, Bohannon staged a fascinating sting operation to report for Science on fake pay-to-publish journals.) After the chocolate “research” was published in a journal, he and his colleagues fed the story to the media via press release.
The press release, Bohannon wrote, had “everything. … there’s no need to even read the scientific paper because the key details are already boiled down.” That’s unfortunately typical of much health and science reporting: reporters rely on press releases rather than looking at the underlying research, and fail to look for other experts to critique the research.
Of course a few “key details” were omitted from the chocolate press release, which didn’t mention the laughably small number of 15 dieters, the short three-week duration of the “experiment,” or the very small weight differences between the groups. The media that published the chocolate “research” failed to check the credentials of the journal or of Dr. Bohannon, which would have been enough to discredit the story even without looking at the underlying research.
To floss or not to floss
Recent news articles say flossing doesn’t have proven health benefits. Have my dentists been lying to me for decades? Can I stop feeling guilty about not flossing? Or, to put it another way, are claims about the benefits of flossing true, false, or unproven? I’ll take this as one opportunity to analyze health reporting.
First, I look at the source. Sure, the New York Times and AP are mainstream news sources. But where did they get this information? AP says it
“… looked at the most rigorous research conducted over the past decade, focusing on 25 studies that generally compared the use of a toothbrush with the combination of toothbrushes and floss. The findings? The evidence for flossing is “weak, very unreliable,” of “very low” quality, and carries “a moderate to large potential for bias.”
That sounds impressive. Of course, I have to rely on AP’s judgment about the value of the studies. The New York Times report is a little better, giving me a link to a review of 12 “randomized, controlled trials.”
Second, I look at the expert testimony. Both articles quote a number of dentists and experts who say that, while the research may not be there, they still think everyone should floss. Dentists explained to Snopes.com that studies are tough: there’s not a lot of money for flossing research; research would have to cover many years; research would rely on individual people’s self-reporting about how often they floss, as well as on their variable skill in flossing; and research might be unethical as putting a control population of people who don’t floss at risk for periodontal disease.
Third, I think about the state of the research. I can’t really read the 25 reports that AP says it consulted, or even all 12 that the NYT linked to. But the conclusions are not that flossing doesn’t work: the conclusions are that the evidence is not there. Flossing might work to prevent periodontal disease. Or it might not. We just don’t know. The claim about the benefits of flossing is unproven.
Reading behind the hype
Other health issues have much more clearcut answers. Take the social media flap about whether doctors recommend HPV vaccine. The story is that Pediatrics journal reported that one-third of doctors don’t recommend it. The first step in evaluating a social media story like this is to look at the original source, in this case, the Pediatrics article. Snopes.com quickly debunks the social media report:
“In short, it was true that researchers surveyed 600 doctors and that the result of that survey appeared in a February 2016 Pediatrics article about Gardasil vaccination rates. However, the study found the only thing about which doctors were ‘hesitant’ was parental objection, not vaccine safety, and that a secondary cause of their hesitation was an underestimation of teen sexual activity rates. A number of social media items inaccurately suggested that a third of doctors had expressed reservations the efficacy and/or safety of Gardasil itself, even though the Pediatrics article plainly stated that was not the case.”
If you’d like more information on evaluating science and health reporting, here are some good places to start, beginning with the most accessible, and going on to the most complex:
- Jon Oliver took on scientific studies in a May episode, skewering “bullshit masquerading as science” in his inimitable style. While he treats the topic with humor, he conveys accurate and important information in a way that will stay with you.
- This summer, Vox published a critique of health journalism with solid recommendations for better reporting from both researchers and journalists. These include evidence-based press releases from researchers and better training in research methods for journalists. Vox also has a new Show Me the Evidence section that analyzes research on various subjects, from e-cigarettes to exercise.
- In August, The Center for Health Journalism analyzed reporting on statistically nonsignificant results, specifically in regard to breast cancer treatments.
- Finally, Health News Review cites its ten years of experience in evaluating 2,000 health stories, finding “a clear pattern of imbalanced media messages – often exaggerating or emphasizing potential benefits, minimizing or ignoring potential harms, and ignoring cost issues.” They set out ten criteria for reviewing and evaluating health stories.