A SKEPTIC'S GUIDE TO HEALTH NEWS AND DIET FADS; A BOGUS STUDY TREATED AS REAL

In this weekend's NPR on the media, "a skeptic's guide to health news and diet fads" was discussed. It mentioned a story that Johannes Bohannon deliberately designed a bad bogus study to test how the bogus study results was published and cited in the news. See the blog article "I Fooled Millions Into Thinking Chocolate Helps Weight Loss. Here's How" and the news "Study showing that chocolate can help with weight loss was a trick to show how easily shoddy science can make headlines". Just today, I read an article titled "Could Too Many Refined Carbs Make You Depressed?", which is most like a bogus study.

Here is the story from Wikipedia:

Publishing under the name Johannes Bohannon, he produced a deliberately bad study to see how the media would pick up their findings. He worked with a film-maker Peter Onneken who was making a film about junk science in the diet industry with fad diets becoming headline news despite terrible study design and almost no evidence.

Bohannon designed a deliberately bad study with a small sample size, many variables that naturally fluctuate in participants, and a statistician told to deliberately "massage the data" using overfitting and p-hacking. The study's sample size was tiny, measuring 18 different measurements from only 15 participants, who were split into three groups. The purported finding of the study was that eating chocolate could assist weight loss. The GP running the study sums up his dislike of food pseudoscience as a "religion" that teaches “Bitter chocolate tastes bad, therefore it must be good for you.” Two thirds of the participants were female, and natural weight changes due to menstrual cycles were greater than the observed difference between chocolate and low-carb groups. The group who were assigned to the "control" were not asked what their diet contained.

He submitted the manuscript to 20 open access publishers well known for their predatory journals; the article ended up published in the International Archives of Medicine. He invented a fake "diet institute" that lacks even a website, and used the pen name, "Johannes Bohannon," a name that does not have any publications or appear on any website. Bohannon fabricated a press release which was picked up on the front cover of German tabloid Bild, as well as "the Daily Star, the Irish Examiner, Cosmopolitan’s German website, the Times of India, both the German and Indian site of the Huffington Post, and even television news in Texas and an Australian morning talk show."

The few journalists who contacted the scientist asked puff piece questions and no reporter published how many subjects were tested, or quoted independent researchers. Most outlets sought to maximise page views by including "vaguely pornographic images of women eating chocolate." He argues that diet fads are covered like gossip columnists "echoing whatever they find in press releases" rather than evaluating the accuracy of scientific papers.

Bohannon argues that because of the large number of factors in diet and lifestyle, large scale studies are frequently inconclusive, even when billions of dollars have been spent on well-designed studies by government agencies that label obesity an epidemic.

The original paper "chocolate with high cocoa content as a weight-loss accelerator" can be read at: http://www.scribd.com/doc/266969860/Chocolate-causes-weight-loss. The statistics section of the paper is below. Looks real, right?

A t-test for independent samples was used to assess differences in baseline variables between the groups. The analysis was a repeated-measures analysis of variance in which the baseline value was carried forward in the case of missing data. One subject (low-carbohydrate) had to be excluded from the analysis, because of a weight measure is-sue within the trial

Unfortunately, in today's world, a lot of published studies were based on the bad science. The bogus studies can be written and published as if it is a real study. The news media would pick it up, disseminate and broadcast it like the great news.

The podcast from PBS is available below.