A SKEPTIC'S GUIDE TO HEALTH NEWS AND DIET FADS; A BOGUS STUDY TREATED AS REAL

In this weekend's NPR on the media, "a skeptic's guide to health news and diet fads" was discussed. It mentioned a story that Johannes Bohannon deliberately designed a bad bogus study to test how the bogus study results was published and cited in the news. See the blog article "I Fooled Millions Into Thinking Chocolate Helps Weight Loss. Here's How" and the news "Study showing that chocolate can help with weight loss was a trick to show how easily shoddy science can make headlines". Just today, I read an article titled "Could Too Many Refined Carbs Make You Depressed?", which is most like a bogus study.

Here is the story from Wikipedia:

Publishing under the name Johannes Bohannon, he produced a deliberately bad study to see how the media would pick up their findings. He worked with a film-maker Peter Onneken who was making a film about junk science in the diet industry with fad diets becoming headline news despite terrible study design and almost no evidence.

Bohannon designed a deliberately bad study with a small sample size, many variables that naturally fluctuate in participants, and a statistician told to deliberately "massage the data" using overfitting and p-hacking. The study's sample size was tiny, measuring 18 different measurements from only 15 participants, who were split into three groups. The purported finding of the study was that eating chocolate could assist weight loss. The GP running the study sums up his dislike of food pseudoscience as a "religion" that teaches “Bitter chocolate tastes bad, therefore it must be good for you.” Two thirds of the participants were female, and natural weight changes due to menstrual cycles were greater than the observed difference between chocolate and low-carb groups. The group who were assigned to the "control" were not asked what their diet contained.

He submitted the manuscript to 20 open access publishers well known for their predatory journals; the article ended up published in the International Archives of Medicine. He invented a fake "diet institute" that lacks even a website, and used the pen name, "Johannes Bohannon," a name that does not have any publications or appear on any website. Bohannon fabricated a press release which was picked up on the front cover of German tabloid Bild, as well as "the Daily Star, the Irish Examiner, Cosmopolitan’s German website, the Times of India, both the German and Indian site of the Huffington Post, and even television news in Texas and an Australian morning talk show."

The few journalists who contacted the scientist asked puff piece questions and no reporter published how many subjects were tested, or quoted independent researchers. Most outlets sought to maximise page views by including "vaguely pornographic images of women eating chocolate." He argues that diet fads are covered like gossip columnists "echoing whatever they find in press releases" rather than evaluating the accuracy of scientific papers.

Bohannon argues that because of the large number of factors in diet and lifestyle, large scale studies are frequently inconclusive, even when billions of dollars have been spent on well-designed studies by government agencies that label obesity an epidemic.

The original paper "chocolate with high cocoa content as a weight-loss accelerator" can be read at: http://www.scribd.com/doc/266969860/Chocolate-causes-weight-loss. The statistics section of the paper is below. Looks real, right?

A t-test for independent samples was used to assess differences in baseline variables between the groups. The analysis was a repeated-measures analysis of variance in which the baseline value was carried forward in the case of missing data. One subject (low-carbohydrate) had to be excluded from the analysis, because of a weight measure is-sue within the trial

Unfortunately, in today's world, a lot of published studies were based on the bad science. The bogus studies can be written and published as if it is a real study. The news media would pick it up, disseminate and broadcast it like the great news.

The podcast from PBS is available below.

An early clinical trial with N=2

In the late 18th century, King Gustav III of Sweden decided that coffee was poison and ordered a clinical trial.

• The King condemned a convicted murderer to drink coffee every day.
• The control was another murderer who was condemned to drink tea daily.
• The outcome measure is 'death'.
• Two physicians were appointed to determine the outcome.

Results:

• The two doctors died first
• The king was murdered
• Both convicts enjoyed long life until the tea drinker died at age 83. (no age was given for the coffee drinker)

J Int Med. Oct 1991:289 - introduction to editorial from Nordic School of Public Health, Goteborg Sweden

Reprinted in Annal of Internal Medicine 1992: 117:30

A joke about statistician

Even thoug this is a joke, I find sometimes it is very true.

A manager wanted to hire someone to work in her department and she had applications from a lawyer, an archeologist, and a statistician. She invited the lawyer for an interview and asked just a single question, "What is one plus one?" The lawyer explained "In matters of commercial law, the Supreme Court precedent in 1867 established that, in matters of commercial law, one plus one is considered to be prima facie evidence of equaling two." The manager was very impressed. Then she invited the archeologist for an interview and asked the same question. The archeologist replied, "Although many ancient cultures had solid concepts of numbers, the concept of addition was first established in the Egyptian culture, circa 2000 B.C. and it was at the tomb of Amenhotep that the first written evidence was found to establish that one plus one was equal to two." Again the manager was very impressed. She then invited the statistician in and asked "What is one plus one?" The statistician looked around very carefully, then went up and closed the door, lowered the window shades, and asked very quietly "What do you want it to be?"