How Feedback Biases Give Ineffective Medical Treatments a Good Reputation

This paper is in the following e-collection/theme issue:

Physician and Health Services Rating by Consumers 

Advertisement: Preregister now for the Medicine 2.0 Congress

Post to Mendeley
Post to Cite U Like

Article Thumbnail

Members can download this full article for Adobe PDF Format.
Membership provides unlimited access to all PDF files.
Or, ask your department head to become an institutional member.

For tax purposes please select your country and if applicable state/province of residence:

Buy Now (Pay-per-download for non-members):

Download Price (USD): $22.00

Original Paper

How Feedback Biases Give Ineffective Medical Treatments a Good Reputation

Mícheál de Barra, PhD;
Kimmo Eriksson, PhD;
Pontus Strimling, PhD

Centre for the Study of Cultural Evolution, Stockholm, Sweden

Corresponding Author:
Mícheál de Barra, PhD

Centre for the Study of Cultural Evolution
Wallenberg Laboratory
Stockholm, WC1E 7HT
Phone: 46 7531327690
Fax: 46 7531327690

1,2], traditional medicines [3,4], and historical “establishment” medicine like bloodletting [5]. It is also likely to be true of some contemporary medical treatments [6-8]. Treatments may be harmful either to the patient directly or cause harm because they replace other effective treatments, or result in broader environmental harms, as in the case of drugs derived from endangered species [2-4].

Medical treatments are very much cultural traits: rather than being invented anew by each individual, they spread from person to person through cultural processes. The prevalence of poor medical treatments is an anomalous outcome of cultural evolution because culturally acquired information in other domains of life is generally reliable and beneficial. Indeed, the extraordinary ecological success of the human species is, in part, due to our reliance on adaptive cultural information [9]. It is clearly true that humans routinely use cultural information to solve complex problems that, like medicine, entail delayed and/or stochastic feedback. The adaptive value of cultural information is thought to result from a number of mechanisms, such as learning heuristics whereby people selectively imitate more successful people, filtering whereby people evaluate the quality of socially acquired traits through experimentation, and natural selection whereby people with more beneficial cultural traits have more children who then learn these traits [10-12].

Some traditional medicines did have a direct benefit for the patient. Effective variolation, for example, was surprisingly common. For example, Yorba healers in West Africa carried smallpox scabs that could be used to induce a non-lethal infection and resultant immunity [13]. A number of vaccination techniques were being employed in 17th century India and China, and Edward Jenner’s vaccination was long a part of English folk medicine [14]. Some globally important pharmaceutical products have their origins in traditional medicine; Artemisinin, a key anti-malaria drug, was part of ancient Chinese medicine [15]. Moreover, medicine—be it allopathic, traditional, or ancient—is not just about altering the course of disease. Medical experts will often have seen many people with similar diseases and thus they can help patients to understand what their illness is (diagnosis) and how it will play out over time (prognosis). For an anxious patient and his or her family, these are important services and they were probably carried out with some sophistication throughout history and across cultures. Moreover, by identifying and validating illness, medical experts may help the ill to garner social support and thus enable crucial rest and recuperation.

It is also clearly true that patients have undergone surgeries, ingested substances, and been subjected to a litany of other treatments with the explicit expectation that they would be helped. These expectations were not justified: the disease course was unaffected and/or the patient was directly harmed by the treatment. Ineffective treatments were common and remain common, and they warrant study [5]. Why then do harmful and non-beneficial medical treatments spread and persist?

We propose the following explanation. Irrespective of effectiveness, medical treatments typically result in a distribution of outcomes with some people improving, some deteriorating, and others experiencing little change. Suppose that the people who have more positive outcomes are more inclined to tell other people about their experience of the treatment than people who have poorer outcomes. This may occur because people recall their successes better than their failures, because people believe others’ success stories, or because people are embarrassed to have adopted an ineffective treatment. Whatever the cause, such a bias would systematically distort the information available to other naive individuals who are seeking an effective treatment—the reputation of a treatment will exceed its real effect.

This hypothesis is assessed using a variety of methods. First, we compared clinical data on weight loss diets with weight loss reported in reviews of books on these diets. Reviews were taken from Amazon, a popular online marketplace where consumers can post reviews of products. We also made a similar comparison for unproven fertility treatments based on herbs and vitamins. In both cases, we predicted that people with positive outcomes are more inclined to post reviews. In a series of experimental studies, we then tested whether the bias of such reviews is sufficient to influence preferences for treatments. We predicted a preference for weight loss diets accompanied by typical reviews (as sampled from Amazon) over diets accompanied by undistorted reviews (ie, reviews that are representative of the diet’s true effect obtained by purposefully sampling and/or editing of the review). Finally, we used a mathematical model to explore some implications of such reputational distortion.

Study 1: Weight-Loss Diets

In order to make the Amazon and clinical data directly comparable, we made several assumptions and simplifications. Readers interested in conducting alternative analyses or comparisons can access the raw data and analysis syntax from the figshare data repository [16].

The Atkins Diet has been tested in several clinical trials and is the most commonly reviewed diet book on the Amazon online bookstore. We downloaded the 1359 reviews written on or before November 18, 2012. We extracted the duration of the diet and the total weight change from each diet review where this information was provided. If weight change at two time-points was mentioned (eg, 1 kg loss after 1 week and a 3 kg after 1 month), only the longer duration and associated weight change was recorded. If the review described the experiences of more than one person, only information about the author was recorded. If the review only discussed the weight change of a person besides the author, then that person’s weight change was recorded. In total, 587 reviews included both a weight change and a time period over which this change occurred. The median diet duration was 42 days. To calculate an average weight loss at 1, 2, 3, 4, 5, 6, 9, and 12 months, we averaged the reports nearest each of these points in time. We excluded reviews of diets that lasted less than 2 weeks or more than 15 months.

The “true” effects of the Atkins diet were assessed using three clinical trials [17-19] in which participants received the Atkins diet book. In two of these trials [18,19], the intervention also entailed meeting a dietitian to discuss the diet and the participant’s progress. Basic information about average weight loss in the Atkins diet arm could be extracted from the published manuscript, but to assess the distribution of outcomes, individual level data were needed. Only Gardner et al [18] were willing and able to share their raw data. The Gardner trial examined weight change among 311 premenopausal overweight and obese women, 77 of which were randomly allocated to the Atkins diet. Participants received the Atkins book and met in groups of six once per week for 8 weeks to discuss the diet and book with a dietitian. Although Amazon reviewers are not all premenopausal women, Figure 1 shows that the average effect of the Atkins diet is broadly similar in several different populations. Moreover, given that the intervention involved reading the books and meeting with a dietitian, the clinical trial weight loss levels are likely to exceed that found in the general population. We compared the clinical weight change at 2, 6, and 12 months with Atkins reviews written between 1.5 and 2.5 months, 5 and 7 months, and 9 and 15 months respectively.

Comments Off on How Feedback Biases Give Ineffective Medical Treatments a Good Reputation

Tags: ,