Damned statistics and the nutrition literature

Posted: January 3, 2011 in Uncategorized
Tags: , ,

A guide for consumers and the media

For Mark Twain’s hierarchy of lies, damned lies and statistics, we should really add epidemiological lies, those reports showing that brown rice or trans-palmitoleic acid will prevent diabetes and diet soda will make you fat, which appear every week or so in ABCNews.  (I mean the generic media, but ABCNews and I have a close relationship: sometimes they even print what I tell them).  If you’ve been eating white rice instead of brown rice and you develop diabetes ten years later, it is the fault of your choice of rice. Everybody knows that this is ridiculous but the data are there showing an almost 4-fold increased risk, so how can you argue with the numbers.

These kinds of studies are always based on associations and the authors are usually quick to tell you that association doesn’t mean causality even as they interpret the data as a clear guide to action (“Substitution of whole grains, including brown rice, for white rice may lower risk of type 2 diabetes.”)   In fact, to most scientists, association can be a strong argument for causality.  That is not what’s wrong with them. Philosophically speaking, there are only associations.  All we really know is that there is a stream of particles and there is an association between the presence of a magnet and the appearance of a spot on a piece of photographic paper (anybody remember photographic paper?).  God does not whisper in your ear that the particle has a magnetic moment.  It is the strength of the idea behind the association and the presentation of the idea that determines whether the association implies causality.  What most people really mean is that “association does not necessarily imply causality.  You may need more information.” What’s wrong with the rice story is that the idea is lacking in common sense.  The idea that the type of rice you eat has any meaningful impact by itself, or even whether one can guess whether it has a positive or negative impact on a general lifestyle, is absurd.  But what about the statistics? Here the problem is really presentation of the data.  The number of papers in the literature pointing out the errors in interpretation of statistics is very large although it is still less than the number of papers making those errors.  There are numerous problems and many examples but let’s look at the simplest case: limitations of reporting relative risk and alternatives.

images-1Here’s a good example cited in a highly recommended popular statistics books, Gerd Gigerenzer’s “Calculated Risks.” He discusses a real case, the West of Scotland Coronary Prevention Study (WOSCOPS) comparing the statin drug, pravastatin to placebo in people with high cholesterol.  The study was started in 1989 and went on for about 5 years.  (These days, I think you can only compare different statins; everybody is so convinced that they are good that a placebo would be considered unethical):

1. First, the press release: “People with high cholesterol can rapidly reduce… their risk of death by 22 per cent by taking…pravastatin.”

2. Now, ask yourself what this means? If 1000 people with high cholesterol take pravastatin, how many people will be saved from a heart attack that might have otherwise killed them?  Think about this, then look at the data, the data that should have  been reported in the media.

3. The data:

Treatment        deaths during 5 years (per 1000 people with high cholesterol)

pravastatin             32

placebo                  41

Right off, it doesn’t look as good as you might have thought.  Overall, death from a heart attack is a major killer, but if you take a thousand people and watch them for five years, not that many people die from a heart attack. Now there are three standard ways of representing the data.

4. Data presentation – Relative risk reduction.

Risk is the number of cases divided by total number of people in the trial (or risk per total number). So you calculate a risk for 1000 people on the drug = 32/1000 = 03.2 % and similarly for people on the statin. Risk reduction for comparing treatments is\ the difference between the two risks.  The relative risk reduction here  is just the reduction in risk divided by the risk for the placebo:

Risk reduction (number of people saved per thousand)  = 41-32 = 9. Saving 9 lives doesn’t sound that great but lets get the per cent as reported.

Relative risk reduction = 9/41 = 22 % as indicated, and it does sound like a big deal but there are other ways to look at the data.

5.  Data presentation – Absolute risk reduction.  Again, you start with risk, the number of cases divided by total number but you calculate the actual fraction.  The absolute risk reduction is the difference between these two fractions.

For pravastatin, risk = 32/1000

For placebo, risk = 41/1000

Absolute risk reduction = (41/1000) – (32/1000) = 9/1000 = 0.9 % (less than 1 %)

6.  Data presentation – Number needed to treat (NNT): This is a good indicator of outcomes.  If you treat 1000 people, 9 will survive who might have otherwise died. So,

number that you have to treat  to save one life = NNT  =  1000/9 = 111 people .

7. Conclusion: 22 % risk reduction is true enough but it seems like it didn’t really tell you what you want to know.  Cutting to the chase, would you take a statin if you had high cholesterol (more than about 250 mg/dl) and, as in WOSCOPS, no history of heart attacks. On the basis of this study alone, it’s not clear.  First, the risk is low.  There is clearly a benefit but how predictable is that benefit?  In the study, 99 % of the people had no benefit.  Of course, if you are the one out of a hundred, the drug would be a good thing.  The question is not easy to answer but the point of what’s written here is that the statistics as reported in the media might have led you to jump to conclusions.  Before you jump, though, you might ask about side-effects.  This is a complicated subject because although the side-effects are rare, their incidence is not zero and they can be severe but this post is only about the statistics.


Comments
  1. […] kind of numbers and what kind of variation we are talking about.” It should be noted that Dr. Feinman was one of the co-authors of the critical review I alluded to in the previous paragraph, “In […]

  2. Mie says:

    “These days, I think you can only compare different statins; everybody is so convinced that they are good that a placebo would be considered unethical”

    A large number of successful RCT’s might have something to do with that. 🙂

    • rdfeinman says:

      I am not sure what the point is here. I don’t see that all statin studies are so well done or so compelling that we don’t need placebo. The ethics is determined by informed consent. I wouldn’t volunteer because I might get the statin and I see little evidence that I am the sort of person who might benefit.

    • James says:

      “A large number of successful RCT’s might have something to do with that”
      Care to elaborate on that. I know of none that stand up to closer scrutiny and seem to somehow or other always tainted by other ‘interests’ or really do nothing that an aspirin and extra Vitamin D would not accomplish. Without the nasty side effects.

      • rdfeinman says:

        I don’t know where the quotation is from. Did I say that? I am certainly not a fan of RCTs but, again, it is what you do with it. Can you explain the quotation source?
        …Oh, I see. It is somebody else’s comment. I am in agreement that “successful RCTs” are smaller in number than one might think. That is the point of the post.

  3. Mie says:

    “I don’t see that all statin studies are so well done or so compelling that we don’t need placebo.”

    You are entitled to your opinion. However, it isn’t really valid. The large statin trials stand up to scrutiny. You are of course free to explain why you think differently.

    “I wouldn’t volunteer because I might get the statin and I see little evidence that I am the sort of person who might benefit.”

    Well, then don’t volunteer. I fail to see the relevance of this in … well, anything.

    The same goes for this post, which doesn’t really contain anything anyone concerned with treating CVD patients doesn’t already know. But if you wanted to explain ARR and RRR to laymen, that’s fine too.

    • Colleen says:

      “The same goes for this post, which doesn’t really contain anything anyone concerned with treating CVD patients doesn’t already know.”

      I’m glad you are so well-informed. Maybe it’s a good idea for the rest of us to understand this also.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s