Richard David Feinman

TIME: You’re partnering with, among others, Harvard University on this. In an alternate Lady Gaga universe, would you have liked to have gone to Harvard?

Lady Gaga: I don’t know. I am going to Harvard today. So that’ll do.

— Belinda Luscombe, Time Magazine, March 12, 2012

There was a sense of déja-vu about the latest red meat scare and I thought that my previous post as well as those of others had covered the bases but I just came across a remarkable article from the Harvard Health Blog. It was entitled “Study urges moderation in red meat intake.” It describes how the “study linking red meat and mortality lit up the media…. Headline writers had a field day, with entries like ‘Red meat death study,’ ‘Will red meat kill you?’ and ‘Singing the blues about red meat.”’

What’s odd is that this is all described from a distance as if the study by Pan, et al (and likely the content of the blog) hadn’t come from Harvard itself but was rather a natural phenomenon, similar to the way every seminar on obesity begins with a slide of the state-by-state development of obesity as if it were some kind of meteorologic event.

When the article refers to “headline writers,” we are probably supposed to imagine sleazy tabloid publishers like the ones who are always pushing the limits of first amendment rights in the old Law & Order episodes.  The Newsletter article, however, is not any less exaggerated itself. (My friends in English Departments tell me that self-reference is some kind of hallmark of real art). And it is not true that the Harvard study was urging moderation. In fact, it is admitted that the original paper “sounded ominous. Every extra daily serving of unprocessed red meat (steak, hamburger, pork, etc.) increased the risk of dying prematurely by 13%. Processed red meat (hot dogs, sausage, bacon, and the like) upped the risk by 20%.” That is what the paper urged. Not moderation. Prohibition. Who wants to buck odds like that? Who wants to die prematurely?

It wasn’t just the media. Critics in the blogosphere were also working over-time deconstructing the study.  Among the faults that were cited, a fault common to much of the medical literature and the popular press, was the reporting of relative risk.

The limitations of reporting relative risk or odds ratio are widely discussed in popular and technical statistical books and I ran through the analysis in the earlier post. Relative risk destroys information.  It obscures what the risks were to begin with.  I usually point out that you can double your odds of winning the lottery if you buy two tickets instead of one. So why do people keep doing it?  One reason, of course, is that it makes your work look more significant.  But, if you don’t report the absolute change in risk, you may be scaring people about risks that aren’t real. The nutritional establishment is not good at facing their critics but on this one, they admit that they don’t wish to contest the issue.

Nolo Contendere.

“To err is human, said the duck as it got off the chicken’s back”

 — Curt Jürgens in The Devil’s General

Having turned the media loose to scare the American public, Harvard now admits that the bloggers are correct.  The Health NewsBlog allocutes to having reported “relative risks, comparing death rates in the group eating the least meat with those eating the most. The absolute risks… sometimes help tell the story a bit more clearly. These numbers are somewhat less scary.” Why does Dr. Pan not want to tell the story as clearly as possible?  Isn’t that what you’re supposed to do in science? Why would you want to make it scary?

The figure from the Health News Blog:

	Deaths per 1,000 people per year
	1 serving unprocessed meat a week	2 servings unprocessed meat a day
Women	7.0	8.5
	3 servings unprocessed meat a week	2 servings unprocessed meat a day
Men	12.3	13.0

Unfortunately, the Health Blog doesn’t actually calculate the  absolute risk for you.  You would think that they would want to make up for Dr. Pan scaring you.   Let’s calculate the absolute risk.  It’s not hard.Risk is usually taken as probability, that is, number cases divided by total number of participants.  Looking at the men, the risk of death with 3 servings per week is equal to the 12.3 cases per 1000 people = 12.3/1000 = 0.1.23 = 1.23 %. Now going to 14 servings a week (the units in the two columns of the table are different) is 13/1000 = 1.3 % so, for men, the absolute difference in risk is 1.3-1.23 = 0.07, less than 0.1 %.  Definitely less scary. In fact, not scary at all. Put another way, you would have to drastically change the eating habits (from 14 to 3 servings) of 1, 429 men to save one life.  Well, it’s something.  Right? After all for millions of people, it could add up.  Or could it?  We have to step back and ask what is predictable about 1 % risk. Doesn’t it mean that if a couple of guys got hit by cars in one or another of the groups whether that might not throw the whole thing off? in other words, it means nothing.

Observational Studies Test Hypotheses but the Hypotheses Must be Testable.

It is commonly said that observational studies only generate hypotheses and that association does not imply causation.  Whatever the philosophical idea behind these statements, it is not exactly what is done in science.  There are an infinite number of observations you can make.  When you compare two phenomena, you usually have an idea in mind (however much it is unstated). As Einstein put it “your theory determines the measurement you make.”  Pan, et al. were testing the hypothesis that red meat increases mortality.  If they had done the right analysis, they would have admitted that the test had failed and the hypothesis was not true.  The association was very weak and the underlying mechanism was, in fact, not borne out.  In some sense, in science, there is only association. God does not whisper in our ear that the electron is charged. We make an association between an electron source and the response of a detector.  Association does not necessarily imply causality, however; the association has to be strong and the underlying mechanism that made us make the association in the first place, must make sense.

What is the mechanism that would make you think that red meat increased mortality.  One of the most remarkable statements in the original paper:

“Regarding CVD mortality, we previously reported that red meat intake was associated with an increased risk of coronary heart disease^{2, 14} and saturated fat and cholesterol from red meat may partially explain this association.  The association between red meat and CVD mortality was moderately attenuated after further adjustment for saturated fat and cholesterol, suggesting a mediating role for these nutrients.” (my italics)

This bizarre statement — that saturated fat played a role in increased risk because it reduced risk— was morphed in the Harvard News Letters plea bargain to “The authors of the Archives paper suggest that the increased risk from red meat may come from the saturated fat, cholesterol, and iron it delivers;” the blogger forgot to add “…although the data show the opposite.” Reference (2) cited above had the conclusion that “Consumption of processed meats, but not red meats, is associated with higher incidence of CHD and diabetes mellitus.” In essence, the hypothesis is not falsifiable — any association at all will be accepted as proof. The conclusion may be accepted if you do not look at the data.

The Data

In fact, the data are not available. The individual points for each people’s red meat intake are grouped together in quintiles ( broken up into five groups) so that it is not clear what the individual variation is and therefore what your real expectation of actually living longer with less meat is.  Quintiles are some kind of anachronism presumably from a period when computers were expensive and it was hard to print out all the data (or, sometimes, a representative sample).  If the data were really shown, it would be possible to recognize that it had a shotgun quality, that the results were all over the place and that whatever the statistical correlation, it is unlikely to be meaningful in any real world sense.  But you can’t even see the quintiles, at least not the raw data. The outcome is corrected for all kinds of things, smoking, age, etc.  This might actually be a conservative approach — the raw data might show more risk — but only the computer knows for sure.

Confounders

“…mathematically, though, there is no distinction between confounding and explanatory variables.”

  — Walter Willett, Nutritional Epidemiology, 2^o edition.

You make a lot of assumptions when you carry out a “multivariate adjustment for major lifestyle and dietary risk factors.”   Right off , you assume that the parameter that you want to look at — in this case, red meat — is the one that everybody wants to look at, and that other factors can be subtracted out. However, the process of adjustment is symmetrical: a study of the risk of red meat corrected for smoking might alternatively be described as a study of the risk from smoking corrected for the effect of red meat. Given that smoking is an established risk factor, it is unlikely that the odds ratio for meat is even in the same ballpark as what would be found for smoking. The figure below shows how risk factors follow the quintiles of meat consumption.  If the quintiles had been derived from the factors themselves we would have expected even better association with mortality.

The key assumption is that the there are many independent risk factors which contribute in a linear way but, in fact, if they interact, the assumption is not appropriate.  You can correct for “current smoker,” but biologically speaking, you cannot correct for the effect of smoking on an increased response to otherwise harmless elements in meat, if there actually were any.  And, as pointed out before, red meat on a sandwich may be different from red meat on a bed of cauliflower puree.

This is the essence of it.  The underlying philosophy of this type of analysis is “you are what you eat.” The major challenge to this idea is that carbohydrates, in particular, control the response to other nutrients but, in the face of the plea of nolo contendere,  it is all moot.

Who paid for this and what should be done.

We paid for it. Pan, et al was funded in part by 6 NIH grants.  (No wonder there is no money for studies of carbohydrate restriction).  It is hard to believe with all the flaws pointed out here and, in the end, admitted by the Harvard Health Blog and others, that this was subject to any meaningful peer review.  A plea of no contest does not imply negligence or intent to do harm but something is wrong. The clear attempt to influence the dietary habits of the population is not justified by an absolute risk reduction of less than one-tenth of one per cent, especially given that others have made the case that some part of the population, particularly the elderly may not get adequate protein. The need for an oversight committee of impartial scientists is the most important conclusion of Pan, et al.  I will suggest it to the NIH.