Stepping back and looking at the recent scientific literature, I am struck with how life is a miracle. How could humans have evolved in the face of threats from red meat, from eggs, even from the dangers of shaving? (If you write about nutrition you have to create a macro that types out “I’m not making this up:” the Caerphilly Study [1] shows you the dangers of shaving… or is it the dangers of not shaving?). With 28% greater risk of diabetes here, 57 % greater risk of heart disease there how could our ancestors have ever come of child-bearing age? With daily revelations from the Harvard School of Public Health showing the Scylla of saturated fat and the Carybdis of sugar between which our forefathers sailed, it is amazing that we are here.
These studies that the media writes about, are they real? They are certainly based on scientific papers. If the media is not always able to decipher them, reporters do generally talk to the researchers. And the papers must have gone through peer review and yet many actually defy common sense. Can the medical literature have such a high degree of error? Could there be a significant number of medical researchers who are not doing credible science? How can the consumer decide? I am going to try to answer these questions. When people ask questions like “could the literature be wrong?,” the answer is usually “yes” and I will try to explain what’s wrong and how to read the nutritional literature in a practical way. I am going to try to make it simple. It is science, but it is pretty simple science. I am going to illustrate the problem with the example of a paper by Djoussé [2]. But first, a joke.
It was a dumb joke. In my childhood, there was the idea, probably politically incorrect, that Indians, that is, Native Americans, always said “how” as a greeting. The joke was about an Indian with a great memory who is asked what he had for breakfast on New Years day the previous year. He says “eggs.” They are then interrupted by an earthquake or some natural disaster and the interviewer and the Indian don’t meet again for ten years. When they meet, the interviewer says “how.” The Indian answers “scrambled.”
If the interviewer had been an epidemiologist he might have asked if he had developed diabetes. Djoussé, et al. [2] asked participants about how many eggs they ate and then ten years later, if they developed diabetes it was assumed to be because of the eggs. Is this for real? Do eggs cause changes in your body that accumulate until you develop a disease, a disease that is, after all, primarily one of carbohydrate intolerance? The condition is due either to the inability of the pancreas to produce insulin in response to carbohydrate (type 1) or to impaired response of the body to the insulin produced and a deterioration of the insulin-producing cells of the pancreas (type 2). Common sense says that there is something suspicious about the idea that eggs would play a major role. It is worth trying to understand the methodology and see if there is a something beyond common sense, and whether this is a problem in other studies besides Djoussé’s.
What did the experimenters actually do. First, people were specifically asked “to report how often, on average, they had eaten one egg during the past year,” and “classified each subject into one the following categories of egg consumption: 0, < 1 per week, 1 per week, 2-4 per week, 5-6 per week, and 7+ eggs per week.” They collected this data every two years for ten years. With this baseline data in hand they then followed subjects “from baseline until the first occurrence of a) type 2 diabetes, b) death, or c) censoring date, the date of receipt of the last follow-up questionnaire” which for men was up to 20 years. Thinking back over a year: is there any likelihood that you might not be able to remember whether you had 1 vs. 2 eggs on average during the year? Is there any possibility that some of the men who were diagnosed with diabetes ten years after their report on eggs changed their eating pattern in the course of ten years? Are you eating the same food you ate ten years ago? Quick, how many eggs/week did you eat last year?
Reading a scientific paper: the Golden Rule.
Right off, there is a problem in people reporting what they ate but this is a limitation of many nutritional studies and, while a source of error, it is depends on how you interpret the data. All scientific measurements have error. It is not a matter of ignoring the data but rather not interpreting results beyond measurement. So, here’s how I read a scientific paper. First, I look for the pictures. What? A professor of biochemistry looks for the pictures first? In a scientific paper, of course, they are called figures but it’s not just saving a thousand words. (I get a thousand emails every couple of weeks). It’s about presentation of the data.
The principle is that a scientific paper is supposed to explain. The principle is laid out in what I call the golden rule of scientific papers. It comes from the Book PDQ Statistics by Geoffrery Norman and David Streiner. PDQ stands for Pretty Darned Quick and some of the humor is pretty sophomoric (e.g. it has Convoluted Reasoning or Anti-intellectual Pomposity detectors) but it is an excellent introductory statistics book. Here’s the Golden Rule:
“The important point…is that the onus is on the author to convey to the reader an accurate impression of what the data look like, using graphs or standard measures, before beginning the statistical shenanigans. Any paper that doesn’t do this should be viewed from the outset with considerable suspicion.”
— Norman & Streiner, PDQ Statistics [3]
In other words: teach. Make it clear. Eye-balling Djoussé, et al., we see that there are no figures. A graph of number of eggs consumed vs number of cases of diabetes is what would be expected of the golden rule. The results, instead are stated in the Abstract of the paper as the following mind-numbing statistics. (You don’t really have to read this)::
“Compared with no egg consumption, multivariable adjusted hazard ratios (95% CI) for type 2 diabetes were 1.09 (0.87-1.37), 1.09 (0.88-1.34), 1.18 (0.95-1.45), 1.46 (1.14-1.86), and 1.58 (1.25-2.01) for consumption of <1, 1, 2-4, 5-6, and 7+ eggs/week, respectively, in men (p for trend <0.0001). Corresponding multivariable hazard ratios (95% CI) for women were 1.06 (0.92-1.22), 0.97 (0.83-1.12), 1.19 (1.03-1.38), 1.18 (0.88-1.58), and 1.77 (1.28-2.43), respectively (p for trend <0.0001).”
What does all this mean? I will just state what the statistics mean because it is worth considering the conclusion as stated by the authors.
The meaning of the statistics is that there was no risk of consuming 1 egg/week compared to eating none. Similarly, there was no risk in eating 2-4 eggs/week or 5-6 eggs/week. But when you up your intake to 7 eggs or more per week, that’s it. Now, you are at risk for diabetes. The relative risk is small but there it is. You are now at greater risk.
Since I like pictures, I will try to illustrate this with a modified still from the movie, The Seventh Seal directed by Ingemar Bergman. Very popular in the fifties and sixties, these movies had a captivating if pretentious style: they sometimes seemed to be designed for Woody Allen’s parodies. One of the famous scenes in The Seventh Seal is the protagonist’s chess game with Death. A little PhotoShop and we have a good feel for what happens if you go beyond 5-6 eggs/week.
1. Ebrahim S, Smith GD, May M, Yarnell J: Shaving, coronary heart disease, and stroke: the Caerphilly Study. Am J Epidemiol 2003, 157(3):234-238.
2. Djoussé L, Gaziano JM, Buring JE, Lee IM: Egg consumption and risk of type 2 diabetes in men and women. Diabetes Care 2009, 32(2):295-300.
3. Norman GR, Streiner DL: PDQ statistics. 3rd edition. Hamilton, Ont.: B.C. Decker; 2003.
Richard David Feinman 