Stepping back and looking at the recent scientific literature, I am struck with how life is a miracle. How could humans have evolved in the face of threats from red meat, from eggs, even from the dangers of shaving? (If you write about nutrition you have to create a macro that types out “I’m not making this up:” the Caerphilly Study [1] shows you the dangers of shaving… or is it the dangers of not shaving?). With 28% greater risk of diabetes here, 57 % greater risk of heart disease there how could our ancestors have ever come of child-bearing age? With daily revelations from the Harvard School of Public Health showing the Scylla of saturated fat and the Carybdis of sugar between which our forefathers sailed, it is amazing that we are here.
These studies that the media writes about, are they real? They are certainly based on scientific papers. If the media is not always able to decipher them, reporters do generally talk to the researchers. And the papers must have gone through peer review and yet many actually defy common sense. Can the medical literature have such a high degree of error? Could there be a significant number of medical researchers who are not doing credible science? How can the consumer decide? I am going to try to answer these questions. When people ask questions like “could the literature be wrong?,” the answer is usually “yes” and I will try to explain what’s wrong and how to read the nutritional literature in a practical way. I am going to try to make it simple. It is science, but it is pretty simple science. I am going to illustrate the problem with the example of a paper by Djoussé [2]. But first, a joke.
It was a dumb joke. In my childhood, there was the idea, probably politically incorrect, that Indians, that is, Native Americans, always said “how” as a greeting. The joke was about an Indian with a great memory who is asked what he had for breakfast on New Years day the previous year. He says “eggs.” They are then interrupted by an earthquake or some natural disaster and the interviewer and the Indian don’t meet again for ten years. When they meet, the interviewer says “how.” The Indian answers “scrambled.”
If the interviewer had been an epidemiologist he might have asked if he had developed diabetes. Djoussé, et al. [2] asked participants about how many eggs they ate and then ten years later, if they developed diabetes it was assumed to be because of the eggs. Is this for real? Do eggs cause changes in your body that accumulate until you develop a disease, a disease that is, after all, primarily one of carbohydrate intolerance? The condition is due either to the inability of the pancreas to produce insulin in response to carbohydrate (type 1) or to impaired response of the body to the insulin produced and a deterioration of the insulin-producing cells of the pancreas (type 2). Common sense says that there is something suspicious about the idea that eggs would play a major role. It is worth trying to understand the methodology and see if there is a something beyond common sense, and whether this is a problem in other studies besides Djoussé’s.
What did the experimenters actually do. First, people were specifically asked “to report how often, on average, they had eaten one egg during the past year,” and “classified each subject into one the following categories of egg consumption: 0, < 1 per week, 1 per week, 2-4 per week, 5-6 per week, and 7+ eggs per week.” They collected this data every two years for ten years. With this baseline data in hand they then followed subjects “from baseline until the first occurrence of a) type 2 diabetes, b) death, or c) censoring date, the date of receipt of the last follow-up questionnaire” which for men was up to 20 years. Thinking back over a year: is there any likelihood that you might not be able to remember whether you had 1 vs. 2 eggs on average during the year? Is there any possibility that some of the men who were diagnosed with diabetes ten years after their report on eggs changed their eating pattern in the course of ten years? Are you eating the same food you ate ten years ago? Quick, how many eggs/week did you eat last year?
Reading a scientific paper: the Golden Rule.
Right off, there is a problem in people reporting what they ate but this is a limitation of many nutritional studies and, while a source of error, it is depends on how you interpret the data. All scientific measurements have error. It is not a matter of ignoring the data but rather not interpreting results beyond measurement. So, here’s how I read a scientific paper. First, I look for the pictures. What? A professor of biochemistry looks for the pictures first? In a scientific paper, of course, they are called figures but it’s not just saving a thousand words. (I get a thousand emails every couple of weeks). It’s about presentation of the data.
The principle is that a scientific paper is supposed to explain. The principle is laid out in what I call the golden rule of scientific papers. It comes from the Book PDQ Statistics by Geoffrery Norman and David Streiner. PDQ stands for Pretty Darned Quick and some of the humor is pretty sophomoric (e.g. it has Convoluted Reasoning or Anti-intellectual Pomposity detectors) but it is an excellent introductory statistics book. Here’s the Golden Rule:
“The important point…is that the onus is on the author to convey to the reader an accurate impression of what the data look like, using graphs or standard measures, before beginning the statistical shenanigans. Any paper that doesn’t do this should be viewed from the outset with considerable suspicion.”
— Norman & Streiner, PDQ Statistics [3]
In other words: teach. Make it clear. Eye-balling Djoussé, et al., we see that there are no figures. A graph of number of eggs consumed vs number of cases of diabetes is what would be expected of the golden rule. The results, instead are stated in the Abstract of the paper as the following mind-numbing statistics. (You don’t really have to read this)::
“Compared with no egg consumption, multivariable adjusted hazard ratios (95% CI) for type 2 diabetes were 1.09 (0.87-1.37), 1.09 (0.88-1.34), 1.18 (0.95-1.45), 1.46 (1.14-1.86), and 1.58 (1.25-2.01) for consumption of <1, 1, 2-4, 5-6, and 7+ eggs/week, respectively, in men (p for trend <0.0001). Corresponding multivariable hazard ratios (95% CI) for women were 1.06 (0.92-1.22), 0.97 (0.83-1.12), 1.19 (1.03-1.38), 1.18 (0.88-1.58), and 1.77 (1.28-2.43), respectively (p for trend <0.0001).”
What does all this mean? I will just state what the statistics mean because it is worth considering the conclusion as stated by the authors.
The meaning of the statistics is that there was no risk of consuming 1 egg/week compared to eating none. Similarly, there was no risk in eating 2-4 eggs/week or 5-6 eggs/week. But when you up your intake to 7 eggs or more per week, that’s it. Now, you are at risk for diabetes. The relative risk is small but there it is. You are now at greater risk.
Since I like pictures, I will try to illustrate this with a modified still from the movie, The Seventh Seal directed by Ingemar Bergman. Very popular in the fifties and sixties, these movies had a captivating if pretentious style: they sometimes seemed to be designed for Woody Allen’s parodies. One of the famous scenes in The Seventh Seal is the protagonist’s chess game with Death. A little PhotoShop and we have a good feel for what happens if you go beyond 5-6 eggs/week.
1. Ebrahim S, Smith GD, May M, Yarnell J: Shaving, coronary heart disease, and stroke: the Caerphilly Study. Am J Epidemiol 2003, 157(3):234-238.
2. Djoussé L, Gaziano JM, Buring JE, Lee IM: Egg consumption and risk of type 2 diabetes in men and women. Diabetes Care 2009, 32(2):295-300.
3. Norman GR, Streiner DL: PDQ statistics. 3rd edition. Hamilton, Ont.: B.C. Decker; 2003.
Richard David Feinman 
Could it be that egg consumption is accompanied by consumption of refined-carb staples like bread and jelly? Ergo, the more eggs consumed, the more bread and jelly consumed.
If there really is a correlation it could be for the reason you say but I don’t think the correlation has been shown. I didn’t go into everything that is wrong in the assumptions in this paper but the method is to collect data into quintiles (losing much information), to incorrectly assume that the trajectory of the population (going from low to high) reflects individual trajectories rather than, say, some people reducing risk while others increasing risk and, in the end, you have to go to the extremes of egg consumption to see their claimed increase in risk and so they could have said: there is no risk for type 2 diabetes associated with egg consumption except at the extreme difference between very low and very high. Most of all, there is no reason to look at eggs in the context of diabetes and the lack of mechanism suggests the data would only tell you something if the numbers were big which they are not. Somebody should be funded to re-evaluate this data with an open mind and see what dietary factors show strong associations with diabetes. This paper does not show anything at all, either way.
[…] […]
“there is no risk for type 2 diabetes associated with egg consumption except at the extreme difference between very low and very high.”
Which is what comes across very clearly from the extract quoted from the paper. Academic language is not a synonym for statistical shenanigans.
“Most of all, there is no reason to look at eggs in the context of diabetes”
Well, how do you know that?
“and the lack of mechanism suggests the data would only tell you something if the numbers were big which they are not.”
But surely you understand that a study of this kind cannot show the exact mechanism? This can be inspected with a well-designed RCT. And how do you think the possible study design for a particular RCT could/should be formed, if not on the basis of issues discovered in cohort data?
First, this was a blog post, even included a joke. In fact, I think that is appropriate because, although I tried to present my arguments in a logical way, I think this is not an acceptable paper as it stands. If you think this paper was subjected to vigorous cross-examination, that is appropriate peer review, and that the conclusions are established without need to see more data, then you should quit eating all those eggs.
If I had reviewed the paper, I would have asked for individual data to be shown graphically. Why do we use quintiles (or quartiles, or whatever)? It is an anachronism from when computers were not on everybody’s desk. Quintiles obscures data. What you want to know is if you look at the people who eat more than 7 eggs a week, how many get diabetes?
“‘Most of all, there is no reason to look at eggs in the context of diabetes’
Well, how do you know that?”
If you think there is a reason, what is it? How do you know there is no reason to test pineapples? You do a correlation to test a hypothesis not, as is usually said, the other way around. I previously quoted Einstein as saying that your theory determines what you measure. Of course, sometimes you have a hunch and you see a previously unexpected strong association. But there is nothing strong here. In fact, I haven’t checked, but my recollection is that when d’Jousse checked a different group there was no association.
I’m just a chemist. If you think you have nothing to learn from me about experimental design, ok, but you might consider that my criticisms of standards in the medical literature are worth turning around in your mind. There is really no such thing as training in science but many people feel that, whatever it might be, physicians are not well-trained and the medical literature is not where the high standards are. I though my Mozart analogy was a good one.
“If you think this paper was subjected to vigorous cross-examination, that is appropriate peer review, and that the conclusions are established without need to see more data, then you should quit eating all those eggs.”
You do realize that the paper doesn’t advice people not to eat eggs, nor does it claim to have established that eggs are dangerous etc., right?
“If I had reviewed the paper, I would have asked for individual data to be shown graphically.”
Yes, from what I’ve gathered you do like your graphs …
“Why do we use quintiles (or quartiles, or whatever)?”
Umm, since they serve a purpose?
“If you think there is a reason, what is it?”
One shouldn’t answer a question with a question. Anyway: I do not know. That’s THE point in science: nothing is certain. Some of the most important discoveries have been a result of a “leap in the dark”. Ok, that’s certainly more than far-fetched in this case, but the main point is that the absence of a credible mechanism at this point doesn’t guarantee that there isn’t one – or that the issue should not be subjected to testing.
“You do a correlation to test a hypothesis not, as is usually said, the other way around.”
And I thought that looking at data for correlations is quite normal and serves its purpose … How do you think that e.g. the relevance of lipoprotein concentrations in CVD risk was first discovered?
“I’m just a chemist. If you think you have nothing to learn from me about experimental design, ok, but you might consider that my criticisms of standards in the medical literature are worth turning around in your mind.”
You do have many good points. But your posts and ideas aren’t above criticism, either.
“There is really no such thing as training in science but many people feel that, whatever it might be, physicians are not well-trained and the medical literature is not where the high standards are.”
Undoubtedly yes. Yet, many of abovementioned “people” cite studies of a particular design as convincing evidence in point A and trash the very same study design in point B. Go figure …
[…] are at high risk as a result of genetic or lifestyle risk factors. As SUNY professor and biochemist Richard David Feinman put it, “this is a limitation of many nutritional studies and, while a source of error, it is depends on […]
I’m curious–I went and read the original study. Male participants were followed up at 2, 4, 6, 8, and 10 years post-baseline assessment, and repeated self-reports of egg consumption were collected at each of these points. Would this change your opinion that diet could have changed over 10 years, skewing results? Apparently they took the additional data into consideration in determining association of egg consumption with development of Type 2 diabetes–it wasn’t simply a baseline assessment and then 10 years later, a diabetes test.
Second, curious why you say there are no figures or tables? I saw several in the article.
Thanks,
janelle
I said that there were no figures. There are several mind-numbing tables. But that is my opinion. I think graphics helps a lot and if you don’t do it, I am suspicious. If you look at the tables, you can find interesting things: across sextiles, Smoking 10.8 12.1 11.8 15.3 19.8 23.5, twice as much smoking in the big egg eaters. Well, since smoking seems more serious than eggs so when you correct egg-effects for smoking, you might want to show us the data, like in a figure. Might even want to correct the effect of smoking for egg consumption (correcting for confounders is symmetrical, (cf. Willett, Nutritional Epidemiology) and some people might say that you can correct for being a smoker but you can’t correct for the effect of smoking on the risk of other variables, that is, these always assume a linear relationship. But, the real thing, is that this is just taking (theoretically) things off the shelf and trying. Why eggs? There is nothing less likely to give you diabetes as shown by the pathetic attempt in the paper to justify the conclusions by dragging in cholesterol which even the USDA has given up on. The problem here is are you willing to consider the possibility that his is utterly and completely meaningless, generated by no reasonable hypothesis.
If you think that I did not read the paper very carefully before writing the blogpost, you are probably right. Looking at it now, I should have emphasized that it was a study of people with diabetes. It does not have a negative control. That is, we don’t know what percentage of the people WITHOUT diabetes, consumed more or less eggs that the highest consumer in this group. You might ask the authors. As I recall they are at Harvard.