Richard David Feinman

Doctor:    Therein the patient

  Must minister to himself.

Macbeth: Throw physic [medicine] to the dogs; I’ll none of it.

— William Shakespeare, Macbeth

The quality of nutrition papers even in the major scientific and medical journals is so variable and the lack of restraint in the popular media is so great that it is hard to see how the general public or even scientists can find out anything at all. Editors and reviewers are the traditional gate-keepers in science but in an area where rigid dogma has reached Galilean proportions, it is questionable that any meaningful judgement was made: it is easy to publish papers that conform to the party line (“Because of the deleterious effects of dietary fructose, we hypothesized that…”) and hard to publish others: when JAMA published George Bray’s “calorie-is-a-calorie” paper and I pointed out that the study more accurately supported the importance of carbohydrate as a controlling variable, the editor declined to publish my letter.  In this, the blogs have performed a valuable service in providing an alternative POV but if the unreliability is a problem in the scientific literature, that problem is multiplied in internet sources. In the end, the consumer may feel that they are pretty much out there on their own. I will try to help.  The following was posted on FatHead’s Facebook page:

 How does one know if a study is ‘flawed’? I see a lot of posts on here that say a lot of major studies are flawed. How? Why? What’s the difference if I am gullible and believe all the flawed studies, or if I (am hopefully not a sucker) believe what the Fat Heads are saying and not to believe the flawed studies — eat bacon.

Where are the true studies that are NOT flawed…. and how do I differentiate? : /

 My comment was that it was a great question and that it would be in the next post so I will try to give some of the principles that reviewers should adhere to.  Here’s a couple of guides to get started. More in future posts:

 1. “Healthy” (or “healthful”) is not a scientific term. If a study describes a diet as “healthy,” it is almost guaranteed to be a flawed study.  If we knew which diets were “healthy,” we wouldn’t have an obesity epidemic. A good example is the paper by Appel, et al. (2005). “Effects of protein, monounsaturated fat, and carbohydrate intake on blood pressure and serum lipids: results of the OmniHeart randomized trial,” whose conclusion is:

“In the setting of a healthful diet, partial substitution of carbohydrate with either protein or monounsaturated fat can further lower blood pressure, improve lipid levels, and reduce estimated cardiovascular risk.”

 It’s hard to know how healthful the original diet, a “carbohydrate-rich diet used in the DASH trials … currently advocated in several scientific reports” really is if removing carbohydrate improved everything.

Generally, understatement  is good.  One of the more famous is from Watson & Cricks’s 1953 paper in which they proposed the DNA double helix structure. They said “It has not escaped our notice that the specific pairing we have postulated immediately suggests a possible copying mechanism for the genetic material.”  A study with the word “healthy” is an infomercial.

2. Look for the pictures (figures).  Presentation in graphic form usually means the author wants to explain things to you, rather than snow you.  This is part of the Golden Rule of Statistics that I mentioned in my blogpost “The Seventh Egg”  which discusses a very flawed study from Harvard on egg consumption. The rule comes from the book PDQ Statistics:

“The important point…is that the onus is on the author to convey to the reader an accurate impression of what the data look like, using graphs or standard measures, before beginning the statistical shenanigans.  Any paper that doesn’t do this should be viewed from the outset with considerable suspicion.”

The Watson-Crick  paper cited above had the diagram of the double-helix  which essentially became the symbol of modern biology.  It was drawn by Odile, Francis’s wife, who is described as being famous for her nudes, only one of which I could find on the internet.

Krauss, et. al. Separate effects of reduced carbohydrate intake and weight loss on atherogenic dyslipidemia.

The absence of a figure may indicate that the authors are not giving you a chance to actually see the results, that is, the experiment may not be flawed but the interpretation may be misleading, intentionally or otherwise.  An important illustration of the principle is a paper published by Krauss. It is worth looking at this paper in detail because the experimental work is very good and the paper directly — or almost directly — confronts a big question in diet studies: when you reduce calories by cutting out carbohydrate, is the effect due simply  to lowering calories or is there a specific effect of carbohydrate restriction.  The problem is important since many studies compare low-carbohydrate and low-fat diets where calories are reduced on both. Because the low-carbohydrate diet generally has the better weight loss and better improvement in HDL and triglycerides, it is said that it was the weight loss that caused the lipid improvements.

So Krauss compared the effects of carbohydrate restriction and weight loss on the collection of lipid markers known collectively as atherogenic dyslipidemia.  The markers of atherogenic dyslipidemia, which are assumed to predispose to cardiovascular disease, include high triglycerides (triacylglycerol), low HDL and high concentrations of the small dense LDL.

Here is how the experiment was set up: subjects first consumed a baseline diet of  54% of energy as carbohydrate, for 1 week. They were then assigned to one of four groups.  Either they continued the baseline diet, or they kept calories constant but reduced carbohydrate by putting fat in its place.  The three lower carbohydrate diets had 39 % or 26 % carbohydrate or 26 % carbohydrate with higher saturated fat.  After 3 weeks on constant calories but reduced carbohydrate, calories were decreased by 1000 kcal/d for 5 week and, finally, energy was stabilized for 4 weeks and the features of atherogenic dyslidemia were measured at week 13.  The protocol is shown in the figure from Krauss’s paper:

The Abstract of the paper describes the outcomes and the authors’ conclusions.

Results: The 26%-carbohydrate, low-saturated-fat diet reduced [atherogenic dylipidemia]. These changes were significantly different from those with the 54%-carbohydrate diet. After subsequent weight loss, the changes in all these variables were significantly greater…(my italics)

Now there is something odd about this.  It is the last line of the conclusion that is really weird. If you lose weight, the effect of carbohydrate is not significant?  As described below, Jeff Volek and I re-analyzed this paper so I have read that line a dozen times and I have no idea what it means.  In fact, the whole abstract is strange.  It will turn out that the lower (26 %) is certainly “significantly different from.. the 54%-carbohydrate diet” but it is not just different but much better. Why would you not say that?  The Abstract is generally written so that it sounds negative about low carbohydrate effects but it is already known from Krauss’s previous work and others that carbohydrate restriction has a beneficial effect on lipids and the improvements in lipid markers occur on low-carbohydrate diets whether weight is lost or not.  The last sentence doesn’t seem to make any sense at all.    For one thing, the experiment wasn’t done that way.  As set up, weight loss came after carbohydrate restriction.  So, let’s look at the data in the paper.  There are few figures in the paper and Table 2 in the paper presents the results in a totally mind-numbing layout.  Confronted with data like this, I sometimes stop reading.  After all, if the author doesn’t want to conform to the Golden Rule of Statistics, if they don’t want to really explain what they accomplished, how much impact is the paper going to have.  In this case, however, it is clear that the experiment was designed correctly and it just seems impossible from previous work that this wouldn’t support the benefits of carbohydrate restriction and the negative tone of the Abstract needs to be explained.  So we all had to slog through those tables.  Let’s just look at the triglycerides since this is one of the more telling attributes of atherogenic dyslpidemia.  Here’s the section from the Table:

Well this looks odd in that the biggest change is in the lowest carb group with high SF but it’s hard to tell what the data look like.  First it is reported as logarithms. You sometime take logs of your data in order to do a statistical determination but that doesn’t change the data and it is better to report the actual value.  In any case, it’s easy enough to take antilogs and we can plot the data.  This is what it looks like:

It’s not hard to see what the data really show: Reducing carbohydrate has an overwhelming effect on triglycerides even without weight loss.  When weight loss is introduced, the high carbohydrate diets still can’t equal the performance of the carbohydrate reduction phase.  (The dotted line in the figure are data from Volek’s earlier work which Krauss forgot to cite).

The statements in the Conclusion from the Abstract are false and totally misrepresent the data.  It is not true as it says “carbohydrate restriction and weight loss provide equivalent…” effects. The carbohydrate-reduction phase is dramatically better than the calorie restriction phase and it is not true that they are “non-additive”  Is this an oversight?  Poor writing?  Well, nobody knows what Krauss’s motivations were but Volek and I plotted all of the data from Krauss’s paper and we published a paper in Nutrition & Metabolism providing an interpretation of Krauss’s work (with pictures).  Our conclusion:

Summary Although some effort is required to disentangle the data and interpretation, the recent publication from Krauss et al. should be recognized as a breakthrough. Their findings… make it clear that the salutary effects of CR on dyslipidemia do not require weight loss, a benefit that is not a feature of strategies based on fat reduction. As such, Krauss et al.  provides one of the strongest arguments to date for CR as a fundamental approach to diet, especially for treating atherogenic dyslipidemia.

An important question in this experiment, however, is whether even in the calorie reduction phase, calories are actually the important variable.  This is a general problem in calorie restriction studies if for no other reason than that there is no identified calorie receptor.  When we published this data, Mike Eades pointed out that in the phase in which Krauss reduced calories, it was done by reducing macronutrients across the board so carbohydrate was also reduced and that might be the actual controlling variable so we plotted the TAG against carbohydrate in each phase (low, medium and high carb (LC, MC, HC) without or with weight loss (+WL) and the results are shown below

This is remarkably good agreement for a nutrition study. When you consider carbohydrates as the independent variable, you can see what’s going on.  Or can you?  After all, by changing the variables you have only made an association between carbohydrate and outcome  of an experiment. So does this imply a causal relation between carbohydrate and triglycerides or not?  It is widely said that observational studies do not imply causality, that observational studies can only provide hypothesis for future testing. It certainly seems like causality is implied here.  It will turn out that a more accurate description is that observational studies do not necessarily imply causality and I will discuss that in the next posts.  The bottom line will be that there is flaw in grand principles like “Random controlled trials are the gold standard.” “Observational studies are only good for generating hypotheses,”  “Metabolic Ward Studies are the gold standard.” Science doesn’t run on such arbitrary rules.