We are very gratified and enthusiastic about getting our study funded and we thank all our backers. Beyond the research, we think that the campaign made a statement about the ability to get organized without having to have an organization. Thanks to everyone.

We got good feedback and good questions and we want to continue the scientific interaction and keep the community intact, initially on this blog. The “lab notes” are continued here and you can leave comments. We will recapitulate some of the points made during the Experiment campaign and you can “ask the researchers” in comments.

“What makes you think ketone bodies will help?”

We and others have carried out experiments that show the effects of ketone bodies on cancer cells in culture, as diet for patients with advanced cancer or as adjuncts to other modalities. Most direct experimental studies, however, must be considered preliminary and it is reasonable to ask why we thought ketone bodies might help.

The evidence supporting carbohydrate restriction, or specifically ketogenic diets in cancer remains largely indirect and speculative. Our recent perspective  summarized some of the relevant evolutionary and mechanistic factors: the central theme rests with the role of the glucose-insulin axis in promoting growth and proliferation, the predominant characteristic of cancer sells. So it has been observed for some time that patients with diabetes have higher risk of cancer. Epidemiological and other kinds of studies are generally consistent with the idea although different cancers are more or less closely associated with diabetes. Drugs employed as diabetes therapy, particularly metformin, have been found to have beneficial effects in cancer as well. Metformin reduces the risk of developing cancer although the effects on mortality are not clear cut. We made the case, in our critical review that dietary carbohydrate restriction is the first line of treatment for type 2 diabetes and the best adjunct for pharmacology in type 1 diabetes.


The association between cancer and diabetes in combination with the benefits of carbohydrate restriction in diabetes constitute one big connection. In dietary approaches, however, it is total caloric reduction that has received the most attention and, in fact, experiments show that if implemented as stated, calorie restriction represents a reliable approach to prevention and treatment of cancer, particularly in animal models. It is unknown how much of the effect is due to de facto reduction in particular macronutrients but when tested, carbohydrate reduction as the means of reducing calories prove most effective. We cited an important study by Tannenbaum. He found, in 1945 (!) that a carcinogen-induced sarcoma in mice was repressed by reduction in total calories but if  reduced by specifically lowering the carbohydrate intake, there was an enhanced response.


Impressive cancer prevention with calorie restriction in animal models has been repeated many times. Oddly, the protocol is most often presented as caloric restriction.  Odd in that this appears in sophisticated scientific papers where the downstream effects of the stimulation may pinpoint twenty molecular components and where the molecular targets of the “nutrients” are characterized and may specifically be the insulin receptor and the related IGF-1 (insulin-like growth factor -1) receptor. (Insulin is probably most important in that it stimulates IGF-1 activity by reducing the levels of the associated binding proteins). In these studies, where total caloric reduction is the independent variable, the involvement of insulin and the insulin-dependent downstream pathways have been shown to be involved.

It is now appreciated that the Warburg effect, the apparent reliance of tumors on glucose for fuel, is a key observation that has been insufficiently explored. The effect provides motivation and clues for exploring the metabolic approach to cancer. Warburg thought that all cancers showed this phenotype which is not true but a large number do; of significance is that one that does not, prostate cancer, is the outlier in the figure above on relation to diabetes. The next post will start from some basic biochemistry and explain why (and how) we think that the Warburg effect points to the potential value of ketogenic diets.

NOTE: You can still back the research on ketogenic diets for cancer at nmsdocs.com. Click on DONATE. You can also get an autographed copy of Feinman’s book, for $ 20 shipping included.  Enter the coupon code SEPT2016 (Still good even though it is already October).

We have a good deal of enthusiasm in the keto/paleo/low-carb community. We have the real sense that we can we use carbohydrate restriction to take advantage of the characteristic metabolic features of cancer — inflexible reliance on glucose. Enthusiasm may have outstripped the data, however, and several groups are trying to fill the gap. The barrier rests with the difficulty for anybody to obtain funding from NIH or other major government or private agencies. On top of this looms the long-standing resistance to low-carbohydrate diets making things particularly difficult. Our group is carrying out some good experiments and we employ a dedicated technician and we can efficiently use limited funds. Your backing can help.  A $ 15 donation gets us several days of supplies for the in vitro experiments that provide the biochemical underpinnings for attacking cancer in the clinic. Our project at experiment.com provides background, a place for discussion and reports from the lab.

The current metabolic point of view in cancer — emphasizing flexibility of fuel choices —  derives from renewed interest in the Warburg effect. Warburg saw that many cancer cells were producing lactic acid, the product of glycolysis. In other words, the tumors were not using the more efficient aerobic metabolism even when oxygen was present in the environment. The tumor cell’s requirement for glucose suggests the possibility of giving the host an advantage by restricting carbohydrate and offering ketone bodies as an alternative fuel.constant_ATP_UCP2

We showed previously that we could inhibit the growth of 7 different cancer cell lines and repress the production of ATP (the”‘energy molecule” in cell culture by adding acetoacetate (one of the ketone bodies) to the medium.  Control normal cell lines were not affected. In addition, we showed that ATP reduction was associated with the level of a molecule called uncoupling protein-2 (UCP-2).  I explain in other posts what “uncoupling” means and how it figures into energy efficiency. First, the big picture..

What is the context of inhibition by ketone bodies ?
The real context, of course, is human cancer. Our 28 day pilot human trial of 10 subjects with advanced cancers on a very low carbohydrate ketogenic diet (KD) was published in Nutrition (Elsevier) in 2012.  A small study — nominally just to show safety and feasibility, it was nonetheless well received. Of note, is that we found that those patients with the greatest extent of ketosis had stable cancers or partial remission, while those with the least ketosis showed continued progressive cancer. Despite a favorable editorial and the Metabolism Award from the journal. Unfortunately, our proposal to scale up to 65 patients was rejected by the NIH/NCI which we find very discouraging perhaps related to a commitment to drug therapy. In any case, we appeal now for help in supporting dietary cancer research.
You can help. To donate:  Our project ia at experiment.com

Image  —  Posted: August 22, 2016 in Uncategorized


One of my favorite legal terms, collateral estoppel, refers to procedures to prevent re-litigation of issues that have already been settled in court. From the same root as stopper, that is, cork, it prevents harassment and wasting of the court’s time. The context is the recent flap over a poster presented by Kevin Hall which has started re-trying the case of whether all diets have the same metabolic efficiency, a question which, in my view, has been adjudicated several times. I put it this way because frequently I have made an analogy between evidence-based-medicine (EBM) and evidence as presented in a court of law. My main point has been that, in the legal system, there are rules of evidence and there is a judge who decides on admissibility. You can’t just say, as in EBM, that your stuff constitutes evidence.  My conclusion is usually that EBM is one of the self-congratulatory procedures that allows people to say anything that they want without having to defend their position. EBM represents of of the many corruptions of research procedure now under attack by critics (perpetrators ?) as in the recent editorial by Richard Horton, editor of The Lancet. One thing that I  criticize medical nutrition for is its inability to be estopped from funding and endlessly re-investigating whether saturated fat causes heart disease, whether high protein diets hurt your kidneys, and whether a calorie is a calorie. It seems that the issue is more or less settled — there are dozens of examples of variable energy expenditure in the literature. It would be reasonable to move on by investigating the factors that control energy balance, to provide information on the mechanisms that predict great variability and, most important, the mechanisms that make it so small in biological systems — most of the time, a calorie is a calorie, at least roughly. Funding and performing ever more expensive experiments to decide whether you can lose more or less weight on one diet or another, as if we had never done a test before, is not helpful.

Several bloggers discussed Hall’s study which claims that either a calorie is a calorie or it is not depending on whether, as described by Mike Eades, you look at the poster itself or at a video of Kevin Hall explaining what it is about. Mike’s blog is excellent but beyond the sense of déja-vu, the whole thing reminded me of the old joke about the Polish mafia. They make you an offer that you can’t understand.  So, because this is how I got into this business, I will try to explain how I see the problem of energy balance and why we might want this trial estopped.

I have taught nutrition and metabolism for many years but I got into nutrition research because the laws of thermodynamics were, and still are, invoked frequently in the discussion. Like most chemists, I wouldn’t claim to be a real expert but I like the subject and I teach the subject at some level. I could at least see that nobody in nutrition knew what they were talking about. I tried to show that the application of thermodynamics, if done correctly, more or less predicts that different diets will have different efficiencies (from the standpoint of storage, that is, weight gained per calorie consumed).

But you don’t really need thermodynamics to see this. Prof. Wendy Pogozelski at SUNY Geneseo pointed out that if you think about oxidative metabolic uncouplers, that is all you need to know. “Coupling,” in energy metabolism, refers to the sequence of reactions by which the energy from the oxidation of food is converted to ATP, that is, into useful biologic energy. The problem in energy metabolism is that the fuel, as in many “combustion engines,” is processed by oxidation — you put in oxygen and get out CO2 and water . The output, on the other hand  is a phosphorylation reaction — generation of ATP from ADP, its low energy form. The problem is how to couple these two different kins of reactions. It turns out that the mitochondrial membrane couples the two processes (together called oxidative phosphorylation). A “high energy” state is established across the membrane by oxidation and this energy is used to make ATP. Uncouplers are small molecules or proteins that disengage the oxidation of substrate (food) from ATP synthesis allowing energy to be wasted or channeled into other mechanisms, generation of reactive oxygen species, for example.

BLOG_car_analogy_May_16The car analogy of metabolic inhibitors. Figure from my lectures. Energy is generated in the TCA cycle and electron transport chain (ETC). The clutch plays the role of the membrane proton gradient, transmitting energy to the wheels which produce forward motion (phosphorylation of ADP). Uncouplers allow oxidation to continue — the TCA cycle is “racing” but to no effect. Other inhibitors (called oxidative phosphorylation inhibitors) include oligomycin which blocks the ATP synthase, analogous to a block under the wheels: no phosphorylation, no utilization of the gradient; no utilization, no gradient formation; no gradient, no oxidation. The engine “stalls.”

In teaching metabolism, I usually use the analogy of an automobile where the clutch connects the engine to the drive train . The German word for clutch is Kupplung and when you put a car in neutral your car is uncoupled, can process many calories of gasoline ‘in,’ but has zero efficiency, so that none of the ‘out’ does the useful work of turning the wheels. Biological systems can be uncoupled by external compounds — the classic is 2, 4-dinitrophenol which, if you are familiar with mitochondrial metabolism, is a proton ionophore, that is, destroys the proton gradient that couples oxidation to ADP-phophorylation.  There are natural uncouplers, the uncoupling proteins, of which there are five, named UCP-1 through UCP-5. Considered a family because of the homology to UCP-1, a known uncoupler, it has turned out that at least two others clearly have uncoupling activity. The take-home message is that whatever the calories in, the useful calories out (for fat storage or whatever) depends on the presence of added or naturally occurring uncouplers as well.

This is one of many examples of the mechanisms whereby metabolic calories-out per calorie-in could be variable.  The implication is that when somebody reports metabolic advantage (or disadvantage), there is no reason to disbelieve it. Conversely, this is one of the mechanisms that can reduce variability.

In fact, homeostatic mechanisms  are usually observed. You don’t have to have a metabolic chamber to know that your intake is variable day-to-day but your weight may be quite stable. The explanation is not in the physics which, again,  predicts variation, but rather in the biological system which is always connected in feedback so as to resist change. However strong the homeostasis (maintenance of steady-state), conversely, everybody has the experience of being in a situation where it doesn’t happen. “I don’t understand. I went on this cruise and I really pigged out on lobster and steak but I didn’t gain any weight.”  (It is not excluded, but nobody ever says that about the pancake breakfast). In other words, biochemistry and daily experience tells us that black swans are to be expected and, given that the system is set up for variability, the real question is why there are so many white swans.

So it is physically predicted that a calorie is not a calorie. When it has been demonstrated, in animal models where there is control of the food intake, or in humans, where there are frequently big differences that cannot reasonably be accounted for by the error in food records, there is no reason to doubt the effect. And, of course, a black swan is an individual. Kevin Hall’s study, as in much of the medical literature, reported group statistics and we don’t know if there were a few winners in with the group. The work has not been reviewed or published but, either way, I think it is likely to waste the court’s time.


The attack was quite sudden although it appeared to have been planned for many years. The paper was published last week (Augustin LS, Kendall CW, Jenkins DJ, Willett WC, Astrup A, Barclay AW, Bjorck I, Brand-Miller JC, Brighenti F, Buyken AE et al: Glycemic index, glycemic load and glycemic response: An International Scientific Consensus Summit from the International Carbohydrate Quality Consortium (ICQC). Nutr Metab Cardiovasc Dis 2015, 25(9):795-815.


As indicated by the title, responsibility was taken by the self-proclaimed ICQC.  It turned out to be a continuation of the long-standing attempt to use the glycemic index to co-opt the obvious benefits in control of the glucose-insulin axis while simultaneously attacking real low-carbohydrate diets. The authors participated in training in Stresa, Italy.

The operation was largely passive aggressive. While admitting the importance of dietary carbohydrate in controlling post-prandial glycemic,  low-carbohydrate diets were ignored. Well, not exactly. The authors actually had a strong attack.  The Abstract of the paper said (my emphasis):

Background and aims: The positive and negative health effects of dietary carbohydrates are of interest to both researchers and consumers.”

Methods: International experts on carbohydrate research held a scientific summit in Stresa, Italy, in June 2013 to discuss controversies surrounding the utility of the glycemic index (GI), glycemic load (GL) and glycemic response (GR).”

So, for the record, the paper is about dietary carbohydrate and about controversies.

The Results in Augustin, et al were simply

“The outcome was a scientific consensus statement which recognized the importance of postprandial glycemia in overall health, and the GI as a valid and reproducible method of classifying carbohydrate foods for this purpose…. Diets of low GI and GL were considered particularly important in individuals with insulin resistance.”

A definition is always a reproducible way of classifying things, and the conclusion is not controversial: glycemia is important.  Low-GI diets are a weak form of low-carbohydrate diet and they are frequently described as a politically correct form of carbohydrate restriction. It is at least a subset of carbohydrate restriction and one of the “controversies” cited in the Abstract is sensibly whether it is better or worse than total carbohydrate restriction. Astoundingly, this part of the controversy was ignored by the authors.  Our recent review of carbohydrate restriction in diabetes had this comparison:




A question of research integrity.

It is considered normal scientific protocol that, in a scientific field, especially one that is controversial, that you consider and cite alternative or competing points of view. So how do the authors see low-carbohydrate diets fitting in? If you search the pdf of Augustin, et al on “low-carbohydrate” or “low carbohydrate,” there are only two in the text:

“Very low carbohydrate-high protein diets also have beneficial effects on weight control and some cardiovascular risk factors (not LDL-cholesterol) in the short term, but are associated with increased mortality in long term cohort studies [156],”


“The lowest level of postprandial glycemia is achieved using very low carbohydrate-high protein diets, but these cannot be recommended for long term use.”

There are no references for the second statement but very low carbohydrate diets can be and frequently are recommended for long term use and have good results. I am not aware of “increased mortality in long term cohort studies” as in the first statement. In fact, low-carbohydrate diets are frequently criticized for not being subjected to long-term studies. So it was important to check out the studie(s) in reference 156:

[156] Pagona L, Sven S, Marie L, Dimitrios T, Hans-Olov A, Elisabete W. Low carbohydrate-high protein diet and incidence of cardiovascular diseases in Swedish women: prospective cohort study. BMJ 2012;344.

Documenting increased mortality.

The paper is not about mortality but rather about cardiovascular disease and, oddly, the authors are listed by their first names. (Actual reference: Lagiou P, Sandin S, Lof M, Trichopoulos D, Adami HO, Weiderpass E: . BMJ 2012, 344:e4026). This minor error probably reflects the close-knit “old boys” circle that functions on a first name basis although it may also indicate that the reference was not actually read so it was not discovered what the reference was really about.

Anyway, even though it is about cardiovascular disease, it is worth checking out. Who wants increased risk of anything. So what does Lagiou, et al say?

The Abstract of Lagiou says (my emphasis) “Main outcome measures: Association of incident cardiovascular diseases … with decreasing carbohydrate intake (in tenths), increasing protein intake (in tenths), and an additive combination of these variables (low carbohydrate-high protein score, from 2 to 20), adjusted for intake of energy, intake of saturated and unsaturated fat, and several non-dietary variables.”

Low-carbohydrate score? There were no low-carbohydrate diets. There were no diets at all. This was an analysis of “43, 396 Swedish women, aged 30-49 years at baseline, [who] completed an extensive dietary questionnaire and were followed-up for an average of 15.7 years.” The outcome variable, however, was only the “score” which the authors made up and which, as you might guess, was not seen and certainly not approved, by anybody with actual experience with low-carbohydrate diets. And, it turns out that “Among the women studied, carbohydrate intake at the low extreme of the distribution was higher and protein intake at the high extreme of the distribution was lower than the respective intakes prescribed by many weight control diets.” (In social media, this is called “face-palm”).

Whatever the method, though, I wanted to know how bad it was? The 12 years or so that I have been continuously on a low-carbohydrate diet might be considered pretty long term. What is my risk of CVD?

Results: A one tenth decrease in carbohydrate intake or increase in protein intake or a 2 unit increase in the low carbohydrate-high protein score were all statistically significantly associated with increasing incidence of cardiovascular disease overall (n=1270)—incidence rate ratio estimates 1.04 (95% confidence interval 1.00 to 1.08), 1.04 (1.02 to 1.06), and 1.05 (1.02 to 1.08).”

Rate ratio 1.04? And that’s an estimate.  That’s odds of 51:49.  That’s what I am supposed to be worried about. But that’s the relative risk. What about the absolute risk? There were 43 396 women in the study with 1270 incidents, or 2.9 % incidence overall.  So the absolute difference is about 1.48-1.42% = 0.06 % or less than 1/10 of 1 %.

Can such low numbers be meaningful? The usual answers is that if we scale them up to the whole population, we will save thousands of lives. Can we do that? Well, you can if the data are strong, that is, if we are really sure of the reliability of the independent variable. The relative risk in the Salk vaccine polio trial, for example, was in this ballpark but scaling up obviously paid off. In the Salk vaccine trial, however, we knew who got the vaccine and who didn’t. In distinction, food questionnaire’s have a bad reputation. Here is Lagiou’s description (you don’t really have to read this):

“We estimated the energy adjusted intakes of protein and carbohydrates for each woman, using the ‘residual method.’ This method allows evaluation of the “effect” of an energy generating nutrient, controlling for the energy generated by this nutrient, by using a simple regression of that nutrient on energy intake.…” and so on. I am not sure what it means but it certainly sounds like an estimate. So is the data itself any good? Well,

“After controlling for energy intake, however, distinguishing the effects of a specific energy generating nutrient is all but impossible, as a decrease in the intake of one is unavoidably linked to an increase in the intake of one or several of the others. Nevertheless, in this context, a low carbohydrate-high protein score allows the assessment of most low carbohydrate diets, which are generally high protein diets, because it integrates opposite changes of two nutrients with equivalent energy values.”

And “The long interval between exposure and outcome is a source of concern, because certain participants may change their dietary habits during the intervening period.”

Translation: we don’t really know what we did here.

In the end, Lagiou, et al admit “Our results do not answer questions concerning possible beneficial short term effects of low carbohydrate or high protein diets in the control of body weight or insulin resistance. Instead, they draw attention to the potential for considerable adverse effects on cardiovascular health of these diets….” Instead? I thought insulin resistance has an effect on CVD but if less than 1/10 of 1 % is “considerable adverse effects” what would something “almost zero” be.?

Coming back to the original paper by Augustin, et al, what about the comparison between low-GI diets and low-carbohydrate diets. The comparison in the figure above comes from Eric Westman’s lab. What do they have to say about that?


They missed this paper. Note: a comment I received suggested that I should have searched on “Eric” instead of “Westman.” Ha.

Overall, this is the evidence used by ICQC to tell you that low-carbohydrate diets would kill you. In the end, Augustin, et al is a hatchet-job, citing a meaningless paper at random. It is hard to understand why the journal took it. I will ask the editors to retract it.

As the nutrition world implodes, there are a lot of accusations about ulterior motives and personal gain. (A little odd, that in this period of unbelievable greed — CEO’s ripping off public companies for hundreds of millions of dollars, congress trying to give tax breaks to billionaires — book authors are upbraided for trying to make money). So let me declare that I am not embarrassed to be an author for the money — although the profits from my book do go to research, it is my own research and the research of my colleagues. So beyond general excellence (not yet reviewed by David Katz), I think “World Turned Upside Down” does give you some scientific information about red meat and cancer that you can’t get from the WHO report on the subject.

The WHO report has not yet released the evidence to support their claim that red meat will give you cancer but it is worth going back to one of the previous attacks.  Chapters 18 and 19 discussed a paper by Sinha et al, entitled “Meat Intake and Mortality.”    The Abstract says “Conclusion: Red and processed meat intakes were associated with modest increases in total mortality, cancer mortality, and cardiovascular disease mortality,” I had previously written a blogpost about the study indicating how weak the association was. In that post, I had used the data on men but when I incorporated the information into the book, I went back to Sinha’s paper and analyzed the original data. For some reason, I also checked the data on women. That turned out to be pretty surprising:


I described on Page 286: “The population was again broken up into five groups or quintiles. The lower numbered quintiles are for the lowest consumption of red meat. Looking at all cause mortality, there were 5,314 deaths [in lowest quintile] and when you go up to quintile 05, highest red meat consumption, there are 3,752 deaths. What? The more red meat, the lower the death rate? Isn’t that the opposite of the conclusion of the paper? And the next line has [calculated] relative risk which now goes the other way: higher risk with higher meat consumption. What’s going on? As near as one can guess, “correcting” for the confounders changed the direction….” They do not show most of the data or calculations but I take this to be equivalent to a multivariate analysis, that is, red meat + other things gives you risk. If they had broken up the population by quintiles of smoking, you would see that that was the real contributor. That’s how I interpreted it but, in any case, their conclusion is about meat and it is opposite to what the data say.

So how much do you gain from eating red meat? “A useful way to look at this data is from the standpoint of conditional probability. We ask: what is the probability of dying in this experiment if you are a big meat‑eater? The answer is simply the number of people who both died during the experiment and were big meat‑eaters …. = 0.0839 or about 8%. If you are not a big meat‑eater, your risk is …. = 0.109 or about 11%.” Absolute gain is only 3 %. But that’s good enough for me.

Me, at Jubilat, the Polish butcher in the neighborhood: “The Boczak Wedzony (smoked bacon). I’ll take the whole piece.”


Boczak Wedzony from Jubilat Provisions

Rashmi Sinha is a Senior Investigator and Deputy Branch Chief and Senior at the NIH. She is a member of the WHO panel, the one who says red meat will give you cancer (although they don’t say “if you have the right confounders.”)

So, buy my book: AmazonAlibris, or

Direct:  Personalized, autographed copy $ 20.00 free shipping USA only.  Use coupon code: SEPT16


…what metaphysics is to physics. The old joke came to mind when a reporter asked me yesterday to comment on a paper published in the BMJ. “Intake of saturated and trans unsaturated fatty acids and risk of all cause mortality, cardiovascular disease, and type 2 diabetes: systematic review and meta-analysis of observational studies” by de Souza, et al.

Now the title ““Intake of saturated and trans unsaturated fatty acids…” tells you right off that this is not good news; lumping together saturated fat and trans-fat is an indication of bias. A stand-by of Atkins-bashers, it is a way of vilifying saturated fat when the data won’t fit.  In the study that the reporter asked about, the BMJ provided a summary:

“There was no association between saturated fats and health outcomes in studies where saturated fat generally replaced refined carbohydrates, but there was a positive association between total trans fatty acids and health outcomes Dietary guidelines for saturated and trans fatty acids must carefully consider the effect of replacement nutrients.”

“But?” The two statements are not really connected. In any case the message is clear: saturated fat is Ok. Trans-fat is not Ok. So, we have to be concerned about both  saturated fat and trans-fat. Sad.

And “systematic” means the system that the author wants to use. This usually means a meta-analysis. Explained by the overly optimistic “What is …?” series:


The jarring notes are “precise estimate” in combination with “combining…independent studies.” In practice, you usually only repeat an experiment exactly if you suspect that something was wrong with the original study or if the result is sufficiently outside expected values that you want to check it. Such a systematic examination involves an analysis of the experimental details. The idea underlying the meta-analysis, however, usually unstated is that the larger the number of subjects in a study, the more compelling the conclusion. One might make the argument, instead, that if you have two or more studies which are imperfect, combining them is likely to lead to greater uncertainty and more error, not less.  I am one who would make such an argument. So where did meta-analysis come from and what, if anything, is it good for?

I am trained in enzyme and protein chemistry but I have worked in a number of different  fields including invertebrate animal behavior. I never heard of meta-analysis until very recently, that is, until I started doing research in nutrition. In fact, in 1970 there weren’t any meta-analyses, at least not with that phrase in the title, or at least not as determined by my PubMed search. By 1990, there were about a 100 and by 2014, there were close to 10, 000 (Figure 1).

Meta-anal_Year_Mar9Figure 1. Logarithm of the number of papers in PubMed search with the title containing “meta-analysis” vs. Year of publication

This exponential growth suggests that the technique grew by reproducing itself. It suggests, in fact, that its origins are in spontaneous generation. In other words, it is popular because it is popular. (It does have obvious advantages; you don’t have to do any experiments). But does it give any useful information?


If you have a study that is under-powered, that is, if you only have a small number of subjects, and you find a degree of variability in the outcome, then combining the results from your experiment with another small study may point you to a consistent pattern. As such, it is a last-ditch, Hail-Mary kind of method. Applying it to large studies that have statistically meaningful results, however, doesn’t make sense, because:

  1. If all of the studies go in the same direction, you are unlikely to learn anything from combining them. In fact, if you come out with a value for the output that is different from the value from the individual studies, in science, you are usually required to explain why your analysis improved things. Just saying it is a larger n won’t cut it, especially if it is my study that you are trying to improve on.
  2. In the special case where all the studies show no effect and you come up with a number that is statistically significant, you are, in essence saying that many wrongs can make a right as described in a previous blog post on abuse of meta-analyses.  In that post, I re-iterated the statistical rule that if the 95% CI bar crosses the line for hazard ratio = 1.0 then this is taken as an indication that there is no significant difference between the two conditions that are being compared. The example that I gave was the meta-analysis by Jakobsen, et al. on the effects of SFAs or a replacement on CVD outcomes (Figure 2). Amazingly, in the list of 15 different studies that she used, all but one cross the hazard ratio = 1.0 line. In other words, only one study found that keeping SFAs in the diet provides a lower risk than replacement with carbohydrate. For all the others there was no significant difference.  The question is why an analysis was done at all.  What could we hope to find? How could 15 studies that show nothing add up to a new piece of information? Most amazing is that some of the studies are more than 20 years old. How could these have had so little impact on our opinion of saturated fat?  Why did we keep believing that it was bad?


Figure 2. Hazard ratios and 95% confidence intervals for coronary events and deaths in the different studies in a meta-analysis from Jakobsen, et al.Major types of dietary fat and risk of coronary heart disease: a pooled analysis of 11 cohort studies. Am J Clin Nutr 2009, 89(5):1425-1432.

3. Finally, suppose that you are doing a meta-analysis on several studies and that they have very different outcomes, showing statistically significant associations in different directions. For example, if some studies showed substituting saturated fat for carbohydrate increased risk while some showed that it decreased risk. What will you gain by averaging them? I don’t know about you but it doesn’t sound good to me. It makes me think of the old story of the emerging nation that was planning to build a railroad and didn’t know whether to use a gauge that matched the country to the north or the gauge of the country to the south. The parliament voted to use a gauge that was the average of the two.

Following up from chapters in The World Turned Upside Down, I will try to provide a guide to getting through the medical literature. I will talk about relative risk first with an excerpt from the book and a quick way to get a feel for what relative risk really is.

It has been frequently pointed out that to succeed at major league baseball all you have to do is screw up no more than 70 % of the time. In fact ,it is almost 75 years since someone was able to come up with a failure rate as low as 60 %. (Ted Williams hit .406 in 1941). The point is that statistics is about interpretation. It is how you describe things. As in Paulos’s Once Upon a Number, mathematics and especially statistics is a story, and how you tell it.  The potential for being misled is obviously great. The idea that statistics is a cut-and-dried standard set of manipulations into which you pour your data and from which you are automatically given significance, importance and truth, is one of the major components in the failure of the medical nutrition literature. Read the rest of this entry »