…what metaphysics is to physics. The old joke came to mind when a reporter asked me yesterday to comment on a paper published in the BMJ. “Intake of saturated and trans unsaturated fatty acids and risk of all cause mortality, cardiovascular disease, and type 2 diabetes: systematic review and meta-analysis of observational studies” by de Souza, et al.

Now the title ““Intake of saturated and trans unsaturated fatty acids…” tells you right off that this is not good news; lumping together saturated fat and trans-fat is an indication of bias. A stand-by of Atkins-bashers, it is a way of vilifying saturated fat when the data won’t fit.  In the study that the reporter asked about, the BMJ provided a summary:

“There was no association between saturated fats and health outcomes in studies where saturated fat generally replaced refined carbohydrates, but there was a positive association between total trans fatty acids and health outcomes Dietary guidelines for saturated and trans fatty acids must carefully consider the effect of replacement nutrients.”

“But?” The two statements are not really connected. In any case the message is clear: saturated fat is Ok. Trans-fat is not Ok. So, we have to be concerned about saturated fat and trans-fat. Sad.

And “systematic” means the system that the author wants to use. This usually means meta-analysis. Explained by the overly optimistic “What is …?” series:

Meta-anal_WhatIs

The jarring notes are “precise estimate” in combination with “combining…independent studies.” In practice, you usually only repeat an experiment exactly if you suspect that something was wrong with the original study or if the result is sufficiently outside expected values that you want to check it. This kind of systematic examination involves analysis of the experimental details. That the larger the number of subjects in a study, the more compelling the conclusion, underlies the practice. One might make the argument, however, that if you have two or more studies which are imperfect, combining them is likely to lead to greater uncertainty and more error.  I am one who would make such an argument. So where did meta-analysis come from and what, if anything, is it good for?

I am trained in enzyme and protein chemistry but I have worked in a few other fields including invertebrate animal behavior. I never heard of meta-analysis until very recently. In fact, in 1970 there weren’t any meta-analyses, at least not with that phrase in the title as determined by a PubMed search. By 1990, there were about a 100 and by 2014, there were close to 10, 000 (Figure 1).

Meta-anal_Year_Mar9Figure 1. Logarithm of the number of papers in PubMed search with the title containing “meta-analysis” vs. Year of publication

This exponential growth suggests that the technique grew by reproducing itself. In other words, it is popular because it is popular. (It does have obvious advantages. You don’t have to do any experiments). But does it give any useful information?

Meta-analysis

If you have a study that is under-powered, that is, if you only have a small number of subjects and you find a degree of variability in the outcome, combining the results from your experiment with another small study may point you to a consistent pattern. As such, it is a last-ditch, Hail-Mary kind of method. Applying it to large studies that have statistically meaningful results, however, doesn’t make sense, because:

  1. If all of the studies go in the same direction, you are unlikely to learn anything from combining them. In fact, if you come out with a value for the output that is different from the value from the individual studies, in science, you are usually required i to explain why your analysis improved things. Just saying it is a larger n won’t cut it, especially if it is my study that you are trying to improve on.
  2. In the special case where all the studies show no effect and you come up with a number that is statistically significant, you are, in essence saying that many wrongs can make a right as described in a previous blog post on abuse of meta-analyses.  In that post, I re-iterated the statistical rule that if the 95% CI bar crosses the line for hazard ratio = 1.0 then this is taken as an indication that there is no significant difference between the two conditions that are being compared. The example that I gave was the meta-analysis by Jakobsen, et al. on the effects of SFAs or a replacement on CVD outcomes (Figure 2). Amazingly, in the list of 15 different studies that she used, all but one cross the hazard ratio = 1.0 line. In other words, only one study found that keeping SFAs in the diet provides a lower risk than replacement with carbohydrate. For all the others there was no significant difference.  The question is why an analysis was done at all.  What could we hope to find? How could 15 studies that show nothing add up to a new piece of information? Most amazing is that some of the studies are more than 20 years old. How could these have had so little impact on our opinion of saturated fat?  Why did we keep believing that it was bad?

SFA_Jakobsen_Sub_AJCN-2_2009

Figure 2. Hazard ratios and 95% confidence intervals for coronary events and deaths in the different studies in a meta-analysis from Jakobsen, et al.Major types of dietary fat and risk of coronary heart disease: a pooled analysis of 11 cohort studies. Am J Clin Nutr 2009, 89(5):1425-1432.

3. Finally, suppose you are doing a meta-analysis on several studies that have very different outcomes, showing statistically significant associations in different directions, for example. What will you gain by averaging them? I don’t know about you but it doesn’t sound good to me. It makes me think of the old story of the emerging nation that was planning to build a railroad and didn’t know whether to use a gauge that matched the country to the north or the gauge of the country to the south. The parliament voted to use a gauge that was the average of the two.

Following up from chapters in The World Turned Upside Down, I will try to provide a guide to getting through the medical literature. I will talk about relative risk first with an excerpt from the book and a quick way to get a feel for what relative risk really is.

It has been frequently pointed out that to succeed at major league baseball all you have to do is screw up no more than 70 % of the time. In fact ,it is almost 75 years since someone was able to come up with a failure rate as low as 60 %. (Ted Williams hit .406 in 1941). The point is that statistics is about interpretation. It is how you describe things. As in Paulos’s Once Upon a Number, mathematics and especially statistics is a story, and how you tell it.  The potential for being misled is obviously great. The idea that statistics is a cut-and-dried standard set of manipulations into which you pour your data and from which you are automatically given significance, importance and truth, is one of the major components in the failure of the medical nutrition literature. Read the rest of this entry »

This post follows from two publications. The 26-author review “Dietary carbohydrate restriction as the first approach in diabetes management: Critical review and evidence base” is a kind of manifesto on carbohydrate restriction for diabetes. It has received some attention but we are still in the process of bringing out its main points.

TWTUD_COVERmy book The World Turned Upside Down. The Second Low-Carbohydrate Revolution.  gives a very basic introduction into organic chemistry and metabolism and I add my voice to the critiques of the low-fat hypothesis and the sorry state of nutritional science. I also provide specific strategies on how to analyze reports in the literature to find out whether the main point of the paper is valid or not. The deconstructions of traditional nutrition, the “also bought” on the Amazon page, are numerous and continuing to proliferate as more and more people become aware of how bad things are. To me, the “surprise” in Nina Teicholz’s “Big Fat Surprise” is that, after all the previous exposés and my own research, there were deceptive practices and poor science that even I didn’t know about.

In addition to the exposés, even establishment voices are beginning to perceive how bad things are. So, with all these smoking guns why doesn’t anybody do anything? Why doesn’t somebody blow the whistle on them? It’s not like we are dealing with military intelligence.  What are they going to do? Not fund my grant, not publish my paper? Ha.

Whistle-blowing

“When you go to work today, imagine having a tape recorder attached to your body, a second one in your briefcase, and a third one in a special notebook, knowing that you will be secretly taping your supervisors, coworkers, and in some cases, your friends.” These are the opening lines of Mark Whitacre’s remarkable confession/essay/exposé (later a movie with Matt Damon) describing his blowing the whistle on Archer Daniels Midland (ADM) one of the largest food companies in the world; their motto at the time “ADM. Supermarket to the world.”

informant_lMatt Damon in The Informant!

It turned out that ADM had been colluding with its competitors to fix prices, in particular on the amino acid, lysine. Whitacre’s story is fascinating in detail.  Although relatively young, he was high up in the company, a division manager (“I lived in a huge home, which had an eight car garage filled with eight cars, and indoor horse-riding stables for my children”). He travelled around the world to big corporate meetings. At some point, encouraged by his wife whose ethical standards were quite a bit higher than his own, he became an FBI informant. Accompanying him in his business trips was a green lamp, housing a video feed. ”It is a good thing that all of the co- conspirators were men. A woman would have immediately noticed that this green lamp did not match the five star décor of some of the finest hotels, such as the Four Seasons in Chicago.” Ultimately, the lysine trial resulted in fines and three-year prison sentences for three of the executives of ADM as well as criminal fine for foreign companies worth $105 million, a record at the time. At the trial, things really went down-hill for the company when Whitacre produced a tape recording of the President of ADM telling executives that the company’s competitors were their friends and their customers were the enemy. Wits at the time suggested a new motto “ADM. Super mark-up to the world.” In the end, in a remarkable twist in the story, his whistle-blowing was compromised by the fact that he was on the take himself.

“I concluded that I would steal my own severance pay, and decided upon $9.5 million, which amounted to several years of my total compensation. …And I also considered what would happen if ADM learned of this theft. If they accused me, I thought that I had the perfect answer. How can you prosecute me for stealing $9.5 million when you are stealing hundreds of millions of dollars each year in the price fixing scheme? …. I decided to submit several bogus invoices to ADM, until I accumulated $9.5 million, which was meant to be my family’s financial security when I would be fired at a future date for being a whistleblower.”

As it turned out, a number of food and beverage companies, who had won hundreds of millions in settlements against ADM  were the ones who actually provided financial security for his family while Mark Whitacre spent nine years in prison.

Whistle blowing and imperial deshabillement

If it is not hidden, is it whistle-blowing? Did the kid “blow the whistle on the emperor’s new clothes?” If it is right out in the open, what is the scandal?  Well, there is open and there is open. Leaving out information may be a sign of a cover-up. I described, in my book, the case of the paper by Foster, et al., the conclusion of which was that “neither dietary fat nor carbohydrate intake influenced weight loss.”  I admitted, in the book, that:

“I had not read Foster’s paper very carefully before making the pronouncement that it was not very good. I was upbraided by a student for such a rush to judgment. I explained that that is what I do for a living. I explained that I usually don’t have to spend a lot of time on a paper to see the general drift…. but I was probably not totally convincing. So I read the paper, which is quite a bit longer than usual. The main thing that I was looking for was information on the nutrients that were actually consumed since it was their lack of effect that was the main point of the paper.…

In a diet experiment, the food consumed should be right up front but I couldn’t find it at all…. The data weren’t there. I was going to write to the authors when I found out…that this paper had been covered in a story in the Los Angeles Times. As reported by Bob Kaplan: ‘Of the 307 participants enrolled in the study, not one had their food intake recorded or analyzed by investigators. The authors did not monitor, chronicle or report any of the subjects’ diets. No meals were administered by the authors; no meals were eaten in front of investigators. There were no self‑reports, no questionnaires. The lead authors, Gary Foster and James Hill, explained in separate e-mails that self‑reported data are unreliable and therefore they didn’t collect or analyze any.’

I confess to feeling a bit shocked. I don’t like getting scientific information from the LA Times.  How can you say “neither dietary fat nor carbohydrate intake influenced weight loss” if you haven’t measured fat or carbohydrate? …. in fact, the whole nutrition field runs on self‑reported data. Is all that stuff from the Harvard School of Public Health, all those epidemiology studies that rely on food records, to be chucked out?”

So was this a breach of research integrity? It might be considered simply an error of omission. If you didn’t measure food consumed, you might think that you don’t necessarily have to put it in the methods. Was it just dumb not to realize that if you write a study of a diet comparison, you can’t leave out what people ate or at least admit that you didn’t measure what they ate. So can you blow the whistle on them for not telling the whole truth?  The authors were all well-known researchers, if party-liners.

The Office of Research Integrity is set up to police serious infractions in federally funded grants but it usually has to be clear cut and, sometimes, there is a whistle-blower. The Baltimore case is one of the better known if somewhat embarrassing cases for the agency — there was nothing to the whistle-blower’s allegations. In any case, there is a big gray area. If you falsify your data on a government research grant, you can go to jail.  If you make a dumb interpretation, however, if you say the data mean X when they show not-X, well, research is about unknowns, and you may have slipped up. Even Einstein admitted to the need to offer “sacrifices at the altar of Stupidity.” The NIH is supposed to not fund stuff like that. Editors and reviewers are supposed to see through the omission. What if they fell down on the job too? What if you have a field like nutrition where the NIH study sections are on the same wavelength as the researchers. There is, however, the question of the total impact. A lot of stuff is never cited and never does any harm. I enquired with the ORI, in a general way, about Foster’s paper. They said that if it is widely quoted, it could be an infraction. It has, in fact, been cited as evidence against low-carb diets. So am I going to be a whistle-blower? I don’t think so.

The problem is that only an insider can blow the whistle and although cooperation and collegiality remain very weak in the nutrition field, it is still our own nest and whistle-blowing makes everybody look bad. The “long blue line” does not form because the police think that corruption is okay. The problem is not just that there can be retribution, as in Serpico, but that it makes everybody look bad. It is simply that it reflects poorly on the whole police force. And while it is probable that, as Mark Whitacre said, “almost all of their 30,000 employees went to work each day doing the right thing morally and ethically,” the statement that “ADM was not a bad company” does not ring true. If we call attention to what is tolerated in medical nutrition, we are all looking like fools. And, of course, Foster’s paper is one of the more egregious but there is a lot of competition for worst. And it reflects badly on all of us in the field. “Is that what you do when you go to work?”

The parable of the big fish

I received an email from a physician in England. He has had consistently good results with low-carbohydrate diets.

“There is never a day when I don’t see the deleterious effects of too many carbs on those with the metabolic syndrome. And yet most doctors carry on as if it doesn’t exist !! …

Only yesterday I saw a man I have known for over 15 years. His GGT [gamma-glutamyl transferase; marker for liver disease] had always been about double normal. Embarrassingly I had assumed that he was a drinker, despite repeated denial, thinking his big belly was evidence!  He chose low carb on March 2013 and never looked back. Liver function normal now and an easy 7 Kg weight loss.”

He said that the information had been used in the production of the ABC Catalyst TV documentary from Australia, but:

“I am a very, very small fish! As smaller fish we GPs specialise in getting ideas across to ordinary folk. The Internet is democratising medicine faster than some big fish realise. I wrote my practical diabetes piece partly for the educated general public and insisted on open access.

Big fish will scoff at my small numbers (70) and lack of double blindness anyway.”

I assured him that he was making an impact, that n = 70 was fine and not to worry about the big fish. I related a story told to me by one of my colleagues in graduate school: he had gone fishing in the Gulf of Mexico and they had caught a very big fish (I no longer remember the kind) which was thrashing around on the deck and they could not contain it. There happened to be a rifle on board and somebody shot the fish. The bullet went through the bottom of the boat which sank.

Carrot_Nation-3c

I was walking on a very dark street and I assumed the guy was talking on a cell phone, apparently about a dinner party. The voice was saying “Remember, I don’t eat red meat.” Just a few years ago, such a statement would have sounded strange. Of course, a few years ago a man talking to himself on the street would have been strange. He would have been assumed to be insane. Even more insane if he told you that he was actually talking on the telephone. But yesterday’s oddity pops up everywhere today. Neo-vegetarianism affects us all. Described very well by Jane Kramer’s excellent review of veggie cookbooks in the April 14 New Yorker,

“…from one chili party to the next, everything changed. Seven formerly enthusiastic carnivores called to say they had stopped eating meat entirely, and would like to join my vegetarians for the pesto. Worse, on the night of that final party, four of the remaining carnivores carried their plates to the kitchen table, ignoring the cubes of beef and pancetta, smoky and fragrant in their big red bean pot, and headed for my dwindling supply of pasta. “Stop!” I cried. “That’s for the vegetarians!”

Read the rest of this entry »

The  SBU (Swedish Council on Health Technology Assessment) is charged by the Swedish government with assessing health care treatments. Their recent acceptance of low-carbohydrate diets as best for weight loss is one of the signs of big changes in nutrition policy.  I am happy to reveal the next bombshell, this time from the American Diabetes Association (ADA) which will finally recognize the importance of reducing carbohydrate as the primary therapy in type 2 diabetes and as an adjunct in type 1.  Long holding to a very reactionary policy — while there were many disclaimers, the ADA has previously held 45 – 60 % carbohydrate as some kind of standard — the agency has been making slow progress. A member of the writing committee who wishes to remain anonymous has given me a copy of the 2014 nutritional guidelines due to be released next year, an excerpt from which, I reproduce below. Read the rest of this entry »

tarnowerhermanThe only person definitely known to have died as a consequence of an association with a low-carbohydrate diet is Dr. Herman Tarnower, author of the Scarsdale diet, although, as they used to say on the old TV detective shows, the immediate cause of death was lead poisoning. His girlfriend shot him. Not that folks haven’t been looking for other victims. The Atkins diet is still the bête noire of physicians, at least those who aren’t on it — a study published a few years ago said that physicians were more likely to follow a low carbohydrate diet when trying to lose weight themselves, while recommending a low fat diets for their patients.

Read the rest of this entry »

Paris. The summer of 1848. Mobs filled the streets, building barricades just like in Les Mis. If they’d had cars, they probably would have been set on fire.  In February of that year, the King, Louis-Phillipe, had abdicated in yet another French Revolution.  There was a new government, what is called the Second Republic, but whatever it tried to do, it didn’t make anybody happy and there was more unrest. At the Collège de France, faculty complained that it had “slackened the zeal for research among all of the chemists, and all of their time … is absorbed by politics.”

Horace_Vernet-Barricade_rue_Soufflot

 Figure 1. The Revolution of 1848. Barricades on the Rue Soufflot (Horace Vernet)

Bernard Read the rest of this entry »