Archive for the ‘Evidence Based Medicine’ Category

…the association has to be strong and the causality has to be plausible and consistent. And you have to have some reason to make the observation; you can’t look at everything.  And experimentally, observation may be all that you have — almost all of astronomy is observational.  Of course, the great deconstructions of crazy nutritional science — several from Mike Eades blog and Tom Naughton’s hysterically funny-but-true course in how to be a scientist —  are still right on but, strictly speaking, it is the faulty logic of the studies and the whacko observations that is the problem, not simply that they are observational.  It is the strength and reliability of the association that tells you whether causality is implied.  Reducing carbohydrates lowers triglycerides.  There is a causal link.  You have to be capable of the state of mind of the low-fat politburo not to see this (for example, Circulation, May 24, 2011; 123(20): 2292 – 2333).

It is frequently said that observational studies are only good for generating hypotheses but it is really the other way around.  All studies are generated by hypotheses.  As Einstein put it: your theory determines what you measure.  I ran my post on the red meat story passed April Smith  and her reaction was “why red meat? Why not pancakes” which is exactly right.  Any number of things can be observed. Once you pick, you have a hypothesis.

Where did the first law of thermodynamics come from?

Thermodynamics is an interesting case.  The history of the second law involves a complicated interplay of observation and theory.  The idea that there was an absolute limit to how efficient you could make a machine and by extension that all real processes were necessarily inefficient largely comes from the brain power of Carnot. He saw that you could not extract as work all of the heat you put into a machine. Clausius encapsulated it into the idea of the entropy as in my Youtube video.

©2004 Robin A. Feinman

 The origins of the first law, the conservation of energy, are a little stranger.  It turns out that it was described more than twenty years after the second law and it has been attributed to several people, for a while, to the German physicist von Helmholtz.  These days, credit is given to a somewhat eccentric German physician named Robert Julius Mayer. Although trained as a doctor, Mayer did not like to deal with patients and was instead more interested in physics and religion which he seemed to think were the same thing.  He took a job as a shipboard physician on an expedition to the South Seas since that would give him time to work on his main interests.  It was in Jakarta where, while treating an epidemic with the practice then of blood letting, that he noticed that the venous blood of the sailors was much brighter than when they were in colder climates as if “I had struck an artery.” He attributed this to a reduced need for the sailors to use oxygen for heat and from this observation, he somehow leapt to the grand principle of conservation of energy, that the total amount of heat and work and any other forms of energy does not change but can only be interconverted. It is still unknown what kind of connections in his brain led him to this conclusion.  The period (1848) corresponds to the point at which science separated from philosophy. Mayer seems to have had one foot in each world and described things in the following incomprehensible way:

  • If two bodies find themselves in a given difference, then they could remain  in a state of rest after the annihilation of [that] difference if the  forces that were communicated to them as a result of the leveling of  the difference could cease to exist; but if they are assumed to be indestructible,  then the still persisting forces, as causes of changes in relationship,  will again reestablish the original present difference.

(I have not looked for it but one can only imagine what the original German was like). Warmth Disperses and Time Passes. The History of Heat, Von Baeyer’s popular book on thermodynamics, describes the ups and downs of Mayer’s life, including the death of three of his children which, in combination with rejection of his ideas, led to hospitalization but ultimate recognition and knighthood.  Surely this was a great observational study although, as von Baeyer put it, it did require “the jumbled flashes of insight in that sweltering ship’s cabin on the other side of the world.”

It is also true that association does imply causation but, again, the association has to have some impact and the proposed causality has to make sense.  In some way, purely observational experiments are rare.  As Pasteur pointed out, even serendipity is favored by preparation.  Most observational experiments must be a reflection of some hypothesis. Otherwise you’re wasting tax-payer’s money; a kiss of death on a grant application is to imply that “it would be good to look at.…”  You always have to have something in mind.  The great observational studies like the Framingham Study are bad because they have no null hypothesis. When the Framingham study first showed that there was no association between dietary total and saturated fat or dietary cholesterol, the hypothesis was quickly defended. The investigators were so tied to a preconceived hypothesis, that there was hardly any point in making the observations.

In fact, a negative result is always stronger than one showing consistency; consistent sunrises will go by the wayside if the sun fails to come up once.  It is the lack of an association between the decrease in fat consumption during the epidemic of obesity and diabetes that is so striking.  The figure above shows that the  increase in carbohydrate consumption is consistent with the causal role of dietary carbohydrate in creating anabolic hormonal effects and with the poor satiating effects of carbohydrates — almost all of the increase of calories during the epidemic of obesity and diabetes has been due to carbohydrates.  However, this observation is not as strong as the lack of an identifiable association of obesity and diabetes with fat consumption.  It is the 14 % decrease in the absolute amount of saturated fat for men that is the problem.  If the decrease in fat were associated with decrease in obesity, diabetes and cardiovascular disease, there is little doubt that the USDA would be quick to identify causality.  In fact, whereas you can find the occasional low-fat trial that succeeds, if the diet-heart hypothesis were as described, they should not fail. There should not be a single Women’s Health Initiative, there should not be a single Framingham study, not one.

Sometimes more association would be better.  Take intention-to-treat. Please. In this strange statistical idea, if you assign a person to a particular intervention, diet or drug, then you must include the outcome data (weight loss, change in blood pressure) for that person even if the do not comply with the protocol (go off the diet, stop taking the pills).  Why would anybody propose such a thing, never mind actually insist on it as some medical journals or granting agencies do?  When you actually ask people who support ITT, you don’t get coherent answers.  They say that if you just look at per protocol data (only from people who stayed in the experiment), then by excluding the drop-outs, you would introduce bias but when you ask them to explain that you get something along the lines of Darwin and the peas growing on the wrong side of the pod. The basic idea, if there is one, is that the reason that people gave up on their diet or stopped taking the pills was because of an inherent feature of the intervention: made them sick, drowsy or something like that.  While this is one possible hypothesis and should be tested, there are millions of others — the doctor was subtly discouraging about the diet, or the participants were like some of my relatives who can’t remember where they put their pills, or the diet book was written in Russian, or the diet book was not written in Russian etc. I will discuss ITT in a future post but for the issue at hand:  if you do a per-protocol you will observe what happens to people when stay on their diet and you will have an association between the content of the diet and performance.  With an ITT analysis, you will be able to observe what happens when people are told to follow a diet and you will have an association between assignment to a diet and performance.  Both are observational experiments with an association between variables but they have different likelihood of providing a sense of causality.

“In the Viking era, they were already using skis…and over the centuries, the Norwegians have proved themselves good at little else.”

–John Cleese, Norway, Home of Giants.

With the 3-foot bookshelf of popular attacks on the low-fat-diet-heart idea it is pretty remarkable that there is only one defense.  Daniel Steinberg’s Cholesterol Wars. The Skeptics vs. The Preponderance of Evidence is probably more accurately called a witness for the prosecution since low-fat, in some way or other is still the law of the land.

The Skeptics vs. the Preponderance of Evidence

The Skeptics vs. the Preponderance of Evidence

The book is very informative, if biased, and it provides an historical perspective describing the difficulty of establishing the cholesterol hypothesis. Oddly, though,  it still appears to be very defensive for a witness for the prosecution.  In any case, Steinberg introduces into evidence the Oslo Diet-Heart Study [2] with a serious complaint:

“Here was a carefully conducted study reported in 1966 with a statistically significant reduction in reinfarction [recurrence of heart attack] rate.  Why did it not receive the attention it deserved?”

“The key element,” he says, “was a sharp reduction in saturated fat and cholesterol intake and an increase in polyunsaturated fat intake. In fact. each experimental subject had to consume a pint of soybean oil every week, adding it to salad dressing or using it in cooking or, if necessary, just gulping it down!”

Whatever it deserved, the Oslo Diet-Heart Study did receive a good deal of attention.  The Women’s Health Initiative (WHI), liked it.  The WHI was the most expensive failure to date. It found that “over a mean of 8.1 years, a dietary intervention that reduced total fat intake and increased intakes of vegetables, fruits, and grains did not significantly reduce the risk of CHD, stroke, or CVD in postmenopausal women.” [3]

The WHI, adopted a “win a few, lose a few” attitude, comparing its results to the literature, where some studies showed an effect of reducing dietary fat and some did not — this made me wonder: if the case is so clear, whey are there any failures.  Anyway, it cited the Oslo Diet-Heart Study as one of the winners and attributed the outcome to the substantial lowering of plasma cholesterol.

So, “cross-examination” would tell us why, if  “a statistically significant reduction in reinfarction  rate”  it did “not receive the attention it deserved?”

First, the effect of diet on cholesterol over five years:

The results look good although, since all the numbers are considered fairly high, and since the range of values is not shown, it is hard to tell just how impressive the results really are. But we will stipulate that you can lower cholesterol on a low-fat diet. But what about the payoff? What about the outcomes?

The results are shown in Table 5 of the original paper:   Steinberg described how in the first 5 years: “58 patients of the 206 in the control group (28%) had a second heart attack” (first 3 lines under first line of blue-highlighting) but only

“…  32 of the 206 in the diet (16%)…”  which does sound pretty good.

In the end, though, it’s really the total deaths from cardiac disease.  The second blue-highlighted line in Table 5 shows the two final outcome.  How should we compare these.

1. The odds ratio or relative risk is just the ratio of the two outcomes (since there are the same number of subjects) = CHD mortality (diet)/ CHD mortality control) = 94/79 =  1.19.  This seems strikingly close to 1.0, that is, flip of a coin.  These days the media, or the report itself, would report that there was a 19 % reduction in total CHD mortality.

2, If you look at the absolute values, however, the  mortality in the controls is 94/206 = 45.6 % but the diet group had reduced this  to 79/206 = 38.3 % so the change in absolute risk is  45.6 % – 38.3 % or only 7.3 % which is less impressive but still not too bad.

3. So for every 206 people, we save 94-79 = 15 lives, or dividing 206/15 = 14 people needed to treat to save one life. (Usually abbreviated NNT). That doesn’t sound too bad.  Not penicillin but could be beneficial. I think…

Smoke and mirrors.

It’s what comes next that is so distressing.  Table 10 pools the two groups, the diet and the control group and now compares  the effect of smoking: on the whole population,  the ratio of CHD deaths in smokers vs non-smokers is 119/54 = 2.2 (magenta highlight) which is somewhat more impressive than the 1.19 effect we just saw.  Now,

1. The absolute difference in risk is (119-54)/206 = 31.6 % which sounds like a meaningful number.

2. The number needed to treat is 206/64 = 3.17  or only about 3 people need to quit smoking to see one less death

In fact, in some sense, the Oslo Diet-Heart Study provides smoking-CHD risk as an example of a meaningful association that one can take seriously. If only such a significant change had actually been found for the diet effect.

So what do the authors make of this? Their conclusion is that “When combining data from both groups, a three-fold greater CHD mortality rate is demonstrable among the hypercholesterolemic, hypertensive smokers than among those in whom these factors were low or absent.”  Clever but sneaky. The “hypercholesterolemic, hypertensive” part is irrelevant since you combined the groups. In other words, what started out as a diet study has become a “lifestyle study.”  They might has well have said “When combining data from fish and birds a significant number of wings were evident.” Members of the jury are shaking their heads.

Logistic regression. What is it? Can it help?

So they have mixed up smoking and diet. Isn’t there a way to tell which was more important?  Well, of course, there are several ways.  By coincidence, while I was writing this post, April Smith posted on facebook, the following challenge “The first person to explain logistic regression to me wins admission to SUNY Downstate Medical School!” I won although I am already at Downstate.  Logistic regression is, in fact, a statistical method that asks what the relative contribution of different inputs would have to be to fit the outcome and this could have been done but in this case, I would use my favorite statistical method, the Eyeball Test.  Looking at the data in Tables 5 and 10 for CHD deaths, you can see immediately what’s going on. Smoking is a bigger risk than diet.

If you really want a number, we calculated relative risk above. Again, we found for mortality, CHD (diet)/ CHD (control) = 94/79 =  1.19. But what happens if you took up smoking: Figure 10 shows that your chance of dying of heart disease would be increased by 119/54 = 2.2  or more than twice the risk.  Bottom line: you decided to add saturated fat to your diet, your risk would be 1.19 what it was before which might be a chance you could take faced with authentic Foie Gras.

Daniel Steinberg’s question:

“Here was a carefully conducted study reported in 1966 with a statistically significant reduction in reinfarction  rate.  Why did it not receive the attention it deserved?”

Well, it did. This is not the first critique.  Uffe Ravnskov described how the confusion of smoking and diet led to a new Oslo Trial which reductions in both were specifically recommended and, again, outcomes made diet look bad [4].  Ravnskov gave it the attention it deserved. But what about researchers writing in the scientific literature. Why do they not give the study the attention it deserves. Why do they not point out its status as a classic case of a saturated fat risk study with no null hypothesis.  It certainly deserves attention for its devious style. Of course, putting that in print would guarantee that your grant is never funded and your papers will be hard to publish.  So, why do researchers not give the Oslo-Diet-Heart study the attention it deserves?  Good question, Dan.

Bibliography

1. Steinberg D: The cholesterol wars : the skeptics vs. the preponderance of evidence, 1st edn. San Diego, Calif.: Academic Press; 2007.

2. Leren P: The Oslo diet-heart study. Eleven-year report. Circulation 1970, 42(5):935-942.

3. Howard BV, Van Horn L, Hsia J, Manson JE, Stefanick ML, Wassertheil-Smoller S, Kuller LH, LaCroix AZ, Langer RD, Lasser NL et al: Low-fat dietary pattern and risk of cardiovascular disease: the Women’s Health Initiative Randomized Controlled Dietary Modification Trial. JAMA 2006, 295(6):655-666.

4. Ravnskov U: The Cholesterol Myths: Exposing the Fallacy that Cholesterol and Saturated Fat Cause Heart Disease. Washington, DC: NewTrends Publishing, Inc.; 2000.

In 1985 an NIH Consensus Conference was able to “establish beyond any reasonable doubt the close relationship between elevated blood cholesterol levels (as measured in serum or plasma) and coronary heart disease” (JAMA 1985, 253:2080-2086).

I have been making an analogy between scientific behavior and the activities of the legal system and following that idea, the wording of the conference conclusion suggests a criminal indictment. Since the time of the NIH conference, however, data on the role of cholesterol fractions, the so-called “good (HDL)” and “bad (LDL)” cholesterols and, most recently, the apparent differences in the atherogenicity of different LDL sub-fractions would seem to have provided some reasonable doubt. What has actually happened is that the nutrition establishment, the lipophobes as Michael Pollan calls them, has extended the indictment to include dietary fat, especially saturated fat at least as accessories on the grounds that, as the Illinois Criminal Code put it “before or during the commission of an offense, and with the intent to promote or facilitate such commission, … solicits, aids, abets, agrees or attempts to aid… in the planning or commission of the offense. . . ..”

A major strategy in the indictment of saturated fat has been guilt by association.  The American Heart Association (AHA), which had long recommended margarine (the major source of trans-fats), has gone all out in condemning saturated fatty acids by linking them with trans-fats.  The AHA website has a truly deranged cartoon film of the evil brothers: “They’re a charming pair, Sat and Trans.  But that doesn’t mean they make good friends.  Read on to learn how they clog arteries and break hearts — and how to limit your time with them by avoiding the foods they’re in.”. While the risk of trans-fats is probably exaggerated — they are a small part of the diet — they have no benefit and nobody wants to defend them; dietary saturated fat, however, is a normal part of the diet, is made in your body and is less important in providing saturated fatty acids in the blood, than dietary carbohydrate.  Guilt by association is a tricky business in courts of law — just having a roommate who sells marijuana can get you into a good deal of trouble — but it takes more than somebody saying that you and the perpetrator make a charming pair.

The failure of the diet-cholesterol-heart hypothesis in clinical trials as been documented by numerous scientific articles and especially in popular books that document the original scientific sources. It is unknown what the reaction of the public is to these books.  However, amazingly, there is only one book I know of that takes the side of the lipophobes and that is Daniel Steinberg’s Cholesterol Wars. The Skeptics vs. the Preponderance of Evidence. A serious book with careful if slightly biased documentation and an uncommon willingness to answer the critics,  it is worth reading.  I will try to discuss it in detail in this and future posts.  First, the title indicates a step down from criminal prosecution.  “Preponderance of the evidence” is the standard for conviction in a civil court and is obviously a far weaker criterion.  One has to wonder why it is that the skeptics have the preponderance of the popular publications — if the scientific evidence is there and health agencies are so determined that the public know about this, why are there so few —  maybe only this one — rebutting the critics.

The Skeptics vs. the Preponderance of Evidence

In any case, what is Steinberg’s case?  The indictment on page 1 is somewhat different than one would have thought.

“….the [lipid] hypothesis relates to blood lipids not dietary lipids as the putative directly causative factor. Although diet, especially dietary lipid is an important determinant of blood lipid levels, many other factors play important roles. Moreover, there is a great deal of variability in response of individuals to dietary manipulations. Thus, it is essential to distinguish between the indirect “diet-heart” connection and the direct “blood lipid — hard” connection failure to make this distinction has been a frequent source of confusion. (his italics)”

What?  Are we really supposed to believe that diet is an incidental part of the lipid hypothesis?  Are we supposed to believe that our cholesterol is just a question of the variability of our response to diet.  Has the message really been that diet is not critical and that heart-disease is just the luck of the draw (until we start taking statins)?  This is certainly the source of confusion in my mind.  Of course by page 5, we are confronted with this:

“In 1966, Paul Leren published his classic five-year study of 412 patients who had had a prior myocardial infarction. He showed that substitution of polyunsaturated fat and saturated fat-rich butter-cream-venison diet favored by the Norwegians reduced their blood cholesterol by about 17 per cent and kept it down.  The number of secondary current events in the treated group was reduced by about one-third and the result was significant at the p < 0.03 level.”

In a future post, I will describe Paul Leren’s classic five-year study which, by 1970, had a follow-up to eleven years and the results will turn out not to be as compelling as described by Steinberg.  For the moment, it is worth considering that, given the strong message, from the AHA, from the American Diabetes Association, from the NIH Guidelines for Americans, the criterion really should be beyond a reasonable doubt. There shouldn’t be even a single failure like the Framingham Study or the Women’s Health Initiative. In fact, the preponderance of the evidence when you add them all up, isn’t there.

The phrase “Evidence-based Medicine” (EBM) guarantees its proponents a certain degree of protection. After all, who would be against medicine that is based on the data, on hard facts rather than opinion. On the other hand, a study that needs to cloak itself in such a self-aggrandizing phrase must raise a few eyebrows; as usual, the Dietary Guidelines, moves to the top of the list in that category but there are many examples.  Martin Tobin, professor of Medicine at Loyola College provided an excellent deconstruction of evidence based medicine [1]. Some of his points were that the grading system has divorced itself from basic science.  For example, he points out that:

  • “ homeopathy uses drugs in which less than one molecule of active agent is present. … A meta-analysis of 89 placebo-controlled trials revealed a combined odds of 2.45 in favor of homeopathy. EBM grades meta-analysis as level 1 evidence but completely ignores scientific theory. There is nothing necessarily wrong with this particular meta-analysis, but the example illustrates how a system that grades findings of all meta-analyses as level 1 evidence is inherently flawed.  A grading system that ranks homeopathy as sounder evidence than centuries of pharmacologic science commits the reductio ad absurdum fallacy in logic.” [1]

Among the things that we found in our critique of the USDA dietary guidelines Report [2] was that the cited evidence did not meet their own standards. They were critical of low-carbohydrate diets on the basis of studies that their own analysis gave a “neutral” quality rating, even those that took dietary assessment at baseline and then assessed  cardiovascular mortality up to 12 years later.

But it is really the idea that there is some set of systematic definitions of science that everybody agrees on. My last post mentioned, by analogy with courts of law, the Frye standard which accepts as evidence, opinions supported by  “general acceptance’ in the scientific community.  While still accepted in some state courts, the federal courts have tried to go beyond trust in such narrow descriptions of science. In 1975, Congress established Federal Rules of Evidence.  The rules are quite general and the major impact is to broaden the range of evidence that could be considered.  Rule 401, defined relevance as  “evidence having any tendency to make the existence of any fact that is of consequence to the determination of the action more probable or less probable than it would be without the evidence,”  in other words, whatever works.  In a future post, I will discuss Daubert v. Merrell Dow, Inc. (pr. Dow-burt as in English), an outgrowth of the Rules of Evidence and generally considered the key judgment in the modern interaction of science and the law.  In the real world of jurisprudence, ideas on what constitutes scientific evidence have become problematical and Daubert may have had the paradoxical effect of restricting admissible data but, in the analogy with evidence in medicine, the Federal Rules of Evidence and Daubert have better captured the real quality of science in recognizing the need for flexibility. The kinds of absolute criteria — association does not imply causality, random controlled trials are a “gold standard,” etc. are at least different from the spirit of Daubert.

More important, nobody in any physical science would recognize the tables of levels of evidence.  A random controlled trial may be good for one kind of experiment but not for another and EBM is critical of “observational studies” but all of astronomy is observational.  In the end, most scientists would agree with the physicist Steven Weinberg, echoing Judge Potter Stewart’s famous take on pornography:

  •  “There is no logical formula that establishes a sharp dividing line between a beautiful explanatory theory and a mere list of data, but we know the difference when we see it — we demand a simplicity and rigidity in our principles before we are willing to take them seriously [3].”

So where do these arbitrary guidelines in EBM come from?  They were set up by the  medical community, a community that is stereotyped as being untrained in science. I hate stereotypes, especially medical stereotypes since I think of myself as coming from a medical family (my father and oldest daughter are physicians) but stereotypes come from someplace and, of course, it is well known that physicians never study nutrition.  In the end, it makes me think of the undoubtedly apocryphal story about Mozart.

  • A man comes to Mozart and wants to become a composer.  Mozart says that they have to study theory for  a couple of years, they should study orchestration and become proficient  at the piano, and goes on like this.  Finally, the man says “but you wrote your first symphony when you were 8 years old.”  Mozart says “Yes, but I didn’t ask anybody.”

Bibliography 

1. Tobin MJ: Counterpoint: evidence-based medicine lacks a sound scientific base. Chest 2008, 133(5):1071-1074; discussion 1074-1077.

2. Hite AH, Feinman RD, Guzman GE, Satin M, Schoenfeld PA, Wood RJ: In the face of contradictory evidence: report of the Dietary Guidelines for Americans Committee. Nutrition 2010, 26(10):915-924.

3. Weinberg S: Dreams of a final theory, 1st edn. New York: Pantheon Books; 1992.

According to the Journal of the American Medical Association (JAMA), the principle of “evidence-based medicine (EBM),” arose in the 1990s [1]. It is widely invoked in the medical literature as a kind of certification that the conclusions of the author are not mere opinions but are backed up by compelling information in biomedical science.  It sounds good. Or does it? It is certainly self-serving and a little bit suspicious, somewhat like Nixon assuring us that he was not a crook.  Evidence based medicine?  What were we doing before?  How was Pasteur able to function in the absence of such an idea?  One thing to think about is that evidence is what is introduced into courts of law.  But not all evidence is admissible. A judge decides what is admissible and there are many precedents, in particular, on what constitutes scientific evidence in a legal proceeding.

EBM relies on a hierarchy of levels of evidence (e.g. Table 1) with the random controlled trial (RCT) as the highest and expert opinion as the lowest.  Recommendations from health agencies and awarding of research grants are frequently justified on conformity to EBM or at least on their placing primary importance on RCTs.

Evidence from the USDA

Early in 2011, the USDA released its 2010 Dietary Guidelines for Americans [2].  With the dates suggesting the backward-looking nature of the Guidelines, they were nonetheless based on the Report of a prestigious committee (DGAC) [3] who, in turn, made much of their reliance on a new Nutrition Evidence Library (NEL). I and my colleagues were invited to submit a critique of the Report by the journal Nutrition. The editor, Michael Meguid indicated that the journal wanted a balanced report, pros and cons.  I called Dr. Meguid:

RDF: You know, the report is not particularly balanced. I’m not sure how you write a balanced review of an unbalanced report.

MM:  You can make the critique as strong as you like as long as you carefully document everything. But what’s your main problem with the Report?

RDF: Well, it makes very strong recommendations in the face of contradictory evidence.

MM: Make that the title of your article.

So we wrote an article called “In the face of contradictory evidence: Report of the Dietary Guidelines for Americans Committee” [4]. The journal was kind enough to make it an open access article and it’s available on this blog. In the end, on titles, we were one-upped by Steven Malanga, whose article in the New York Post was called “Fed’s Food Fog.”

For sure, both the Report and the final Guidelines were the proverbial camel-like production of a committee, tedious, repetitive and stylistic dreadful.  But what about the NEL?  What about the evidence?  Style aside, wasn’t this evidence based medicine?

Where do these guidelines come from? The assumption is that evidence follows its etymologic roots, stuff that is visible, stuff that comes from the sensible and true avouch of our own eyes. In fact, it is most often applied, as in the case of the DGAC, to the most controversial and contentious subjects. Calling something evidence is not enough. So what happens in courts of law? In a court of law, a judge decides on whether the jury can hear the evidence.  Who decides admissibility of the evidence in EBM?

Conflict resolution in science.

Science is a human activity. Conflict, controversy and a resistance to new ideas are well known even in the so-called hard, that is, more mathematical, sciences, and even where there are no outside forces as there was in the case of Galileo.  In the twentieth century, conflicts do not generally impede progress for long. Especially in the physical sciences, there is usually agreement on basic assumptions and on the rules of logic, allowing ultimate acceptance of strong evidence. Competing theories may coexist and supporters of both are likely to admit that they are awaiting reconciliation.

What happens when the spontaneous process of conflict resolution in science breaks down?  What happens in conditions where scientific disagreement is strong and a majority position becomes so dominant that it controls the funding and publication of scientific work and can ignore or repress contradictory evidence and repress exposition of alternative theories.  In essence, how do we deal with a recapitulation of the case of Galileo?

There is no system to decide on the admissibility in the cases considered by EBM.   I am not the first person to point out that EBM is largely the position of experts on one side of a scientific conflict [5], the lowest level of evidence on traditional EBM scales (e.g. “Level III: Opinions of respected authorities… of the US Preventive Services Task Force Systems,” Table 1).  EBM is sustained by those who want to use its particular criteria but these have never been subjected to outside affirmation.

In this situation, where science cannot police itself, we have to look for some outside guidance.  What do the courts do?  As one would expect there, is a long and extensive history of the legal system’s  attempt to deal with what constitutes scientific evidence.  On the chance that the legal perspective may help, I will discuss some of the issues.

Frye and the need for rules.

A key decision in the history of science in the courts is Frye v. United States.  In 1923, a Federal Appeals court ruled that the opinions of experts have to be supported by a scientific community. Frye had been convicted of second-degree murder but appealed on the grounds that he had successfully passed a lie-detector test.  At that time the device was a simple blood pressure machine and an expert witness testified as to the results. The court ruled that the lie-detector test “has not yet gained such standing and scientific recognition among physiological and psychological authorities as would justify the courts in admitting expert testimony‚” affirming the judgment of the lower court.

The ruling in Frye gave rise to the idea of “general acceptance,” and, by analogy, this appears to be the main principle in the admissibility of evidence in the nutrition world.  Sufficiently well established that it could be included in a biochemistry text is the idea that “consumption of saturated fats is positively associated with high levels of total plasma cholesterol and LDL cholesterol and an increased risk of CHD”[6] Known to students as “the Lippincott Book,” Harvey and Ferrier is the best selling biochemistry book in the world and it is correct when it states “Most experts strongly advise limiting intake of saturated fats.”

Most, but not all.  A small but not insignificant minority hold otherwise and whereas they agree that dietary saturated fat may raise blood cholesterol, they can provide overwhelming evidence that it is not associated with cardiovascular disease. This has been demonstrated in almost every large trial.

The problem is described in Marcia Angell’s Science on Trial [7].  Angell explains that Frye was not without its critics ([7], page 126).  Opponents, she wrote,

“claimed somewhat improbably, that it would tend to exclude novel, far-sighted testimony by modern-day Galileos. There is no record of this happening once, let alone often.  Furthermore, even if a modern-day Galileo did not make into court at first, that fact should not stop him from prevailing in the scientific community.  Courts do not determine scientific acceptance, as implied by the argument that we need to keep our courts open to the hidden Galileos in our midst.”

But isn’t this exactly what has happened in nutrition and maybe, in general, in the medical community?  The “experts” control editorial boards, granting agencies and academic departments and are as powerful as the Catholic Church in repressing dissent.  They have prestige and, in many cases, undisputed accomplishments, but does science run on general acceptance? Does majority (of experts) rule?

One of the problems with Frye that lawyers have addressed is a question of identifying the field of academic or scientific field in which the general acceptance is to be considered.  Different disciplines hold to different standards.  In the case at hand, many ideas in nutrition would be dismissed out of hand by biochemists. Many methodologies would be considered absurd by physical scientists: Intention-to-treat is perhaps the most absurd.  It has been pointed out that the question of who is an expert might have applied to the techniques in the original Frye case, at least as it might be implemented today: “If polygraph examiners are selected as the relevant field, polygraph results would be admissible.” (http://law.jrank.org/pages/2006/Scientific-Evidence-Frye-v-United-States.html).

The epidemic of obesity and diabetes stands as a testament to the failure of the experts.  A small library can be assembled of books attacking establishment medical nutrition. Uffe Ravnskov’s classic Cholesterol Myths is updated in Ignore the Awkward. Gary Taubes’s recent Good Calories, Bad Calories is the most compelling and James Le Fanu’s Rise and Fall of Modern Medicine, the most succinct but just sitting at my desk now I can see a dozen others on the book shelf.  Surprisingly, there has been only one rebuttal, Steinberg’s Cholesterol Wars, the subject of the next post .

Table 1.  Examples of Levels of Evidence from Various Sources. 

US Preventive Services Task Force Systems for ranking evidence about the effectiveness of treatments or screening:

Level I: Evidence obtained from at least one properly designed randomized controlled trial.

Level II-1: Evidence obtained from well-designed controlled trials without randomization.

Level II-2: Evidence obtained from well-designed cohort or case-control analytic studies, preferably from more than one center or research group.

Level II-3: Evidence obtained from multiple time series with or without the intervention. Dramatic results in uncontrolled trials might also be regarded as this type of evidence.

Level III: Opinions of respected authorities, based on clinical experience, descriptive studies, or reports of expert committees.

Bibliography

1. Torpy JM, Lynm C, Glass RM: JAMA patient page. Evidence-based medicine. JAMA 2009, 301(8):900.

2. Dietary Guidelines for Americans, 2010 [http://www.dietaryguidelines.gov.]

3. US Department of Agriculture and US Department of Health and Human Services: Report of the Dietary Guidelines Advisory Committee on the dietary guidelines for Americans, 2010. June 15, 2010. In.; 2010.

4. Hite AH, Feinman RD, Guzman GE, Satin M, Schoenfeld PA, Wood RJ: In the face of contradictory evidence: report of the Dietary Guidelines for Americans Committee. Nutrition 2010, 26(10):915-924.

5. Marantz P, Bird E, Alderman M: A Call for Higher Standards of Evidence for Dietary Guidelines. Am J Prev Med 2008, 34(3):234-239.

6. Harvey R, Ferrier D: Biochemistry, 5th edn. Baltimore and Philadelphia: Lippincott Williams & Wilkins; 2011.

7. Angell M: Science on Trial. New York: W. W. Norton & Co.; 1996.

I don’t believe in time travel, of course, so when somebody sent me the following article that was supposed to be a chapter from a Study of the History of Diabetes published in 2018, I didn’t think about it much.  Then I read an article about a woman who had been charged with neglect in the death of her son from complications due to diabetes.  It seems she “was trying to live by faith and felt like God would heal him.”

For some reason, that made me think of the Future History, so here is a chapter from the History.

Chapter IV.  ACCORD to The Court

We have seen how, early in the history of medicine, diabetes was recognized as a disease of carbohydrate intolerance and how, until the discovery of insulin, removing carbohydrate from the diet became the major treatment (Chapters I and II).  We chronicled the shift away from this medical practice under the influence of low fat recommendations and the ascendancy of pharmacology that followed the discovery of insulin.  Nonetheless, it persisted in the popular mind that you don’t give candy to people with diabetes, even as health agencies seemed to encourage sucrose (sugar) consumption.

The rather sudden reappearance of carbohydrate restriction, the so-called modern era in diabetes treatment, is usually dated to 2008, the precipitating event, publication of the ACCORD study in which a group undergoing  “intensive treatment” to lower blood glucose showed unexpected deaths [1].  ACCORD concluded that “These findings identify a previously unrecognized harm of intensive glucose lowering in high-risk patients with type 2 diabetes.” The intensive treatment turned out to be intensive pharmacologic therapy and this flawed logic lead to a popular uprising of sorts, a growing number of patients claiming that they had been hurt by intensive drug treatment and typically that they had only been able to get control of their diabetes by adherence to low carbohydrate diets. Blogs compared the ACCORD conclusion to an idea that alleviating headaches with intensive aspirin led to bleeding and we should therefore not treat headaches.

The conflict culminated in the large judgment for the plaintiff in Banting v. American Diabetes Association (ADA) in 2017, affirmed by the Supreme Court in 2018.  Dalton Banting, coincidentally a distant relative of the discoverer of insulin, was an adolescent with diabetes who took prescribed medications and followed a diet consistent with ADA recommendations.  He experienced worsening of his symptoms and ultimately had a foot amputated. At this point his parents found a physician who recommended a low carbohydrate diet which led to rapid and sustained improvement.  The parents claimed their son should have been offered carbohydrate-restriction as an option.  The case was unusual in that Banting had a mild obsessive-compulsive condition, expressed as a tendency to follow exactly any instructions from his parents or other authority figures.  Banting’s lawyers insisted that, as a consequence, one could rely on his having complied with the ADA’s recommendations.  Disputed by the defense, this was one of several issues that made Banting famous for vituperative courtroom interactions between academics.

Banting was a person with type 2 diabetes.  Unlike people with type 1 diabetes, he was able to produce insulin in response to dietary (or systemic) glucose but his pancreas was progressively dysfunctional and his body did not respond normally, that is, he was insulin-resistant.  Although most people with type 2 diabetes are at least slightly overweight, Banting was not, although he began gaining weight when treated with insulin.

The phrase “covered with insulin…” rocked the court: the president of the ADA, H. Himsworth, Jr., was asked to  read from the 2008 guidelines [2], never rescinded: “Sucrose-containing foods can be substituted for other carbohydrates in the meal plan or, if added to the meal plan, covered with insulin or other glucose lowering medications.”

Jaggers (attorney for Banting): “Are there other diseases where patients are counseled to make things worse so that they can take more drugs.”

Himsworth: “We only say ‘can be.’  We don’t necessarily recommend it.  We do say that ‘Care should be taken to avoid excess energy intake.’”

It soon became apparent that Himsworth was in trouble.  He was asked to read from the passage explaining the ADA’s opposition to low carbohydrate diets:

“Low-carbohydrate diets might seem to be a logical approach to lowering postprandial glucose. However, foods that contain carbohydrate are important sources of energy, fiber, vitamins, and minerals and are important in dietary palatability.”

Jaggers: “Important sources of energy?  I thought we wanted to avoid excess energy,” and “would you say that taking a vitamin pill is in the same category as injecting insulin?”

Finally,

Jaggers: “Dr. Himsworth, as an expert on palatability, could you explain the difference between Bordelaise sauce and Béarnaise sauce?” [laughter]

Damaging as this testimony was, the tipping point in the trial is generally considered to have been the glucometer demonstration.  Banting consumed a meal typical of that recommended by the ADA  and glucometer readings were projected on a screen for the jury, showing, on this day, so-called “spikes” in blood glucose.  The following day, Banting consumed a low carbohydrate meal and the improved glucometer readings were again projected for the jury.  Defense argued that one meal did not prove anything and that one had to look at the whole history of the lifestyle intervention but was unable to show any evidence of harm from continued maintenance of low blood sugar despite testimony of several expert witnesses.  In the end, the jury agreed that common sense overrides expert testimony and that Banting should have been offered the choice of a carbohydrate-restricted diet.

Banting was held in New York State which adheres to the Frye standard: in essence, the idea that scientific evidence is determined by “general acceptance.” The explicit inclusion of common sense was, in fact, a legal precedent [3].   The Supreme Court ultimately concurred and held that the more comprehensive standards derived from Daubert v. Merrill-Dow, could sensibly be seen to encompass common sense.

The final decision in Banting lead to numerous law suits.  The ADA and other agencies changed their tactics claiming that they never were opposed to low carbohydrate diets and, in fact, had been recommending them all along [4].  This is discussed in the next chapter.

References

1. Gerstein, H. C. et al., Effects of intensive glucose lowering in type 2 diabetes. N Engl J Med 358 (24), 2545 (2008).

2. American Diabetes Association, Nutrition Recommendations and Interventions for Diabetes–2008. Diabetes Care 31 (Suppl 1), S61 (2008).

3. Berger, M, Expert Testimony: The Supreme Court’s Rules Issues in Science and Technology (2000).

4. American Diabetes Association, Nutrition Recommendations and Interventions for Diabetes–2018. Diabetes Care 40 (Suppl 1), S12 (2018).