Evidence-based Medicine: Who Decides Admissibility? I. The Frye Standard

Posted: May 6, 2011 in Evidence Based Medicine, The Nutrition Story
Tags: , , , , ,

According to the Journal of the American Medical Association (JAMA), the principle of “evidence-based medicine (EBM),” arose in the 1990s [1]. It is widely invoked in the medical literature as a kind of certification that the conclusions of the author are not mere opinions but are backed up by compelling information in biomedical science.  It sounds good. Or does it? It is certainly self-serving and a little bit suspicious, somewhat like Nixon assuring us that he was not a crook.  Evidence based medicine?  What were we doing before?  How was Pasteur able to function in the absence of such an idea?  One thing to think about is that evidence is what is introduced into courts of law.  But not all evidence is admissible. A judge decides what is admissible and there are many precedents and, in particular what constitutes scientific evidence is widely studied in the law.

EBM relies on a hierarchy of levels of evidence (e.g. Table 1) with the random controlled trial (RCT) as the highest and expert opinion as the lowest.  Recommendations from health agencies and awarding of research grants are frequently justified on conformity to EBM or at least on their placing primary importance on RCT.

Evidence from the USDA

Early in 2011, the USDA released its 2010 Dietary Guidelines for Americans [2].  With the dates suggesting the backward-looking nature of the Guidelines, they were nonetheless based on the Report of a prestigious committee (DGAC) [3] who, in turn, made much of their reliance on a new Nutrition Evidence Library (NEL). I and my colleagues were invited to submit a critique of the Report by the journal Nutrition. The editor, Michael Meguid indicated that the journal wanted a balanced report, pros and cons.  I called Dr. Meguid:

RDF: You know, the report is not particularly balanced. I’m not sure how you write a balanced review of an unbalanced report.

MM:  You can make the critique as strong as you like as long as you carefully document everything. But what’s your main problem with the Report?

RDF: Well, it makes very strong recommendations in the face of contradictory evidence.

MM: Make that the title of your article.

So we wrote an article called “In the face of contradictory evidence: Report of the Dietary Guidelines for Americans Committee” [4]. The journal was kind enough to make it an open access article and it’s available on this blog. In the end, we were one-upped by Steven Malanga, whose article in the New York Post was called “Fed’s Food Fog.”

For sure, both the Report and the final Guidelines were the proverbial camel-like production of a committee, tedious, repetitive and stylistic dreadful.  But what about the NEL?  What about the evidence?  Style aside, wasn’t this evidence based medicine?

Where do these guidelines come from? The assumption is that evidence follows its etymologic roots, stuff that is visible, stuff that comes from the sensible and true avouch of our own eyes. In fact, it is most often applied, as in the case of the DGAC, to the most controversial and contentious subjects. Calling something evidence is not enough. So what happens in courts of law? In a court of law, a judge decides on whether the jury can hear the evidence.  Who decides admissibility of the evidence in EBM?

Conflict resolution in science.

Science is a human activity. Conflict, controversy and a resistance to new ideas are well known even in the so-called hard, that is, more mathematical, sciences, and even where there are no outside forces as there was in the case of Galileo.  In the twentieth century, conflicts do not generally impede progress for long. Especially in the physical sciences, there is usually agreement on basic assumptions and on the rules of logic, allowing ultimate acceptance of strong evidence. Competing theories may coexist and supporters of both are likely to admit that they are awaiting reconciliation.

What happens when the spontaneous process of conflict resolution in science breaks down?  What happens in conditions where scientific disagreement is strong and a majority position becomes so dominant that it controls the funding and publication of scientific work and can ignore or repress contradictory evidence and repress exposition of alternative theories.  In essence, how do we deal with a recapitulation of the case of Galileo?

There is no system to decide on admissibility is applied in the cases considered in EBM.   I am not the first person to point out that EBM is largely the position of experts on one side of a scientific conflict [5], the lowest level of evidence on traditional EBM scales (e.g. Level III: Opinions of respected authorities… of the US Preventive Services Task Force Systems, Table 1).  EBM is sustained by those who want to use its particular criteria and these have never been subjected to outside affirmation.

In this situation, where science cannot police itself, we have to look for some outside guidance.  What do the courts do?  As one would expect there, is a long and extensive history of the legal system’s  attempt to deal with what constitutes scientific evidence.  On the chance that the legal perspective may help, I will discuss some of the issues.

Frye and the need for rules.

A key decision in the history of science in the courts is Frye v. United States.  In 1923, a Federal Appeals court ruled that the opinions of experts have to be supported by a scientific community. Frye had been convicted of second-degree murder but appealed on the grounds that he had successfully passed a lie-detector test.  At that time the device was a simple blood pressure machine and an expert witness testified as to the results. The court ruled that the lie-detector test “has not yet gained such standing and scientific recognition among physiological and psychological authorities as would justify the courts in admitting expert testimony‚” affirming the judgment of the lower court.

The ruling in Frye gave rise to the idea of “general acceptance,” and this appears to be the main principle in the admissibility of evidence in the nutrition world.  Sufficiently well established that it could be included in a biochemistry text is the idea that “consumption of saturated fats is positively associated with high levels of total plasma cholesterol and LDL cholesterol and an increased risk of CHD”[6] Known to students as “the Lippincott Book,” Harvey and Ferrier is the best selling biochemistry book in the world and it is correct when it states “Most experts strongly advise limiting intake of saturated fats.”

Most, but not all.  A small but not insignificant minority hold otherwise and whereas they agree that dietary saturated fat raises cholesterol, they can provide overwhelming evidence that it is not associated with cardiovascular disease. This has been demonstrated in almost every large trial.

The problem is described in Marcia Angell’s Science on Trial [7].  Angell explains that Frye was not without its critics ([7], page 126).  Opponents, she wrote,

“claimed somewhat improbably, that it would tend to exclude novel, far-sighted testimony by modern-day Galileos. There is no record of this happening once, let alone often.  Furthermore, even if a modern-day Galileo did not make into court at first, that fact should not stop him from prevailing in the scientific community.  Courts do not determine scientific acceptance, as implied by the argument that we need to keep our courts open to the hidden Galileos in our midst.”

But isn’t this exactly what has happened in nutrition and maybe, in general, in the medical community?  The “experts” control editorial boards, granting agencies and academic departments and are as powerful as the Catholic Church in repressing dissent.  They have prestige and, in many cases, undisputed accomplishments but does science run on general acceptance? Does majority (of experts) rule?

One of the problems with Frye that lawyers have addressed is a question of identifying the field of academic or scientific field in which the general acceptance is to be considered.  Different disciplines hold to different standards.  In the case at hand, many ideas in nutrition would be dismissed out of hand by biochemists. Many methodologies would be considered absurd by physical scientists: Intention-to-treat in a future post.  It has been pointed out that this applies to the techniques as in the original Frye case, at least as implemented today: “If polygraph examiners are selected as the relevant field, polygraph results would be admissible.” (http://law.jrank.org/pages/2006/Scientific-Evidence-Frye-v-United-States.html).

The epidemic of obesity and diabetes stands as a testament to the failure of the experts.  A small library can be assembled of books attacking establishment medical nutrition. Uffe Ravnskov’s classic Cholesterol Myths is updated in Ignore the Awkward. Gary Taubes’s recent Good Calories, Bad Calories is the most compelling and James Le Fanu’s Rise and Fall of Modern Medicine, the most succinct but just sitting at my desk now I can see a dozen others on the book shelf.  Surprisingly, there has been only one rebuttal, Steinberg’s Cholesterol Wars, the subject of the next post .

Table 1.  Examples of Levels of Evidence from Various Sources. 

US Preventive Services Task Force Systems for ranking evidence about the effectiveness of treatments or screening:

Level I: Evidence obtained from at least one properly designed randomized controlled trial.

Level II-1: Evidence obtained from well-designed controlled trials without randomization.

Level II-2: Evidence obtained from well-designed cohort or case-control analytic studies, preferably from more than one center or research group.

Level II-3: Evidence obtained from multiple time series with or without the intervention. Dramatic results in uncontrolled trials might also be regarded as this type of evidence.

Level III: Opinions of respected authorities, based on clinical experience, descriptive studies, or reports of expert committees.

Bibliography

1. Torpy JM, Lynm C, Glass RM: JAMA patient page. Evidence-based medicine. JAMA 2009, 301(8):900.

2. Dietary Guidelines for Americans, 2010 [http://www.dietaryguidelines.gov.]

3. US Department of Agriculture and US Department of Health and Human Services: Report of the Dietary Guidelines Advisory Committee on the dietary guidelines for Americans, 2010. June 15, 2010. In.; 2010.

4. Hite AH, Feinman RD, Guzman GE, Satin M, Schoenfeld PA, Wood RJ: In the face of contradictory evidence: report of the Dietary Guidelines for Americans Committee. Nutrition 2010, 26(10):915-924.

5. Marantz P, Bird E, Alderman M: A Call for Higher Standards of Evidence for Dietary Guidelines. Am J Prev Med 2008, 34(3):234-239.

6. Harvey R, Ferrier D: Biochemistry, 5th edn. Baltimore and Philadelphia: Lippincott Williams & Wilkins; 2011.

7. Angell M: Science on Trial. New York: W. W. Norton & Co.; 1996.

Comments
  1. Pål Jåbekk says:

    Scientifically speaking, trusting in authority is a fallacy. Unfortunately, many scientists, and sometimes the majority, put authority before reason. If most scientists didn’t trust in authority people could at least trust the scientists.

    • rdfeinman says:

      The problem is that we don’t really have a definition of science. The tip-off that were are dealing with a very limited amount of science in medical nutrition is the way they are making rules of the type in Table 1. Scientists don’t do that. They don’t think that there is one kind of evidence for every scientific question. It is said often that physicians do not study nutrition. That may not be a bad thing but they don’t really study science. The real definition of science is that it is what scientists do. One thing that they don’t do is make charts of levels of evidence.

  2. JP Fanton says:

    I really enjoyed this thoughtful piece. Many thoughts came to mind while reading it. I’ll mention a few.

    The field of nutrition reminds me a lot of politics. It seems to me that many so-called experts hold a dogmatic point of view that influences the way they interpret data. For instance, in politics you’ll have someone like Paul Krugman evaluate the current economic outlook in one way while Thomas Sowell will come to a completely different conclusion – based on the same or similar figure and statistics. These are both learned and well respected individuals who presumably want good things for the country. Who are we to believe?

    I spend a lot of time looking through abstracts and full text studies – mostly pertaining to the field of alternative and complementary medicine. The thing I find most frustrating is the way studies are (frequently) designed in the first place. Even if the results are positive, they’ll almost certainly be discounted because the design is seriously flawed.

    As a non-scientist, here’s what I would like to see more of:

    If a study is being considered on the efficacy of a low-carb or low-fat diet, invite skeptics to offer suggestions about how to best design the study. Allow both sides to establish what success would be beforehand. In short, make sure the methodology of the trial is “bullet proof” rather than defending it’s shortcomings after the results are published. There’s only so much money to go around. Why not spend it more wisely by ensuring the highest quality trials before commencing them?

    Finally, as you pointed out in your excellent blog about statistics, it would be helpful if publications in medical journals included more figures and graphs so they could be practically interpreted by laypeople and scientists alike. I believe that much of the change that needs to occur in medicine will come from the bottom up – via the participation of informed consumers/patients. Making science more accessible can only help, IMO.

  3. [...] Evidence-based Medicine: Who Decides Admissibility? i. The Frye … [...]

  4. Mary Saucier Choate says:

    Not sure Gary Taubes’ work and science belong in the same sentence. Maybe misrepresented science would be more like it, according to the misquoted scientists he interviewed, who complained about this.

    • rdfeinman says:

      I cited Taubes’s work in the context of the low-fat fiasco and as I indicated, he has the most compelling description of this scandal (still going on) — the first 150 pages of “Good Calories, Bad Calories” is a masterpiece. Having interviewed some of the people he cited, I would not jump to the conclusion that they were misquoted. The reigning principles are cognitive dissonance and self-delusion. I doubt they know what they said or even what they think.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s