Top ten studies in EBM, Part 4 (September 14, 2006) Category: Teaching resources

About a year ago, I wanted to give a talk on the ten studies that anyone who teaches EBM needs to know. These studies should be

My previous writings on the topic were a bit rambling,

so I am going to try to organize things a bit better.

Study #1. Historical firsts in clinical trials. You can go way back if you want to, and the study of limes for British sailors (citation needed) is an interesting tale. I want to highlight the first published trial that formally used randomization:

Streptomycin treatment for pulmonary tuberculosis. Medical Research Council Streptomycin in Tuberculosis Trials Committee. BMJ 1948; ii: 769-782.

which showed that streptomycin was superior to bed rest. Actually, a randomized trial testing the efficacy of immunization against whooping cough was started first, but did not get reported until 1951. Two historical perspectives on the streptomycin trial appear in a special 1998 theme issue of the BMJ celebrating 50 years of progress in randomized trials:

Study #2. Spectrum bias. When you are evaluating a diagnostic test, you need a broad range of disease severity. If you only test people who have obvious manifestations of a disease versus patients who are obviously healthy, you are omitting a large number of patients in the middle of the spectrum. Such a situation occurs commonly in a case-control design, for example. This leads to estimates of sensitivity and specificity that are too good to be true, and this problem is called spectrum bias. This first paper that identified spectrum bias is

Problems of spectrum and bias in evaluating the efficacy of diagnostic tests. Ransohoff DF, Feinstein AR. N Engl J Med 1978: 299(17); 926-30. [Medline] [Abstract]

Another early paper that discusses spectrum bias is

Spectrum bias is just one of several issues that you face when you make a critical evaluation of a research article on diagnosis. The STARD initiative promotes good quality reporting for these type of articles and will help you evaluate diagnostic tests properly.

Study #3. Evaluating alternative medicine. There is a huge controversy over how to apply evidence based medicine to alternative and complementary medicine. A science fair project by a fourth grader, Emily Rosa, evaluated Therapeutic Touch (TT) and eventually end up being published in a major medical journal.

A close look at therapeutic touch. Rosa L, Rosa E, Sarner L, Barrett S. Jama 1998: 279(13); 1005-10. [Medline] [Abstract] [PDF]

Emily Rosa's experiment was very simple, perhaps too simple. If practitioners of Therapeutic Touch are able to manipulate energy fields, they must first be able to detect energy fields. She would hold her hand above either the left or right hand of the practitioner and ask him/her to tell which hand. The choice of hand was randomly determined by a coin flip. A screen with two holes in it prevented the practitioner from seeing what was going on.

Emily Rosa got 21 experienced practitioners to agree to the test. They were right only 44% of the time. Did this simple experiment disprove the healing power of TT? Perhaps not. TT is a complex intervention and this experiment only looked at a single aspect of it.

But this experiment gets at the heart of the claim by many in the alternative medicine community that medical research, especially randomized trials, are reductionist and fail to represent how alternative medicine is practiced in the real world. On the other hand, a simple experiment like this one raises the question: how can proponents of TT manipulate an energy field that they can't even detect at a 50% success rate?

A good balanced view of the controversy of the reductionist nature of clinical trials is

A more sharply skeptical view appears in

where the authors write

It is time for the scientific community to stop giving alternative medicine a free ride. There cannot be two kinds of medicine - conventional and alternative. There is only medicine that has been adequately tested and medicine that has not, medicine that works and medicine that may or may not work.

Studies #4A and 4b. Placebo surgery trials. Two widely cited early randomized double blind trials for surgery are

These trials examined a treatment for angina that involved cutting an artery that leads away from the heart to encourage more blood flow to the heart itself through the narrowed coronary arteries. While anecdotal evidence for this procedure accumulated in the 1950s, these two randomized double blind trials showed that the surgery was ineffective.

These two studies are frequently cited as rationale for use of placebos and blinding. See, for example,

Some additional placebo surgery trials include

Again, these trials showed no additional effectiveness of the therapies over a placebo surgery. A placebo surgery trial is highly controversial, because patients in the placebo arm undergo all of the risks of surgery, (side effects of anesthesia, increased chance for infection, side effects associated with subsequent antibiotic use, etc.).

Study #5. The danger in using a surrogate endpoint. Three drugs, encainide, flecainide, and moricizine, were thought to prevent heart attacks by suppressing asymptomatic ventricular arrhythmias (actually, ventricular premature depolarization or VPD), a condition associated with arrhythmic death. A large scale randomized trial, however, showed that rather than reducing your risk of arrhythmic death, these drugs actually increased the risk of death.

Preliminary report: effect of encainide and flecainide on mortality in a randomized trial of arrhythmia suppression after myocardial infarction. The Cardiac Arrhythmia Suppression Trial (CAST) Investigators. N Engl J Med 1989: 321(6); 406-12. [Medline]

In 1989, the researchers ended the two arms of the study associated with encainide and flecainide early because of a higher rate of arrhythmia deaths, of nonfatal cardiac arrests, and a higher rate of overall mortality. The study was then redesigned to compare only the third drug, moricizine, to placebo, with some changes in the entry criteria to enroll only the more seriously ill patients. This redesigned study, CAST-II, was also ended early, because of excessive cardiac deaths during the first two weeks of drug exposure, and a futility analysis. The futility analysis showed that the current data held out little hope that additional data would accumulate to the point where there would be evidence of long term survival.

Why did this happen. One possible explanation is that only a few people with VPD will die from the condition, limiting the effectiveness of these drugs, but all those who take these drugs are exposed to potential side effects. The CAST trials illustrate the important principle that a change in a surrogate outcome (reduction in VPD) does not always translate into a reduction in mortality. Here are some additional references for the CAST trials.

Study #6. Meta-analysis skewers a long held assumption about placebos. It has been common knowledge that patients placed on a placebo will get better at an amazing rate. This common knowledge, if you trace it back, all goes back to a study done in 1955. Hrobjartsson and colleagues undertook a careful review of 130 studies that included both a placebo arm and a no treatment arm.

Is the placebo powerless? An analysis of clinical trials comparing placebo with no treatment. Hrobjartsson A, Gotzsche PC. N Engl J Med 2001: 344(21); 1594-602. [Medline] [Abstract] [Full text] [PDF]

This study showed that the placebo effect was absent or equivocal to a large extent. One possible explanation is that the natural course of the disease, regression to the mean, and other factors might produce effects that are confused with the placebo effect.

Study #7A and 7B. Randomized studies trump observational studies. Another commonly held belief was that if you gave estrogen supplements to women who are going through menopause, they would have better health outcomes. There was some observational evidence to support this contention as well as a plausible biological mechanism. But a carefully conducted randomized trial in 1998

Randomized Trial of Estrogen Plus Progestin for Secondary Prevention of Coronary Heart Disease in Postmenopausal Women. Hulley S, Grady D, Bush T, Furberg C, Herrington D, Riggs B, Vittinghoff E. JAMA 1998: 280(7); 605-613.

and another randomized study in 2002

Risks and benefits of estrogen plus progestin in healthy postmenopausal women: principal results from the Women's Health Initiative randomized controlled trial. Jacques E. Rossouw, Garnet L. Anderson, Ross L. Prentice, Andrea Z. LaCroix, Charles Kooperberg, Marcia L. Stefanick, Rebecca D. Jackson, Shirley A. A. Beresford, Barbara V. Howard, Karen C. Johnson, Jane Morley Kotchen, Judith Ockene. Jama 2002: 288(3); 321-33. [Medline] [Abstract] [Full text] [PDF]

showed that women receiving hormone supplements in a randomized double blind trial showed slightly worse health outcomes. A meta-analysis of randomized trials that excluded observational studies

also gave evidence that HRT was not effective.

 There is still active debate about why the randomized trial showed different results. Some commentaries on the HRT controversy are

and the impact of these studies on prescribing patterns is shown at

Study #8. Where does meta-analysis stand on the hierarchy of evidence.

Issues in Comparisons between Meta-analyses and Large Trials. Ioannidis J, Cappelleri J, Lau J. Jama 2002: 279(14); 1089-93. [Medline]

Study #9. Financial conflicts of interest.

Conflict of interest in the debate over calcium-channel antagonists. Stelfox HT, Chua G, O. Rourke K, Detsky AS. N Engl J Med 1998: 338(2); 101-6. [Medline]

Study #10. Nonrandomized trials are evidence also.

The disappearance of Reye's syndrome--a public health triumph. Monto AS. N Engl J Med 1999: 340(18); 1423-4. [Medline]

Other possible candidates.

[Prevention of the first occurrence of anencephaly and spina bifida with periconceptional multivitamin supplementation (conclusion)] [Article in Hungarian]. Czeizel E, Dudas I. Orv Hetil. 1994 Oct 16;135(42):2313-7. [Medline]

The existence of publication bias and risk factors for its occurrence. Dickersin K. Jama 1990: 263(10); 1385-9.

A Randomized, Controlled Trial of the Effects of Remote, Intercessory Prayer on Outcomes in Patients Admitted to the Coronary Care Unit. Harris WS, Gowda M, Kolb JW, Strychacz CP, Vacek JL, Jones PG, Forker A, O’Keefe JH, McCallister BD. Archives of Internal Medicine 1999: 159(19); 2273-2278. [Medline]

Final report on the aspirin component of the ongoing Physicians' Health Study. Physicians' Health Study Research Group. N Engl J Med 1989: 321(3); 129-35. [Medline]

Randomised trials, human nature, and reporting guidelines. Schulz KF. Lancet 1996: 348(9027); 596-8. [Medline]

This webpage was written by Steve Simon and was last modified on 07/08/2008.