Critical appraisal. Critical appraisal is the ability to judge the persuasiveness of the evidence in a research study. You have to strike the proper balance between being too harsh and being too accepting of research findings. Articles are arranged by date with the most recent entries at the top. You can find the theme and closely related categories and other resources at the bottom of this page.
Stats: The post-modern assault on evidence-based medicine, part 3 (January 7, 2008). I have volunteered to give a talk for a group of statisticians which discusses an article I am preparing: "The post-modern assault on evidence-based medicine." I might give this talk on the first Monday in February or the first Monday in March. Here is a tentative abstract.
Stats: The post-modern assault on evidence-based medicine, part 2 (December 18, 2007). As mentioned in a December 7, 2007 weblog entry, I sent an email to Michael Shermer, editor of Skeptic Magazine. In it, I included a brief pitch for an article, "The post-modern assault on evidence-based medicine" and he gave me some encouragement to work on this. Guidelines for contributions are at http://www.skeptic.com/the_magazine/contribute.html and they accept articles of varying lengths (500 to 5000 words) though they prefer shorter articles. the current draft is already 2700 words with a lot left to be written. So I'll have to find a way to condense many of these thoughts. Here are some of the talking points in the article. I'm including heading in this draft, but they are just to help me organize my thoughts. This draft still needs a lot of work, but I wanted to have something to show to people.
Stats: The post-modern assault on evidence-based medicine (December 7, 2007). I sent a brief email to Michael Shermer, editor of Skeptic Magazine. In it, I included a brief pitch for an article. The title would be "The post-modern assault on evidence-based medicine."
Stats: Which expert should you believe? (August 3, 2007). There's a common saying in research circles that goes something like this: "For every PhD, there is an equal and opposite PhD." That saying is certainly true in my experience. For just about any scientific controversy, you can find an expert on either side of the issue. Quite often the experts on both sides know far more about the controversy than you or I will ever get a chance to know. So the question becomes: "Which expert should you believe?"
Stats: Ignore weak evidence at your own peril (March 13, 2007). I ran across an interesting article recently: Incorporating quality of evidence into decision analytic modeling. R. S. Braithwaite, M. S. Roberts, A. C. Justice. Ann Intern Med 2007: 146(2); 133-41. I have not yet read the full article, but the message seems to be that using evidence from weak data sources is better than ignoring it.
Stats: Ambiguous nature of the word "risk" (September 13, 2006). A recent discussion on the Evidence-Based Health email discussion group focused on the word "risk". It means harmful event (risks versus benefits) as well as a probability (risk of heart attack).
Stats: Contradictory research (March 30, 2006). The Washington Post published an article that gets to the heart of the difficulty with Evidence Based Medicine, Fat or Fiction? Is There a Link Between Dietary Fat and Cancer Risk? Why Two Big Studies Reached Different Conclusions. Lisa M. Schwartz, Steven Woloshin, H. Gilbert Welch, Published March 14, 2006 in The Washington Post, Page HE01. It starts out with a rather provocative statement, but one that is very hard to argue with, "The public is bombarded with messages about diet and cancer prevention. Unfortunately, the advice is pretty inconsistent. One day a diet prevents cancer, the next day it doesn't. In the early '90s, beta carotene (a vitamin A precursor present in fruits and vegetables) was said to prevent lung cancer. But several years later, headlines read, "Beta carotene pills yield no benefit" (The Post, 1996). And while people have been told for years to eat a high-fiber diet to reduce the risk of colon cancer, recently we were told "High-fiber diets are not anti-cancer miracle" (Montreal Gazette, 2005)."
Stats: Integrating internal and external evidence (March 17, 2006). Evidence based medicine involves the combining of the best available evidence with your clinical knowledge and the patient's individual values. A good article that emphasizes how this should be done is on the web at: Evidence-based decision making--the six step approach. Franz Porzsolt, Andrea Ohletz, Anke Thim, David Gardner, Helmuth Ruatti, Horand Meier, Nicole Schlotz-Gorton, Laura Schrott. Evid Based Med 2003: 8(6); 165-166.
Stats: More on the Emily Rosa experiment (March 10, 2006). One of the more interesting research studies from an Evidence-Based Medicine perspective started out as a simple science fair project by a fourth grade student. Emily Rosa wanted to see if practitioners of Therapeutic Touch could detect the energy fields in a carefully controlled condition. The topic of this project was not too surprising, since her parents both worked for the QuackWatch website, but Emily came up with the idea entirely on her own. The science project received a lot of publicity and Emily was encouraged to publish here results in a medical journal. With the assistance of several adults, the publication, A close look at therapeutic touch. L. Rosa, E. Rosa, L. Sarner, S. Barrett. Jama 1998: 279(13); 1005-10, appeared, giving Emily Rosa something nice to put on her resume when she applies to college. I'm still waiting for my first publication in an "A journal" like JAMA, so I am quite jealous.
Stats: A nice definition of anecdotal evidence (October 24, 2005). Robert Todd Carroll, author of The Skeptics Dictionary and creator of the website Skepdic.com, mentioned in a recent newsletter that he has updated his definition for anecdotal evidence. He points out that anecdotes are unreliable because they are "prone to contamination by beliefs, later experiences, feedback, selective attention to details, and so on." and although these are "scientifically worthless", their vivid details make them popular. But Dr. Carroll wisely does not totally rule out their use.
Stats: Effective communication about randomized clinical trials (February 22, 2005). The most recent issue of BMJ has a nice article evaluating a training program for health care professionals. This is an unusual thing to do; most of the time, training programs are just put together with the assumption that because intelligent people are designing the program, it must be effective. I'm just as guilty of this as anyone else, of course. The particular training class is of great interest to me, because it works on how health care professionals communicate to patients about randomized clinical trials.
Stats: How good is your intuition? (January 21, 2005). One definition of Evidence Based Medicine is "the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients. The practice of evidence based medicine means integrating individual clinical expertise with the best available external clinical evidence from systematic research. Sackett et al 1996." In the same article, the authors point out that "Good doctors use both individual clinical expertise and the best available external evidence, and neither alone is enough. Without clinical expertise, practice risks becoming tyrannised by evidence, for even excellent external evidence may be inapplicable to or inappropriate for an individual patient. Without current best evidence, practice risks becoming rapidly out of date, to the detriment of patients." A recently published book by Malcolm Gladwell highlights the value of individual expertise, which you might also call clinical judgment or simply intuition.
Stats: Entrenched beliefs (December 1, 2004). A member of the Evidence Based Health listserv wrote about a workshop he taught on Evidence Based Medicine (EBM). It went pretty well, he thought until he overheard a conversation during one of the breaks. A doctor was recommending an herbal remedy to a friend with the comment: "It cleared my problem up really quickly." This demonstrated to him how readily we still accept anecdotal evidence instead of EBM. I think it is just a fundamental characteristic of human nature to apply critical thinking skills to everything except a core set of beliefs that you hold near and dear to your heart.
Stats: Craniosacral therapy (September 24, 2004). An educational email circulated at our hospital during Pain Awareness Week has two interesting questions that draw a sharp contrast between traditional medicine and alternative medicine.
Stats: Statistical nihilism (July 6, 2004). There's an enormous mistrust of statistics in the real world. To the extent that it makes people skeptical, that's good. To the extent it turns them cynical, that's bad. There's a viewpoint, championed by too many people, that statistics are worthless. I call this viewpoint statistical nihilism.
Stats: The research world is not black and white (July 7, 2007). I was asked to review a commentary about studies involving remote healing (which includes prayer studies as a subgroup). It is not available yet on the web, but I'll try to link to it when it becomes available. One of the documents that this commentary did refer to, however, is available on the web. Mistakes in Experimental Design and Interpretation. Peter Norvig, norvig.com/experiment-design.html. This is a good commentary, but it falls into a philosophical trap that plagues most of the discussion on how to critically evaluate research, including (I have to admit) some of my own commentaries.
Stats: The Seven Deadly Sins of Researchers (April 4, 2007). I was reading an article in written by Steven Goodwin, The Seven Sins of Programmers, published in issue 17 of the Free Software Magazine and thought it would be fun to use a similar theme in research. So here are the seven deadly sins of researchers. Lest I be accused of the sin of pride, let me admit that everyone, including myself has been tempted by and has indulged in some of these sins at one point in their research career.
Stats: Why do we need research? (May 31, 2006). I help a lot of people to perform research and it's important to understand why research is important. We do research because good research drives out bad medical practices.
Stats: What is critical thinking? (May 17, 2005). A lot of people use the term "critical thinking" in an offhand way. Usually the writer who uses this term is trying to imply that anyone who disagrees with the writer's theory or belief is naive. Critical thinking is also produced as an excuse to attack a particular theory or to promote "equal time" for a competing belief. Critical thinking, however, is much more complex than this. It involves more than just fault finding.
Stats: No tolerance for ambiguity (May 10, 2006). I was at a meeting tonight and put in a plug for my book, Statistical Evidence in Medical Trials, by mentioning that it was intended to help people understand the controversies and the seemingly contradictory research that appears in the medical journals. A woman talked to me afterwards and wanted to know what I thought about a particular author who had written about hormone replacement therapy. I had to defer any comments because I was unfamiliar with this particular author. She then informed me that she had taken hormone replacement therapy and it gave her breast cancer. Thankfully, the cancer has responded well to treatment, but I was struck by the certainty of her comment about how the estrogen supplements caused her cancer.
Stats: Anecdotal information (November 2, 2004). Anecdotal information is problematic for several reasons, and I have highlighted some of these reasons in a speech I gave at the 2004 meeting of the Midwest Society for Pediatric Research. Catherine Fiorello highlights the weakness of the anecdotal argument "I was spanked as a child and I turned out okay." Such an observation ignores the need for a comparison group. Such a group would help answer the question "What would I have been like if I hadn't been spanked as a child?"
Stats: Overlooking one's own flaws (May 3, 2004). Can we ever be truly objective about ourselves? Maybe not. A recent article in Scientific American by Michael Shermer, The Enchanted Glass, talks about the tendency to see ourselves more positively than our peers. For example, when asked the probability that certain people will go to heaven, the surveyors listed Bill Clinton at 52%, Mother Theresa at 79% and so forth. But these same people rated their own probability at 87%.
Stats: Flaws in a research paper (January 27, 2000). Dear Professor Mean: A small group I've been teaching has gotten extremely interested in how to decide when there are too many flaws in a paper which would completely invalidate (and circular file) it.
Stats: Jargon in Statistics (January 27, 2000). Dear Professor Mean: I have to review a paper for journal club and I don't understand all the obscure statistical jargon that the authors use.
Theme and closely related categories:
- AGREE (Appraisal of Guidelines REsearch Evaluation) Collaboration Excerpt: AGREE is an international collaboration of researchers and policy makers who seek to improve the quality and effectiveness of clinical practice guidelines by establishing a shared framework for their development, reporting and assessment.
- Article makes simple errors and could cause unnecessary deaths Description: This article offers a critical review of a critical review (Cleland 2002). Cleland cited issues a large randomized trial of aspirin for prevention of heart attacks and with a meta-analysis. Baigent et al argue that the claims of Cleland are "wrong for trivial reasons and potentially damaging to patients."
- Biases in the interpretation and use of research results Description: This article provides several fascinating examples of people's tendency to be hypercritical of research findings that they dislike and to overlook the flaws of research that they favor.
- Calling all charlatans. A group of researchers puts companies making scientific claims on the spot
- Distinguishing Association from Causation: A Backgrounder for Journalists
- Statistics as Principled Argument
- Survey of claims of no effect in abstracts of Cochrane reviews Description: This article notes that claims about "negative" results need to be phrased cautiously. In a review of 989 Cochrane reviews, the authors found 240 poorly worded interpretations of no difference or no effect.
- The Taxonomy of Logical Fallacies
- Unconventional cancer therapies: What we need is rigorous research, not closed minds Description: This article notes the popularity of many complementary and alternative medicine techniques, but also warns of the lack of methodological rigor in many evaluations of these techniques. Rather than reject wholesale all of these techniques, these authors suggest that rigorous research is needed.
[Return to full topic list] [Read current weblog entries]
This webpage was written by Steve Simon on 2007-07-08, edited by Steve Simon, and was last modified on 2008-07-14. Send feedback to ssimon at cmh dot edu or click on the email link at the top of the page.