Statistical Evidence: Preface

This is an early draft of the preface for "Statistical Evidence."

Who is this book for?

I am writing this book for any health care professional who is making the effort to read and evaluate medical publications. Do you update and modify your clinical practice on the basis of what you read in the research journals? I have guidelines that can help you.

Non medical professionals can also benefit from this book. I do use a few technical medical terms, but as long as words like "myocardial infarction" don't give you a heart attack, you will be just fine. Indeed, many people like me who do not have specialized medical training will still read medical journals. Journalists, for example, have to write about the peer-reviewed literature for the public and they need to know when researchers are overhyping their research findings. Lawyers involved with malpractice suits need to understand which medical practices have been supported by medical research, which practices have been discredited, and which practices still require additional research. More and more patients want to research their own diseases so they can discuss treatment options intelligently with their doctors.

And while I focus mostly on medical examples, the general principles apply to other areas as well. If you work in a non-medical field, but you read peer-reviewed journals and try to incorporate their findings into your job, my guidelines can help you.

I did not write this book to teach you how to conduct good research. I wrote it for consumers of research, not producers of research. Even so, when you plan your research you should try to use a research design that is most likely to be persuasive. To that extent, my book can help.

Finally, I do not discuss how to compute any statistical tests or estimates. There are no formulas in this book because I want to focus on how the data were collected, not how they were analyzed. I've always like formulas and graphs; it was hard for me to keep the formulas out.

How did this all get started?

The original inspiration for this book came from the students in an informal class I was teaching at Children's Mercy Hospital in 1997. In a survey, I asked the students why they were taking the class. My hope was that this information would help me select future topics for discussion. A common response was along the lines of "I want to understand the statistics used in medical journal articles." So I prepared a talk called "How to Read a Medical Journal Article." I expanded the talk into a web page (www.childrensmercy.org/stats/journal.asp). I had the good fortune of being invited to write a series of articles about research for the Lab Corner section of the Journal of Andrology. This allowed me to further refine these ideas. I also was invited to participate in several journal clubs at Children' Mercy Hospital. The journal articles were always interesting and the discussions helped me refine the ideas that I am presenting here.

Outline of this Book

What you are reading is only partially complete, and I have used square brackets to indicate areas where I need to present additional material. I also have to work to get consistency in how I cite journal publications. The language is also a bit inconsistent because these sections were written over a five year period with parts of them updated more recently than others.

This overview that you are reading is about 95% complete. You can find this section on the web at www.childrensmercy.org/stats/journal/overview.asp.

This book is divided into five (maybe six) major sections. The first section: "Apples or Oranges?" is about 95% complete. This section examines the quality of the control group. How carefully the control group was selected and handled relates to credibility of the research. If you want a technical term, this is often called the internal validity of the research. The most current version of this section is on the web at www.childrensmercy.org/stats/journal/apples.asp.

The second section: "Who Was Left Out?" is about 90% complete. This section examines exclusions before the study started, and exclusions during the study. If important segments of the population are left out, then you may have difficulty generalizing the results of the study. This is often called the external validity of the research. The most current version of this section is on the web at www.childrensmercy.org/stats/journal/leftout.asp.

The third section: "Mountain or Molehill?" is about 75% complete. This section examines the clinical relevance of the outcome. The outcome measure has to be properly collected and has to measure something of interest to your patients. The size of the study has to be large enough to produce reasonably precise estimates and the difference between the treatment and control group has to be large enough to have a clinical impact. The most current version of this section is on the web at www.childrensmercy.org/stats/journal/mountain.asp.

I have just barely started writing the fourth section "Is there outside corroboration?". This section will discuss how to look at additional supporting evidence outside the journal article itself. Corroborating evidence is especially important for observational studies, because it is rare that a single observational study provides definitive results entirely by itself. Rather, it is a collection of observational studies, all looking at the problem from a different perspective that can provide persuasive evidence. This section will be loosely based on the nine factors to assess a causal relationship that Sir Bradford Hill developed in 1966. I have a very early draft which outlines the issues at  www.childrensmercy.org/stats/journal/corroboration.asp.

The fifth section "What do all these numbers mean" gives a non-technical explanation for some of the statistics used in hypothesis testing, such as p-values and confidence intervals. It also explains the various measures of risk, like the odds ratio, relative risk, and number needed to treat. This section is less than half complete. The most current version of this section is on the web at www.childrensmercy.org/stats/journal/numbers.asp.

I am also working on closely related materials that discusses meta-analysis (www.childrensmercy.org/stats/journal/meta-analysis.asp). This could be a possible sixth section to the book, but it needs substantial revisions.

Other Resources (I'm just getting this list started.  Please be patient.)

There are a lot of good books, web pages, and research papers out there that can help you.

Statistics as Principled Argument. Robert P. Abelson (1995) Hillsdale, New Jersey: Lawrence Erlbaum Associates. ISBN: 0805805281. Description: There is a wealth of wisdom in this book. The basic theme is that Statistics provides basic principles to argue (debate might be a nicer word) about scientific claims. In the first chapter, Dr. Abelson argues that a persuasive argument has to have MAGIC--Magnitude, Articulation, Generality, Interestingness, and Credibility. Then he describes probability and randomness, illustrates common fallacies about probability, and shows how these principles can be applied to research findings. Chapter 5, On Suspecting Fishiness, describes some wonderful examples of strange numbers that might indicate fraud. This chapter is especially valuable because it is so rarely covered. The remaining chapters describe the MAGIC components of a persuasive argument with frequent citations of real research. This book is more conceptual than computational, which fits in with one of Abelson's Laws "Don't talk Greek if you don't know the English translation."

Damned Lies and Statistics Untangling Numbers from the Media, Politicians, and Activists. Joel Best (2001) Berkeley, California: University of California Press. ISBN: 0520219783. Description: Joel Best captures your attention right from the start by describing the worst statistic ever published: a claim that "every year since 1950, the number of American children gunned down has doubled." If you look at what a yearly doubling over one or two decades implies, you will quickly see how inaccurate this claim has to be. Joel Best goes beyond this example, though, to show how there is a social need to wield statistics as "weapons in political struggles over social problems and social policy." Furthermore, these statistics, even when they start as mere guesses tend to be repeated by different media sources and gain credibility with every repetition. When these statistics relate to controversial social policies, they are often defended not by any objective standard but "by challenging the motives of anyone who disputes the figure." Joel Best cites numerous statistics: the suicide rate, the poverty level, the number of homeless people, the illiteracy rate, and shows a remarkable level of even handedness is describing how different political groups use and abuse these numbers. When you see a statistic, Joel Best suggests that you ask three questions: Who created this statistic? Why was this statistic created? and How was this statistic created? You should critical rather than naive or cynical: "The issue is whether a particular statistic's flaws are severe enough to damage its usefulness."

Evidence-Based Medicine:  How to Practice and Teach EBM. David L. Sackett, MD, Scott W. Richardson, William Rosenberg, Brian R. Haynes (1998) Edinburgh: Churchill Livingstone. ISBN: 0443056862. Description: There are many books on evidence-based Medicine (EBM), but this is the classic text, and it is hard to beat. It is succinct and to the point, and if that weren't enough, the authors summarize the most important points on plastic index cards that you can carry in your pocket. The authors provide a clear and understandable definition of EBM and make a compelling case for the need to use EBM in your practice. EBM starts with asking the right question. The right question, the authors tell us, should have four components: (1) the patient or problem, (2) the intervention, (3) the comparison, and (4) the outcome. The authors then describe how to search for the best evidence and explain some of the technical details of Medline, a database of medical publications from thousands of journals. The authors mention other resources like the ACP Journal Club on Disk and the Cochrane Database of Systematic Reviews. They then provide practical guidance on how to evaluate studies in six major areas: diagnosis, prognosis, therapy, harm, economic analysis, and quality of care. Everywhere you turn, the authors are addressing real problems and providing pragmatic advice. If you are already an expert on EBM, this book is still valuable, because it provides helpful advice on how to teach these methods. (There is a second edition of this book, published in February 2000 that I have not seen yet.)

Evaluating Research Articles from Start to Finish. Ellen R. Girden (2001) Thousand Oaks, CA: Sage Publications. ISBN: 0761922148. Description: This book offers pragmatic advice for people who read research articles and covers a tremendous range of studies, both qualitative and quantitative. Dr. Girden uses examples of case studies, narrative analysis, surveys, correlation studies, regression analysis studies, factor-analytic studies, discriminant analysis studies, two-condition experimental studies, single classification studies, factorial studies, and quasi-experimental studies. For each type of study, Dr. Girden offers some background on the methodology, provides a checklist of caution factors, and then critically reviews two research publications. The actual text of the research studies is included in the book itself.

Critical Appraisal of Epidemiological Studies and Clinical Trails. Mark J. Elwood (1998) Oxford: Oxford University Press. Description: Dr. Elwood describes intervention trials (both randomized and non-randomized), retrospective and prospective cohort studies, case-control studies, and cross-sectional studies. Dr. Elwood then discusses how to select subjects for the various research designs, how to identify sources of bias and error, how to avoid confounding or control for its effects, and how to assess statistical and practical significance. He then outlines nineteen questions you should ask relating to a description of the evidence, internal validity issues, external validity issues, and comparison of the results to other evidence. Then Dr. Elwood provides a critical review of a variety of published research, including excerpts from the research itself.

Interpreting the Medical Literature Third Edition. Stephen H. Gehlbach (1993) New York: McGraw-Hill. ISBN: 0071054510. Description: Dr. Gehlbach discusses case-control designs, cross-sectional designs, follow-up (cohort) studies, and experimental designs. Then he discusses measurement issues (reliability, validity, systematic error, and measurement error) and explains the basics of hypothesis testing and confidence intervals. He also defines terms used in diagnostic testing (sensitivity, specificity, and predictive value) and measures of risk (relative risk, odds ratios, and attributable risk). Then he discusses how to determine cause and effect (strength of the association, dose-response relationship, biological plausibility, and consistency of the observed evidence). Dr. Gehlbach lumps case series, editorials, and reviews together in a chapter as examples of less rigorous research, but he does not mention meta-analysis. (There is a fourth edition, published in May 2002, which I have not seen.)

Studying a Study and Testing a Test: How to Read the Health Science Literature Third Edition. Richard K. Riegelman, Robert P. Hirsch (1996) Boston, MA: Little, Brown and Company. ISBN: 0316745219. Description: The authors define case-control studies, cohort studies, randomized clinical trials, and meta-analysis and offers a series of questions to help you assess the possible flaws of each type of research. He then discusses how to evaluate diagnostic methods (testing a test) and measures of the frequency of disease (rating a rate). He finally reviews the statistical methods that you can choose for a research study (selecting a statistics). Almost all of the examples in this book are hypothetical. (There is a fourth edition, published in January 2000, which I have not seen.)

Statistical Reasoning in Medicine. The Intuitive P-Value Primer. Lemuel A. Moye (2000) New York: Springer-Verlag. ISBN: 0387989331. Description: This book provides an intuitive and conceptual understanding of p-values. Dr. Moye has chapters on observational data, effect sizes, power, one-tailed tests, multiple endpoints, Bayesian p-values, and subgroup analysis. He includes fascinating examples about the use and abuse of statistics in medicine and each example is backed up with the appropriate journal reference.

Users' Guides to Evidence-Based Practice. Center for Health Evidence. Accessed on 2003-09-02. "The following is the complete set of Users' Guides originally published as a series in the Journal of the American Medical Association (JAMA). The CHE continues to maintain the full text pre-publication version of this series on behalf of the Evidence-Based Medicine Working Group with permission from the journal. See the Disclaimer and Copyright for more information." www.cche.net/usersguides/main.asp

What this book would add

If there are so many good resources out there, why I am writing this book? I believe there are three things that I can add to this list.

Extensive use of real world examples. There is a lot of fascinating research papers out there, and they tell a fascinating story. These papers pose interesting questions like "what sort of person would volunteer to have a spinal tap done as part of a research study" and "why would a doctor flip a sterilized coin in the operating room?" I have included hundreds of citations in this book, and many of these examples have the full text on the web for free.

Focus on statistics issues. When you are trying to assess the quality of a medical publication, most of the issues touch directly on Statistics. And yet, Statistics is the one area that medical professionals are intimidated by. Well, Statistics isn't brain surgery, and you are capable of understanding the concepts.

Avoidance of formulas and technical language. People think that Statistics is a bunch of numbers and formulas, but there are a lot of non-quantitative issues in how statistics are applied in research. When you are trying to assess the credibility of a research study, these non-quantitative concerns are far more important than any formulas or statistical calculations.

This webpage was written by Steve Simon on (unknown date), edited by Steve Simon and Linda Foland, and was last modified on 2008-07-08. This page needs minor revisions. Category: Statistical evidence