Introduction to evidence-based health science lessons from medicine

Introduction to evidence-based health science: lessons from medicine Joseph Lau, MD September 19, 2011

To be covered • The arrival of an evidence-based world • Impetus for and a very brief overview of systematic review (SR) and meta-analysis • An example of a SR to inform pubic health policy: dietary reference intakes for calcium and vitamin D • A very brief history and current status of SR in medicine • ? Lessons to learn for conservation medicine

The Year in Ideas New York Times Magazine 12/9/2001 Editors and writers included “ 80 ideas that shook the world (or at least jostled it a little) in 2001. . . ” Covered: politics, medicine, sports, business, fashion, warfare, . . . Under medicine: – – – Evidence-Based Medicine Pharmacogenomics Global antiretroviralism Telesurgery Turning a bad drug good

Evidence-based health care is the conscientious use of current best evidence in making decisions about the care of individual patients or the delivery of health services. Current best evidence is up-to-date information from relevant, valid research about the effects of different forms of health care, the potential for harm from exposure to particular agents, the accuracy of diagnostic tests, and the predictive power of prognostic factors. Sackett et al. BMJ 1996

Conventional paradigm for medical practice and education • See one • Do one • Teach one

Google – September 6, 2011 “evidence-based” “evidence-based medicine” “evidence-based practice” “evidence-based nursing” “evidence-based healthcare” “evidence-based surgery” “evidence-based nutrition” “evidence-based veterinary medicine” “evidence-based conservation” “evidence-based ecology” “evidence-based conservation medicine” “evidence-based management” “evidence-based social” “evidence-based education” “evidence-based marketing” “evidence-based politics” “clinical practice guideline” ”systematic review” “meta-analysis” 22, 400, 000 5, 000 4, 030, 000 740, 000 377, 000 47, 400 506, 000 248, 000 121, 000 4, 330 0 5, 090, 000 1, 900, 000 121, 000 505, 000 32, 900 2, 560, 000 3, 210, 000 18, 400, 000

The weight of medical knowledge David T. Durack, M. B. , D. Phil. ; NEJM 1978 Weight of the Index Medicus According to 10 -Year Periods from 1879 to 1977

Medical Junk Mail. Montauk L. Lancet 2000; 356: 344. Emergency Medicine San Francisco, USA Jan 1, 1999 - Dec 31, 1999 122 kg, 2 m high

Lots of studies doesn’t necessarily translate to lots of clinically useful evidence

The Medical Review Articles: State of the Science Mulrow C. Ann Intern Med 1987; 106: 485 -8. Review of 50 Review Articles in four popular clinical journals over 12 months June 1985 - June 1986 (Ann Intern Med, Arch Intern Med, JAMA, NEJM) Specified Unclear Not Specified Purpose 40 1 9 Data Identification 1 2 47 Data Selection 1 0 49 Validity Assessment 1 1 48 Qualitative Synthesis 43 0 7 Quantitative Synthesis 3 1 46 Summary 37 12 1 Future Directive 21 4 25 Criteria

How health care varies by region USA Today, Tuesday, September 19, 2000, 9 D

Percentage of acute otitis media patients given antibiotics Figure taken from Froom J et al. Diagnosis and antibiotic treatment of acute otitis media: report from International Primary Care Network. BMJ 1990; 300: 582 -6.

The Importance of Beta, the Type II Error and Sample Size in the Design and Interpretation of the Randomized Control Trial Survey of 71 “Negative” Trials Freiman JA, Chalmers TC, Smith H Jr, Kuebler R. N Engl J Med 1978; 299: 690 -4.

Lau J, et al. NEJM 1992

The research of many commentators have already thrown much darkness on this subject, and it is probable that, if they continue, we shall soon know nothing at all about it. Mark Twain

Summary: why we need systematic reviews and meta-analyses? • Too much information for practitioners (including domain experts) to keep up and synthesize • Clinicians and researchers need reliable information; narrative review articles and textbooks often are outdated and biased • Information of variable quality and reliability, practitioners lack training to evaluate and synthesize research findings • Studies examined individually offer only partial answers (the whole is greater than the sum of its parts) • Need for update • Identify research gaps

Principles of systematic review • Prospective research on published data • Reproducible, transparent (clear and thorough documentation) • Rigorous, take steps to minimize bias and errors • Involve stakeholders when feasible • When possible, the practice of EBM should be based on evidence (use practices that are supported by empirical data) • Caveat: many current methods are evolving

General methods of systematic review • Develop analytic framework • Formulate answerable research question(s) • Specify review criteria – – • • Participants Intervention Comparison Outcomes Identify relevant studies Take measures to minimize bias; assess potential biases Synthesize results (qualitative, quantitative [meta-analysis]) Appropriate handling of heterogeneity and interpretation of results • Identify research gaps

Vitamin D and Calcium: A Systematic Review of Health Outcomes Tufts Evidence-based Practice Center August 4, 2009 Information Gathering Workshop Committee to Review DRI for Vitamin D and Calcium

Analytic Framework • Relevant questions can be formulated into a model that analyzes all effects and interactions between intervention or exposure and outcomes; to appreciate relationships of the questions • Can be used to clarify and generate questions (topics) • Can highlight what aspects are known and unknown • Can clarify what study designs may be best to address specific questions 29

Key Question 1. What is the effect of vitamin D, calcium, or combined Key Questioncalcium intakes on clinical vitamin D, or serum or Key Question 2. What is the effect forvitamin D D combined vitamin D and 3. What is the association between Key Question 5. What is the effect of outcomes, including growth, 4. of Vitamin calcium 25(OH)D Analytic Framework and/or combinedconcentrations calcium intakescancer, outcomes? vitamin calcium balance and clinical immuneor D and surrogate or intermediate function, on surrogate cardiovascular or 25(OH)D and diseases, weight on serum concentrationscalcium intakesoutcomes, 25(OH)D vitamin D Calcium as hypertension, blood pressure, intermediate birth outcomes, mortality, fracture, renal outcomes, and pregnancy or outcomes, such Health Outcomes outcomes? concentrations? soft bone mineral density? and tissue calcification?

Analytic Framework for Vitamin D and/or Calcium Safety -related (adverse) Outcomes

PICO Selection Criteria • Population – Generally healthy people with no known disorders – Studies enrolled <20% patients with common diseases allowed – For adverse effects of high intake, any population • Intervention / Exposure – Observational studies: Serum 25(OH)D or 1, 25(OH)2 D • Comparator – Dose relationship • Outcome – 17 outcomes selected by technical expert panel • Study Design – Experimental / Observational, Duration, N; (excluded crosssectional studies and retrospective case-control)

Literature search and selection

Critical Appraisal of Primary Studies We adapted a 3 -category grading system of the AHRQ Methods Reference Guide for Comparative Effectiveness Reviews, which encompasses the principles of CONSORT statement for RCTs, STROBE checklist for observational studies. This system defines a generic grading system that is applicable to each type of study design. A - Least bias; results are valid B - Susceptible to some bias, but not sufficient to invalidate the results C - Significant bias that may invalidate the results

Critical Appraisal of Systematic Reviews • A summary quality grade for systematic review is difficult to interpret • Various dimensions and nuances of the systematic review must be understood • We applied AMSTAR checklist, a tool to assess the quality of reporting of systematic reviews • Items evaluated are made explicit for the reader • Comments provided where appropriate

Reporting of Evidence • Evidence tables – detailed information about each study • Summary tables – summary from each study that address a question (outcome, study design) • Figures, graphs • Meta-analyses (if appropriate) • Narratives, highlight features and limitation of study in answering question

Results

Primary Studies on Vitamin D Intake or Concentration

Availability and Quality of Primary Studies Quality grade of primary studies A (11%) B (50%) C (39%) Vitamin D (94) RCT (30) 3 14 13 Non-RCT (64) 4 33 27 Calcium (108) RCT (23) 1 11 11 Non-RCT (85) 17 41 27 Vitamin D + Calcium (19) RCT (19) Non-RCT (0) 0 12 7 0 Total = 221 (from 165 unique studies); same study may contribute more than one outcome

Organization of Results Chapter • Vitamin D – outcomes • Calcium – outcomes • Vitamin D + calcium – Outcomes • Association with intake and serum Vitamin D level (Question 4) • Adverse or Safety Outcomes

Reporting of Individual Outcome • Synopsis • Detailed presentation (highlight features and limitation of study in answering question) – Findings per vitamin D concentration – Findings per age and sex – Findings by life stage • Summary tables – RCTs • Characteristics • Results – Cohort studies • Characteristics • Results • Figures, graphs • Meta-analyses (if appropriate)

Vitamin D and colorectal cancer: results of RCTs (pg 68)

Vitamin D and colorectal cancer: results of RCTs (pg 70)

Vitamin D and colorectal cancer – observational studies (pg 74)

Vitamin D and colorectal cancer: Results of observational studies (pg 76)

Caveats • YGWYA (you get what you asked) - report provide information only on questions formulated a priori, it does not answer all potential questions of interest • As much as we strive to be objective, some judgment is involved (grading of studies) • No standard tool available to assess quality of nutritional observational studies • Publication bias • Need to rely on existing systematic reviews, but using them is challenging • Studies in general were not designed for DRI issues (e. g. , targeting DRI life stages)

A very brief history and current status of SR in medicine / healthcare • • First meta-analysis Current era (starting in late 1970 s) Criticisms Methods advancements EBM era US government effort (AHRQ EPC) Healthcare reform (comparative effectiveness reviews)

Methodological advances • Empirical research – Validity of meta-analysis – Large studies vs meta-analysis of smaller studies – Understanding of heterogeneity • • Statistical methods Computer software Questions formulation / analytic framework Searching literature Improving quality of primary studies Interpretation and using of results Future research needs

Applications of systematic reviews and metaanalyses in healthcare / biomedical research • • Interventions (many 10, 000’s) Epidemiologic (several 1, 000’s) Diagnostic test accuracy (~1, 000) Genetic tests / Genomics (~2, 000) Health economics (<100) Basic science studies (<100) All share similar principles; unique issues and methods required for each category

Translational Spectrum of Comparative Effectiveness Research Evidence Prioritization Community Based Stakeholders (Qualitative / Quantitative Elicitation) Evidence Generation (Randomized Control Trials, Registries, Observations, N-of-Trials) Academic Medical Centers Evidence Synthesis (Systematic Reviews, Meta-Analysis) Evidence Interpretation and Integration (Cost-Effectiveness Analysis Decision Analysis) Dissemination and Application (Guidelines, Policy, Social Sciences, Implementation Science) Feedback and Assessment (Qualitative Elicitation, Data Monitoring, Quality Monitoring / Measurement) Tufts CTSI