Introduction to Principles of Evidence-based Medicine Benjamin Djulbegovic

Introduction to Principles of Evidence-based Medicine Benjamin Djulbegovic, M. D. , Ph. D. Distinguished Professor Associate Dean for Clinical Research Director of Clinical Translational Science Institute USF

Well-intentioned and “common sense” advice led to tens of thousands avoidable cot deaths Dr Zabriskie’s Mother & Baby Care recommends face up Back to Sleep” Campaigns Dr Spock 1956 edition switches his recommendation to face down 1950 1960 1970 UK 1980 1990 First study Suggests harm 2000 2010 Systematic Review Published Second study Suggests harm Three further studies; two suggest harm Evans, Thornton, Chalmers, Glasziou. Testing treatments. 2 ed, 2011 USA

Courtesy of Dr. Glasziou

Should similar patients with similar conditions be Which rate is right? Inpatient Cholecystectomy similarly? Medicare Enrollees treated per 1, 000 Rates among regions vary by 10 times for shoulders, four times for hip and knees Dramatic practice variations in the management of seemingly similar patients and their conditions are well documented

Percentage of Participants Choosing Each Treatment Option for the Management of Prostate Cancer: an Example of Practice Variation Practice variations occur across large and small areas, among physicians within the same medical settings, in test and treatment use, and in hospital admissions, referral rates, and surgical rates Schwartz R. N Engl J Med 2009; 360: e 4

Which recommendations? “Official” Medicare rules for reimbursment Drug Cancer Off-Label use approved under new Medicare rules Bevacizumab (Avastin) Oxaliplatin (Eloxatin) Irinotecan (Camptosar) Docetaxel (Taxotere) Gemcitabine (Gemzar) Rituximab (Rituxan) Erlotinib (Tarceva) Recommended Compendia Medicare is Using AHFS-DI DRUGDEX CLINICAL PHARAMCOLOGY Breast Lung Breast ------ --- Esophageal Gastric Ovarian Biliary Tract Bladder Ovarian Chronic Lymphocytic Leukemia Head and Neck Pancreatic ----- NCCN --- Advised Against ------- Recommendation unclear ----- ---Not discussed

What Works in Health Care? • Well documented variation in clinical practice • “…actual treatments are often dictated by authority, personal experience, and habit” • Evidence-based medicine movement was born out of an attempt “to find out what works and what doesn’t in medicine”…through systematic, standardized analysis of available evidence. (Science 1996; 272: 22 -24)

“Inconvenient truths” • 30% of US health care is inappropriate, or wasteful • Only 55% of needed health services is delivered • 70, 000 deaths from medical errors • 3 -20 x service variation by geography • $2 trillion spent on health care • 16% of GDP Manchikanti L et al 2010

Part I. Understanding evidence

What is Evidence-Based Medicine? • Evidence is that “which enables the mind to see truth” (Webster’s Dictionary) • Evidence as “ground for belief” , that which justifies belief • proof (Dutch, German, Spanish, Portuguese) • fact (Slavic languages, French) • witness (Hindi) • effectiveness (Italian) • knowledge (Norwegian) • scientific (Japanese)

Use of evidence in health care • Evidence as neutral arbiter among competing views – Response to variation in practice of medicine • “Evidence as a guide to truth” • as a reliable sign, symptoms, or mark of that which it is evidence of i. e. , evidence “points beyond itself” to enhance reasonableness or truthfulness of some particular claim • “Rational thinkers respect their evidence” Djulbegovic, Guayatt, Ashcroft, 2009

EBM-definition • “It is a set of principles and methods to ensure that, to the greatest extent possible, population-based policies and individual decisions are consistent with evidence of effectiveness and benefits” – EBM is primarily focused on health outcomes • “To be called ‘evidence-based’ the decision must at least be consistent with whatever evidence does exist” • Don’t call your decisions/policies ‘evidence-based’ if they are not based on systematic reviews i. e. totality of available evidence. • weight that is attached to any particular information – to support a given decision or policy- is proportional to the underlying reliability of the information – an assumed link between high quality evidence and “truth” Tunis, Eddy D. Health Affairs 2007; June 19; Djulbegovic, Guayatt, Ashcroft, 2009

EBM-historical perspectives Three movements (~1990): • Clinical Practice Guidelines and the need to standardize the practice (US) • Evidence Based Medicine and the need for critical appraisal (Canada) • Systematic reviews and the need for research synthesis-Cochrane Collaboration (UK)

Evidence-based Medicine: an Idea of the Year • A notion that claims of health intervention effects should be founded on scientifically valid empirical evidence was dubbed by the New York Times as one of the “Ideas of the Year” in 2001

Not All Evidence Is Created Equal • Understanding the hierarchy of clinical evidence • Developing rules of evidence

Important Domains and Elements for Systems to Rate Grade the Strength of Evidence Quality: the aggregate of quality ratings for individual studies, predicated on the extent to which bias was minimized. Quantity: magnitude of effect, numbers of studies, and sample size or power. Consistency: for any given topic, the extent to which similar findings are reported using similar and different study designs.

106 Number of Systems to Rate the Quality of Individual Studies *none is considered useful *their continuing use will not help reduce errors in making of recommendations, will not help improve communication between guidelines developers and guidelines users, and thus will not help people make well-informed decisions. AHRQ Report 2002 No. 02 -E 016

New system to unify/ improve on the existing systems of rating the strength of scientific evidence • GRADE (Grading of Recommendation Assessment, Development and Evaluation) system BMJ 2004; June 19

Part II. Type of evidence

The nature of Medical Evidence Direct Evidence (Acting on basis of results of randomized controlled trials, case-control studies, etc) Indirect Evidence (Acting on basis of an understanding of biology of a disease and mechanism of action of the intervention)

Relevance: Type of Evidence • POE: Patient-oriented evidence – mortality, morbidity (disease-free survival), quality of life • DOE: Disease-oriented evidence – pharmacology (CR, PR, level of markers), pathophysiology, etiology

Part III Return on research/health care investment: -spending vs. quality of care/research -Generating False Research Findings or Ignoring Best Research Practices

Systematic flaws in generation of evidence: “Scandal of poor medical research”

Most Published Research Findings Are False PPV (Probability That a Research Finding Is True) PLo. S Med. 2005 August; 2(8): e 124 Power (1 - ß), Ratio of True to Not-True Relationships (R), and Bias (u)

Avoidable waste in the production and reporting of research evidence …US$100 billion is invested every year in supporting biomedical research, which results in an estimated 1 million research publications per year…$100, 000/paper …assuming 50% loss at all 4 stages, then 93. 5% of research represents waste…. or, $95 billions Source: The Lancet 2009; 374: 86 -89 (DOI: 10. 1016/S 0140 -6736(09)60329 -9) Iain Chalmers, DSc and Paul Glasziou, RACGP

Examples of early adoption of evidence based on premature or non-controlled trials that was later contradicted • Hormone-replacement therapy – ↑breast cancer, infarct, stroke, dementia & thrombosis • Anti-arrhythmic drugs (class I) – ↑ infarcts & stroke • NSAIDs (Cox II inhib. e. g. Vioxx) – ↑ infarcts & stroke • Anti-ageing (vit E) – ↑deaths • Bone marrow transpl. (breast ca) – ↑deaths • 3 rd vs. 2 nd vs. 1 st generation of chemo. Rx in NHL – ↑ toxicity • Surgery – Emphysema – spinal fusion – arthroscopy for OA • Anti-diabetic (Rezulin) – Liver toxicity • Anti-obesity(Fen-Phen) – Cardiovascular ADE courtesy of Dr. Brezis

Best Research Practices are often ignored • Cancer outcomes could be improved by 30% with at least 10% reduction in cancer mortality with optimum application of what is currently known • Crystalloids rather colloids • Poor detection, undertreatment, or inadequately controlled hypertension • Underuse of ASA after MI/stroke • Substantial delays between cardiovascular trial results and textbook recommendations – Evidence-based gaps of underuse and overuse • Need for Knowledge Translation ACP March/April 2005; Semin Oncol 1990: 4: 485 ACP March/April 2005

Status of Quality of Evidence in Hematology-Oncology Djulbegovic et al Am J Med 1999; 106: 198 -205

Part IV Practice of EBM

The purpose of evidence-based medicine To learn how to: • distinguish evidence from propaganda (advertisement) • probability from certainty • data from assertions • rational belief from superstitions • science from folklore • theory from dogma (P. D. Hurd, Sci Am 1999; 281: 87 -92)

Evidence-based medicine process EBM is problem-oriented practice that seeks to • Convert information needs regarding patient care into answerable questions • Track down, with maximum efficiency, the best evidence with which to answer them • Critically appraise that evidence for its validity (closeness to the truth), relevance (clinical applicability), and power (precision) • summarize and integrate this appraisal with clinical expertise to apply results in practice • evaluate performance and the impact of the process on clinical practice Sackett, 1995

Process Evidence-based Medicine (5 As) • • • Ask Access Appraise Apply Assess

Part V Knowledge Resources: handling information explosion and bias

Information explosion and brain’s limited capacity for information processing and storage • 6 million articles is published in >20, 000 biomedical journals annually • Medical students are given to read > 11, 000 pages/yr • Practicing physicians need to instantaneously recall more than 2 million facts • MEDLINE Over 11 million indexed citations from~ 4, 500 journals • Randomized controlled trials – There about 60, 000 clinical trials currently under way with more than 200 clinical trials published in the scientific literature every year Djulbegovic B, Lyman GH, Ruckdeschel J. Why Evidence-based Oncology 2000; 1: 2 -5

Searching for evidence Arch Dis Child 2002; 87: 168– 169

Challenge to providers Facts per Decision 1000 Proteomics and other effector molecules 100 Functional Genetics: Gene expression profiles 10 Human Cognitive Capacity Structural Genetics: e. g. SNPs, haplotypes 5 Decisions by Clinical Phenotype 1990 2000 2010 2020 William Stead, IOM Meeting, 8 October 2007. Growth in facts affecting provider decisions versus human cognitive capacity.

Physicians’ unaided mind is not sufficient for optimal decision-making • For every three patients seen physicians have 2 unanswered questions • This means that decisions could have been different 66% of time if the access to evidence (knowledge resources) had been optimal Ann Internal Med 1985; 103: 596 -599 BMJ 1995; 310: 1085 -1086 JAMA 1998; 280: 1347 -1352)

Information explosion and need to filter and synthesize evidence • Need for filtering high-quality evidence – variety of sources of primary and secondary evidence have been developed • Systematic review/ Meta-analyses as a means to provide totality of evidence

Solving information paradox: doable exercise of developing Resources to Integrate Evidence • Data consistently point that only about 1 -2% of published evidence meet criteria for validity and relevance to be of interest for practicing physicians (Evidence-based Oncology 2001; 2: 1)

Mc Master Evidence-based Medicine Service (general medicine) Clinical Relevancy Filter (MORE) ~3, 000 articles/y meet critical appraisal and content criteria (94% noise reduction) ~20 articles/yr for clinicians (99. 96% noise reduction) ~5 -50 articles/y for authors of evidencebased clinical topic reviews Courtesy of Dr. Brian Haynes

Cochrane Database of Systematic Reviews The Cochrane Collaboration - an international network of individuals and institutions committed to preparing, maintaining, and promoting the accessibility of systematic reviews of the effects of health care interventions. Cochrane Systematic Reviews (6, 500) (Feb 2011) Database of Abstracts of Reviews of Effectiveness (14, 018) Registry of Randomized Controlled Trials (641, 406)

Part VI Evidence and decision-making

Evidence and decision-making • Every day we make ~ 2, 000 decisions. – How should we make them? – How do we make them? – Can we improve decisionmaking?

What is diagnosis?

Gut feeling vs. analytical diagnostic approach • Problem: diagnosis of serious infection in children presenting to primary care physicians. • Participants: Consecutive series of 3890 children and young people aged 0 -16 years presenting in primary care • Results: Intuition that something was wrong despite the clinical assessment of non-severe illness substantially increased the risk of serious illness (likelihood ratio 25. 5, 95% confidence interval 7. 9 to 82. 0) and acting on this gut feeling had the potential to prevent two of the six cases being missed (33%, 95% confidence interval 4. 0% to 100%) at a cost of 44 false alarms (1. 3%, 95% confidence interval 0. 95% to 1. 75%). BMJ 2012; 345: e 6144

Accuracy of gut feeling vs. analytical diagnostic approach BMJ 2012; 345: e 6144

Which of these treatments would your recommend to this patient? 65 y/o patient in remission after induction Rx for AML. You are considering consolidation with allo. SCT vs. chemo. Rx? Without Rx, relapse is assumed to be 100% and all patients will die. Transplant: Chemo. Rx: LFS=60%; TRM=20%; B/H=3 LFS=20%; TRM=1%; B/H=20

Optimal choice of consolidation Rx for AML may require mathematical modeling (e. g, decision analysis)

Rational decision-making and architecture of cognition: dual-processing theory -How we think determines how we make decisions, which in turns determines patients’ outcomes - Understanding cognitive processes may help us develop optimal “rules” for decision-making Kahneman D. Maps of bounded rationality: psychology for behavioral economics. Am Econom Review 2003; 93: 1449 -1475 Slovic et al 2004

Health is a decision-making problem as much as a health care or policy problem: Personal Decisions Are the Leading Cause of Death Personal decisions concerning Actual causes of death Medical causes of death Keeney R. Operations Research 56(6), pp. 1335– 1347

Which process is more accurate correct? Intuition vs. deliberation? • A $64 B question

Part VII Improving quality of care

Reasons for suboptimal care • The poor (or the lack of high quality) evidence related to the effects of most health care interventions. – Evidence is necessary but not sufficient for decision-making • The second reason for delivery of inappropriate health care relates to the suboptimal decision-making.

From research-evidence-to-practice pipeline: getting the evidence straight and getting the straight evidence used Need for Knowledge Translation: 80% of transfer between 7 stages of diffusion of evidence result in only 21% patient use (0. 87=21%) ACP J Club 2005 Empirical data: overall recommendations were adhered to 1/3 of the time (Mickan et al)

Linking EBM and clinical quality of improvement: ‘do the right things right’ ‘doing the right things’ (based on external research evidenced) ‘doing things right’ (based on local processes) Glasziou et al, 2011

Time is a key commodity of our times…

Presentation of research findings in real-time is of key importance When knowledge is needed How quickly? What? Speaking with a patient Within 15 seconds (such as in consultation or on ward round) "Bottom line" that can be Reflecting on a patient Within two minutes (such as over coffee when discussing a case with colleagues or when writing to a colleague) Structured summary of article, systematic review, or guideline that can be read in five minutes or During training or Full text linked to other relevant material or open educational resources, allowing users to read as much as they want, usually at home, either for research or continuing professional development Within a week BMJ 1999; 319: 1476 -1479 ( 4 December ) reached within 15 seconds less