How Good Is the Evidence Naomi Aronson Ph

How Good Is the Evidence? Naomi Aronson, Ph. D. Executive Director, Technology Evaluation Center Blue Cross Blue Shield Association Mass. MEDIC April 20, 2006 Copyright 2005 Blue Cross Blue Shield Association

Overview • TEC perspective and process • Evidence quality: inside the RCT • Cost effectiveness and affordability 2 Copyright 2006 Blue Cross Blue Shield Association

Covering America Blue Plans cover every community in the nation • 38 Blue Cross and Blue Shield Plans • 94 million members BCBSA • Largest processor of Medicare claims in the nation • Contract with 90% of hospitals, 80% of doctors • 4 -million member FEP Program – Largest private health insurance product in world 3 Copyright 2006 Blue Cross Blue Shield Association

Technology assessment supports health plans and other stakeholders in developing evidence-based policies Medical Policy • Based on scientific evidence • Costs and coverage NOT considered 4 Coverage Policy • Determined by purchasers of health plan products • Cost effectiveness considered Payment Policy Contract between health plans and medical professionals and providers Copyright 2006 Blue Cross Blue Shield Association

Technology Evaluation Center • Rigorous assessment of clinical evidence, systematic review with quality appraisal • Independent, expert Medical Advisory Panel – Academic clinical researchers (Harvard, Stanford, Johns Hopkins) – Specialty society appointees – Only 4 of 17 votes are Plan M. D. s • Does this technology improve health? 5 Copyright 2006 Blue Cross Blue Shield Association

6 Copyright 2006 Blue Cross Blue Shield Association

Technology Evaluation Center • 300+ technology assessments 3 -year inventory at www. bcbs. com/tec • Publications in medical journals – Annals of Internal Medicine – Journal of the National Cancer Institute – Journal of the American College of Surgeons • Agency for Healthcare Research and Quality (AHRQ) Evidence-based Practice Center 7 Copyright 2006 Blue Cross Blue Shield Association

TEC Leadership • Pharmacogenomics – CYP 450 report sorts hypotheses and facts TEC Report CYP 450 • Imaging – TEC series in Journal of American College of Radiology • Cancer Therapies – ASCO health services research and guidelines committee 8 Copyright 2006 Blue Cross Blue Shield Association

Medical Policy Reference Manual • TEC and Medical Policy are BCBSA’s core clinical information services • Integrated to maximize depth and breadth, rigorous documentation and operational efficiency, public prestige, and proprietary service • Medical Policy Reference Manual is an inventory of about 350 policies updated annually • Incorporates TEC Assessments and many additional policies developed or updated through Medical Policy process • Typical topic cycle is Medical Policy TEC Medical Policy • Medical Policy Panel of BCBS Plan medical directors 9 Copyright 2006 Blue Cross Blue Shield Association

Systematic review vs. narrative review • Use prospectively defined and reproducible methods to control against bias • Treat articles and reported data as subjects in a study • Seek all potentially relevant studies • Rules govern study selection and data abstraction • Relevance and study quality thresholds reduce noise • Assess quality of studies and body of evidence • Summarize abstracted data, not others’ conclusions 10 Copyright 2006 Blue Cross Blue Shield Association

Good clinical studies are like good accounting and financial practices Deviation from standards leads to unreliable results and misinformed decisions 11 Copyright 2006 Blue Cross Blue Shield Association

Quality parameters: sources • U. S. Preventive Services Task Force (USPSTF), Harris et al. (2001). Am J Prev Med, 20(3 suppl): 21 -35 (available at www. ahrq. gov/clinic/ajpmsuppl/harris 1. htm) • Consolidated Standard of Reporting Trials (CONSORT 1996, 2001; available at www. consort-statement. org) • Systems to Rate the Strength of Scientific Evidence, West et al. AHRQ Evidence Report #47 (2002; available at www. ahrq. gov) • “Strength of recommendation taxonomy (SORT): a patientcentered approach to grading evidence in the medical literature, ” Ebell et al. (2004). Am Fam Physician, 69(3): 548 -56 12 Copyright 2006 Blue Cross Blue Shield Association

Inside the randomized controlled trial • Prospectively defined and clinically significant outcome measures • Populations and comparators • Double-blinded is best estimate of effect size • Randomization • Intention-to-treat analysis • Generalizing trial findings See CONSORT Statement 2001 13 Copyright 2006 Blue Cross Blue Shield Association

Inside the randomized controlled trial • Outcome measures should be prospectively defined and clinically significant Extracorporeal Shock Wave Therapy for Plantar Fasciitis. TEC Assessments 2005 (Vol. 19, No. 18) http: //www. bcbs. com/tec/vol 19/19_18. pdf 14 Copyright 2006 Blue Cross Blue Shield Association

Inside the randomized controlled trial • Study populations must be clearly defined • Comparison treatment must be clinically effective and appropriate Artificial Vertebral Disc Replacement. TEC Assessments 2005 (Vol. 19, No. ) www. bcbs. com/tec/Vol 20/20_01. pdf 15 Copyright 2006 Blue Cross Blue Shield Association

Inside the randomized controlled trial • Double-blinding gives best estimate of effect size 16 Copyright 2006 Blue Cross Blue Shield Association

Double-blinding not limited to drug trials • More difficult for surgery, but not impossible – Embryonic cell transplant for Parkinson’s disease (Freed et al. N Engl J Med 2001; 344 (10): 710 -719) – Percutaneous transmyocardial revascularization (Stone et al. J Am Coll Cardiol 2002; 39(10): 1581 -7) – Arthroscopic knee surgery for osteoarthritis (Moseley et al. N Engl J Med 2002; 347 (2): 81 -88) • Blinded, independent outcome assessment 17 Copyright 2006 Blue Cross Blue Shield Association

Inside the randomized controlled trial • Randomization requires generating and concealing unpredictable sequence for allocating subjects to treatment group 18 Copyright 2006 Blue Cross Blue Shield Association

Inside the randomized controlled trial • Results should be reported as intention-to -treat analysis Intradiscal electrothermal therapy (IDET) TEC Assessments 2004 Vol. 18, No. 19 http: //www. bcbs. com/tec/Vol 18/18_19. pdf 19 Copyright 2006 Blue Cross Blue Shield Association

Intention-to-treat analysis • All participants should be included in analysis of outcomes • Analyze by initial group assignment • Avoid bias from nonrandom loss of participants – e. g. , dropouts who experienced no improvement • Application can be complex • If the author says intention-to-treat, check it out 20 Copyright 2006 Blue Cross Blue Shield Association

21

Inside the randomized controlled trial • Guidance on the reporting of RCTs and systematic reviews focus mainly on internal validity and give very little space to external validity. ” Rothwell PM, Lancet (2005) 365: 82 -93. 22 Copyright 2006 Blue Cross Blue Shield Association

Generalizing the trial findings • Many familiar recommendations: outcomes, who measures outcomes, comparators • Health care setting, national differences in practice, recruitment of patients • Enrichment strategies: run-in periods, exclusion of nonresponders to prior treatment • Differences between protocol and practice: timing of treatment, prohibition of non-trial treatments • Duration of treatment and follow-up for chronic conditions (asthma, depression, arthritis) • Severity vs. frequency of adverse effects 23 Copyright 2006 Blue Cross Blue Shield Association

Measuring adverse effects • Variation in what is reported, how classified, lumping and splitting (and how asked) • Frequency vs. severity of effects • RTOG criteria to grade toxicity severity • Withdrawal from therapy is a common summary measure • Overall survival more informative • Net health outcome vs. safety 24 Copyright 2006 Blue Cross Blue Shield Association

Selective reporting and publication bias • The “file drawer” problem: are nonsignificant results less likely to be published? • What about adverse results? • Recent systematic reviews analyze the relationship of sponsorship to outcomes and conclusions 25 Copyright 2006 Blue Cross Blue Shield Association

Selective reporting and influence of sponsorship • Chan AW, Hrobjartsson A, Haahr MT et al. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. [Chan AW, Hrobjartsson A, Haahr MT et al JAMA, 2004; May 26; 291(20): 2457 -65] • American Medical Association, Council on Scientific Affairs. 2004 Influence of Funding Source on Outcome, Validity and Reliability of Pharmaceutical Research. CSA Report 10 -A-04. Available online at: http: //www. ama-assn. org/ama/pub/category/14314. html 26 Copyright 2006 Blue Cross Blue Shield Association

Registration of clinical trials • American Medical Association, June 2004 • International Committee of Medical Journal Editors (ICMJE), September 2004 • 11 prestigious journals, including JAMA, New England Journal, Annals, Lancet • Effective in September 2005, only publish articles from trials prospectively registered in a public database • To date, only www. clinicaltrials. gov qualifies http: //jama. ama-assn. org/cgi/content/full/292. 11. 1363 v 1 27 Copyright 2006 Blue Cross Blue Shield Association

Cost, cost effectiveness and affordability • Clinical effectiveness is cornerstone of plan medical and coverage policy • New technologies may bring small benefit at high cost • Cost effectiveness and affordability are pressing issues • TEC is leading and educating on cost-effectiveness analysis methods • But no clear cost-effectiveness threshold: can you afford everything that is a “good buy”? 28 Copyright 2006 Blue Cross Blue Shield Association

Increases in Health Insurance Premiums Compared to Other Indicators 1988– 2005 * Estimate is statistically different from the previous year shown at p<. 05. No statistical tests were conducted for years prior to 1999. † Estimate is statistically different from the previous year shown at p<. 10. Note: Data on premium increases reflect total health insurance premiums for a family of four. Historical estimates of workers’ earnings have been updated to reflect new industry classifications (NAICS). Source: Kaiser/HRET Survey of Employer-Sponsored Health Benefits: 1999– 2005; KPMG Survey of Employer-Sponsored Health Benefits: 1993, 1996; The Health Insurance Association of America (HIAA): 1988, 1989, 1990; Bureau of Labor Statistics, Consumer Price Index, U. S. City Average of Annual Inflation (April to April), 1988– 2005; Bureau of Labor Statistics, Seasonally Adjusted Data from the Current Employment Statistics Survey (April to April), 1988– Copyright 2005 Blue Cross Blue Shield Association

Percent of Employers Offering Health Benefits to Employees 2000 -2005 There has been a steady decline in employer-offered health benefits since 2000 and large increases in premiums. From 2000 -2005, premiums paid by employers increased by 73%. Source: “Employer Health Benefits 2005 Annual Survey, ” (#7315), The Henry J. Kaiser Family Foundation and Health Research and Educational Trust, September 2005 This information was reprinted with permission from the Henry J. Kaiser Family Foundation. The Kaiser Family Foundation, based in Menlo Park, California, is a nonprofit, independent national health care philanthropy and is not associated with Kaiser Permanente or Kaiser Industries. 30 Copyright 2006 Blue Cross Blue Shield Association

Top Five Employer Priorities for 2005 Employers’ number one employee benefit priority is to control healthcare costs. Source: Deloitte Consulting (2004) Copyright © 2005 Deloitte Development LLC. All rights reserved. 31 Copyright 2006 Blue Cross Blue Shield Association

Summary • TEC process is evidence based and independent • Good clinical studies are like good financial and accounting practices, essential for informed decision-making. • There is no bright line to establish cost effectiveness - ultimately value cannot trump affordability 32 Copyright 2006 Blue Cross Blue Shield Association

Questions? 33 Copyright 2006 Blue Cross Blue Shield Association

Naomi Aronson, Ph. D. Executive Director, Technology Evaluation Center Director, Evidence-based Practice Center Blue Cross Blue Shield Association 312. 297. 5530 naomi. aronson@BCBSA. com 34

Claudia Bonnell, R. N. , M. L. S. Project Manager, Medical Information Technology Evaluation Center (TEC) Blue Cross Blue Shield Association 1310 G Street N. W. Washington, DC 20005 -3000 claudia. bonnell@BCBSA. com 202. 626. 8616 35 Copyright 2006 Blue Cross Blue Shield Association