Slide 1 2003 By Default Methodologies and

Slide 1 © 2003 By Default! Methodologies and Automated Applications for Post-Marketing Outcomes Surveillance of Medical Devices and Medications Michael E. Matheny, MD, MS NLM Biomedical Informatics Fellow Decision Systems Group, Department of Radiology Brigham & Women’s Hospital, Boston, MA A Free sample background from www. powerpointbackgrounds. com

Slide 2 © 2003 By Default! Outline n Post-Marketing Surveillance Background n Statistical Methodology Development n Computer Application Development n Clinical Examples n Future Directions A Free sample background from www. powerpointbackgrounds. com

Slide 3 © 2003 By Default! Background Surveillance Rationale n Phase 3 Trials insufficient to ensure adequate safety of medications and devices – Low frequency events are not detected – Protected populations (pregnant women, children) and more ill populations not represented – Complications delayed by a number of years cannot be detected A Free sample background from www. powerpointbackgrounds. com

Slide 4 © 2003 By Default! Background FDA Medical Devices n 1, 700 types

Slide 5 © 2003 By Default! Background FDA New Drug Applications (NDA) A Free

Slide 6 © 2003 By Default! Background Current Post-Marketing Surveillance n Combination of mandatory and voluntary adverse event reporting – Mandatory reporting by manufacturers and health facilities – Voluntary Med. Watch / MAUDE reports by providers and patients 2004 Drug-Related Adverse Event Reports Total 422, 889 Manufacturer & Facility Reports 401, 396 Med. Watch 21, 493 A Free sample background from www. powerpointbackgrounds. com

Slide 7 © 2003 By Default! Background MAUDE Cypher Reporting Rate 2003 FDA Warning

Slide 8 © 2003 By Default! Background Current Post-Marketing Surveillance n ‘Phase 4’ Trials – Poor Compliance • As of March 2006 report, 797 of 1231 (65%) agreedupon trials had yet to be started – Barriers • Lack of manufacturer incentives – Expensive – Drug already on the market • Lack of regulatory enforcement A Free sample background from www. powerpointbackgrounds. com

Slide 9 © 2003 By Default! Background Medical Device Recalls n Boston Scientific cardiac stent (1998) – Balloon rupture at low pressures n Guidant cardio-defibrillator (2005) – Malfunction due to electrical short n Vioxx (2004) – cardiovascular complications n Tequin (2006) – Hypoglycemia and hyperglycemia A Free sample background from www. powerpointbackgrounds. com

Slide 10 © 2003 By Default! Background FDA Response n Increasing demands for Phase 4 trials n Legislation to increase quality of adverse event reporting n Emphasizing trial registries (clinicaltrials. gov) as way to prevent omission of results n Commissioned IOM report “The Future of Drug Safety” A Free sample background from www. powerpointbackgrounds. com

Slide 11 © 2003 By Default! Background Adverse Event Data Continuum Phase 3 Trials Internal Validity Generalizibility Breadth Immediacy Lack of Bias +++ +/++ --+++ A Free sample background from www. powerpointbackgrounds. com MAUDE / Voluntary Med. Watch Registry -+/+++ --- ++ ++ --- Mandatory Registry +++ ++

Slide 12 © 2003 By Default! Statistical Methods Medical Outcomes Monitoring n Using registry data that tracks all patients allows different types of analysis than used in the FDA’s adverse event reporting systems n No generally accepted methods for monitoring registry data for adverse events – Lack of sufficient discrete electronic data sources to construct registries – Some outcomes are challenging or expensive to track for an entire population A Free sample background from www. powerpointbackgrounds. com

Slide 13 © 2003 By Default! Objective n Develop methodologies and implement an automated computer monitoring system to perform outcomes surveillance of registry data for new medical devices and medications A Free sample background from www. powerpointbackgrounds. com

Slide 14 © 2003 By Default! Statistical Methods Statistical Process Control A Free sample

Slide 15 © 2003 By Default! Statistical Methods Bayesian Updating Statistics A Free sample

Slide 16 © 2003 By Default! Statistical Methods Establishing Baseline Data n Primary Data Sources – Phase 3 trial data – Post-Marketing data from a closely related medication/device n Alternative Data Sources A Free sample background from www. powerpointbackgrounds. com

Slide 17 © 2003 By Default! Statistical Methods Establishing Alerting Thresholds n Use number of events and sample size to calculate proportion with confidence intervals n Typically, medical domains use 95% CI or 1. 96 sigma from the point estimate A Free sample background from www. powerpointbackgrounds. com

Slide 18 © 2003 By Default! Statistical Methods Establishing Alerting Thresholds SPC BUS A

Slide 19 © 2003 By Default! Statistical Methods Establishing Alerting Thresholds A Free sample

Slide 20 © 2003 By Default! Statistical Methods Establishing Alerting Thresholds n Wilson’s method

Slide 21 © 2003 By Default! Statistical Methods Risk Stratification n Allows creating subgroups for separate analyses n Single variable n Logistic regression model with scoring thresholds A Free sample background from www. powerpointbackgrounds. com

Slide 22 © 2003 By Default! Application Development DELTA n Data Extraction and Longitudinal Time Analysis (DELTA) n Design Goals – Generic data import format – Allow both prospective and retrospective analyses – Modular framework to allow sequential addition of statistical methodologies – Multiple alerting methods – Any number of concurrent ongoing analyses A Free sample background from www. powerpointbackgrounds. com

Slide 23 © 2003 By Default! Application Development DELTA Clinical Data Entry Source Database DELTA Database SPC VPN Intranet Web Server Data Dictionary Source IT Manager DELTA Users A Free sample background from www. powerpointbackgrounds. com BUS Statistical Modules

Slide 24 © 2003 By Default! Application Test Data Cypher Drug-Eluting Stent (DES) n Setting: – Brigham & Women’s Hospital (07/2003 – 12/2004) n Population: – All patients receiving a drug-eluting stent (2270) n Outcome: – Post-procedural in-hospital mortality (27) n Baseline: – University of Michigan Data (1997 -1999) A Free sample background from www. powerpointbackgrounds. com

Slide 25 © 2003 By Default! Application Development DELTA A Free sample background from

Slide 26 © 2003 By Default! Application Development DELTA A Free sample background from

Slide 27 © 2003 By Default! Application Development DELTA A Free sample background from

Slide 28 © 2003 By Default! Application Development DELTA A Free sample background from

Slide 29 © 2003 By Default! Application Development DELTA A Free sample background from

Slide 30 © 2003 By Default! Application Development DELTA A Free sample background from

Slide 31 © 2003 By Default! Application Development DELTA A Free sample background from

Slide 32 © 2003 By Default! Application Development DELTA A Free sample background from

Slide 33 © 2003 By Default! Application Development DELTA A Free sample background from

Slide 34 © 2003 By Default! Risk Stratification Potential Solution n Incorporate individual risk prediction models in order to adjust for case mix and illness severity A Free sample background from www. powerpointbackgrounds. com

Slide 35 © 2003 By Default! Possible Risk Prediction Methods n Linear / Logistic Regression n Artificial Neural Networks n Bayesian Networks n Support Vector Machines A Free sample background from www. powerpointbackgrounds. com

Slide 36 © 2003 By Default! LR External Validation Models Model Dates Location Sample NY 1992 1991 NY 5827 NY 1997 1991 – 1994 NY 62670 CC 1997 1993 – 1994 Cleveland, OH 12985 NNE 1999 1994 – 1996 NH, ME, MA, VT 15331 MI 2001 1999 – 2000 Detroit, MI 10796 BWH 2001 1997 – 1999 Boston, MA 2804 ACC 2002 1998 – 2000 National 100253 A Free sample background from www. powerpointbackgrounds. com

Slide 37 © 2003 By Default! LR External Validation n Setting: – Brigham & Women’s Hospital (01/2002 – 09/2004) n Population: – All patients undergoing percutaneous coronary intervention (5216) n Outcome: – Post-procedural in-hospital mortality (71) A Free sample background from www. powerpointbackgrounds. com

Slide 38 © 2003 By Default! LR External Validation Results Deaths AUC HL χ2 HL (p) NY 1992 96. 7 0. 82 31. 1 <0. 001 NY 1997 61. 6 0. 88 32. 2 <0. 001 CC 1997 78. 8 0. 88 27. 8 <0. 001 NNE 1999 56. 2 0. 89 45. 9 <0. 001 MI 2001 61. 8 0. 86 30. 4 <0. 001 BWH 2001 136. 1 0. 89 39. 7 <0. 001 ACC 2002 49. 9 0. 90 42. 0 <0. 001 BWH 2004 70. 5 0. 93 7. 61 0. 473 Curve A Free sample background from www. powerpointbackgrounds. com

Slide 39 © 2003 By Default! LR External Validation Conclusions n Excellent discrimination across all models n Calibration (Hosmer-Lemeshow) poor for all models but recent local one n Addressed categorical risk stratification by keeping all records in one stratum n Calibration problems over time limit application, and require exploration of recalibration methods A Free sample background from www. powerpointbackgrounds. com

Slide 40 © 2003 By Default! OPUS (TIMI-16) n Setting: n Intervention: n Population: n Outcome: n Baseline: – 888 Hospitals in 27 Countries – Oral IIb-IIIa Inhibitor vs Placebo – Intervention Arm [Both arms identical at 30 days] (6867) – 30 day mortality – Trial stopped early due to elevation in intervention arm – Control Arm (3421) A Free sample background from www. powerpointbackgrounds. com

Slide 41 © 2003 By Default! OPUS (TIMI-16) 30 Day Mortality Control Intervention Period Events Patients Event Rate (%) p 1 0 0 0 5 0. 0 * 2 0 17 0. 0 0 30 0. 0 * 3 0 48 0. 0 1 95 1. 1 1. 000 4 0 135 0. 0 7 249 2. 8 0. 102 5 1 268 0. 4 11 529 2. 1 0. 070 6 4 463 0. 9 18 951 1. 9 0. 173 7 5 764 0. 7 33 1, 528 2. 2 0. 008 8 8 1, 089 0. 7 46 2, 161 2. 1 0. 003 9 11 1, 412 0. 8 54 2, 815 1. 9 0. 004 10 16 1, 805 0. 9 78 3, 610 2. 2 <0. 001 11 21 2, 173 1. 0 92 4, 410 2. 1 <0. 001 12 33 2, 701 1. 2 109 5, 447 2. 0 0. 011 13 46 3, 360 1. 4 134 6, 756 2. 0 0. 031 14 46 3, 421 1. 3 134 6, 867 2. 0 0. 031 A Free sample background from www. powerpointbackgrounds. com

Slide 42 © 2003 By Default! OPUS (TIMI-16) Alert Summary A Free sample background

Slide 43 © 2003 By Default! CLARITY (TIMI-28) n Setting: n Intervention: n Population n Outcome: n Baseline – 313 Hospitals in 23 Countries – Oral Anti-Platelet Agent vs Placebo – Intervention Arm (1751) – Major Bleeding – DSMB concerned, but trial did not stop early – Control Arm (1739) A Free sample background from www. powerpointbackgrounds. com

Slide 44 © 2003 By Default! CLARITY (TIMI-28) Major Bleeding Control Intervention Period Events Patients Event Rate (%) p 1 0 4 0. 0 0 1 0. 0 * 2 0 13 0. 0 0 12 0. 0 * 3 1 27 3. 7 0 23 0. 0 1. 000 4 2 40 5. 0 1 49 2. 0 0. 586 5 2 73 2. 7 2 83 2. 4 1. 000 6 3 116 2. 6 5 126 4. 0 0. 724 7 4 168 2. 4 7 173 4. 0 0. 548 8 4 214 1. 9 9 226 4. 0 0. 262 9 7 276 2. 5 10 284 3. 5 0. 624 10 8 337 2. 4 12 361 3. 3 0. 502 11 10 424 2. 4 15 450 3. 3 0. 423 A Free sample background from www. powerpointbackgrounds. com

Slide 45 © 2003 By Default! CLARITY (TIMI-28) Major Bleeding Control Intervention Period Events Patients Event Rate (%) p 12 11 536 2. 1 16 554 2. 9 0. 438 13 13 639 2. 0 18 649 2. 8 0. 468 14 17 776 2. 2 22 780 2. 8 0. 517 15 17 892 1. 9 23 926 2. 5 0. 427 16 18 1, 041 1. 7 28 1, 058 2. 6 0. 180 17 20 1, 192 1. 7 31 1, 195 2. 6 0. 156 18 24 1, 314 1. 8 31 1, 327 2. 3 0. 414 19 25 1, 457 1. 7 31 1, 459 2. 1 0. 500 20 26 1, 584 1. 6 32 1, 606 2. 0 0. 509 21 30 1, 739 1. 7 34 1, 751 1. 9 0. 706 A Free sample background from www. powerpointbackgrounds. com

Slide 46 © 2003 By Default! CLARITY (TIMI-28) Alert Summary A Free sample background

Slide 47 © 2003 By Default! OPUS /CLARITY Conclusions n SPC performed well in the positive study, but did have some false positive alerts in the negative study n LR stratified SPC failed to alert early in the positive study, but performed well in the negative study n BUS was more specific than SPC in both studies A Free sample background from www. powerpointbackgrounds. com

Slide 48 © 2003 By Default! Sensitivity Analysis n Setting: n Population: n Outcome: – Brigham & Women’s Hospital (01/2002 – 12/2004) – All patients undergoing percutaneous coronary intervention (6175) – Post-procedural major adverse cardiac events (403) • • • n Death Post-Procedural Myocardial Infarction Repeat Vascularization Baseline: – Arbitrarily set event rates and sample sizes A Free sample background from www. powerpointbackgrounds. com

Slide 49 © 2003 By Default! Sensitivity Analysis Results A Free sample background from

Slide 50 © 2003 By Default! Clinical Alert n Setting: – Brigham & Women’s Hospital (01/2002 – 12/2004) n Population: – All patients receiving a vascular closure device after percutaneous coronary intervention (3947) n Outcome: – Retroperitoneal Hemorrhage (25) n Baseline: – Stanford University Data (2000 – 2004) A Free sample background from www. powerpointbackgrounds. com

Slide 51 © 2003 By Default! Event Rate Elevation A Free sample background from

Slide 52 © 2003 By Default! Manual Review n Triggered root cause analysis n Manual chart review and multivariable analsysis n Final Result: Not related, confounded by indication A Free sample background from www. powerpointbackgrounds. com

Slide 53 © 2003 By Default! Future Work Methodology n Address Calibration Concerns – Recalibration of Logistic Regression models – Development of Machine Learning Risk Prediction Models n Address BUS Insensitivity – Incorporate data weight decay over time A Free sample background from www. powerpointbackgrounds. com

Slide 54 © 2003 By Default! Future Work Application Brigham and Women’s Hospital Outcomes DB Massachusetts Data Analysis Center (Mass-DAC) DELTA Agent DELTA Server DELTA reports Massachusetts General Hospital Outcomes DB DELTA Agent SMTP E-mail Server E-mail alert St. Elizabeth’s Medical Center Outcomes DB DELTA Agent A Free sample background from www. powerpointbackgrounds. com MA Claims DB MA Death Index

Slide 55 © 2003 By Default! Future Work n New Medication Outcomes Surveillance n Inpatient (versus Outpatient) – More frequent monitoring – Higher quality source data – Outcomes easier to capture A Free sample background from www. powerpointbackgrounds. com

Slide 56 © 2003 By Default! Future Work Develop Data Repository Local Institution Patient Demographics Hospital Billing Computerized Order Entry Medication Administration Laboratory Radiology IS Centralized Data Repository DELTA Server Progress Notes (NLP) State Death Index A Free sample background from www. powerpointbackgrounds. com Other Outcomes

Slide 57 © 2003 By Default! Future Work Initial Framework n New medication laboratory monitoring protocol n Standard measures that are most commonly affected in new medications – AST, ALT, Creatinine, WBC, Platelets n Establish reasonable baselines – Closely Related medication lab results – Unrelated medication lab results – Expert Panel Estimation A Free sample background from www. powerpointbackgrounds. com

Slide 58 © 2003 By Default! Acknowledgements n Mentors n Collaborators n Programming Team n Funding – Lucila Ohno-Machado, MD, Ph. D – Frederic S. Resnic, MD, MS – Nipun Arora, MD – Sharon Lise-Normand, Ph. D – Ewout Steyerberg, Ph. D – – – Richard Cope Barry Coflan Atul Tatke – NLM R 01 -LM-08142 – NLM T 15 -LM-07092 A Free sample background from www. powerpointbackgrounds. com

Slide 59 © 2003 By Default! The End Michael Matheny, MD MS mmatheny@dsg. harvard. edu Brigham & Women’s Hospital Thorn 309 75 Francis Street Boston, MA 02115 A Free sample background from www. powerpointbackgrounds. com