The RE-AIM Framework History Recent Applications and Future

The RE-AIM Framework: History, Recent Applications, and Future Directions RUSSELL E. GLASGOW, PH. D. UNIVERSITY OF COLORADO, DENVER DEPARTMENT OF FAMILY MEDICINE COLORADO RESEARCH IN IMPLEMENTATION SCIENCE PROGRAM KT Canada – November 2014

Outline • Need for and History of RE-AIM • RE-AIM Framework—for planning, implementing, reporting, and reviewing • Example Applications • Challenges and Complexities • Future Directions

Pragmatic Perspective / World View “The importance of an idea or action lies in whether it makes a difference in everyday life. Ideas or actions that correspond to attractive explanations (e. g. , metaphysical theories), but make no difference to outcomes, are problematic. ” Charles Pierce

Challenge #1: Clinical Research is Slow • To most people, randomized controlled trials (RCTs) are the mainstay of clinical research. • But traditional RCTs are slow and expensive—and rarely produce findings that are easily put into practice. • In fact, it takes an average of 17 years before 14% of research findings lead to

Challenge #2: Research is Not Relevant to Practice Traditional RCTs study the effectiveness of treatments delivered to carefully selected populations under ideal conditions. This makes it difficult to translate results to the real world. “If we want more evidence-based practice, we need more practicebased evidence. ” Green LW. Am J Pub Health 2006 Even when we do implement a tested intervention into everyday clinical practice, we often see a “voltage drop”—a Rothwell PM. External validity of randomised controlled trials…Lancet 2005; 365: 82 -93. dramatic decrease in effectiveness.

Bench to Bookshelf

To Enhance External Validity, Fewer Exclusions Allow for a Broader Subset of Settings, Staff and Participants Traditional RCT PCT l Eligible population l Exclusions, nonresponse, etc. l Efficacy, among a defined subset l Eligible population l Exclusions, non -response, etc. l Effectiveness, in a broad subset Figure provided by Gloria Coronado, Ph. D, Kaiser Permanente Center for Health Research

External Validity/ Pragmatic Criteria— Often Ignored • Participant Representativeness • Setting Representativeness • Context and Setting • Community/Setting Engagement • Adaptation/change in Intervention • Sustainability • Costs/Feasibility of Tx • Comparison Conditions

“The significant problems we face cannot be solved by the same level of thinking that created them. ” A. Einstein

Readiness for Translation? RE-AIM • Internal validity perspective § The magnitude of effect as the key indicator of readiness for translation and adheres to the principles of evidence rating for determining efficacy • External validity perspective § Attention to intervention features that can be adopted and delivered broadly, have the ability for sustained and consistent implementation at a reasonable cost, reach large numbers of people, especially those who can most benefit, and produce replicable and long-lasting effects Glasgow RE, Vogt TM, Boles SM. Evaluating the Public Health Impact…Am J Public Health, 1999; 89: 1322 -1327

Original RE-AIM • First published articles in 1999 • Originally intended to increase balance between internal and external validity • First used to evaluate prevention and health behavior change programs • RE-AIM Trivia: was going to be called ARIEM

RE-AIM Outcomes • Reach: The number, percent of target audience, and representativeness of those who participate (I) • Effectiveness: Change in outcomes and impact on quality of life and any adverse outcomes (I) • Adoption: Number, percent, and representativeness of settings and educators who participate (S) • Implementation: Extent to which a program or policy is delivered consistently, adaptations made, and the time and costs of the program (S) • Maintenance: Sustained change in outcomes and impact on quality of life and any adverse outcomes (I) (S) • Maintenance: Extent of discontinuation, modification, or sustainability of intervention (I) (S)

Ultimate Impact of a Weight Management Program Dissemination Step Concept Percent Impacted 8. 8% of Weight Management sites participated Adoption 8. 80% 5. 9% of members participated Reach 0. 52% 91. 4% program components implemented Implementat ion 0. 47% 43. 8% of participants showed weight loss Effectivenes s 0. 21% 21. 2% individuals maintained benefit (individual) Maintenance 0. 04% Abildso CG, Zizzi SJ, Reger-Nash B. Prev Chronic Dis 2010 May; 7(3): A 46 Are they representativ ?

Moral of the Story? • Are multiple issues (steps) in successfully delivering a program • All steps are important—and can impact other steps or RE-AIM dimensions • Focus on the “denominator(s)”

RE-AIM—Health Equity Implications RE-AIM Issue Disparity Overall Impact 30% 70% of benefit 0 (equal) 70% of benefit Adoption 30% 49% of benefit Implementation 30% 34% of benefit Maintenance 30% 24% of benefit Reach Effectiveness www. re-aim. org

RE-AIM Evolution • Used to conduct reviews of the literature— many different content areas, consistent results § Setting level factors reported much less often § Maintenance (sustainability) reported least often • More recently used for diverse areas ranging from education to health policy to community/environmental interventions

RE-AIM Precision (Personalized) Medicine Questions Determine • What percent and types of patients are Reached; • For whom among them is the intervention Effective; in improving what outcomes; with what unanticipated consequences; • In what percent and types of settings and staff is this approach Adopted; • How consistently are different parts of it Implemented at what cost to different parties; Pawson R, how. Jwell are the Policy 2005; 10(S 1): S 21 -S 39. • And et al. Health Serv Res intervention components and their Gaglio B, Glasgow RE. Evaluation approaches…In: Brownson R, Colditz G, Proctor E, (Eds). effects Maintained? Dissemination and implementation research in health: Translating science to practice. New York: Oxford University Press; 2012. Pages 327 -356.

Key Translational Issues • Shift from focus on the numerator to the denominator • Generalizability to target population • Avoid contributing to disparities • Common comparison for decision making including unintended consequences • Robustness when combined with adoption: What works best for whom, and under what conditions* *Pawson R, et al. Realist review. . . J Health Serv Res Policy 2005; 10(S 1): S 21 -S 39.

…Applying RE-AIM to Planning Interventions

Planning and Evaluability • Do initial ESTIMATES of RE-AIM dimensions where do not have data (evaluability)* • Often helpful to compare two or more program or policy options • Expect different programs or interventions to do well on different RE–AIM dimensions • Include multiple perspectives on ongoing basis *Leviton L et al. Annual Review Public Health, 2010, 31, 213 - 233.

Key Strategies in Planning and Design Phase Reach Effectiveness Adoption Implementation Maintenance • Pilot recruitment and talk to early non-participants Design for Minimal Intervention Necessary for Change • Involve relevant stakeholders • Collect information on context • Create standard staff training and intervention delivery materials • Allow for ‘guided adaptation’ of intervention • Plan for ongoing feedback after intervention

Example Application

Be Fit Be Well: 24 -month randomized weight loss and hypertension self-management intervention trial among low-income urban clinics. • RE-AIM used to assess and reduce disparities, as well as report outcomes Bennett et al. Obesity Treatment for Socioeconomically Disadvantaged Patients in Primary Care Practice. Arch Intern Med. 2012; 172(7): 565 -574

IVR and Print

RE-AIM Element Key Inequity Issues Accessing disparate populations Understanding characteristics of those who participated vs. those who declined. Effectivenes Understanding the impact s of the context on results. Considering Minimal Intervention Needed for Change (MINC). Reach How BFBW Design Addressed Issues Intervention(Ix) largely mediated by phone and internet with no added visits and participant options; use of community health workers (CHW) Allowed tailoring, participant choice; use of multiple channels; followup contacts; culturally appropriate and designed for low literacy Adoption Enhancing participation Ix designed with staff for of low resource settings buy-in and used CHWs; Glasgow RE et al. Use of RE-AIM to Address Health Inequities…Transl Behav Med 2013 Jun and a variety of staff. incorporated medication 1; 3(2): 200 -210. adherence for MDs; placed

RE-AIM Element Key Inequity Issues Implementat Monitoring delivery to ion different subgroups and by different staff. Understanding and tracking costs of delivery. Transparently documenting adaptations to original program. Maintenance Individual Assessing long-term results across subgroups and identifying inequities and reasons Providing infrastructure and links to community Setting How BFBW Design Addressed Issues Provided staff training in motivational interviewing and offered certification; Minimized and tracked costs; kept web and IVR novel and fresh Ix for 2 years; addressed social environment barriers and facilitators; website remains for study participants to use. Planning for and supporting Assessed, but more. sustainability of program could have been done

Results • REACH: 60% of eligible population was invited to participate (604), of those 365 (60%) completed baseline and were randomized. Those that participated vs not were younger and had a higher mean BMI. • EFFECTIVENESS: * § § At 24 months, intervention (Ix) participants had greater weight losses compared with those receiving usual care (difference, − 1. 03 kg; 95% CI, − 2. 03 to − 0. 03 kg) Mean systolic blood pressure was significantly lower in the Ix arm compared with the usual care arm. *No differential patterns in outcomes observed for minority vs. nonminority or disparity Bennett et al. Obesity Treatment for Socioeconomically Disadvantaged related sub-groups. Patients…Arch Intern Med. 2012; 172(7): 565 -574

Results (cont. ) • ADOPTION: § All three centers invited participated, 4 centers were excluded for lack of EHR system. § For staff, 19 of 20 primary care physicians (95%) referred their patients to the program. • IMPLEMENTATION: § 70. 6% completion rate of counseling calls and 63. 3% of participants completing more than 70% of their calls. Calls were significantly more likely to be completed with participants making over $10, 000 a year (73. 1% vs. 62. 2%, P<0. 0001). § English speakers were more likely to have goals, barriers and strategies evaluated (P<0. 0001), as were participants making more than $10, 000 (P<0. 001). Glasgow RE et al. Use of RE-AIM to Address Health Inequities…Transl Behav Med 2013 • MAINTENANCE: Strong individual-level maintenance with no sub-group

Diabetes CD-ROM Reach and Effectiveness Study • Innovative Hybrid 2 X 2 design adult diabetes patients • Design § First step: Randomize to Choice or RCT Recruitment methods § Second step: Within these conditions randomize to in-person class or mail CD • Results § CD Had 4 times the reach § Equivalent behavioral and biologic outcomes § No interaction of Choice by Condition Glasgow RE, et al. Reach and effectiveness of DVD…Chron Illn 2009; 5: 2

CURRENT RE-AIM USES • Popularized by Glanz, Rimer book 1 & Larry Green(s) • Used by public health, CDC, HRSA, RWJF, VA & QUERI projects • Used in over 250 publications and many grant proposals • Development of “Calculators, ” “Quizzes, ” Self-test- available at www. re-aim. org • Moved website to KP Colorado (2006), NCI (2010), VT (2012) • Used in NIH, RWJF, Ao. A, CDC grant, Peers for Progress applications • e. Learning Trainings: 1 Glanz K, et al. Health behavior and health education: Theory, research and practice. (3 r http: //www. centertrt. org/index. cfm? fa=webtraining. reaim San Francisco: John Wiley & Sons 30

Integration with Other Models • Integrated with organizational factors in Pragmatic Research Implementation Sustainability Model (PRISM) • Combine with prescriptive models such as Chronic Care Model or recent variants? • Integration with complexity science or systems models? Feldstein, A. C. , Glasgow, R. E. (2008) A practical, robust implementation and sustainab model (PRISM)… Jt Comm J Qual Patient Saf, 34, 228 -243 31

RE-AIM for Decision Makers • For policy makers, admin types, those responsible for selecting programs • Use similar to how Health and Environmental Impact Assessments have been conducted • Use to compare two or more alternatives—produce RE-AIM “profile” 32

Challenges and Complexities Combining Dimensions—no overall summary score Confusing Reach and Adoption Is a framework, not a theory—tells you what to attend to, not HOW or WHAT to do When used to evaluate multi-component programs, how to treat components in terms of denominators and overlap

RE-AIM Summary Points • RE-AIM is an outcomes framework that can be used for planning and evaluation • Each dimension is an opportunity for intervention • RE-AIM can be used for efficacy, effectiveness, and implementation science projects • All dimensions can be addressed within a given study (though likely not all intervened upon) • Methods exist to combine and summarize REAIM outcomes

All Models (and Methods)—Including RE-AIM are WRONG …. Some are useful George Box “To every complex question, there is a simple answer… and it is wrong. ” H. L. Mencken

THE FUTURE OF REAIM? • Application to Comparative Effectiveness Research (CER- T) • Application to aging and disability? • Transparency focus • What it means to “Use RE-AIM” Possible Directions: • Merge with PRECIS model*? • *Thorpe KE, et al. A pragmatic–explanatory continuum indicator summary Your IDEAS WELCOMED! (PRECIS)…CMAJ; MAY 12, 2009; 180(10) Kessler RS, et al. What Does It Mean to ''Employ'' the RE-AIM Model? Eval Health Prof. , 2012 Mar; 36, 44 -46 37

Your Questions and Issues

A New “Bold Standard”? The 5 R’ s • Relevant • Rapid and Recursive • Rigorous (redefined to include robustness and generalizability) • Resource Need Informative • Replication is built in / emphasized Peek, Kessler, Glasgow, Klesges, Purcell, Stange. Submitted 2013— available by request

Key Issues in Research Phases to Report…. in Publication Phase Reach • • Method to Identify target population Inclusion/exclusion criteria Sample size and participation rate Characteristics of participants v. nonparticipants Efficacy/Effectiven ess • • • Measures/results at shortest time point Analytical methods ( e. g. ITTT v. Present at follow up) Quality of Life Measure Effects at longest time point Percent Attrition Adoption Implementation Maintenance • Description of: • Intervention location • Staff who delivered intervention/level of expertise • Method to identify target delivery agent • Setting inclusion/exclusion criteria • Setting and individual adoption rate • Characteristics of adoption/non-adoption • Measures of cost • Intervention type and intensity • Extent protocol delivered as intended • Measures of cost • Whether individual behavior was assessed at least 6 months post-intervention • Whether program is still in place www. re-aim. org

Research to Practice Pipeline Green, LW et al. Diffusion theory and knowledge dissemination…Annu Rev Public Health 2009; 30: 151 -74