Program Evaluation and Operational Research Is There a

Program Evaluation and Operational Research: Is There a Difference – Does It Matter? Marlene Glassman, Ph. D. , MSW Prevention Research Branch Division of HIV/AIDS Prevention Centers for Disease Control and Prevention mug 7@cdc. gov American Evaluation Association November 13, 2010

The findings and conclusions in this presentation are those of the author and do not necessarily represent the views of the CDC

Presentation Outline • What is program evaluation? • What is operational research (OR, traditional and recent perspectives)? • In what ways are evaluation and OR similar? • In what ways do evaluation and OR differ? • How can evaluation and OR work together?

What is Evaluation/Program Evaluation? • Evaluation is the systematic assessment of the worth or merit of some object (Scriven) • Can evaluate programs, consumer products, personnel • Evaluation is the systematic acquisition and assessment of information to provide useful feedback about some object

What is Evaluation/Program Evaluation? • Goal: influence decision making or policy formulation by providing empirically-driven feedback (http: //www. socialresearchmethods. net/ Trochim) • Effective program evaluation is a systematic way to improve and account for public health actions that involves procedures that are useful, feasible, ethical, and accurate (http: www. cdc. gov/eval/over. htm) • Monitoring and evaluation are information gathering and analysis activities that provide answers to important questions about HIV prevention program implementation and outcomes

Types of Evaluation/Program Evaluation • • • Evaluability assessment Process monitoring Process evaluation Outcome monitoring Outcome evaluation Needs assessment Cost-benefit analysis Formative evaluation Summative evaluation

Evaluation History • Ralph W. Tyler (1902 -1994) headed evaluation staff of the "Eight-Year Study" (1933 -1941) on high school curricula • 1958: US Navy employed program evaluation and review technique • 1963: Campbell and Stanley, Experimental and Quasi. Experimental Designs • 1967: Congress asked General Accounting Office to evaluate the Johnson Administration's Great Society anti-poverty efforts • 1967 book by E. A. Suchman, Evaluation Research • 1970 s: growth of books on evaluation and evaluation associations

Examples of Evaluations for HIV Prevention • Evaluation of performance arts based HIV prevention events in London with 13 -16 year olds • Evaluation of HIV/AIDS prevention in South African schools • Impact evaluation study of a community mobilization and HIV prevention programme among gay men • Scaling Up HIV voluntary counseling and testing in Africa • Evaluation of a Latino community health worker HIV prevention program

Traditional Operational Research (OR) • An interdisciplinary branch of applied mathematics that uses advanced analytical methods, such as mathematical and statistical modeling and simulation, to optimize decisionmaking for complex projects in such fields as transportation, delivery systems, factory design, and pricing • Uses analytical and numerical techniques to develop and manipulate mathematical models of organizational systems • Provides rational bases for decision making, predicts system behavior, improves system performance • Assists in management functions, business performance, cost reduction, forecasting, scheduling, planning, pricing

OR History • 1840: Charles Babbage (British) research into the cost of transportation and sorting of mail • 1937: A. P. Rowe (British) analysis to improve early warning radar system • Grew during World War II in Britain: used for military planning and applications • 1967: Stafford Beer (British) characterized the field of management science as "the business use of operations research” • 1974: OR project funded by US Agency for International Development on delivery of family planning and reproductive health services in developing countries

“New” (Morphed) OR Based on OR for HIV/AIDS in international settings • Provides decision-makers with information to improve program performance • Identifies solutions to problems that affect program quality, efficiency, effectiveness • Helps determine alternative delivery strategies for best outcomes • Operational and implementation research should investigate strategies to integrate new HIV prevention technologies and approaches in the broader HIV prevention and service continuum (Global HIV Prevention Working Group)

New (Morphed) OR • Enhances strategic performance of nonprofit organizations • “Interfaces” journal: practical application of OR in diverse areas, including government and education • Humanitarian operations research and public sector OR: efficient allocation of scare resources, planning and response to disease outbreaks, disaster-related operations (supply chain management), fair and sustainable water allocation, reduction of poverty through new market mechanisms, delivery of meals to seniors, adherence to TB medication

Examples of OR for HIV/AIDS Studies in international settings cover • strategies for the allocation of antiretroviral therapy in South Africa • programs to prevent mother-to-child transmission • factors that affect participation in HIV prevention programs and vaccine efficacy trials • the scaling up of treatment and prevention services • HIV stigma • male circumcision • modeling the spread of AIDS among homosexual population in United Kingdom

Thoughts about Changes in OR Rise of OR for domestic HIV prevention best understood in the context of issues related to “turning research into practice” and the rise of evidence-based interventions

Example of Terminology Confusion In 2008, NIH published a funding opportunity announcement, “Dissemination, Implementation, and Operational Research for HIV Prevention Interventions”, to enhance the science of technology transfer, dissemination, implementation, and operational research for evidence-based HIV-prevention interventions in the United States. The document noted, “Whether it is referred to as dissemination and implementation research, operations research, systems science, or technology transfer, the collective understanding of how to best ensure that these interventions are adopted and effectively implemented remains underdeveloped. ”

If names are not correct, language will not be in accordance with the truth of things. Confucius

Has OR Morphed into Implementation Research/Science? • “Implementation Science: ” an online, peer-reviewed journal that publishes research on the scientific study of methods to promote the use of research findings in healthcare clinical and policy contexts (http: //www. implementationscience. com/) • The National Implementation Research Network (NIRN): works to advance the science and practice of implementation of evidence-based programs in such fields as mental health, substance abuse prevention, and education through research and evaluation (http: //www. fpg. unc. edu/~nirn/)

How does CDC’s DHAP Define OR? Operational research is defined as 1) the study of implementation and factors that affect the implementation of HIV prevention programs and 2) the subsequent identification and assessment of recommendations to strengthen programs in order to reduce HIV/AIDS incidence, morbidity, and mortality. Factors include moderating variables that affect implementation and the behavior change outcomes HIV prevention programs seek to accomplish.

Domains for OR Factors/Moderating Variables that can be Studied Agency Strategic Plans and Initiatives Agency Structure, Functions, Operations Agency Policy Reflected in FOAs Grantees' Environmental Factors Grantee Capacity Proximal Moderators for EBI Implementation

Conceptual Relationship between Program Monitoring and Evaluation and DHAP Defined OR

Similarities and Differences OR and Monitoring and Evaluation (M&E)

ACTIVITY EXAMPLES of KEY QUESTIONS Process Monitoring What intervention activities took place and for whom? Process Evaluation To what extent did actual intervention activities match planned activities? Outcome Monitoring Did clients achieve the outcome objectives of the intervention? Outcome Evaluation Is the intervention efficacious when implemented under “real world” conditions?

ACTIVITY EXAMPLES of KEY QUESTIONS Implementation Research Do CBOs implement evidence-based interventions (EBIs) with fidelity to core elements? Operational Research • What factors affect the delivery of EBIs by CBOs? • What factors are associated with intervention success? • How can implementation be improved?

Program Evaluation Traditional OR “New” OR Evaluation vs. research Social science orientation Research vs. evaluation Applied mathematical orientation Focus Public sector focus (public funding) education, social service, prevention programs Focus Private sector focus: business and industry (roots in military planning) Research vs. evaluation Applied mathematical orientation with some social science orientation Focus Humanitarian and public sector focus (public funding), including HIV/AIDS programs disaster relief, non-profit organizations Whole program focus Effectiveness Implementation Outcomes Starts with solution (the program) and identifies problems Less focus on problem solving Focus on stakeholder involvement • Problem and solution focus for organizations vs. programs • Management focus • • Problem and solution focus for organizations & programs Examine alternative strategies, variability, probabilities, prediction, modeling DHAP Defined OR Implementation and factors that affect implementation (moderating variables)

Program Evaluation Inputs Activities Outputs Outcomes Traditional OR Inputs Activities Outputs “New” OR Inputs Activities Outputs Outcomes

Program Evaluation Purposes Accountability Program Improvement Program effectiveness Inform decision making (rational basis) Theories Democratic evaluation Traditional OR Purposes Efficiency Effectiveness Inform decision making (rational basis) Theories Optimal control theory (Mac. Donald) Perishable inventory theory Transformative evaluation Transaction cost theory (Mertens) Von Neumann-Morgenstern (concerns about ethics, social linear utility theory justice, diversity, power, human Theory on total quality rights) Game theory (process to solve social Optimal control theory problems; inform the just distribution of resources) Appreciative inquiry (Preskill) Utilization-focused evaluation (Patton) Empowerment evaluation (Fetterman) Feminist evaluation “New” OR Purposes Program improvement Efficiency Effectiveness Inform decision making (rational basis) Theories Optimal control theory Perishable inventory theory Transaction cost theory Von Neumann-Morgenstern linear utility theory Theory on total quality Game theory Optimal control theory process to solve social problems; inform the just distribution of resources)

Program Evaluation Traditional OR “New” OR Analytical methods Experimental design Quasi-experimental design Monitoring Considerations of internal and external validity Sampling Statistics Network analysis Modeling Analytical methods Simulation Industrial engineering Mathematical modeling Forecasting Supply chain management Simulation Optimization Statistics Network analysis Real-world settings

Conclusions • Traditional OR, “new” OR, DHAP defined OR and M&E share: – use of data for decision making (evidence-based decision making) – concerns about effectiveness • DHAP defined OR and traditional OR vary in focus and analytical methods DHAP defined OR: the study of implementation and factors t hat affect the implementation of HIV prevention programs

Conclusions • DHAP defined OR and M&E are complimentary processes • DHAP defined OR/implementation science/implementation research is similar to process monitoring and process evaluation – Differences are OR’s emphasis on starting with problems and moving to solutions and OR’s perspective of studying factors that affect implementation (factors can be distal or proximal) – Process monitoring and process evaluation can be considered activities that address a “solution” in place; i. e. , a program that is being implemented to “solve” a problem

Conclusions • • • DHAP defined OR can identify need for M&E can identify need for DHAP defined OR Complete cycle of DHAP defined OR includes M&E can be applied to DHAP defined OR (evaluate OR) Evaluators and researchers share commitment to “make things better”