Evaluating System Change Ventures Innovations Toward a More

Evaluating System Change Ventures Innovations Toward a More Dynamic and Democratic Approach Bobby Milstein Measurement Knowledge Network WHO Commission on Social Determinants of Health March 22, 2005

Plan for Today CDC Framework for Program Evaluation Difference between research and evaluation Link to institutional change Steps and standards Innovations for System Change Ventures Syndemic orientation Place of simulation modeling

Appreciating the Unique Character of Evaluative Inquiry “It is easier to find facts than it is to face them. ” -- Anonymous Research Systematic Methods Questions of Fact (descriptions, associations, effects) Evaluation Questions of Values (merit, worth, significance) Centers for Disease Control and Prevention. What procedures are available for planning and evaluating initiatives to prevent syndemics? Syndemics Prevention Network, 2001. Available at .

Institutional Climate for Evaluative Inquiry “The CDC’s senior leaders understood that strengthening evaluation capacity in public health would require a process of culture change, including significant reforms to their own organization. ” Milstein B, Chapel T, Wetterhall S, Cotton D. Building capacity for program evaluation at the Centers for Disease Control and Prevention. New Directions for Evaluation 2002; 2002(93): 27 -47.

The Evaluation Continuum Informal Ways of Evaluating Formal Low Stakes Involved High

Framework for Program Evaluation “Both a synthesis of existing evaluation practices and a standard for further improvement. ” Milstein B, Wetterall S, CDC Evaluation Working Group. Framework for program evaluation in public health. MMWR Recommendations and Reports 1999; 48(RR-11): 1 -40. Available at .

Framework for Program Evaluation “A practical, nonprescriptive tool, designed to summarize and organize the essential elements of program evaluation. ” Milstein B, Wetterall S, CDC Evaluation Working Group. Framework for program evaluation in public health. MMWR Recommendations and Reports 1999; 48(RR-11): 1 -40. Available at .

Standards for Effective Evaluation Utility (7) Serve information needs of intended users Feasibility (3) Be realistic, prudent, diplomatic, and frugal Propriety (8) Behave legally, ethically, and with due regard for the welfare of those involved and those affected Accuracy (12) Reveal and convey technically accurate information Joint Committee on Educational Evaluation, James R. Sanders (Chair). The program evaluation standards: how to assess evaluations of educational programs. 2 nd edition ed Thousand Oaks, CA: Sage Publications, 1994.

Catalyst for Complementary Resources CDC Evaluation Working Group. Framework for program evaluation: adapted versions. Available at .

Are We Posing Questions About Attribution or Contribution? “…if a program’s activities are aligned with those of other programs operating in the same setting, certain effects (e. g. , the creation of new laws or policies) cannot be attributed solely to one program or another. In such situations, the goal for evaluation is to gather credible evidence that describes each program’s contribution in the combined change effort. Establishing accountability for program results is predicated on an ability to conduct evaluations that assess both of these kinds of effects. ” p. 11 -12 Calls into question the conditions in which one focuses on a “program” as the unit of analysis

Acknowledging Plurality “You think you understand two because you understand one. But you must also understand ‘and’. ” -- Sufi Saying Efforts to Reduce Population Health Problems Problem, problem solver, response Efforts to Organize a System that Assures Healthful Conditions for All Dynamic interaction among multiple problems, problem solvers, and responses Bammer G. Integration and implementation sciences: building a new specialisation. Cambridge, MA: The Hauser Center for Nonprofit Organizations, Harvard University 2003.

The Syndemics Prevention Network “You think you understand two because you understand one. But you must also understand ‘and’. ” -- Sufi Saying Health Power to Act Living Conditions A syndemic orientation clarifies the dynamic and democratic character of public health work The word syndemic signals special concern for many kinds of relationships: mutually reinforcing health problems health status and living conditions synergy/fragmentation in the health response system Learning within innovative ventures Comprehensive Community Initiatives Philanthropy Legacy Initiatives State Tobacco Settlements Efforts to Eliminate Health Disparities Government and Philanthropy Responses to Unjust Conditions Broad-based Citizen Organizations Milstein B. Spotlight on syndemics. Centers for Disease Control and Prevention, 2001.

Questioning the Character of Public Health Work PUBLIC HEALTH WORK SOCIAL NAVIGATION (governing movement) Directing Change • Who does the work? • By what means? • According to whose values? Charting Progress • How are conditions changing? • In which directions? SYSTEMS THINKING & MODELING (understanding change) • What causes population health problems? Innovative Health Ventures • How are efforts to protect the public’s health organized? • How and when do health systems change (or resist change)? PUBLIC HEALTH (setting direction) What are health leaders trying to accomplish?

Formalizing an Orientation Joining Concepts and Methods Syndemic Orientation What influences what? Where are we going? Connections Leverage Directed Public Work Network View Systems View Navigational View What links to what? Proximity Data Causal Data Directional Data

Changing (and Accumulating) Ideas in Causal Theory What accounts for population health? God’s will Humors, miasma, atmosphere (“epidemic constitution”) Poor living conditions, immorality (sanitation) 1840 Single disease, single cause (germ theory) 1880 Single disease, multiple causes (heart disease) 1950 Single cause, multiple diseases (tobacco) 1960 Multiple causes, multiple diseases (but no feedback dynamics) (social epidemiology) 1980 Dynamic feedback among afflictions, living conditions, and public strength (syndemic) 2000

The Feedback Thought “When X and Y affect each other, one cannot study the link between X and Y and, independently, the link between Y and X and predict how the system will behave. Only the study of the whole system as a feedback system will lead to correct results. " -- System Dynamics Society. What is system dynamics? System Dynamics Society, 2002. Available at . Richardson GP. Feedback thought in social science and systems theory. Philadelphia: University of Pennsylvania Press, 1991.

A Very Particular Distance

Looking Through the Macroscope “The macroscope filters details and amplifies that which links things together. It is not used to make things larger or smaller but to observe what is at once too great, too slow, and too complex for our eyes. ” -- Joèel de Rosnay J. The macroscope: a book on the systems approach. Principia Cybernetica, 1997.

Toward a Balanced System of Health Protection Public Work Society's Health Response General Protection Targeted Protection Primary Prevention Secondary Prevention Demand for response Becoming safer and healthier Safer Healthier People Becoming vulnerable Tertiary Prevention Vulnerable People Becoming afflicted Afflicted without Complications Developing complications Adverse Living Conditions From: Milstein B, Homer J. The dynamics of upstream and downstream: why is so hard for the health system to work upstream, and what can be done about it? CDC Futures Health Systems Workgroup; Atlanta, GA; 2003. Afflicted with Complications Dying from complications

Balancing Two Major Areas of Emphasis Public Work Healthy Public Policy & Public Work Medical and Public Health Policy Society's Health Response General Protection Targeted Protection Primary Prevention Demand for response Becoming safer and healthier Safer Healthier People Becoming vulnerable Tertiary Prevention Secondary Prevention Vulnerable People Afflicted without Becoming Complications Developing Complications complications afflicted Dying from complications Adverse Living Conditions DEMOCRATIC SELF-GOVERNANCE World of Transforming… By Strengthening… • Deprivation • Dependency • Violence • Disconnection • Environmental decay • Stress • Insecurity • Etc… • Leaders and institutions • Foresight and precaution • The meaning of work • Mutual accountability • Plurality • Democracy • Freedom • Etc… MANAGEMENT OF RISKS & DISEASES World of Providing… • Health education • Screening tests • Disease management • Pharmaceuticals • Clinical services • Physical and financial access • Etc…

Understanding Health as Public Work - Citizen Involvement in Public Life Society's Health Response General Protection Targeted Protection Primary Prevention - Tertiary Prevention Secondary Prevention Demand for response Becoming safer and healthier Safer Healthier People Becoming vulnerable Vulnerable People Adverse Living Conditions Fraction of Adversity, Vulnerability and Affliction Borne by Disadvantaged Sub-Groups (Inequity) Becoming afflicted Afflicted without Complications Developing complications Afflicted with Complications Dying from complications Vulnerable and Afflicted People Social Division Public Strength

How Can Test a Dynamic Hypothesis? Public Work - Citizen Involvement in Public Life Society's Health Response General Protection Targeted Protection Primary Prevention Public Strength - Secondary Prevention Tertiary Prevention Demand for response Becoming safer and healthier Safer Healthier People Becoming vulnerable Vulnerable People Adverse Living Conditions Fraction of Adversity, Vulnerability and Affliction Borne by Disadvantaged Sub-Groups (Inequity) Becoming afflicted Afflicted without Complications Developing complications Afflicted with Complications Dying from complications Vulnerable and Afflicted People Social Division -- How can we learn about the consequences of actions in a system of this kind? -- Could the behavior of this system be analyzed using conventional epidemoiological methods (e. g. , logistic or multi-level regression)?

Tools for Policy Analysis Events Time Series Models Increasing: Describe trends • Depth of causal theory • Degrees of uncertainty Patterns • Robustness for longerterm projection • Value for developing policy insights Structure Multivariate Stat Models Identify historical trend drivers and correlates Dynamic Models Anticipate future trends, and find policies that maximize chances of a desirable path

Redirecting the Course of Change Questions from System Dynamics and Social Navigation re? he W 14% increase Why? Ho Wh o 2005 w? ? 2025 2050 Zack MM, Moriarty DG, Stroup DF, Ford ES, Mokdad AH. Worsening trends in adult health-related quality of life and selfrated health–United States, 1993 -2001. Public Health Reports 2004; 119(September-October): 493 -505.

Simulations for Learning in Dynamic Systems The Problem of Outside Assistance Dynamic Hypothesis (Structure) Outside assistance to alleviate and prevent affliction Affliction cross-impacts R 1 Affliction prevalence & burden B 1 a Effort to alleviate and B 3 a prevent affliction R 4 a Social disparity B 2 Public strength R 2 b Adverse living conditions Key Rectangle: Stock/state variable Blue arrow: same-direction link Green arrow: opposite-direction link Circled “B”: balancing causal loop Circled “R”: reinforcing causal loop Behavior Over Time (Experiments) Effort to build public strength R 2 a At-risk fraction R 2 c Outside assistance to build public strength B 1 b Public work fraction R 3 a R 3 b Magnitude of ameliorative efforts United efforts Divided efforts Effort to improve living conditions R 4 b B 3 b Outside assistance to improve living conditions Homer J, Milstein B. Optimal decision making in a dynamic model of poor community health. Proceedings of the 37 th Hawaii International Conference on System Science; Big Island, Hawaii; January 5 -8, 2004. Available at .

Simulations for Learning in Dynamic Systems Diabetes Dynamics in an Era of Epidemic Obesity Dynamic Hypothesis (Structure) Testing for Pre. Diabetes Pre. Diabete Onset s People with Normal Glycemic Levels Recovering from Pre. Diabete s Testing for Diabetes Pre. Diabetes Detection Developing Diabetes from Undx Developing Pre. D, from People with Complications. People with Dying from Undx diab People with Undiagnosed, Complications Undiagnosed Uncomplicated Complicated Pre. Diabetes Diagnosing Pre. Diabetes People with Diagnosed Pre. Diabetes Risk for Pre. Diabetes & Diabetes Obese Fraction of the Population Access to Preventive Health Services Diabetes Onset Diagnosing Uncomplicated Diabetes Diagnosing Complicated Diabetes Developing People with Complications Diagnosed, Uncomplicated Complicated Diabetes Clinical Management of Diagnosed Diabetes Clinical Management of Pre. Diabetes Adoption of Healthy Lifestyle Caloric Intake Physical Activity Ability to Self Monitor Medication Affordability Deaths per Population 0. 0035 Dying from Complications Diabetes Control Pre. Diabetes Control Behavior Over Time (Experiments) 0. 003 B 0. 0025 Upstream 0. 002 Personal Capacity am tre s wn Do 0. 0015 Living Conditions Mixed e as 1980 1990 2000 Striking an acceptable balance. 2010 2020 Time (Year) 2030 2040 2050 Blue: Base run; Red: Clinical mgmt up from 66% to 90%; Green: Caloric intake down 4% (99 Kcal/day); Black: Clin mgmt up to 80% & Intake down 2. 5% (62 Kcal/day) Jones A, Homer J, Milstein B, Essien J, Murphy D, Sorensen S, Englegau M. Modeling the population dynamics of a chronic disease: the CDC's diabetes system model. American Journal of Public Health (in press).

Iterative Steps in System Dynamics Simulation Modeling Identify a Persistent Problem Graph its behavior over time Create a Dynamic Hypothesis Identify and map the main causal forces that create the problem Convert the Map Into a Simulation Model Formally quantify the hypothesis using all available evidence Run Simulation Experiments Compare model’s behavior to expectations and/or data to build confidence in the model Learn About Policy Consequences Test proposed policies, searching for ones that best govern change Enact Policy Build power and organize actors to establish chosen policies Milstein B, Homer J. Background on system dynamics simulation modeling, with a summary of major public health studies. Atlanta, GA: Syndemics Prevention Network, Centers for Disease Control and Prevention; February 1, 2005.

Learning In and About Dynamic Systems “The complexity of our models vastly exceeds our ability to understand their implications without simulation. " -- John Sterman Complexity Hinders Benefits of Simulation/Game-based Learning Generation of evidence (by eroding the conditions for experimentation) Formal means of evaluating options Learning from evidence (by demanding new heuristics for interpretation) Compressed time Acting upon evidence (by including the behaviors of other powerful actors) Actions can be stopped or reversed Experimental control of conditions Complete, undistorted results Visceral engagement and learning Tests for extreme conditions Early warning of unintended effects Opportunity to assemble stronger support Sterman JD. Learning from evidence in a complex world. American Journal of Public Health (in press). Sterman JD. Business Dynamics: Systems Thinking and Modeling for a Complex World. Boston, MA: Irwin Mc. Graw-Hill, 2000.

A Third Branch of Science “Simulation is a third way of doing science. Like deduction, it starts with a set of explicit assumptions. But unlike deduction, it does not prove theorems. Instead, a simulation generates data that can be analyzed inductively. Unlike typical induction, however, the simulated data comes from a rigorously specified set of rules rather than direct measurement of the real world. While induction can be used to find patterns in data, and deduction can be used to find consequences of assumptions, simulation modeling can be used as an aid to intuition. ” -- Robert Axelrod R. Advancing the art of simulation in the social sciences. In: Conte R, Hegselmann R, Terna P, editors. Simulating Social Phenomena. New York, NY: Springer; 1997. p. 21 -40. .

Revisiting the Framework “Steps in the framework are starting points for tailoring an evaluation to a particular public health effort at a particular time. ” Simulation Modeling Offers Support for multi-stakeholder dialogue A larger conception of the “program” context Another avenue for experimentation and visceral learning Ability to track interrelated indicators (both states and rates) An emphasis on pragmatism (learning through action)

For Additional Information http: //www. cdc. gov/syndemics

Enhancing Learning Through Simulation Sterman J. Business dynamics: systems thinking and modeling for a complex world. Boston, MA: Irwin Mc. Graw-Hill, 2000.

Policy Resistance “The tendency for interventions to be delayed, diluted, or defeated by the response of the system to the intervention itself. ” -- Meadows, Richardson, Bruckman “At least six times since the Depression, the United States has tried and failed to enact a national health insurance program. ” -- Lee & Paxman Lee P, Paxman D. Reinventing public health. Annual Reviews of Public Health 1997; 18: 1 -35. Meadows DH, Richardson J, Bruckmann G. Groping in the dark: the first decade of global modelling. New York, NY: Wiley, 1982.

Flaws in Previous Attempts at Health System Reform in America (and Elsewhere) Piecemeal approaches that do not address the full scope of the problem Comprehensive strategies that are opposed by special interests Assumption that healthcare dynamics are separate from other areas of public concern Conventional analytic methods make it difficult to Observe the health system as a large, dynamic enterprise Craft high-leverage strategies that can overcome policy resistance Been thinking of health and healthcare primarily as nouns (i. e. , commodities distributed to consumers), not as verbs (i. e. , public work to be done by citizens) Heirich M. Rethinking health care: innovation and change in America. Boulder CO: Westview Press, 1999. Kari NN, Boyte HC, Jennings B. Health as a civic question. American Civic Forum, 1994. Available at .

Basic Problem Solving Orientations Event Oriented View Goals Problem Action Results Situation Feedback View Action Delay Goals Delay “Side Effects” Delay Environment Delay Goals of Others Delay “Side Effects” Delay Action of Others Delay Sterman J. Business dynamics: systems thinking and modeling for a complex world. Boston, MA: Irwin Mc. Graw-Hill, 2000.

Building on Decades of SD Health Studies Disease epidemiology heart disease, diabetes, HIV/AIDS, cervical cancer, chlamydia, dengue fever, drug-resistant infections Substance abuse epidemiology heroin, cocaine, tobacco Health care patient flows emergency care, extended care Health care capacity and delivery HMO planning, dental care capacity, mental health care, disaster preparedness Interactions between health capacity and disease epidemiology chronic illness management, syndemics Homer JB, Hirsch GB. System dynamics modeling for public health: background and opportunities. American Journal of Public Health (in press).

Progress in Dynamic Modeling Problem Focus Dynamic Focus Grantmaking strategy in areas challenged by multiple afflictions • Designed to explore interactions between afflictions, living conditions, and public strength Balancing upstream and downstream effort • Designed to understand an observed phenomenon, the 97% -- 3% split in health care expenditures Diabetes in an era of increasing obesity • Most empirically supported Fetal and infant health Health care reform policies Life-course dynamics of obesity

Transforming the Future of Diabetes… "Every new insight into Type 2 diabetes. . . makes clear that it can be avoided--and that the earlier you intervene the better. The real question is whether we as a society are up to the challenge. . . Comprehensive prevention programs aren't cheap, but the cost of doing nothing is far greater. . . " …in an Era of Epidemic Obesity Gorman C. Why so many of us are getting diabetes: never have doctors known so much about how to prevent or control this disease, yet the epidemic keeps on raging. how you can protect yourself. Time 2003 December 8. Accessed at http: //www. time. com/time/covers/1101031208/story. html.

Re-Directing the Course of Change Questions from System Modeling and Social Navigation ? ere h W Ho w? ? W hy Wh o ? 2010 2020

Diabetes System Modeling Project Where is the Leverage for Health Protection? Diabetes Detection Pre. Diabetes Onset People with Normal Glycemic Levels Recovering from Pre. Diabetes Obesity Prevention People with Undiagnosed, Uncomplicated Diabetes People with Undiagnosed Pre. Diabetes Diagnosing Pre. Diabetes People with Diagnosed Pre. Diabetes People with Undiagnosed, Complicated Diabetes Onset People with Diagnosed, Uncomplicated Diabetes Pre. Diabetes Control Developing Complications Diagnosing Diabetes People with Diagnosed, Complicated Diabetes Dying from Complications Diabetes Control Jones A, Homer J, Milstein B, Essien J, Murphy D, Sorensen S, Englegau M. Modeling the population dynamics of a chronic disease: the CDC's diabetes system model. American Journal of Public Health (in press).

Diabetes System Modeling Project Where is the Leverage for Health Protection? Testing for Pre. Diabetes People with Normal Glycemic Levels Access to Preventive Health Services Testing for Diabetes Pre. Diabetes Detection Developing Diabetes from Undx Developing Pre. Diabetes Pre. D, Complications from People with Onset People with Dying from Undx diab People with Undiagnosed, Complications Undiagnosed Uncomplicated Complicated Pre. Diabetes Recovering from Pre. Diabetes Diagnosing Pre. Diabetes Recovering from Pre. Diabetes People with Diagnosed Pre. Diabetes Risk for Pre. Diabetes & Diabetes Obese Fraction of the Population Diabetes Onset Diagnosing Uncomplicated Diabetes Diagnosing Complicated Diabetes Developing People with Complications People with Diagnosed, Uncomplicated Complicated Diabetes Control Pre. Diabetes Control Clinical Management of Pre. Diabetes Adoption of Healthy Lifestyle Caloric Intake Physical Activity Living Conditions Personal Capacity Dying from Complications Clinical Management of Diagnosed Diabetes Ability to Self Monitor Medication Affordability

Deaths per Population 0. 0035 0. 003 Mixed se Ba 0. 0025 Upstream 0. 002 am e str n w Do Striking an acceptable balance. 0. 0015 1980 1990 2000 2010 Time (Year) Blue: Base run; Red: Clinical mgmt up from 66% to 90%; Green: Caloric intake down 4% (99 Kcal/day); Black: Clin mgmt up to 80% & Intake down 2. 5% (62 Kcal/day) 2020 2030 2040 2050

Using Available Data to Calibrate the Model Information Sources U. S. Census National Health Interview Survey National Health and Nutrition Examination Survey Behavioral Risk Factor Surveillance System Professional Literature Data • Adult population and death rates • Health insurance coverage • Diabetes prevalence • Diabetes detection • Prediabetes prevalence • Weight, height, and body fat • Caloric intake • Glucose self-monitoring • Eye and foot exams • Participation in health education • Use of medications • Physical activity trends • Effects of control and aging on onset, progression, death, and costs • Expenditures

Diabetes System Modeling Project Confirming the Model’s Fit to History Obese % of Adults Diagnosed Diabetes % of Adults Jones A, Homer J, Milstein B, Essien J, Murphy D, Sorensen S, Englegau M. Modeling the population dynamics of a chronic disease: the CDC's diabetes system model. American Journal of Public Health (in press).

Explaining the Past What Has Driven the Burden of Diabetes? Huge Growth in Number of People with Diabetes Great Progress in Reducing the Burden for the Average Person with Diabetes Overall, Total Diabetes Burden Held at Bay

Explaining the Past Deaths Due to Diabetes Have Fallen People with Diabetes per Thousand Adults Complications Deaths per Thous People w Diabetes 100 90 80 Mode 40 t Outpu del l Outp 30 Mo ut 20 70 More people with diabetes 60 50 1985 1990 1995 Time (Year) 2000 10 0 1980 2005 2. 5 2 Among people with diabetes, fewer dying every year Deaths from Comps of Diabetes Per Thous Adults Model Output 1. 5 1 0. 5 0 1980 Combine to mean fewer U. S. adults dying 1980 -2004 1985 1990 1995 Time (Year) 2000 2005

Anticipating the Future Prevalence Under ‘Status Quo’ Assumptions Obese Fraction of Adult Population 0. 6 People with Diabetes per Thousand Adults 130 utput del O Mo M 0. 45 110 0. 3 0. 15 t Outpu odel 90 Even if we assume the obesity epidemic has peaked… Prevalence continues to increase. 70 0 1980 1990 2000 2010 2020 2030 Time (Year) 2040 2050 After a delay 50 1980 1990 2000 2010 2020 2030 Time (Year) 2040 2050

Anticipating the Future Deaths Under ‘Status Quo’ Assumptions Complications Deaths per Thous w Diabetes People with Diabetes per Thousand Adults 40 130 30 110 20 90 If prevalence continues to increase, 70 50 1980 And if we can maintain current levels of care but no continued improvement… 10 0 1980 1990 2000 2010 2020 Time (Year) 2030 2040 1990 2000 2050 2010 2020 2030 Time (Year) 2040 Deaths from Complications of Diabetes Per Thousand Adults 2. 5 Then prevalence overwhelms the improved care to boost the burden 1. 25 1980 1990 2000 2010 2020 Time (Year) 2030 2040 2050

Policy Experiments Continued downstream improvements Upstream efforts Downstream & Upstream

Downstream-Only Intervention Deaths per Population 0. 0035 0. 003 s Ba 0. 0025 e m wn o a tre s D 0. 002 Disease control acts quickly but does not slow the growth in deaths. 0. 0015 1980 1990 2000 2010 2020 Time (Year) 2030 2040 2050 Blue: Base run; Red: Clinical mgmt of diagnosed up from 66% to 90%

Upstream-Only Intervention Deaths per Population 0. 0035 0. 003 s Ba 0. 0025 e m 0. 002 0. 0015 1980 n w Do a tre s Upstream Obesity prevention slows the growth but takes a long time to do so. 1990 2000 2010 2020 Time (Year) 2030 2040 2050 Blue: Base run; Red: Clinical mgmt up from 66% to 90%; Green: Caloric intake down 4% (99 Kcal/day)

Mixed Intervention Deaths per Population 0. 0035 0. 003 0. 0025 e as B 0. 002 rea t s wn o D 0. 0015 1980 1990 2000 m Mixed Upstream Striking an acceptable balance. 2010 2020 2030 2040 2050 Time (Year) Blue: Base run; Red: Clinical mgmt up from 66% to 90%; Green: Caloric intake down 4% (99 Kcal/day); Black: Clin mgmt up to 80% & Intake down 2. 5% (62 Kcal/day)

Questions Where might the results of simulation experiments fit in the hierarchies of evidence used for systematic reviews?

Understanding Health as Public Work

How Many Triangles Do You See? Wickelgren I. How the brain 'sees' borders. Science 1992; 256(5063): 1520 -1521.

“Solutions” Can Also Create New Problems Merton RK. The unanticipated consequences of purposive social action. American Sociological Review 1936; 1936: 894 -904. Forrester JW. Counterintuitive behavior of social systems. Technology Review 1971; 73(3): 53 -68.

New Word for a Familiar Phenomenon “We have introduced the term ‘syndemic’ to refer to the set of synergistic or intertwined and mutually enhancing health and social problems facing the urban poor. Violence, substance abuse, and AIDS, in this sense, are not concurrent in that they are not completely separable phenomena. ” -- Merrill Singer M, Snipes C. Generations of suffering: experiences of a treatment program for substance abuse during pregnancy. Journal of Health Care for the Poor and Underserved 1992; 3(1): 222 -34. Singer M. 1994. AIDS and the health crisis of the US urban poor: The perspective of critical medical anthropology. Social Science and Medicine 39(7): 931 -948. Singer M. 1996. A dose of drugs, a touch of violence, a case of AIDS: Conceptualizing the SAVA syndemic. Free Inquiry in Creative Sociology 24(2): 99 -110. Singer M, Clair S. Syndemics and public health: reconceptualizing disease in bio-social context. Medical Anthropology Quarterly 2003; 17(4): 423 -441.

Strengths and Further Frontiers Positioned Evaluation As… Left Unexamined… Essential, not a question of whether, but rather how Focus on a singular “program” as the unit of inquiry (N=1 depth) Continuous, not left to the end Limitations of logic (counterintuitive effects) Inclusive, powerful roles for all stakeholders Situational, decisions change based on participants, context, questions, uses, resources, etc. Practical, judged against explicit standards Humane, primarily about values and learning Dynamic aspects of public health work (e. g. , better-before-worse patterns of change) Democratic aspects of public health work (e. g. , actions by citizens who don’t self-identify as health professionals) Evaluative aspects of planning

Beyond Scapegoating “When we attribute behavior to people rather than system structure the focus of management becomes scapegoating and blame rather than the design of organizations in which ordinary people can achieve extraordinary results. ” “The tendency to blame other people instead of the system is so strong that psychologists call it the fundamental attribution error. ” -- John Sterman J. System dynamics modeling: tools for learning in a complex world. California Management Review 2001; 43(4): 8 -25.

Standards for Effective Evaluation Utility (7) Serve information needs of intended users Feasibility (3) Be realistic, prudent, diplomatic, and frugal Propriety (8) Behave legally, ethically, and with due regard for the welfare of those involved and those affected Accuracy (12) Reveal and convey technically accurate information Joint Committee on Educational Evaluation, James R. Sanders (Chair). The program evaluation standards: how to assess evaluations of educational programs. 2 nd edition ed Thousand Oaks, CA: Sage Publications, 1994.

Institutional Climate for Evaluative Inquiry Links to Cross-Cutting Trends Expanding goals (length, quality, equity) Wider ecological/systems view (assure the conditions for health) Deepening democratic processes/outlook Public accountability Multi-sector action Information systems integration Public health research Knowledge management Systematic reviews Performance standards Ethical and moral considerations Excellence in science and practice Milstein B, Chapel T, Wetterhall S, Cotton D. Building capacity for program evaluation at the Centers for Disease Control and Prevention. New Directions for Evaluation 2002; 2002(93): 27 -47.

Definition: Living Conditions “Living conditions are the everyday environment of people, where they live, play and work. These living conditions are a product of social and economic circumstances and the physical environment–all of which can impact upon health–and are largely outside of the immediate control of the individual. ” -- World Health Organization. Health promotion glossary. World Health Organization, 1998. Accessed July 15 at http: //www. who. int/hpr/NPH/docs/hp_glossary_en. pdf.

Definition: Public Work Public work is sustained, visible, serious effort by a diverse mix of ordinary people that creates things of lasting civic or public significance. -- Center for Democracy and Citizenship Mitchell Siporin. Jane Addams memorial. Illinois Federal Art Project, WPA, 1936. Fine Arts Collection, General Services Administration. Center for Democracy and Citizenship. The concept and philosophy of public work. Center for Democracy and Citizenship, 2001. Available at .

Definition: Public Strength 1. The power of citizens to direct the course of change toward a negotiated set of valued conditions/goals. 2. Vitality of a society’s public sphere, the health of its polis. Lyon D. Tottle House. . . occupied during a sit-in by some of America's most effective organizers. Washington, DC: Library of Congress; 2002.

What are the Prospects for Using a Syndemic Orientation? A way of thinking about public health work that focuses on connections among health-related problems, considers those connections when developing health policies, and aligns with other avenues of social change to assure the conditions in which people can be healthy Complements single-issue prevention strategies, which can be effective for discrete problems but often are mismatched to the goal of assuring conditions for health in its widest sense Incorporates 21 st century systems science and political sensibilities, but the underlying concepts are not new. Still, the implications of adhering to this orientation remain largely unexplored. Milstein B. Syndemic. In: Mathison S, editor. Encyclopedia of Evaluation. Thousand Oaks, CA: Sage Publications; 2004.

Seeing Conditions as Freedoms Adverse living conditions are circumstances that inhibit people's freedom to be safe and healthy and develop their full potential They include, at a minimum, any deviation from prerequisite conditions for life and human dignity (e. g. , physical extremes, violence, deprivation, disconnection) Phenomena like hunger, homelessness, joblessness, illiteracy, war, environmental decay, and various forms of injustice, including racism, are all examples of adverse living conditions

Choice and Non-Choice “Choices are always made from among alternatives presented by the social environment, or by circumstances that were themselves not chosen…When we recognize the elements of non -choice in choice, we can escape the contradiction between social causation and individual responsibility and understand the interactiveness of the two. ” -- Richard Levins & Cynthia Lopez Levins R, Lopez C. Toward an ecosocial view of health. International Journal of Health Services 1999; 29(2): 261 -93.

Learning In and About Dynamic Systems “The complexity of our models vastly exceeds our ability to understand their implications without simulation. " -- John Sterman Barriers to Learning Benefits of Simulation Dynamic complexity Formal means of evaluating options Time delays Experimental control of conditions Inadequate and ambiguous feedback Compressed time Poor reasoning skills Complete, undistorted results Defensive reactions Actions can be stopped or reversed Inability and costs of experimentation Visceral engagement and learning Tests for extreme conditions Early warning of unintended effects Opportunity to assemble stronger support Sterman JD. Learning from evidence in a complex world. American Journal of Public Health (in press). Sterman JD. Business Dynamics: Systems Thinking and Modeling for a Complex World. Boston, MA: Irwin Mc. Graw-Hill, 2000.

Insight Generating Capacity Descriptive Realism Curve-Fitting Models Point Predictive Ability Dynamic Models Formal Correspondence with Data Mode Reproduction Ability Transparency Fertility Ease of Enrichment Relevance Source: Randers JR. Elements of the System Dynamics Method. Cambridge Mass: Productivity Press, 1980. p. xx.

Different Modeling Approaches For Different Purposes Logic Models (flowcharts, maps or diagrams) • Articulate steps between program actions and results System Dynamics (causal loop diagrams and simulation models) Forecasting Models • Improve understanding about the possible effects of a policy over time • Make accurate forecasts of key variables • Focus on patterns of change over time (e. g. , long delays, worse before better) • Focus on precision of point predictions and confidence intervals