Скачать презентацию There s nothing so practical as good theory Evaluating Скачать презентацию There s nothing so practical as good theory Evaluating

7158097ee13a48e67df6bf015a3ad5ae.ppt

  • Количество слайдов: 15

“There’s nothing so practical as good theory” Evaluating ‘Wicked’ Problems Bob Gardner July 2011 “There’s nothing so practical as good theory” Evaluating ‘Wicked’ Problems Bob Gardner July 2011

Clear Starting Points • English evaluation leader Ray Pawson quotes sociologist Kurt Lewin that Clear Starting Points • English evaluation leader Ray Pawson quotes sociologist Kurt Lewin that “there’s nothing so practical as good theory” • Pawson wasn’t arguing for abstract theory, but for ensuring we are always clear about • the assumptions we are making that underpin our work – whether planning a specific service initiative or developing a broad multi-sectoral strategy • the pathways and factors that we assume will lead from the planning through service delivery to the hoped-for impact • how we think all of this will vary depending upon the organizational, social or policy context • there are many approaches and as many terms – theory of change, program theory, framework for change • but the basic idea is to be really clear about starting points and premises when planning any initiative • and it’s this set of assumptions, pathways and objectives in our theory of change that we evaluate 16 March 2018 | www. wellesleyinstitute. com 2

The Problem to Solve = Systemic Disparities in Ontario • there is a clear The Problem to Solve = Systemic Disparities in Ontario • there is a clear gradient in health in which people with lower income, education or other indicators of social inequality and exclusion tend to have poorer health + major differences between women and men • the gap between the health of the best off and most disadvantaged can be huge – and damaging • in addition, there are systemic disparities in access to and quality of care within the healthcare system 16 March 2018 3

‘Wicked’ Policy Problems • • health inequities and their underlying social determinants of health ‘Wicked’ Policy Problems • • health inequities and their underlying social determinants of health are classic ‘wicked’ policy problems: • shaped by many inter-related and inter-dependent factors • in constantly changing social, economic, community and policy environments • action has to be taken at multiple levels -- by many levels of government, service providers, other stakeholders and communities • solutions are not always clear and policy agreement can be difficult to achieve • effects take years to show up – far beyond any electoral cycle have to be able to understand navigate this complexity to develop solutions • need comprehensive strategy to tackle the roots of health inequality in the wider social determinants of health • from high-level national social and policy change to reduce inequality through community-based innovation, cross-sectoral collaboration and mobilization 16 March 2018 4

…Need Theory • for complex social problems – from health inequities, through poverty reduction, …Need Theory • for complex social problems – from health inequities, through poverty reduction, to neighbourhood renewal and comprehensive community initiatives -- we need to be able to: • identify the connections and causal pathways between multiple factors • articulate the mechanisms that we assume drive change in these factors and the problem as a whole • within that, identify the crucial leverage points that will drive significant change in the problem • develop strategies for moving these leverage points • specify the short, intermediate and long-term outcomes expected and the preconditions for achieving them. that is our theory of change 16 March 2018 5

… And Here’s How how to develop theory of change: • basic principle is … And Here’s How how to develop theory of change: • basic principle is to pull group working on the issue together to answer those questions • to clearly articulate our starting points – what we think are the key causal pathways, mechanisms for change and crucial leverage points • these are our assumptions and goals for how we think program, community initiative or policy direction will work • based upon best available evidence, learning and experience and some wrinkles • can’t have all the answers and can’t be too prescriptive – don’t wait for ‘perfect’ theory • be as participatory as possible – especially if community-based evaluation then tests these assumptions about change mechanisms and outcomes → contributes to further learning and development of the initiative 16 March 2018 6

Theories of Change for ‘Wicked’ Problem I: Health Inequities complex multi-level strategies to tackle Theories of Change for ‘Wicked’ Problem I: Health Inequities complex multi-level strategies to tackle health gaps: • • • international frameworks such as WHO Commission on Social Determents European Union and other international initiatives many individual countries key features in all: • • focus on inclusive labour market, antidiscrimination, childcare, affordable housing, social security and other macro policies targets and deliverables for relevant departments equitable access to quality health care is just one part of this broader package emphasized partnerships with community service providers and organizations – in both policy development and service delivery 16 March 2018 Theory of change = all about reducing structured inequality : • • • health inequities arise out of wider social determinants of health won’t solve through health reform alone underlying structures of social and economic inequality need to be addressed by new policies need to be aligned with the incentives and processes that drive government key levers will vary – e. g. could be reducing prevalence and impact of precarious employment has to involve collaboration and coordination across governments and with many community and non-government stakeholders 7

Theories of Change for ‘Wicked’ Problem II: Community Level Action Comprehensive community initiatives: • Theories of Change for ‘Wicked’ Problem II: Community Level Action Comprehensive community initiatives: • • broad partnerships of local residents, community organizations, governments, business, labour and other stakeholders coming together to address deeprooted local problems – poverty, neighbourhood deterioration, health inequities, racism collaborative cross-sectoral efforts – not just for better coordinated service provision but for community development and capacity building Vibrant Communities – 14 communities across the country working to build individual and community capacities to reduce poverty 16 March 2018 Theory of change: • • • poverty, social exclusion, etc are not individual problems but are deeply rooted in fundamental social and economic inequality → even the best programs cannot get at these underlying structures of inequality need to be addressed by new policy directions building community capacities is another direction to address structural inequalities: • drilling down – Vibrant Communities works to build up individual skills and other assets as building block for community change broad collaborations are key to both enhancing community capacities and implementing policy changes locally broad coalitions also can highlight the deep-seated nature of problems → build community and public awareness and focus political advocacy 8

Evaluating Complex Policy and Community Initiatives • • far too complex to pick apart Evaluating Complex Policy and Community Initiatives • • far too complex to pick apart all the causal relations and patterns of influence: • very difficult to attribute particular changes to particular components of the overall initiative • will never meet RCT gold standard of proof – that approach can’t capture complexity • but that doesn't mean initiative is ineffective impact can take many years to show up • • • traditional evaluation of one program in isolation or of a particular population among many will not capture this complexity so we evaluate our theory of change • • and that doesn’t mean nothing is happening Tamarack talks of framework for change – to highlight the breadth of activities and stakeholders involved in comprehensive initiatives we identified levers in our strategy – did they prove to be important in practice? looking for indications that the change mechanisms unfold as we expected, that the direction of causal influence and impact is as we expected , etc looking for evidence that outcomes anticipated are being achieved, and the change mechanisms we anticipated have been important 16 March 2018 9

Build Use and Users Into Evaluation • crucial to identify users of evaluation • Build Use and Users Into Evaluation • crucial to identify users of evaluation • what they need • the context in which they will interpret and use the results of evaluation • policy makers need to: • • solve social problems as they see them determine where to invest scarce resources for maximum impact avoid mixing up accountability requirements with evaluation get beyond the assumption that randomly controlled trials are the ‘gold standard’ of evidence • community leaders and members are interested in: • the same – how to solve social problems as they see them – and live them • how to make case for reform/spending to policy makers • how to ensure programs and services meet the needs of community members – and how to continually improve service delivery • both need to be able to determine – and demonstrate – what types of initiatives work, for whom and in what contexts 16 March 2018 10

Case Study: Health Equity Strategy Into Action • goal is to ensure equitable access Case Study: Health Equity Strategy Into Action • goal is to ensure equitable access to high quality healthcare regardless of social position • can do this through a multi-pronged strategy: 1. • • 2. 3. 4. • 5. building health equity into all health care planning and delivery doesn’t mean all programs are all about equity but all take equity into account in planning their services and outreach aligning equity with system drivers and priorities embedding equity in provider organizations’ deliverables, incentives and performance management targeting some resources or programs specifically to addressing disadvantaged populations or key access barriers looking for investments and interventions that will have the highest impact on reducing health disparities or enhancing the opportunities for good health of the most vulnerable while thinking up-stream to health promotion and addressing the underlying determinants of health 16 March 2018 11

Equity-Focused Planning • a critical component of all these strategic directions is good planning Equity-Focused Planning • a critical component of all these strategic directions is good planning • addressing health inequities requires a solid understanding of: • key barriers to equitable access to high quality care • the specific needs of health-disadvantaged populations • gaps in available services for these populations • great deal of emphasis on tools such as Health Equity Impact Assessment • but can’t see particular tool as ‘magic bullet • to develop effective planning, we need: • • • clear strategy a coherent approach a repertoire of effective tools and techniques support for planners and practitioners to effectively use them good actionable information • and drilling down: what is our ‘theory’ of how equity-focused planning works? 16 March 2018 12

taking account of social constraints & conditions 16 March 2018 not just individual programs taking account of social constraints & conditions 16 March 2018 not just individual programs but coordination, partnerships & collaboration 13

enhanced access to health promotion for most disadvantaged 16 March 2018 up-stream heath conditions enhanced access to health promotion for most disadvantaged 16 March 2018 up-stream heath conditions & opportunities improve fastest for those in greatest need 14

Complexities At All Stages • processes and constraints are complex, and outcomes uncertain and Complexities At All Stages • processes and constraints are complex, and outcomes uncertain and unpredictable, at each of these junctures • and all of this varies by context: • particular communities or neighbourhoods – with their different health challenges and needs • particular population health and service landscape in specific areas • community resilience, connectedness, organizing and traditions • further specified by health condition or concern (e. g. the determinants and dynamics of mental health are different) • we need to know what works best at each of these junctures (and cumulatively), what programs and frameworks work for particular populations, and what works in varying contexts → need to build evaluation in from the start to learn 16 March 2018 15