Challenging Evaluation An Introduction to Outcome Mapping Amy

Challenging Evaluation: An Introduction to Outcome Mapping Amy Etherington & Rebecca Lee Mini-training for IDRC Interns & PDAs December 7 & 8, 2005

Session overview: § Evaluation context – where does OM fit? § Introduction to OM methodology – main concepts and tools § OM book – it’s useful! § How is OM being used – when is it appropriate?

Warm-up question: What words come to your mind when you think of “evaluation”?

IDRC’s Evaluation System § Promotes ownership & use of findings at all levels § Decentralized § Focus on evaluation processes § Monitors quality § Accountability for results at program & project levels

IDRC’s Evaluation Unit § Strategic evaluations § Cross-cutting issues § Corporate reporting § To Bo. G on performance and results § Information systems § Capacity building § With Southern partners § Promoting M&E with programs § Tools & methods § Organizational Assessment § Outcome Mapping

Monitoring & evaluation challenges: 1. Establishing cause & effect in open systems 2. Sharing ownership & participation 3. Recognizing the contributions of others 4. Tracking progress 5. Encouraging iterative learning 6. Measuring development results of research 7. Timing

Influence Shifting influence over time Time

Are we having an impact yet ?

ØA framework that allows researchers to plot human behaviour and actions and assess their contribution to the aims of research projects and programs IDRC’s Evaluation Unit (1998 -2000) ØOutcomes as changes in behaviour, relationships, activities or actions of the people, groups and organizations with whom a program works directly

OM bumper-stickers: The map is not the territory! § Be prepared for surprises along the way Keep your eyes wide open! §Being attentive along the journey… §This is as important as the destination

OM t-shirts CONTRIBUTION ATTRIBUTION

OM t-shirts INFLUENCE IMPACT

OM recognizes that in a development context, change is: § § § Complex (multiple actors and factors) Continuous (not limited to the life of the project) Non-linear (unexpected results occur) Beyond the control of the project (but subject to its influence) Two-way (program also changes)

Principles of use ü Flexible: modular - use adapted to circumstances ü Participatory: seeks dialogue and collaboration with boundary partners in P, M, &E ü Evaluative thinking; culture of reflection: promotes social and organizational learning

Step 1: Vision

I have a dream! Martin Luther King, Jr. August 28, 1963

Vision is a guide: North Light house Distant star

Vision statement: Reflects the broad human, social, and environmental betterment in which the program is engaged and to which it is contributing ØWritten in present tense as if the vision were already a reality

“Those who dream make a difference!” Ms. Kalpana Pant -Chaitanya

Women and girls in rural India enjoy full access to appropriate health care, education, food and water security and freedom from violence. They earn their own livelihoods with free access to markets and to banking and municipal services. Women knowledgeably use drudgery reduction technologies and agricultural practices that support human and ecological wellbeing. Villages are served by public transport, are well lit at night and have police forces that enforce laws fully and equitably. Girls attend school full time and, with their families, have the information and resources to make informed decisions regarding their personal health, safety and social needs. Gender equity is evident in the allocation of household labor and household decision-making; and men in the community support genderresponsive initiatives. Schools reinforce the role of women as educated, valuable and empowered community members. Through community-based organizations, women participate in and influence decisions, policies and programs affecting their well-being and share their experiences with others.

Vision: facilitation questions § What is the human, social and/or environmental condition that you hope to see in the region in which the program is being carried out? § Who is contributing to this situation - and how? Dream boldly !

Step 2: Mission

The mission is that “bite” of the vision statement on which the program is going to focus. ØWritten in future tense - as something the program will do

Your mission is your “business” § What do you do? § Who do you do it with? § Why do you do it?

Mission: facilitation questions § How can the program best support or contribute to the achievement of the vision? § What areas do you need to work in? § Where do you have credibility? § Who can you work with?

Step 3: Boundary Partners

Boundary Partners: definition Those individuals, groups, and organizations with whom the program: § interacts directly to effect change § anticipates opportunities for influence § engages in mutual learning

= partners The rest of the world Program

Sphere of influence The rest of the world Program = partners

Boundary Partners have Boundary Partners program’s bp bp’s bp

Families PHCs Banks Police Community Leaders State NGO SHG State NGO State NGO BAIF IDRC CIDA Swayamsiddha

Boundary Partners: Facilitation Questions § In which individuals, groups, or organizations is our program trying to encourage change so that they can better contribute to the vision? § With whom will we work directly? § Are we choosing X BP because we want to influence their behaviour and actions, or because they will influence others? Or both? § What behavioural changes do we (the project and BPs, collectively) want to see in the BP that will contribute to the vision?

Step 4 & 5: Outcome Challenges and Progress Markers

Outcome Challenge: definition § One OC is about a single boundary partner § Describes the ideal behavioural changes, relationships, actions and interactions in this partner § Describes how these changes will contribute to the vision.

Outcome Challenge: characteristics § One for each boundary partner § Does NOT describe program strategies § Written like this: “The program intends to see [boundary partner] who [description of behaviours in the active present tense]”

Outcome Challenges: facilitation questions §Ideally, how would your boundary partner be acting? §With whom would they be interacting? § What would they be doing?

Progress Markers: description § A graduated set of statements describing a progression of changed behaviours in the boundary partner § Describe changes in actions, activities and relationships leading to the ideal outcome § Articulate the complexity of the change process § Can be monitored & observed § Permit on-going assessment of partners’ progress (including unintended results)

3 Levels of Progress Markers The program sets out what it would: § Expect to see the boundary partner doing? § Like to see the boundary partner doing? § Love to see the boundary partner doing?

Progress markers = ladder of change Love to see Truly transformative Set quite high Likeactive learning, to see More engagement Expect to see Early response to program’s basic activities

X

Why graduated progress markers? Taken as a set, the progress markers: § are graduated from easier to more difficult to achieve changes in behaviour § describe the change process of a single boundary partner § are more complete than a single indicator § help the program think about how it can intentionally contribute to the most profound transformation possible § facilitate mid-course corrections and improvement

Influencing Outcomes Program relevance & viability Org. Practices Boundary Partner Program Delivery Strategies Program Results Progress Markers

Step 6: Strategy Map

Strategy Map § Outlines the program`s approach in working with the boundary partner § Indicates the relative influence the program is likely to have on boundary partner § Helps pinpoint strategic gaps in the approach or if the program is overextended

6 Types of Strategies Strategy Persuasive Supportive I-1 Aimed at the Boundary Parnter Causal I-2 I-3 • Direct Output • Arouse New Skills/ Thinking • Supporter who guides change over time E-2 E-3 • Modify the information system • Create / Strengthen a Peer Network E-1 Aimed at the Boundary • Alter physical or regulatory Partner`s environment Environment

Facilitation Questions:

Strategy images

Step 7: Organizational Practices

Organizational Practices: definition What the program does to: ü look within ü stay fresh, sharp, effective, healthy ü better serve its partners ü learn and change

Why Organizational Practices? § Important to how the program is going to function to effectively fulfill its mission § Supporting change in its boundary partners requires that the program be able to change and adapt

8 Organizational Practices 1. Prospecting for new ideas, opportunities, and resources 2. Seeking feedback from key informants 3. Obtaining the support of your next highest power 4. Assessing and (re)designing products, services, systems, and procedures

8 Organizational Practices 5. Checking up on those already served to add value 6. Sharing your best wisdom with the world 7. Experimenting to remain innovative 8. Engaging in organizational reflection

Step 8 -11: Monitoring priorities and Journals

Count every " F " in the following text: FINISHED FILES ARE THE RE SULT OF YEARS OF SCIENTI FIC STUDY COMBINED WITH THE EXPERIENCE OF YEARS. . .

M & E involves making choices “You can’t always get what you want… But if you try sometimes, you might find You get what you need… ahhhhhhh, yeah…” The Rolling Stones

Choosing WHAT to monitor § Who will use the monitoring information? § What will it be used for? § When is it needed?

Choosing HOW to monitor § Which components will be monitored? § How and when will data be collected? § Who will collect it? § Who will analyze, collate, package data? § Where and when will it be discussed and used?

Monitoring needs to be: ü Useful ü Affordable ü Light in work ü Understandable

Outcome Journal Work Dating From/To: Contributors to Monitoring Update: Outcome Challenge: Low = Medium = High = Expect to see: LMH 1 2 3 Who?

Strategy Journal Working Dating From/To: Contributors to Monitoring Update: Strategy to be Monitored: Effectiveness? (How did it help the boundary partner? ) Outputs

Performance Journal Working Dating From/To: Contributors to Monitoring Update: Practice 1: Prospecting for New Ideas, Opportunities, & Resources Example or Indicators: Sources of Evidence: Lessons: Practice 2: Seeking Feedback from Key Informants Example or Indicators: Sources of Evidence: Lessons:

Step 12: Evaluation Plan

Evaluation Plan: § A short description of the main planning elements of an evaluation study § For Whom? What? Where? How? By Whom? When? How Much?

Elements of an Evaluation Plan

Setting Evaluation Priorities Criteria with which to prioritize include: § Learning needs § Reporting requirements § Partners’ information needs

How can OM be used? § Designing and articulating the program’s logic § Recording internal and external monitoring data § Indicating cases of positive performance and areas for improvement § Evaluating intended and unexpected results § Gathering data on the contributions that a program made to bringing about changes in its partners § Establishing evaluation priorities and an evaluation plan

When is OM best used? § Once strategic direction or primary program areas are established § Particularly effective for larger projects § Best used at the start, but can also be used as a midway or final assessment tool § Activities must be sufficiently specific to identify key groups who will be influenced

Is OM appropriate? Not always! § § May require changes that are not possible Able to focus on “OM” outcomes Commitment to change and self-assessment Not intended for technical evaluations § (assessing relevance of a programming area or a costeffectiveness comparison) § Credible, and compatible with donor reporting requirements § Team consensus § Resource commitment