Evaluating Community Coalitions Partnerships Challenges Approaches

Evaluating Community Coalitions & Partnerships: Challenges, Approaches & Methods CDC Summer Evaluation Institute Atlanta, GA June 4 -6, 2012 Frances D. Butterfoss, Ph. D Yorktown, VA

Evaluation Myths Myth 1: Evaluation is complex - we might not do it "right“! Myth 2: We don’t have the time or resources! Myth 3: We never evaluated our coalition or strategies – it’s too late to start now! Myth 4: We don't need evaluation to know if our coalition & strategies are working!

When ? ? The 4 “Ws” of Coalition Evaluation What ? ? Why ? ? Who ? ?

Why Should We Evaluate? 1. Builds capacity within organization & community Do partners use approaches or tools for other issues? 2. Determines whether objectives are achieved Did organization achieve goals? 3. Improves strategies interventions, practices & policies Were strategies effective? 4. Provides accountability to partners & funders Are staff/members satisfied & contributing? Were resources used well?

When Do We Evaluate? • Change in resources • Change in membership, leadership or staff • Change in structure • Expand topics, population, geographic focus • Begin new project or hold important event • Before marketing or fund-raising initiative

Who Evaluates? • Evaluation team -Tap experts, staff or coalition members to help coordinate evaluation efforts • In-house evaluator - coalition makes decisions & controls process • External consultant plans/collects data & engages coalition as appropriate • External evaluation by funder

What will it Cost? 7 -10% of program budget q Salary & benefits: Staff time & training q Consultants: Varied expertise & perspectives q Travel: Variable - depends on program & methods q Communications: Postage & phone/fax q Printing: Data-collection tools, reports & other documents q Supplies & equipment: Computers, software & licenses q Incentives: Meals, babysitting, transportation & gift cards

What Do We Evaluate? • What about coalition works/doesn’t work well? How can we make it work better? • Who are key members? Are we meeting their needs? Are we building their capacity? • What resources have been most helpful or are needed? What are associated costs? • Do strategies fit coalition’s mission & goals? • Are strategies successful? Reaching intended groups? • What have we accomplished?

Coalitions are viewed in 2 ways: View 1: Coalition results are insufficient & weak - lack sufficient data to support continued use View 2: Coalitions are responsive, innovative & effective – they lead to significant health improvements (Birkby, 2003)

Coalitions are Ineffective • Kreuter et al (2000): 68 health promotion coalitions - marginal evidence that activities changed health status & systems • Roussos & Fawcett (2000): 34 studies/252 coalitions - results insufficient to make strong conclusions about pop. outcomes • Hallfors et al (2003): Well-funded coalition initiatives (e. g. , Fighting Back ATOD abuse prevention) costly to maintain & didn’t implement effective, comprehensive strategies • Zakocs & Edwards (2006): 26 studies - only 2 measured community behavior change; others focused on coalition function

Coalitions Change Policy, Programs & Services • Alcohol & drug abuse prevention (Shaw et al, 1997; Yin et al, 1997) • Immunization promotion (Morrow et al, 1998) • Prenatal care access, Disabled housing (Berkowitz, 2001) • Arson prevention (Wandersman & Florin, 2003) • Lead poisoning prevention (Klitzman et al, 2005) • Tobacco tax, air quality & access (RWJF, 2005) • Pregnancy prevention (Kegler et al, 2005) • Asthma management: health care quality, air quality & housing (Lara et al, 2006; Nicholas et al, 2006)

Challenges of Evaluation

Evaluation not built into coalition’s planning process • Evaluation can be costly (time & resources) - more likely to spend on programs (visible to partners) • Staff compromised on making coalition “effective” to sustain funding & jobs • Evaluation not based on logic model/theory • Coalitions often use “in-house” evaluators who may lack experience, methods or tools to evaluate coalitions

Each coalition is unique • Coalitions are embedded within communities & respond to variety of cultural contexts • Number of coalitions (N) is unknown; Many are unpublished & only recognized locally; Limited sample in any one health-issue area • Finding similar coalitions that address the same issue or comparative communities makes generalization about findings difficult

Design & Methods Issues • Experimental designs (randomization, control/comparison groups) impractical; case studies produce weak conclusions about impact on population (e. g. , changes in health status) • Methodology & instruments vary across studies • Confounding variables (e. g. , new programs/funding & demographics) vary across communities; hard to control for • Measurable outcomes limited & inconsistent • • Baseline data unavailable and/or outcomes not specified; Short & intermediate outcomes not linked long-term ones Distinguishing cause/effect or attribute outcome to certain coalition strategy is problematic

Overcome Challenges • Get training/technical assistance • Enlist partners’ help • Begin where coalition is • Be realistic; evaluate something • Use innovative methods • Evaluate promising strategies • Create measurable outcomes at all levels

Part I Get Ready

Help Members Value Evaluation • Demystify evaluation - Integrate it into everyday work – recognize its multiple purposes • View evaluation as reflection: How can it … improve our work? Influence stakeholders? Ensure funding? • Share impact that other evaluations have had on funding decisions or public visibility

Evaluation Pitfalls • Evaluation results may be used for unintended purposes, i. e. , to support political agenda or conceal real problems • Evaluation process may upset fragile relationships • Powerful players tend to influence evaluation outcomes; all players not included or heard • Evaluation results are ignored Taylor-Powell, Rossing & Geran, 1998

Are You Ready to Evaluate? q Are you committed to evaluate coalition & its work? q Are coalition’s goals realistic? q Are strategies consistent with coalition’s goals? q Are desired results specific & clear enough? q Are strategies well-grounded in theory/evidence? q Do you know what kinds of data will be needed? q Is data available? Can you access it?

Are You Ready to Evaluate? • Will evaluation be used to make coalition/policy decisions? • Will findings help you decide whether to expand strategies to new populations/settings? • Are adequate resources available? • Will evaluation focus on key questions & provide “lessons learned”? • Will you heed results, even if coalition must change? • Will evaluator work w/coalition to design evaluation & report & interpret findings?

Plan the Evaluation • Engage partners • Obtain buy-in/commitment to evaluate • Establish evaluation team & resources q Implementers – initiate & monitor program q Partners - actively support program q Participants - served/affected by program q Decision-makers – make strategic decisions

Participatory Evaluation for Coalitions • Values multiple perspectives; builds community capacity • Based on local community circumstances & issues • Multidisciplinary • Adapts to needs of coalition, strategies & populations Cousins & Whitmore, 1998; Kellogg Foundation, 1998

Evaluation Plan • Goals & measurable objectives? • Design: Pre-post, time series, case study? Process, outcome, or both? • Documentation (participant information, type & number activities completed) & how it will be used? • Methods (surveys, interviews, focus groups)? • Major tasks, responsible persons & timeline?

Describe Coalition or Program Stage of coalition & strategies • Planning or formation • Implementation • Maintenance Problem Statement • Nature & magnitude? • Causes & trends? • Consequences? Create logic model

Coalition Logic Models • Describes core components of coalition or strategy • Shows connection between coalition, strategies & expected outcomes • Includes key contextual (influencing) factors

Logic Model Shows… Inputs/ Resources Strategies/ Activities Available resources: time, staff, volunteers, money, material Actions to be taken: Events, training, curricula, marketing, advocacy r ou y … ow orts H ff e Outputs Tangible results or products of activities: # people reached, sessions held Short-term Outcomes Changes in learning, knowledge, skills Intermediate Outcomes Changes in behavior or practice Long-Term Outcomes Changes in health, policy, systems, or environments o Influencing Factors - Surrounding environments in which coalition exists (politics, socioeconomics, social norms & conditions, history, stage of development, staff turnover, other initiatives) that can positively or negatively affect its success t ad ge le n … ha c

Logic Model: Tobacco Policy Change Outcomes Inputs Coalition members Time Funding Activities Establish baseline for policy change in community with help from youth Educate youth and adults on policy change options and how to achieve them Partners ·Local ·Regional ·State Identify partners, including youth serving organizations and schools, for engaging youth in policy change Effective practice strategies Develop strategy for and promote engagement of youth in policy change Local media outlets Assist with development of youth advocacy skills Promote community support for youth involvement in community affairs/ policy change Reach Community organization s, businesses policy makers Adults Youth serving organizations Short Increased # youth, community members who: ·Understand tobacco use issues in their communities ·Know how to advocate for policy change Increased # youth wanting to be involved in advocating for policy changes Schools Youth Increased # youth skilled in being able to advocate for policy change Increased support for youth involvement in policy change Intermediate Long Increased # of youth actively engaged in advocating for policy change in community Increased # of activities or increased intensity of activities that involve youth to accomplish policy change Increased adoption of policies that involve youth in the policy change Counter industry influence Promote clean indoor air Decrease availability of tobacco products in community Increased number of tobacco policies in community Social norms less supportive of youth tobacco use Delayed average at first use; reduced initiation

Community Coalition Action Theory (CCAT) Butterfoss & Kegler, 2008 ty ni u m xt m e Co ont C Leadership & Staffing Processes Collaborative Synergy Member Engagement Lead Agency/ Convener Group Coalition Membership Leadership & Staff Structures Pooled Resources Assessment & Planning Implementation of Strategies Community Change Outcomes Structure s Community Capacity Formation Maintenance Health/ Social Outcomes Institutionalization

PART II GET SET. . .

Focus the Evaluation Ask partners what they want to know about: • Coalition • Strategies & activities • Outcomes • Members & participants • Coalition impact • Influencing factors

Coalition Evaluation Questions • • • Are we meeting members’ needs? Do work groups function well? Which strategies are effective? Are we accomplishing goals & objectives? Are we reaching our priority audiences? Are we influencing policies & practices? Are we building community capacity? Could we improve? Do we make a difference? What unintended outcomes are occurring?

Ecological Levels of Outcomes • Individual outcomes: Changes in status, quality of life, attitude, behavior, knowledge, skills & transfer of skills • Organizational outcomes: Changes in policies, practices, programs, services, resources, lives/career options, perceptions/skills of staff, mission, direction, or partners • Inter-organizational outcomes: Changes in linkages, relationships, collaboration, interaction, civic engagement & participation, authority & responsibility among organizations • Community or systems outcomes: Changes in health, social or physical environment due to changes in policies, practices, norms & sense of community

Levels of Coalition Evaluation • LEVEL 1: Process - coalition LEVEL 1 effectiveness • LEVEL 2: Impact - coalition strategies • LEVEL 3: Outcome – Change in community policies, systems & environments – Change in health/social status LEVEL 2 LEVEL 3

Process, Impact & Outcome Evaluation • Process evaluation: Short-term Ø Coalition activities (budget, recruitment, meetings) Ø Program implementation (context, reach, doses delivered/ received & fidelity) • Impact evaluation: Intermediate Ø Creation/modification of programs, services, or policies Ø Change knowledge, attitudes & practices • Outcome evaluation: Long-term Ø Change health status indicators Ø Change community capacity/institutionalization

Outcomes are more than just numbers • Membership: Increase # of members, not as important as what sectors they represent & how they contribute to coalition (e. g. , taking leadership role, being active in WGs) • Attendance: Numbers of members attending meetings not as meaningful as whether attendance is consistent (i. e. , core of members who regularly attend) • Strategies: Number of people reached not as important as whether they represent priority/intended population & whether strategies were sufficiently intense to cause change

PART III EVALUATE !

Choose Data Collection Methods & Sources • Use mixed methods to assess coalition or projects (qualitative & quantitative) • Use well-defined, measurable indicators • Collect enough data to be reliable, but reduce response burden • Don’t try to measure it all – Plan to use data you collect!

Sources of Data Existing Data √ Coalition Documents (Reports, newsletters, minutes, rosters) √ Existing databases √ Business records or websites People √ Coalition members √ Participants or participants √ General public √ Key informants (funders, officials) Other √ Actual events, activities, practices √ Before/after pictures (photovoice) √ Maps, charts

A Dozen Data Collection Methods 1. Survey - mailed, electronic, in-person, phone 2. Interviews - in-person, phone, focus group 3. Group Assessment - forum, Delphi technique, Nominal Group Process 4. Observation - use trained interviewers 5. Case Study - multiple data sources & methods 6. Formal Review - by peers, panel, experts, review committee 7. Portfolio Review - work samples to rate quality & scope 8. Document Review - content analysis to assess & summarize 9. Testimonials - responses/reactions by key participants 10. Tests - standard measures of knowledge, skill, performance, physiology 11. Photographs/videos - visual images & stories 12. Diaries/Journals/Logs - chronological records; personal views

Selecting Methods q What data do you need to make decisions about your strategies or coalition? q What data can be collected & analyzed practically? q What methods will be used if more data is needed? q Will results be credible to decision makers? q Are methods acceptable - will respondents engage in surveys/interviews/focus groups? Allow record access? q Who has skills to use methods? Is training required? q How will data be analyzed?

Instruments & Tools – Coalition Effectiveness Inventory (CEI) – Meeting Effectiveness Inventory (MEI) – Teamwork surveys – Collaboration & Inclusivity Checklists – Member Satisfaction Surveys – Member & Leader Interviews

Coalition Evaluation Measures Level 1. Infrastructure, function & process Member representation Expectations Member skills/experience Perceived effectiveness Recruitment Leadership Participation Staff performance Costs/benefits Organizational structure Role clarity Planning products/quality Satisfaction Task focus Commitment Decision making Organizational culture Conflict Sense of ownership Group relationships Communication Resources

Short-term Outcome Measures • # of coalition meetings in past year • # members from priority population attending training • # staff certified in communication/media training • Level of satisfaction of coalition members • # hours contributed by volunteers

Coalition Evaluation Measures Level 2. Strategies Implementation Media coverage New or modified services & programs Community actions taken New or modified policies & practices

Intermediate Outcome Measures • Increased knowledge about diabetes management • Improved cultural awareness of priority population • Increased self-efficacy about physical activity • Improved skill in advocacy • Improved behaviors (e. g. , attending prenatal classes) • New policy - point of purchase tobacco sales • New clinic referral practice • Increased treatment options

Coalition Evaluation Measures Level 3. Health status & community change Community coalitions Community capacity & competence Organization viability Health status indicators Availability Access Use Prevention Treatment

Long-term Outcome Measures • Sister coalition formed in adjacent community • Long-term funding obtained by coalition • Institutionalization of coalition strategy in partner agency • Decreased rates of substance use, decreased initiation of teen smoking, decreased rates of diabetes & heart disease

Free/Low Cost Evaluation Manuals & Websites • Community Toolbox: http: //www. ctb. edu • Empowerment Evaluation Website: http: //www. stanford. edu/empowermentevaluation • CDC. Physical Activity Evaluation Handbook. (2002). US Dept of Health & Human Services • W. K. Kellogg Foundation. (1998). Evaluation Handbook. Battle Creek, MI http: //www. wkkf. org/Pubs/Tools/Evaluation/Pub 770. pdf • Coalitions Work. http: //coalitionswork. com

Free/Low Cost Evaluation Tools • Survey Monkey: $200. /yr http: //www. surveymonkey. com/ • Innovation Network: Tools for evaluation plans, logic models & more http: //www. innonet. org/index. php • Online Evaluation Resource Library: Create surveys, interviews, observation guides: http: //www. oerl. sri. com/ • United Way Outcome Measurement Resource Network: Downloadable or low cost (< $50. tools & instruction) http: //national. unitedway. org/outcomes/resources/

Analyze & Interpret Data Analyze & Summarize data • Tabulate/stratify data - check errors • Summarize/present data to partners for feedback • Re-circulate draft report to ensure brevity, clarity & understanding Interpret results • Are results as expected? • Compare to other programs? • Limitations of evaluation?

Communicate & Ensure Use of Findings • Final Report: Methods, partners’ roles, strengths & weaknesses & results (tables/ graphs) • Recommend: Actionable, relevant & useful - Create action plan, roles & responsibilities for acting on findings • Share Results: Media, mail, websites, list serves, forums, newsletters; Reception, social or press event to legislators, funders & supporters

Parting Tips • Your questions about coalition & strategies should drive evaluation – not the other way around • Enlist partners’ help - builds buy-in & cooperation • Start small – plan 1 or 2 evaluation activities/year • Take advantage of data you already have • Verify partner roster to maximize response rates • Reduce respondent burden – short, frequent surveys • Use & adapt existing tools

So, take your mark & start evaluating!

References • Butterfoss, FD. (2007). Coalitions and Partnerships in Community Health. San Francisco, CA: Jossey Bass. • Butterfoss, FD & Francisco, VT. (2004). Evaluating community partnerships & coalitions with practitioners in mind. Health Promotion Practice, 5(2), 108 -114. • CDC. (1999). Framework for Program Evaluation in Public Health. MMWR 1999; 48 (No. RR-11), updated 2001. Washington, DC: Superintendent of Documents, U. S. Government Printing Office. http: //www. cdc. gov/mmwr/preview/mmwrhtml/rr 4811 a 1. htm • Olson, S. (2007). Partnership Tool Kit: Public Health Excellence Through Partnership, Draft Version 2. 0. Centers for Disease Control and Prevention.