Evaluation Planning with Tobacco Prevention Education Networks

Evaluation Planning with Tobacco Prevention & Education Networks: A Community-Based Participatory Approach NWHF Community-Based Participatory Research Conference Portland, OR Friday, July 20, 2007 www. npcresearch. com This project was funded by the Oregon DHS Public Health Division

Panel Members n R. E. Szego, OR LGBTQ Coalition Against Tobacco n Jennifer Kue, Asian & Pacific Islander TPEP Coordinator n Luci Longoria, Disparities Liaison for the Oregon Tobacco Prevention & Education Program n Carrie Furrer & Scott Burrus, NPC Research July 2007 2

NPC Project Team n Beth Green, Ph. D. , Principal Investigator n Carrie Furrer, Ph. D. , Project Director n Scott Burrus, Ph. D. , Project Coordinator n Jelani Greenidge, BA Research Assistant NPC Research July 2007 3

Acknowledgements n TPEN Program Coordinators n Evaluation Teams at each of the 5 TPENs n Oregon DHS Public Health Division NPC Research July 2007 4

Overview of Oregon TPEP n Tobacco Prevention & Education Program n Comprehensive, statewide public health effort aimed at reducing the toll of tobacco-related death and disease n TPEP-funded programs work to: – Create smoke-free and tobacco-free environments through public policy – Counter pro-tobacco influences – Promote quitting NPC Research July 2007 5

What is a TPEN? Tobacco Prevention & Education Network n Focus on groups facing tobacco disparities: n – – – n African American Asian and Pacific Islander Hispanic/Latino Urban American Indian Lesbian/Gay/Bisexual/Transgender/Questioning (LGBTQ) Develop and implement culturally-reflective best practice strategies NPC Research July 2007 6

Purpose of Project n Assist TPENs in the development of community-driven evaluation plans and tools n Build capacity for evaluation planning and eventual evaluation activities NPC Research July 2007 7

What is CBPR? n Community-Based Participatory Research n Engages program stakeholders n Shared learning and power n Evaluators as “coaches” n Values community expertise, traditions, priorities NPC Research July 2007 8

CBPR Evaluation Planning n Community “owns” the evaluation process – Articulates priorities – Crafts questions – Involvement in tool development – Defines success n Culturally mindful planning process n Culturally appropriate products NPC Research July 2007 9

CBPR Framework n Step 1: Information Gathering n Step 2: Information Synthesis n Step 3: Information Sharing & Feedback n Step 4: Develop Goals n Step 5: Complete Evaluation Plan n Step 6: Review Progress NPC Research July 2007 10

Evaluation Planning Activities n Create community buy-in for evaluation n Identify community needs n Identify TPEN practices & whether they address community needs n Develop logic model for TPEN practices n Craft evaluation questions & data sources for each step of the logic model n Design tools to answer evaluation questions NPC Research July 2007 11

Steps 1 & 2: Information Gathering and Synthesis n Interview DHS key stakeholders n Interview each TPEN Program Coordinator n Review each TPEN’s work plan n Review CDC Best Practices & related literature n Review other state Tobacco Control Plans NPC Research July 2007 12

Steps 1 & 2: TPEN Program Coordinator Questions n What was your initial reaction to the idea of evaluation planning? n Did you have any concerns? n Was there anything about evaluation that you were particularly interested in? NPC Research July 2007 13

Steps 1 & 2: DHS Project Officer Questions n Why was CBPR important to this process? n What did you hope that TPEN Program Coordinators would get out of the evaluation planning process? NPC Research July 2007 14

Step 3: Information Sharing & Feedback n Pre-planning meeting – Statement of purpose – Logistics – Review preliminary logic model – Recruitment NPC Research July 2007 15

Step 3: Information Sharing & Feedback n Initial Evaluation Team Meeting – Form collaborative partnership – Introduce basic evaluation principles – Establish structure to guide collaboration – Identify tobacco-related community needs – Select core TPEN activity – Develop logic model for selected activity NPC Research July 2007 16

Step 3: TPEN Program Coordinator Questions n How prepared were you for what was covered in your initial Evaluation Team Meeting? n What was the most important thing that you learned from your initial Evaluation Team Meeting? n What do you wish you would have learned but did not? NPC Research July 2007 17

Step 4: Develop Goals n Initial & Second Evaluation Team Meetings – Thinking through short-term and long-term goals – Identifying what is important for the community to know about the work – Linking evaluation questions to logic model – Authenticating different ways of knowing NPC Research July 2007 18

Step 4: TPEN Program Coordinator Questions n What challenges did you face going into your second Evaluation Team Meeting? n What was the most important thing that you learned from your second Evaluation Team Meeting? n What do you wish you would have learned but did not? NPC Research July 2007 19

Step 5: Complete Evaluation Plan n Second & Third Evaluation Team Meetings – Identify key evaluation questions – Revise preliminary logic model – Identify data sources, measures and data collection methods – Revise current data collection plans (if appropriate) – Develop evaluation tools NPC Research July 2007 20

Step 5: TPEN Program Coordinator Questions n Do you believe that the tools developed (logic models, evaluation questions, tracking forms, assessments, etc. ) reflect the needs, values, traditions, and priorities of your community? n Do you now have the capacity to: – revise your current evaluation plan – execute evaluation activities, and – develop an evaluation plan for other activities? NPC Research July 2007 21

Step 5: DHS Project Officer Questions n What was the most beneficial technical assistance that you provided during this process? n What other kinds of technical assistance would have been useful? n What were the top 2 accomplishments of the evaluation planning process? n What do you wish would have been accomplished but was not? NPC Research July 2007 22

Step 6: Review Progress n Iterative process n Opportunities for feedback n Review at each community meeting n One-on-one meetings with Program Coordinator n Readiness for change NPC Research July 2007 23

Step 6: TPEN Program Coordinator Questions n Do you think that you were provided enough opportunities to provide feedback? n What kind of technical assistance was most helpful to you? n What else in the way of technical assistance would you have liked to receive? n How responsive was the process to your/your community’s needs? NPC Research July 2007 24

Accomplishments n Program advocacy n Build evaluation capacity n Design culturally-appropriate tools n Program documentation n Define community-specific processes n Develop a community knowledge base NPC Research July 2007 25

Lessons Learned n TPEN readiness for evaluation planning n Community readiness for evaluation n Community stakeholder recruitment n Versatile evaluation planning team n Flexible evaluation planning process n Need for clear next steps NPC Research July 2007 26

More Information n Carrie Furrer, NPC Research furrer@npcresearch. com n Full report online at: www. npcresearch. com n Oregon DHS Public Health Division: www. oregon. gov/DHS/ph/tobacco/index. shtml n R. E. Szego, OR LGBTQ Coalition Against Tobacco reszego@cascadiabhc. org n Jennifer Kue, API TPEN Coordinator jenniferk@mail. irco. org NPC Research July 2007 27