Evaluating a Moving Target Early Lessons Learned from

Evaluating a Moving Target: Early Lessons Learned from the Community Alliance Against Violence (CAAV) Project Anita Baker, Evaluation Services, anitabaker@evaluationservices. co Jamie Bassell, Evaluation Services, jamie@evaluationservices. co Tracy Hobson, Center for Anti-Violence Education, tracy@caeny. org AEA 2012 Minneapolis, MN Panel Session 112 Evaluation Services

Session Overview • CAAV description; Role of evaluation in program and fund development; Evaluation partnership Tracy Hobson • Evaluation design, evaluation partnership Anita Baker • First year findings, features and challenges; how CAAV has provided a meaningful real-world evaluation learning opportunity Jamie Bassell AEA Session 112, 2012: Baker, Bassell, Hobson 1

Evaluating a Moving Target: Maximizing Opportunities to Learn From and About the Community Alliance Against Violence (CAAV) Project Tracy Hobson Center for Anti-Violence Education tracy@caeny. org AEA Session 112, 2012: Baker, Bassell, Hobson

CAAV Background • The Community Alliance Against Violence (CAAV) works to reduce violence in the lives of LGBTQI youth/young adults, many of whom are homeless • Goal: building skills to prevent, counter, and heal from – – intimate partner violence interpersonal violence street harassment bias crimes AEA Session 112, 2012: Baker, Bassell, Hobson 3

CAAV Background: Two-Pronged Approach • Provide 20 session self‐defense and violence prevention courses for LGBTQI youth/YA at partner organizations that combines physical and verbal self‐defense techniques, discussions, and role‐plays • Specific training for staff of three partner organizations who work with the youth/YA providing residential and other services AEA Session 112, 2012: Baker, Bassell, Hobson 4

CAAV Background: Critically Important Skills for Youth/YA • Stress‐reduction • Dealing with trauma • De‐escalation strategies • Self‐defense These are all critically important skills for youth and LGBTQI citizens living in volatile situations including the streets, shelters, and often hostile communities. AEA Session 112, 2012: Baker, Bassell, Hobson 5

Evaluative Partner: Benefits • Building our capacity • Helping in the proposal process – Framing what is realistic – Language to discuss benchmarks – Clarity for reporting and expectations • Increasing quality of programming • Sharing information with funders and donors AEA Session 112, 2012: Baker, Bassell, Hobson 6

Evaluative Partner: Challenges • Resources and staff time • Evaluators not in the office – timeliness of communication (to be refined year 2) AEA Session 112, 2012: Baker, Bassell, Hobson 7

Things Evaluators Did That Made it Easier for Us • Evaluation partners have a deep understanding and sensitivity to target population • Willingness to work with us – meetings, by phone, in person • Answered many questions and helped with pre‐planning and continued interactions with funders and other stakeholders • Helped with implementation – Coding system guide – Meeting with staff AEA Session 112, 2012: Baker, Bassell, Hobson 8

Partner Organizations: Benefits • Legitimacy conferred on our project by working with partner organizations – AFC – MCCNY – SI LGBT CC • Staff at partner organizations were an asset – Taking pulse of how workshops were benefitting youth – Asking questions about incentives – Helping get youth engaged and focused AEA Session 112, 2012: Baker, Bassell, Hobson 9

Partner Organizations: Challenges • Logistics of travelling, scheduling and space • Maintaining focus/attention/engagement difficult at other organizations – Chaotic spaces – Staff distracted by multiple issues – Challenges of working in shelters (enforcing ground rules while creating a welcoming environment) • “Stuff happens” (agencies can’t/don’t always do what they say they are going to do) – – Personnel changes Insufficient personnel buy‐in (initially) Emergencies (staff pulled away) Staff changes result in inconsistency for youth AEA Session 112, 2012: Baker, Bassell, Hobson 10

Working with LGBT Youth: Benefits and Challenges • Reaching a population in serious need • Resilient population • Transient population with chaotic lives. They were used to people coming and going during sessions, but instructors weren’t. • Others had abandoned programming and the agencies and participants indicated they were glad CAE “stayed. ” AEA Session 112, 2012: Baker, Bassell, Hobson 11

What Youth Worry About 12 AEA Session 112, 2012: Baker, Bassell, Hobson

Evaluating a Moving Target: Evaluation Design and Technical Assistance for the Community Alliance Against Violence Project Anita Baker Evaluation Services anitabaker@evaluationservices. co AEA Session 112, 2012: Baker, Bassell, Hobson

Evaluation T. A. * 1) Project Development 2) Project Initiation 3) Project Implementation 4) Lessons Learned — Outcomes and Target Identification — Proposal Review — Funder Site Visit Prep — Evaluation and CAAV Workplan Revisions — Needs Assessment — Instrument Development — Data Collection (design and participatory administration) — Interim Reporting — Staff Presentation — Workplan Revisions * CAE is a long-term Evaluation Services client 14 AEA Session 112, 2012: Baker, Bassell, Hobson

CAAV Evaluation Context 1) Project Overview Year 1 (completed) = 3 agencies, 20 workshops for youth/YA at each agency, 2‐ 3 staff workshops at each agency Year 2 (just beginning) = 4 agencies and 5 sites, same number of workshops 2) Evaluation focused on learning about anti‐ violence, self‐defense and self‐care techniques, and on behavior changes 3) Evaluation and evaluation‐related T. A. included from start 4) Evaluation team worked closely with CAE decision‐makers to design and conduct evaluation AEA Session 112, 2012: Baker, Bassell, Hobson 15

CAAV Evaluation Questions 1 a. How and to what extent has the CAAV project helped participating youth achieve desired outcomes? 1 b. What are the challenges and effective strategies related to delivery of the CAAV training for youth? 16 AEA Session 112, 2012: Baker, Bassell, Hobson

CAAV Evaluation Questions 2. How and to what extent are participating youth able to apply what they have learned through CAAV to life choices they make while involved in the program? 3. How and to what extent has the CAAV project helped staff at the three participating agencies achieve desired staff outcomes? 17 AEA Session 112, 2012: Baker, Bassell, Hobson

Desired Outcomes for Participants • Learn three de-escalation strategies • Learn three anger management techniques • Learn safety planning • Identify healthy and abusive relationships • Increase awareness of surroundings • Learn self-defense rights and responsibilities • Learn how to access support • Develop better communication skills and ability to negotiate 18 AEA Session 112, 2012: Baker, Bassell, Hobson

Evaluation Strategies: Staff Workshop Surveys • Closed and open‐ended questions about learning — self‐defense techniques — avoiding and de‐escalating violence — reducing stress • Items about usefulness and applicability of training • Overall rating of workshop quality • 2‐ 3 pages long, administered and collected immediately following the training 19 AEA Session 112, 2012: Baker, Bassell, Hobson

Evaluation Strategies: Participant Workshops • Participant Surveys ― Open & closed‐ended questions (including demographics) ― Content‐specific questions about learning/usefulness/use of SD topics and anti‐violence strategies ― Hard copy, comprehensive, administered after every other session • CAE Instructor Survey — Delivery/Support — Engagement • Partner Agency Staff Surveys — Delivery/Support — Engagement AEA Session 112, 2012: Baker, Bassell, Hobson 20

Evaluation Strategies Participant Workshops Cont’d • Participant Sign‐in Forms – Unique identifiers* – Name – Age Note: Only ID number included on surveys, attendance and demographics merged into one data base Individual databases developed for each survey (includes demographics and sessions responses) 21 AEA Session 112, 2012: Baker, Bassell, Hobson

Evaluation Strategies: Site Visits • Session Observation — Conducted at mid‐point, 1 at each site — Session delivery including opening, verbal SD, physical SD, discussion, and closing — Participant engagement • Participant Interviews — Small group (2‐ 5 select participants) — 1 hour, structured protocol, incentives • Staff Interviews 22 AEA Session 112, 2012: Baker, Bassell, Hobson

Year 1 Evaluation Implementation Summary • Rigorous and complete data collection from all three sites – Multiple methods – Multiple points in time – Multiple perspectives • Consistent collection of attendance data, staff, instructor and participant surveys • High survey completion rates including response to open ended questions. – Rich and compelling feedback about CAAV progress – Triangulated data 23 AEA Session 112, 2012: Baker, Bassell, Hobson

Evaluation Challenges • Instrument/item validity • Timing/scheduling of data shipments • Site data collection • Modified case studies • Participant unique IDs/instructions • Participant attendance at more than one site • Participants repeating the same training in year 2 • Lots of paper used! AEA Session 112, 2012: Baker, Bassell, Hobson 24

Evaluating a Moving Target: Early Findings from the Community Alliance Against Violence Project Jamie Bassell Evaluation Services jamie@evaluationservices. co AEA Session 112, 2012: Baker, Bassell, Hobson

Summary of Staff Workshop Participant Feedback about CAAV Training ALL SITES: AFC, MCCNY, & SI LGBT CC (n=55) Somewhat Applicable Very Applicable Workshop was applicable to everyday work 21% 79% No Yes Would recommend training to a colleague 4% 96% Overall Rating* Okay 16% Very Good / Excellent 83% * Please note that the overall rating scale included options of poor (1 participant at AFC selected this), okay, very good, and excellent AEA Session 112, 2012: Baker, Bassell, Hobson 26

Staff Learned CAAV Topics All Workshop Participants, All Sites, All Sessions % Reported learning about CAAV topics: How to offer clients new ways to de‐ escalate situation Understand the importance of self‐care/ stress reduction Access additional strategies for self‐ care/stress reduction How to provide positive interventions for clients The class supported your healing/ strengthening process Inform the work done with clients regarding the violence they are experiencing in their lives n Not at all Not sure Some A lot 31 3% 3% 23% 74% 65 5% 0 31% 66% 68 3% 2% 32% 63% 61 3% 5% 33% 59% 29* 10%** 7% 45% 38% 30* 7% 13% 53% 27% *This question was only asked at 2 workshops, both conducted at AFC **3 participants at AFC AEA Session 112, 2012: Baker, Bassell, Hobson 27

Participants Indicated They Would Apply What They Learned When clients get into arguments, I feel much better equipped to step in and intervene • A total of 95% of staff participants indicated they were somewhat or very likely to use the strategies and/or tools they got in the workshop to reduce their own stress, and 96% were somewhat or very likely to use those strategies with their clients to help reduce stress. • Most participants (80%+) reported that they were somewhat or very likely to use the self‐defense techniques/tools to make themselves and their clients feel safer. AEA Session 112, 2012: Baker, Bassell, Hobson 28

Additional Workshop Participant Feedback All participants at AFC, MCCNY, & LGBT CC (n=62) Somewhat Very Likely Will use strategies or tools to reduce stress for self 27% 68% Will use strategies or tools to reduce stress for clients 48% Will use self‐defense strategies or tools to help feel safer themselves 42% 44% Will use self‐defense strategies or tools to help clients feel safer 36% 44% AEA Session 112, 2012: Baker, Bassell, Hobson 29

Participants Reported That Staff Training Was Well-presented and Well-facilitated Instructor was very organized and curriculum well structured. Very informative. Written exercise was very practical and applicable to real life situations. Trainer immediately made me feel very comfortable. Trainer was familiar with population I work with. Training was empowering. I especially liked the calming, welcoming style of the trainer. Her very presence made my stress level go down. AEA Session 112, 2012: Baker, Bassell, Hobson 30

Suggestions for Improving Staff Workshops: More Time, Interaction, and Information Quotes From 3 AFC Workshop Participants on 2 -1 -2012 Shorter surveys. . . Giving more information at the beginning of what the training is about and will cover. Making it less lecture/talking intensive and create activities, questions, ways it can be more intensive and participants can engage and be engaged and be active in the presentation. Ice breakers at the beginning. Use role plays or media. Setting the training up so I know where you are going, will cover, and how it’s applicable to me. We need more time because there is a lot of info and we had to rush through. More interactive. . . It would be great to get us to talk more about what we or our residents deal with then speak to and direct your training and segues to that and then having different ways to engage us so it’s not just lecturing and stimulates different ways people 31 learn. AEA Session 112, 2012: Baker, Bassell, Hobson

Cross-Agency Implementation Findings • All three organizations successfully implemented all 20 sessions of the CAAV curriculum – Each workshop session included most or all of the five key segments: go around circle, physical SD moves, verbal SD exercise, discussion and the closing circle • Implementation happened differently at each agency – differing physical spaces – age of participants – background characteristics of participants • Delivery of the training by the two key instructors was described very positively by: – youth/YA participants – staff from the partner agencies – instructors AEA Session 112, 2012: Baker, Bassell, Hobson 32

Cross-Agency Implementation Findings: Engagement and Communication/Support • Engagement – described as very good by instructors, staff and youth – varied by agency – definitely improved over time • All partner agencies consistently described communication with CAE staff very positively. • CAE instructors described positive relations and support at the agencies, as well as continued need. – Support was somewhat varied by agency – Instructors needed more and regular support to help keep participants focused and engaged. 33 AEA Session 112, 2012: Baker, Bassell, Hobson

Cross-Agency Implementation Findings: Instructors and Training Elements • Most elements of the training went well at every session, at each of the three agencies • Some training components not done consistently at every agency/session – verbal self‐defense techniques – closing circle • At MCCNY, scheduling irregularities caused the sessions not to happen every week, resulting in a gap of several months between sessions 8 and 9. 34 AEA Session 112, 2012: Baker, Bassell, Hobson

Key Findings: Youth/YA Workshops • CAAV is serving the desired population • Attendance is intermittent but improving • Very well implemented despite varied and sometimes trying circumstances • Having the desired effect on participants, especially those that participate regularly, as reported by youth, instructors and staff. What did I learn? Don’t always look for a fight. The main goal is to get out & make sure everyone is safe. 19 year old gay male with a history of getting into fights. 35 AEA Session 112, 2012: Baker, Bassell, Hobson

Who Participated in the Youth/YA Workshops? • A total of 234 youth/YA participated in CAAV – 86 at AFC – 70 at MCCNY – 78 at SI LGBT CC – these numbers based on survey counts (possibly a different number on attendance sheets) • About half of the participants attended two or more workshops – 25% who attended between 3 and 5 – 7% who attended between 6 and 10 – few at each agency who attended 11 or more AEA Session 112, 2012: Baker, Bassell, Hobson 36

Who Participated in the Youth/YA Workshops? • Many participants had been affected by violence in the three months preceding their involvement in CAAV – about two‐thirds of the participants at AFC – about two‐thirds of participants at SI LGBT CC – slightly less than half at MCCNY AEA Session 112, 2012: Baker, Bassell, Hobson 37

Summary of Participant Findings • Instructor, staff and participants at all three partner agencies: – – provided uniformly positive feedback reported workshops were helpful and useful • At all three agencies, youth/YA and staff: – could give detailed examples of application of the trainings both in their daily lives and at partner agencies – reported that relatively large proportions of youth/YA learned how to keep themselves safer in multiple settings 38 AEA Session 112, 2012: Baker, Bassell, Hobson

Participants at Each Site Learned Safety Strategies for Various Locations AFC (n=69) MCCNY (n=63) SI LGBT CC (n=114) ALL SITES (n=246)* Street 90% 86% 95% 91% Subway 92% 85% 86% 88% Center 91% 88% 84% 87% Home 88% 84% 86% School 89% 83% 86% Work 85% 83% 79% *For this table, n represents the number of survey responses across all sessions at each site, but does not represent the total number of unduplicated participants AEA Session 112, 2012: Baker, Bassell, Hobson 39

Summary of Participant Findings • Many participants, especially those that attended more regularly, accomplished all of the desired outcomes. • Many participants learned the key topics at each session. – Varied by session and site – Possible improvements at later sessions AEA Session 112, 2012: Baker, Bassell, Hobson 40

AFC (n=69) MCCNY (n=63) Learned physical self-defense strategies 83% 76% 88% 83% Found the workshops to be useful 93% 94% 100% 96% FINAL SESSION SURVEY** n=12 n=6 n=12 n=30 At least one strategy to de-escalate a situation 75% 60% 91% 79% How to defend yourself if attacked 67% 80% 92% 79% Any negotiation strategies to keep you safer 75% 80% 73% 75% Your self-defense rights and responsibilities 58% 80% 73% 68% How to identify healthy and abusive behaviors in relationships 67% 60% 67% 66% How to get help/support when you need it 58% 60% 75% 66% At least one strategy to help you manage your anger 58% 60% 73% 64% How to be more aware of your surroundings 50% 20% 83% 59% Learned about the following across all sessions SI LGBT CC ALL SITES (n=114) (n=246)* * n represents the number of survey responses across all sessions at each site, but does not represent the total number of unduplicated participants **responses are from participants who attended the last session and not representative of participants at all other sessions AEA Session 112, 2012: Baker, Bassell, Hobson 41

Youth/YA Participants Confirmed That They Learned About Anti-violence and Self-defense I’ve been pretty calm and not violent in the last three months. Before three months ago, I was pretty violent, but I was doing mind training in high school so it was helping me calm down, so I’ve been a lot calmer, knowing how to breathe and count to 20— 20 because 10 is too close!—and I step away from the situation and ignore it. I was a violent person. The last fight was right before Christmas, it was with a girl here at the training today, she was the last incident I had but now we’re good friends and I didn’t think that was possible. But no, nothing violent in the last 3 months. For example, a week ago, two friends who were in the class today were arguing. I saw one of them said they needed to take a break from the argument and walked away, cooled off and came back and they talked it out. No one gets physical [fighting] lately here. --“Participant N” From SI LGBT CC AEA Session 112, 2012: Baker, Bassell, Hobson 42

One SI LGBT CC staff member also shared her perception of N’s change in attitude as a result of the workshops: [Participant N] is a fighter and has a history of being an aggressor, but when she listens she learns. She’s changing her attitudes. She’s less of a bully now. She was in that conflict last week and had some self-control. She was willing to listen and communicated better. AEA Session 112, 2012: Baker, Bassell, Hobson 43

Key Program Challenges • Attendance —Multiple sessions —Incomplete session • Covering verbal self‐defense topics • Physical space • Different subpopulations suitable for training • Workshop supervision/assistance • Incentives and voluntary participation • More time to deliver content at one agency than others AEA Session 112, 2012: Baker, Bassell, Hobson 44

A Real–World Learning Opportunity for a Newbie Evaluator “How long will this take? ” Planning for data entry, report writing High tech versus low tech (paper vs. e‐surveys, handwritten vs. laptop note taking at observations and interviews) You can’t always get what you want. . . learning how to go with the flow & working smartly with what you have when faced with limitations Interfacing with clients – best strategies for effective communication AEA Session 112, 2012: Baker, Bassell, Hobson 45

Next Steps Year 2 study continues. . . • More administrative involvement (at CAE and partner agencies) • Other data maintenance strategies? (emailing, faxing, pre-paid, scanning forms? ) • E-Surveys for staff/instructor? • Modified survey strategy • Modified case study participant selection 46 AEA Session 112, 2012: Baker, Bassell, Hobson

Questions/Thoughts/Ideas 47 AEA Session 112, 2012: Baker, Bassell, Hobson