Research and Evaluation Data A Chat with the

Research and Evaluation Data: A Chat with the Mentoring Research Experts 2011 National Conference • Dallas, Texas • June 14 - 16

Panelists David Du. Bois, Ph. D. , Professor, Institute for Health Research and Policy, University of Illinois at Chicago Timothy Cavell, Ph. D. , Professor and Director of Clinical Training, Department of Psychology, University of Arkansas Michael Karcher, Ph. D. , Professor of Education and Human Development, University of Texas, San Antonio.

David Du. Bois Findings of Recent Meta-Analysis (Du. Bois, Portillo, Rhodes, Silverthorn, & Valentine, in press)—Not for external distribution or citation at this point without permission Analyzed results from 73 evaluations conducted 1999 - 2010 The “Good News” o Positive program impacts on youth in multiple domains: behavioral, social, emotional, and academic (including school attendance, grades, academic achievement test scores) o Individual programs often have made in-roads in two or more outcome domains (e. g. , social and academic) o Two-dimensional benefits – preventing declines in youth outcomes that might otherwise occur and promoting improvements o Benefits generalize across key dimensions such as age of youth, format (1 to-1 vs. group), and age of mentor (older peers vs. adults) o Magnitude of effects generally within range of those found for related types of child and youth interventions 2011 National Conference • Dallas, Texas

Comparison of Mean Post-Treatment Effect Sizes for Mentoring Programs in the Current Meta-Analysis to Effect Sizes Reported in Other Meta-Analyses of School- and Community-Based Interventions for Children and Adolescents Type of outcome Current Other meta-analyses Attitudinal/Motivational 0. 19 0. 23 r, 0. 25 b Social/Relational 0. 17 0. 15 a, 0. 17 i, 0. 24 r, 0. 29 b, 0. 39 g Psychological/Emotional 0. 15 0. 10 a, 0. 17 p, 0. 19 d, 0. 24 r, 0. 37 b Conduct problems 0. 21 0. 02 j, 0. 07 k, 0. 14 h, 0. 15 s, 0. 21 a, 0. 21 e, 0. 22 r, 0. 30 b, 0. 30 c, 0. 41 l Academic/School 0. 21 0. 11 a, 0. 23 n, 0. 27 r School attendance 0. 19 0. 14 b Grades 0. 24 0. 22 b Achievement test scores 0. 18 0. 11 a, 0. 20 b, 0. 24 f, 0. 30 c Physical health 0. 06 0. 08 m, 0. 17 t, 0. 29 q, 0. 41 o

David Du. Bois The “Challenging News” o No evidence of improved effectiveness over prior generation of programs o Too few studies to evaluate impacts on several key outcomes (e. g. , school drop-out, juvenile offending) o Same largely true for longer-term, “follow-up” effects The “Informative News” o Several program practices associated with greater effectiveness, including: ü Targeting “at risk” youth (exception: populations high on both individual and environmental risk) ü Utilizing mentors with educational/occupational backgrounds that are a good fit with program goals ü Matching youth and mentors based on similarity of interests ü Supporting mentors in adopting teaching and advocacy roles 2011 National Conference • Dallas, Texas

David Du. Bois Bottom-Line Assessment o Value in continued investment in youth mentoring as an intervention strategy within the policy arena o Strongest argument can be made for utilization of mentoring when interest is in promoting outcomes across multiple areas of a young person’s development o Policy recommendations to maximize ROI 1) Ensure adherence to core practices (e. g. , mentor screening and training) essential to program quality 2) Facilitate research-informed development of program innovations 3) Foster stronger practitioner-researcher collaboration in design, implementation, evaluation, and ongoing refinement of programs 2011 National Conference • Dallas, Texas

Michael Karcher TEAM Framework: Understanding Relationship Activities Suggests why we should train mentors to consider: Focus: How relational vs. directive are the mentoring interactions (activities, discussions)? Authorship—How collaborative the conversation or activity decision is? Who authors their story? Purpose: Do interactions serve serious, futureoriented, adult or playful, youth-oriented goals? From Karcher, M. J. & Nakkula, M. J (2010). Youth mentoring with a balanced focus, a shared purpose, and collaborative interactions. In “Play, talk, learn: Promising Practices in Youth Mentoring, ” Jossey-Bass.

Michael Karcher TEAM Framework: Understanding Relationship Activities What should Baloo and Bagheera do to get Mowgli where he needs to be? casual conversation talk about family Baloo-type (Relational Arrival in the Man Village (Success: mentor-mentee relationship quality) Conversations talk about friends listening & learning talk about school Bagheera (goal-directed conversations) discuss attendance discuss behavior talk about the future Karcher, M. J. , Herrera, C. , & Hansen, K. , (2010). “I dunno, what do you wanna do? ”: Testing a framework to guide mentor training and activity selection. New Directions in Youth Development, 126.

Michael Karcher

Michael Karcher

Timothy Cavell What do we know about bullying • Prevalence – 30 -40% of youth involved in bullying – Peaks in middle school – # of victims drops, but they are more visible • Bullying can be physical, verbal, or relational • Victims = unpopular and friendless children

Peers Seldom Intervene

Timothy Cavell Anti-bullying Interventions • School-wide programs can reduce overall # of children being bullied • But success requires whole school buy-in and effects tend to fade with time • We need programs for chronically bullied children

Timothy Cavell School-Based Mentoring: An Indirect Way to Help • Why indirect help? • Many bullied children… – Reluctant to ask for help or resist help – Doubt whether adults can or will help – Believe adult help will make things worse

Lunch Buddy Mentoring: One type of SBM • Mentors visit twice/week for 30 minutes during lunch period • College student mentors who sit with mentee and peers

Timothy Cavell Recent Pilot Study • Method – Lunch Buddy children (n = 12) – 2 matched control groups • Same Controls (n = 12) – same school • Different Controls (n = 12) – different school • Results – LB children significantly less bullied after 1 semester (peer reports) – Significantly less than Different Controls (but not Same Controls) – Parents & teachers: high satisfaction with LB mentoring – No evidence of harm to bullied children (child, parent, teacher ratings) Elledge, Cavell, Ogle, & Newgent (2010). Journal of Primary Prevention, 31, 171 -187.

Timothy Cavell What One Lunch Buddy Mentor Said: ØThere were kids who would ask why I sat with him. ØI’d say he was my friend and it was pretty cool to sit with him. ØThey looked shocked, but then they began to sit next to us on every visit. ØIt was amazing how their little attitudes towards my mentee changed by me saying I was his friend and I like sitting with him.

Questions for Panelists 1. Why is fidelity to a program model so important? 2. Why is our Outcome Evaluation System critical in communicating effectiveness? 3. What are some simple things agencies can do with their AIM data to help improve practice and demonstrate value to stakeholders? 4. What do you feel is the most important change mentoring programs can make to their CBM programs to improve youth impact?