Dialogues between Internal and External Evaluators Evaluating the

Dialogues between Internal and External Evaluators: Evaluating the Program Impact on Academic Achievement of Homeless/Highly Mobile Children in Minneapolis Public Schools Chair: Leah Goldstein Moses Becky Stewart The Improve Group, Inc. Chi-Keung (Alex) Chan & Elizabeth Hinz Minneapolis Public Schools Presentation at the 2009 American Evaluation Association Annual Meeting November 13, 2009, Orlando, Florida

Introductions l Presenters: l Discussant: • Alex Chan, Minneapolis Public Schools • Rebecca Stewart, the Improve Group • Leah Goldstein Moses, the Improve Group

Background of the Program l Partnership (since 2001): • City of Minneapolis, Lutheran Social Services, Minneapolis Public Schools, Minneapolis Public Housing l Target population: • Families experiencing homelessness or housing instability. Starting in 200607, the program began targeting chronically homeless families. l Services offered: • Housing vouchers • Case management • Family support services, including parenting and mental health support • Educational support services

Background of the Program Race and ethnicity of head of household (Participant Group September 2006 -August 2007, N=29) Average household size is five people, with three or more children. Families entering the program are struggling financially: Two-thirds were unemployed and even more (72%) were receiving welfare assistance (Minnesota Family Investment Program or MFIP). The average household income was $11, 788 at program entry—less than 50% of the criteria for “extremely low income. ” Source: Lutheran Social Service One-half (52%) have a household member who struggles with mental health issues; the same share of households have experienced domestic abuse at one point or another. A smaller percentage report chemical health issues.

Background of the Evaluation l l Evaluation is guided by a logic model and each partner contributes data Evaluation results are used to identify needs, plan services, manage the program, and build continued support Evaluation questions and priorities are set both externally (funder priorities) and internally (identified needs) A steering committee representing all of the partners provides guidance for the evaluation

Let’s Begin the Dialogue The first question faced by the internal and external evaluator was how to evaluate the Kids’ Collaborative program impact on academic achievement of homeless/highly mobile children for the homeless/highly mobile families

Internal and External Evaluators Internal External Context Within the system (MPS - one Outside of the system of the partners) Flexibility Data Accessible to individual-level Able to collect in-depth qualitative data quantitative data Able to compile data from multiple sources Knowledge of adv. statistics Logic model Familiar with H/HM data Mixed-method evaluation Relative Strength Relative Limited time for collecting in- Limited access to de-identified quantitative data Limitation depth qualitative data

Applying Multiphase Longitudinal Analysis to Evaluate the Program Impact on the Achievement Growth of Homeless/Highly Mobile Students Alex Chan Internal Evaluator

Data The data was provided the LSS and consisted of 196 kids from 79 active participating families between June 2006 and July 2009 (received in Sept 2009). The sample for the analyses included 72 students who enrolled at grades 1 -8 (at the time the families moved in) with at least one pre- or post- intervention achievement data point.

Probability Matching Since the participation of the program is not randomly assigned, probability matching method (propensity score matching) was adopted to match the participant kids to a comparison group of homeless/highly mobile students clustering within school and grade level and control for the demographic characteristics and prior achievement.

Multi-phase Longitudinal Analyses Since participating families got the housing vouchers and moved into the house at different time point during the year, a better approach to address the intervention impact of the program across cohorts and families who moved in different time-point. References: Cudeck, R. , & Klebe, K. (2002). Multiphase mixed-effects models for repeated measures data. Psychological Methods, 7, 41 -63. Verbeke, G. , & Molenberghs, G. (2000). Linear mixed-models for longitudinal data. New York: Springer.

Intervention Began

Strengths versus Limitations Strengths Limitations Data Able to handle Management complicated data Time-consuming for data manipulation & analyses Context Need more in-depth qualitative evidence Overall program effectiveness Evaluation Evaluate multi-cohort Methodology multi-phase programs Use along with other evaluation approaches Policy Implications Consider political, funding, and timing factors Short-term & long-term program impact

Kids Collaborative: An Independent Program Evaluation of How Family Support Impacts Homeless Student Achievement Rebecca Stewart External Evaluator

Background: Client Needs l l Interest in helping funders and partners understand the day -to-day reality for families Interest in deepening understanding of impact; results were not as strong as desired on initial basic measures • Capture more of an understanding about families’ path to stability • Investigate more school outcomes (less interest in school stability measures)

Evaluation Plan l l Created a logic model • Home, family and student impacts identified that will support improved student achievement • Quantitative and qualitative data Defined cohorts to reflect program experience Worked with Minneapolis Public Schools’ Evaluator Data collected from: partner records, parent interviews, teacher interviews/surveys, staff focus groups and interviews

Findings: Successes l l l Families live in safer, more stable housing About 9 in 10 families are making progress on their goals Staff are making referrals for mental health services for most of the families facing these issues Average annual school moves decreased Elementary students improved discipline and attendance Teachers say that students who had struggled with peer interactions had improved over the school year

Findings: Remaining Challenges l l l Percent of students making expected one-year growth on reading and math tests dropped; no change on a normal curve equivalent Low rate of follow-through on mental health referrals Referrals for chemical dependency or child behavioral issues are not documented at the same level as the need for these services Longer case management time is needed Education Support Advocate is stretched to provide services to all children who need it 15% of those students who teachers say were inattentive did not show improvement over the school year

Findings “[My children] attended school everyday, but they really didn’t want to go, because the area stressed the kids out, they loved school but they were unhappy and they were depressed. It started to affect them; they didn’t seem interested in school anymore. I think they would see me stressed out, they were too busy trying to focus on things I should been worried about. That wasn’t healthy at all… With the atmosphere that was going on outside, it was hard for them to focus [on their school work] with all the things that were going on. ” “My son and my oldest daughter, my two biggest ones, oh my goodness, they started hanging around with the kids in the neighborhood and they were, I won’t even lie, I love my kids but yes they were to the point where police were coming up to our driveway and asking questions about them. It was embarrassing. It really was. They had to pretend to be tough, they didn’t talk proper. It was hard for me to even discipline them because I knew I was the one that put them in that situation. [Now] they are good. They are getting better. My daughter, my oldest one, has gotten a whole lot better. My son is getting better but experiencing more of a physiological thing with him. I know it’s gotten a little better, because I think it would have gotten a whole lot worse. I wouldn’t have been able to keep track of them. It just would have been terrible. You can’t raise kids in that neighborhood. If you are around that atmosphere, then your kids eventually are going to have to adapt to that. ”

Findings l l Almost all (94%) have an Individualized Educational Plan – indicating significant learning challenges Families are taking well over a year to make significant changes; achievement changes may also take longer Students’ educational challenges vary widely – some are struggling a lot, others are doing fine (staff report) More success at elementary level

What we were able to do: l l l l Document staff hunches Document more about the change process, and the time needed for it, in families Document repercussions of city policy Document prevention rather than intervention nature of program (first year evaluation) Document low follow-through on mental health referrals and prevalence of these issues Share the realities of how housing impacts schooling Explain the relative “fit” of the comparison group Provided case file review spreadsheet

Benefits l l l Can adjust research questions (flexibility) Assist in communication between partners and compile data for more complete picture Can gather qualitative data that leads to the formalization of important indicators Can provide whole family focus/understanding Can structure evaluation to answer funder questions and well as identifying research questions of all partners

Limitations l l l Struggle to add more depth about student achievement challenges Cannot work with individual-level student data Limited opportunity to do data mining

Comments and questions l l Discussant comments Initial questions: • What, if any, points of tension existed • between the external and internal evaluators? What, if any, areas of success were made possible through the collaboration between the external and internal evaluators?