Assessment and Progress Monitoring in School-Based Mental Health

Assessment and Progress Monitoring in School-Based Mental Health A collaboration between Public Health of Seattle-King Country, the Seattle Public Schools, and the University of Washington School of Medicine/Seattle Children’s Hospital Aaron Lyon, Ph. D (UW/SCH) Jessica Knaster Wasse, MPH (PHSKC) Kristy Ludwig, Ph. D (UW/SCH) October 16, 2012

Acknowledgments Collaborators: • • • Ann Vander Stoep Elizabeth Mc. Cauley Eric Bruns Evalynn Romano Jane Koltracht Kelly Thompson Seattle Public Schools TJ Cosgrove School-based practitioners! Funding: • National Institute of Mental Health (K 08 MH 095939) • Institute of Education Sciences (R 305 A 120128) • American Psychological Foundation • Public Health of Seattle & King County • Bill and Melinda Gates Foundation Dr. Lyon is an investigator with the Implementation Research Institute (IRI), at the George Warren Brown School of Social Work, Washington University in St. Louis; through an award from the National Institute of Mental Health (R 25 MH 080916 -01 A 2) and the Department of Veterans Affairs, Health Services Research & Development Service, Quality Enhancement Research Initiative (QUERI).

Overview 1. Background & goals of standardized assessment / progress monitoring initiative in Seattle’s school-based health centers (SBHCs) 2. Public Health of Seattle & King County’s (PHSKC) interest in / support for progress monitoring in SBHCs 3. Current training approach / activities 4. Future directions

School-Based Health Centers (SBHCs) • Integrated care clinics situated in schools • Operate in nearly 2, 000 schools in the US (NASBH, 2008) • Typically provide primary care and mental health services (Brown & Bolen, 2003) • Well-substantiated as a mechanism to increase service accessibility to underserved and under/uninsured (Gance-Cleveland & Yousey, 2005; Kaplan et al. , 1999; Wade et al. , 2008)

Evidence-Based Practice in SBMH • EBP utilized inconsistently in SBMH (Evans & Weist, 2004) • Implementation of EBP in SBMH should focus on lowcost, high-yield practice improvement targets • Disseminate / Implement key competencies instead of full Tx packages (Beidas et al. , 2011; Rotheram-borus et al. , 2012) • Evidence-based assessment may be the “low-hanging fruit” of EBP implementation • School-based providers unlikely to use standardized assessment and progress monitoring routinely • Previous work has demonstrated good uptake of SA following training (Lyon et al. , 2011)

Evidence-Based Assessment (EBA): “Assessment methods and processes that are based on empirical evidence in terms of both their reliability and validity as well as their clinical usefulness for prescribed populations and purposes” (Mash & Hunsley, 2005, p. 364) • Evidence-based interventions commonly rely on EBA Standardized assessment (SA), or the use of measurement tools with empirical support for their reliability, validity, etc, is at the core of EBA (Jensen-Doss & Hawley, 2005)

Value of Standardized Assessment Initial Assessment • Rating scales can increase the ease and accuracy of clinical diagnosis (e. g. , Jenkins et al. , 2011; Youngstrom et al. , 2005) • Psychological assessment carries positive, clinically meaningful effects (Posten & Hanson, 2010) Progress Monitoring • Clinicians are often not able to detect client deterioration (Hannan et al. , 2005) • Providing assessment results to clinicians can result in improved outcomes (Bickman et al. , 2011; Lambert et al. , 2003)

Steps of Standardized Assessment Use 1. 2. 3. 4. 5. 6. 7. 8. Selection Administration Scoring Interpretation Feedback to clients Treatment planning Monitoring (over time w/ feedback) Treatment plan adjustment

SA / Progress Monitoring: SBMH v. CMH SBHC (n = 16) CMH (n = 58) Percent of Caseload: 0% 1 -39% Gave a SA measure in initial 1 -2 meetings 47% 27% 16% 32% Gave a SA measure at termination 63% 36% 52% 23% Gave a SA measure 38% 44% 42% Gave feedback about a SA measure 50% 31% 46% 30% Changed Tx plan based on SA data 69% 31% 64% 32% Changed indiv. session plan based on SA 56% 38% 64% 27%

Progress Monitoring & Feedback • Progress monitoring may use SA tools or ideographic monitoring targets • Value of ideographic targets (Weisz et al. , 2011)… • • Add specificity to ID’d problems Focus on client concerns / priorities Give clients a voice Enhance rapport / alliance Provide foci for ongoing assessment Provide info about treatment termination Easy to fit into everyday practice • Combination of SA and ideographic is likely optimal

Progress Monitoring & Feedback • Feedback Intervention Theory (Kluget & De. Nisi, 1996): Bx is regulated by comparisons of feedback to goals • Two levels (at least!): (1) feedback to the clinician & (2) feedback to the client • Most research has focused on feedback to clinicians • Feedback is associated with higher rapport with provider, client self-understanding (Allen et al. , 2003), and increased engagement in therapy (Ackerman et al. , 2000)

Monitoring School Outcomes in Tx • Academic outcomes are an essential domain for youth success, but rarely included in MH research (Atkins et al. , 2010; Franklin et al. , 2009; Hoagwood et al. , 2007) • Youth w/ combined MH and academic risk have a greater likelihood of long-term negative outcomes (Roeser et al. , 1999) • School-based clinicians rate school attendance as the most clinically-useful academic indicator (Lyon, Ludwig, & Mc. Cauly, in prep)

Potential Assessment / Monitoring Targets Mental Health • Depression symptoms • Disruptive behavior • Self-injurious behavior School Engagement • Attendance • Homework completion • Class participation • School connectedness Academic Outcomes • Grades • Credits earned • Standardized test scores Social Functioning • Interpersonal conflicts • Positive social experiences • Disciplinary events Physical Health • Sleep • Diet & Exercise Services • Satisfaction with treatment • Engagement in intervention

Current Goals & Activities 1. Engage stakeholders in planning / adapting a computerized measurement feedback system (MFS) to support SBMH providers 2. Provide training to SBMH providers in the use of standardized and ideographic assessment / progress monitoring 3. Implement, evaluate, and iterate the MFS, incorporating the ability to monitor academic outcomes

Engaging Stakeholders in the Adaptation of a Measurement Feedback System for SBHCs

SBHCs in Seattle / King County Policy/funding context • Seattle Families & Education Levy • Property tax levy, renewed for 4 th 7 year term starting 2012 • SBHC funding managed by PHSKC • Levy performance goals are academic in nature • Successfully lobbied for Levy $ for MH enhancement • Goal: Enhance the academic impact of Levy health investments by improving the quality of school mental health services • Increase use of evidence-based practice, with focus on standardized assessment and outcome monitoring

SBHCs in Seattle / King County Vision: Collaborative stepped care model • SA data used to support clinical decision making • Track clinical improvement and identify patients who are not improving as expected Aligns with Seattle Public Schools’ Response to Intervention (Rt. I) strategy Graphic from the National Center on Response to Intervention

Stakeholder Committee • “When developing an evaluation system, two of the most challenging components are getting buy-in from all partners involved and then selecting the measurement tools” (Sander et al. , 2011, p. 15). • Gates Foundation funding for one-year planning process • Stakeholder committee comprised of: school-based MH providers, supervisors, evaluators, program managers, computer programmers, school district

Stakeholder Committee Monthly meetings designed to: • Build initial consensus about data fields and other user-interface issues • Identify initial battery of standardized assessment measures • Identify SA toolkit for various MH domains • Develop and refine implementation / evaluation plan • Enhance provider buy-in

Stakeholder Committee Challenges/hurdles • • Differing documentation requirements Duplicate data entry EHR and MFS Varying degrees of technical knowledge Evaluation design: impact of randomization

Stakeholder Committee Results/Successes • • Data sharing agreement with Seattle Public Schools Specifications completed on schedule Negotiated acceptable evaluation plan Providers engaged and excited

Training in Standardized Assessment and Progress Monitoring

Training in SA & Progress Monitoring In promoting data-based decision making, SBMH professionals often struggle to (Kelly, 2011)… 1. Clearly define the reason for collecting the data 2. Ensure that data-collection procedures are manageable / user-friendly 3. Map out a clear system to use the data for continuous improvement

Training in SA & Progress Monitoring • Over the past 2 years, our group has provided a training and consultation series to local SBHC MH providers covering… • Standardized assessment principles, psychometrics, and normative data • Initial assessment using standardized tools • Feedback to students on the basis of assessment • Identifying and tracking progress monitoring targets

Training in SA & Progress Monitoring Principles of progress monitoring: • Monitoring of some type is a good idea for ALL youth receiving services • Only track those targets that are meaningful to the specific case/match treatment goals • Graphical feedback increases understanding and makes the feedback more memorable (Kluger & Di. Nisi, 1996; Miller & Watkins, 2010) • Can be used as a tool to validate changes in a student’s experience

Training in SA & Progress Monitoring Moving from the WHAT to WHY… • Progress monitoring is the first step and gives you the WHAT • Next steps include identifying the WHY (i. e. , why is a target changing or not changing? ) • Moving beyond just “success” or “failure” • The WHY is important to the kid feeling successful • Kid’s feelings of agency and control (therapy isn’t being done to them or for them but something they are willfully and actively participating in)

Assessment Flow Chart SA Student Presentation Assess 1) Decrease anxiety symptoms 2) Improved school engagement 3) Improved sleep hygiene Treatment Goal ID Ideographic SA: MASC-10 IG: Anxiety experienced when arriving at school? SA: School Connectedness Scale IG: # on-time arrivals for 1 st period SA: None? IG: Nights/wk w/o FB or texting after 8 pm

Current Findings and Future Evaluation Directions

Focus Groups • Focus groups conducted to evaluate the context in which the MFS will be implemented. • Transcribed and evaluated using a conventional content analysis approach. • EMERGING THEME: Factors which facilitate or inhibit the utility of a technology in SBMH (Lyon et al. , in prep) • • Accessibility Impact on Youth Engagement Confidentiality Knowledge or Attitudes (Provider) Integration of Aspects of Care Relevance to MH Care Speed of the Technology

Pilot of the Un-adapted MFS • Anonymous surveys of youth receiving SBMH services supported by the un-adapted MFS (n = 34) • 71% of youth respondents stated that answering questionnaires about emotions made it easier to tell their providers how they were feeling • 55% said answering the questionnaires (and receiving feedback) made the time spent with their SBMH providers either a little or much easier/better. • 45% said it did not affect their interactions (no students reported that it made it worse)

Many Questions Remain… Selected findings from Key Informant individual interviews from a related project… • Who should provide feedback about progress? • Teachers, students, school teams • Who should receive / review feedback? • In addition to clinician, principal, teachers, school staff, the academic counselor, school teams • Allow student to determine who will review • Other challenges • Unique data systems and data sharing rules • May require individualized approach district-by-district • Providers not used to collecting/using data systematically • Transitioning from health/MH to academic outcomes

Next Steps 1. Complete initial adaption of the MFS 2. Implement the MFS with randomly-selected providers 3. Compare provider behavior (e. g. , use of standardized or ideographic assessment/monitoring) between MFS and non-MFS groups 4. Conduct independent evaluations of youth outcomes by recruiting from the caseloads of MFS and non-MFS providers

Questions and Discussion