Fidelity of Implementation and Program Impact Dr Bonnie

Fidelity of Implementation and Program Impact Dr. Bonnie J. Faddis & Dr. Margaret Beam RMC Research

PPS Striving Readers Partners • Grantee: Portland Public Schools • Professional Developer: University of Kansas, Center for Research on Learning • Evaluator: RMC Research 2

Targeted Intervention Overview • Goals: – To increase reading achievement – To increase reading motivation • Intervention features: – Xtreme Reading curriculum – Small class size (15 students/teacher) – Ongoing professional development 3

Eligibility for Intervention • Students enrolled in Grades 7– 10 • Students reading 2 or more years below grade level on the Oregon State Assessment Test (OSAT) • Students without OSAT scores who are reading 2 or more years below grade level on the GRADE test 4

Intervention Characteristics Year 1 • 9 schools – 5 middle schools – 4 high schools • 18 targeted intervention teachers (2 per school) Year 2 • 9 schools – – – 1 middle school 2 K– 8 schools 1 6– 9 girls’ school 1 8 th grade academy 4 high schools • 18 targeted intervention teachers 5

Intervention Characteristics Year 3 • 10 schools – – 2 middle schools 3 K-8 schools 1 6 -10 girls’ school 4 high schools • 14 targeted intervention teachers 6

Intervention Staffing Plan • 25 -30 students per grade level assigned to intervention class at each school • Intervention class paired with language arts or social studies class in high school • Intervention class integrated with LA/SS block in middle school • 2 teachers co-taught intervention class and LA/SS class (Years 1 -2) 7

Counterfactual Middle Schools High Schools • Xtreme Reading taught during LA/SS block • Xtreme Reading was an elective course • Control group students received LA/SS instruction with higher studentteacher ratios • Control group students enrolled in a range of other electives across content areas 8

Xtreme Reading Model 9

8 Stages of Instruction • • Describe rationale and strategy steps Model strategy through think-aloud Verbally rehearse strategy steps Guided reading practice Paired reading practice with fluency tests Independent practice w/comprehension check Apply strategy during oral reading with teacher • Integrate strategy with text from other classes 10

Implementation Results • • • Professional Development Classroom Implementation Teacher Buy-In Factors that Facilitated Implementation Barriers to Implementation 11

Professional Development Participation • Group session attendance > 75% – Year 1: 40% of middle school and 25% of high school Xtreme teachers – Year 2: 89% of middle school and 75% of high school Xtreme teachers – Year 3: 100% of middle school and 71% of high school Xtreme teachers • Added in Year 2 & 3: – Monthly meetings of Xtreme teachers 12

Classroom Implementation Average Fidelity Year 1 Year 2 Year 3 Alpha 92% 89% 79%* Beta 96% 90% 92% Gamma 72% 84% 93% Delta 93% 91% 94%* Epsilon 94% 88% — Phi — — 95% Zeta — — 80%* Kappa 83% 63% 92% Lambda 84% 58% 82% Sigma 82% 78% 74%* Theta 45% —* 80%* School Middle Schools High Schools 13

Teacher Buy-In Year 1 Year 2 Year 3 Professional Perceived Development & Program Support* Effectiveness** Middle High 4. 2 3. 6 4. 1 3. 6 4. 2 4. 4 3. 2 4. 0 4. 6 3. 7 4. 0 * Rating Scale: 1 = Strongly disagree; 5 = Strongly agree ** Rating Scale: 1 = Not at all helpful; 5 = Very helpful 14

Factors that Facilitated Implementation • Xtreme teachers liked the curriculum • Students liked the books • Teachers liked the professional development • Curriculum pacing schedules improved • 80% of Year 2 teachers had experience • 62% of Year 3 teachers had experience 15

Barriers to Implementation • Organization of Xtreme materials was confusing • Teacher skills with low achievers • Changes in middle school configurations reduced target population • Small school organization in 2 high schools created scheduling problems • High school counselors resistant 16

Impact Evaluation • Questions – Was the Striving Readers Xtreme intervention effective in improving students’ reading achievement? – Were the effects of the Striving Readers Xtreme intervention similar for middle schools and high schools? 17

Impact Evaluation • Random Assignment Eligible students were randomly assigned to treatment or control (stratified by school and grade level) • Analytic Design Intent-to-treat statistical model using 2 -level HLM (students clustered within schools) 18

Impact Evaluation • Outcome Measure Spring GRADE test normal curve equivalent (NCE) scores • Covariates – Fall GRADE pretest normal curve equivalent (NCE) scores – Cohort – Ethnicity (Black, Hispanic) – English language proficiency – Grade level 19

Statistical Power • 2 -level HLM cluster randomized trial • N = 1, 273 • Power = 80% • Alpha =. 05 • Minimum detectable effect (MDE) =. 27 20

Random Assignment 21

Student Sample Grade Level 7 8 9 10 Total N 347 313 342 271 1, 273 22

Student Characteristics Characteristic Gender Male Female Ethnicity White American Indian Hispanic African American Asian Special Education Services ELL Services Treatment Percent Control Percent 53 47 25 2 30 28 15 25 27 25 2 31 29 12 25 25 Note. Treatment total N = 599; Control total N = 674. 23

Impact Results Estimated Impact ( ) S. E. Effect p. Size value Group n Overall 849 2. 58 0. 79 . 15 . 002 Middle School 443 4. 84 0. 90 . 29 . 000 High School 406 0. 16 1. 31 . 01 . 901 24

Impact Results • Additional Finding Significant school level variability in treatment effects in both the overall analysis and in the middle school analysis 25

Effects of Implementation • Question To what extent do teacher level variables explain school level variability in treatment effects? 26

Analytic Design • Statistical model using 2 -level HLM (students clustered within schools) • Teacher level variables aggregated up to school level • Separate models by implementation year 27

Level 2 Variables • Percent fidelity of implementation • Percent of professional development training attended • Years of teaching experience 28

Results • Fidelity of implementation significantly contributed to school level variability in treatment effects in Years 1 and 3, but not in Year 2 29

Summary • Program was more effective in middle than in high schools • Variability in student outcomes partially explained by fidelity of classroom implementation • Teaching experience and amount of professional development not related to student outcomes 30