Implementation of Randomized Trials David Myers American Institutes

Implementation of Randomized Trials David Myers American Institutes for Research Washington, DC Prepared for IES/NCER Summer Research Training Institute, 2008 1

Context • Conducting effectiveness trials and not efficacy trials • Design and implementation of random assignment should not distort the program/intervention or the population served • Programs should run at capacity 2

Illustrative Studies: Personal Involvement • Upward Bound – Nationally representative study – 67 sites and more than 3, 000 students – Student RA within site – http: //www. mathematica-mpr. com/publications/PDFs/upboundimpact. pdf • Workplace Literacy – 3 sites and about 500 adult learners – Learner RA within site – "Addressing Literacy Needs at Work: Implementation and Impact of Workplace Literacy Programs. Final Report. " Washington, DC: Mathematica Policy Research, Inc. , October 1998 3

Illustrative Studies: Personal Involvement • 21 st Century Community Learning Centers – 12 grantees (elementary school study) – Student RA within site – http: //www. mathematica-mpr. com/publications/PDFs/elementaryschools. pdf • NYC Voucher Experiment – 2000 students – Family RA – http: //www. mathematica-mpr. com/publications/PDFs/nycfull. pdf 4

Illustrative Studies: Personal Involvement • Reading Comprehension – 10 districts and 40 schools – School RA – http: //www. mathematica-mpr. com/publications/PDFs/readcomp. pdf • Closing the Reading Gap – 32 schools (4 interventions) – Student RA to reading groups within schools – http: //ies. ed. gov/ncee/pdf/20084013. pdf 5

Illustrative Studies: Personal Involvement • Quantum Opportunities – 7 sites and about 1000 students – Student RA within site – http: //www. mathematica-mpr. com/publications/PDFs/QOPfinalimpacts. pdf 6

Practical Problems in Implementation • Technical • Political and Ethical Challenges • Recruitment (not independent of P and E Challenges) 7

Further Exploration: Technical • Cross-overs – Members of C get into T or something like T (use UB example and James Comer example) • Into T -- straight forward adjustment (AIR, 1996) • Something like T -- ? ; affects interpretation • Attrition – Post baseline -- “straight forward” adjustment (e. g. , MI) – Before baseline -- problematic, no information – How to fix? ? ? 8

Further Exploration: Technical • Unbalanced designs – Don’t demand additional recruitment • Large variance in selection probabilities (weights) – “over subscription” didn’t meet expectations (UB example) • Dishonest assignment -- post randomization – Sites don’t tell all students they have been selected for the program (UB example) – Sites ignore RA and move controls into T 9

Further Exploration: Ethical and Political Challenges • Random assignment isn’t fair – Programs afraid of denying services to students • Is it fair to never give a student a chance? (UB example) • Random assignment will force a program/teacher to serve a different population – Role of stratification to serve the desired mix • Some students (units) must be served – Role of the “wild card” before randomization – Wild cards excluded from analysis 10

Further Exploration: Ethical and Political Challenges • All “seats” must be filled – Role of the waiting list • Random selection within strata, if needed and desired by program operators • T and C groups analyzed as implemented at initial randomization • When programs believe recruitment will distort the population – Identify “most likely” and “least likely” to serve – Prior to randomization and stratify in analysis 11

Further Exploration: Ethical and Political Challenges • New treatments/interventions are hard to sell – Core programs vs. supplemental programs – Does the core program align with other curriculum and state assessments? • Concerns about making AYP – Will a supplemental program reduce hours of instruction in a core area such as ELA? 12

Strategies for Recruitment: Schools • Start with assistant superintendent for instruction (reading, math, science) or someone of similar stature • Quickly develop a relationship with an office and not just an individual -- staff come and go • Determine who needs to approve and to buy into participation – – Superintendent -- they may need to go to the school board Principals Teachers Parents and community groups 13

Strategies for Recruitment: Schools • Be prepared to meet with them – Have a recruitment team • Technical expertise in the design – Be prepared to tell audience why RA is valuable and not in a technical sense! • Deep knowledge about the intervention -- most important • Example of a meeting (next slide) 14

Irrefutable Evidence 15

Strategies for Recruitment: Schools • Establish expectations for the researchers and the schools – Minimize legal talk • Will bring in their general counsel and …. – Responsibilities and timelines • • Obtaining consent Data collection Administering instruments and tests Answering questions from parents and others 16