Assessment of Teacher Preparation Issues and Illustrations 2005

Assessment of Teacher Preparation: Issues and Illustrations 2005 -2008 Louisiana Value Added Assessment George H. Noell, Ph. D.

Overview of today’s talk n Early work that lead to VAM initiative n n Evolution of the early work n n Data, funding, flaws, and relationships Ramping up to a sustained effort n n How connections & opportunities can emerge Coordination, funding, flaws, and relationships Some interesting findings

Some consideration of the complexity of the journey n The scope of the effort in Louisiana n The complexity of the work n n IT, models, IRB, software Ramping up to a sustained effort n Coordination, funding, flaws, and relationships

Acknowledgements n Commissioner of Higher Education E. Joseph Savoie, Past State Superintendent Cecil Picard, Current State Superintendent Paul Pastorek, and Assoc. Commissioner Jeanne Burns. n Louisiana Department of Education Division of Planning, Analysis, and Information Resources (David Elder – Division Director, Allen Schulenberg, Robert Kaufman, Kelvin La. Croix, Steve Gunning, Sam Pernici, Bobby Franklin, and Roth Aymond) n LSU Research Team (Veronica Gulley, Bethany Porter, Anna Beth Ball, Maria Patt, Amanda Dahir)

Redesign in Louisiana & the Blue Ribbon Commission

How redesign lead to Louisiana’s Assessment n The principle of meaningful assessment n Putting all of the key players in the same room n Creating a climate of striving and mission n Creativity and shared resources ($)

Challenges in Assessing Teacher Preparation n The challenge of measures n n n Achievement versus opinions Geography Heterogeneous schools & classes Data management Technical issues The plausible counter factual (Rubin)

General Assessment Approach 1. Establish empirical expectations 2. Measure Performance 3. Compare Expectations & Performance 4. Act on the Results Detailed follow-up & policy issues

Moving to sustained work n Funding then and now n Establishing a small work team n n Establishing the big statewide team as well Working in public view and listening

Selected methodological issues with VAM in Ed. n Meaning of the data you have n Teacher experience variable n Scaling of your tests n Who “taught” the student versus tested n Reliability of identifiers: students vs. teachers n Modeling issues n “Just fixed models” because that is the right thing to do n Just drop the Special Ed. kids, what a mess n Of course you need cross year school fixed effects

Selected issues 2 n How big is a BIG effect? n n How important is 0. 1 SD or 5 points Poverty SLD and 3 to 12 points n What is a reasonable apportioning of the variance n What would that mean n What do demographic effects “mean” is a massively specified VAM? n Learning disability and free lunch effects & change in time

Selected Issues 3 Value added models, growth models, and their union and disunion When effects don’t make sense Is it too complicated yet or already too complicated

2004 -2008 Study Breakdown of Data n Years 2004 -2005, 2005 -2006, and 2006 -07 n Student Grade Levels: Grades 4 -9 n Content Areas: Mathematics; Science; Social Studies; Language Arts; and Reading n Tests: Iowa Test of Basic Skills, i-LEAP, and LEAP-21 n Pathways for New Teachers: Undergraduate and Alternate Certification Programs (Master of Arts in Teaching, Practitioner Teacher Program, and Non-Master’s/Certification-Only)

New & Experienced Teachers n New Teachers: n n Experienced Teachers n n 1 st and 2 nd year teachers Regular certificates; Completed TPP within 5 years 3 rd or subsequent year teacher with a regular certificate Teaching within area of certification.

Important Decisions Defining New Teachers (2007 study data)

Criteria for Inclusion of New Teachers in Study n Inclusion for each content area n n n 25 or more new teachers in grades 4 -9 At least 10 new teachers per year Teaching within certification Remained with student full academic year Not all universities had sufficient new teachers to be included in the analysis. Another year of data will be added and will increase the number that exceed 25 graduates.

Size of the Data Base for the Study Per Year n Districts: All school districts in Louisiana n Students: approximately 250, 000 n Teachers: 7, 000+ n Schools: 1, 300+

Hierarchical Linear Models: Nesting and Interacting by Year

Teacher Preparation Effect Estimates 355 Prior (2005) Achievement Current (2006) Achievement 350 Gifted African American Free Lunch 10 days absent Male

Post-Redesign Programs & Pre-Redesign Programs n Post-Redesign Programs: Programs that were redesigned for grades PK-3, 1 -5, 4 -8, and 6 -12 and began admitting pre-service teachers on July 1, 2003. n Pre-Redesign Programs: Universities stopped admitting candidates to the programs on July 1, 2003. A phase out period is occurring for preredesign programs.

Effect Estimates for Post-Redesign Alternate Certification Program Mathematics Performance Bands Level 1: Programs more effective than experienced teachers. Effect Estimates The New Teacher Project (+3. 1) Level 2: Programs comparable in University of Louisiana at Monroe (+1. 1) effectiveness to experienced teachers. Northwestern State University (+0. 8) Level 3: Programs comparable in effectiveness to new teachers. Louisiana College (-2. 7) University of Louisiana at Lafayette (-2. 9) Louisiana Resource Center (-3. 2) (Note: Mean for new teachers = -2. 7) Level 4: Programs less effective than new teachers. Level 5: Programs significantly less effective.

Effect Estimates for Post-Redesign Alternate Certification Program Language Arts Performance Bands Level 1: Programs more effective than experienced teachers. Effect Estimates University of Louisiana at Monroe (+2. 7) The New Teacher Project (+1. 6) Level 2: Programs comparable in Louisiana College (+1. 5) effectiveness to experienced teachers. Northwestern State University (+0. 5) Nicholls State University (-0. 3) Level 3: Programs comparable in effectiveness to new teachers. Louisiana Resource Center (-1. 8) (Note: Mean for new teachers = -1. 8) Level 4: Programs less effective than new teachers. Level 5: Programs significantly less effective. University of Louisiana at Lafayette (-4. 6)

Effect Estimates for Post-Redesign Alternate Certification Program Reading Performance Bands Level 1: Programs more effective than experienced teachers. Effect Estimates The New Teacher Project (+2. 2) Louisiana College (+2. 1) Level 2: Programs comparable in Northwestern State University (+0. 6) effectiveness to experienced teachers. Level 3: Programs comparable in effectiveness to new teachers. University of Louisiana at Lafayette (-2. 4) (Note: Mean for new teachers = -1. 8) Level 4: Programs less effective than new teachers. Level 5: Programs significantly less effective. Louisiana Resource Center (-6. 2)

ACT Scores of New Teachers n ACT scores of new teachers within programs did not account for variance in teacher preparation program effectiveness. n ACT mathematical scores of individual new teachers across programs was a modest predictor of teacher effectiveness in mathematics.

Impact of Teachers Who Are Not Content Certified Teachers who are certified in the content area they are teaching are more effective than those who are not certified to teach that content. Content Coefficient (CI) Mathematics -3. 50 (-4. 70, -2. 28) Reading -1. 27 (-1. 72, -0. 82) Language Arts -4. 09 (-4. 70, -2. 28) Science -1. 58 (-2. 34, -0. 82) Social Studies -3. 32 (-4. 61, -2. 03)

Placing Results in Context: Mathematics from 2007 study

Summary Observations from the Data • Some TPP prepare new teachers contribution to achievement is similar to experienced teachers. • • Effectiveness varies across and within TPP. • ACT mathematics scores of individual teachers is a modest predictor of teacher effectiveness in mathematics. • Certified teachers are more effective than teachers who are not certified to teach the content. ACT scores of new Louisiana teachers do not account for variance in teacher preparation program effectiveness.

Summary Observations from the Process n Sustaining system change and/or the long term research agenda n n n Working in the public view n n Data & Policy Relationships & funding Visibly responding to criticism Sustaining institutional support n n The many agendas in play Louisiana’s statewide research team