Use and Impact of the Student Progress Monitoring

Use and Impact of the Student Progress Monitoring System Mississippi Department of Education Office of Instructional Programs and Services Office of Student Assessment MASS January 10, 2006

Assessment that is explicitly designed to promote learning is the single most powerful tool we have for raising standards and empowering life long learning. – Assessment Reform Group (1999) 2

Overview n n n Student Progress Monitoring System (SPMS) is designed to offer a web-based approach to help districts track which students are mastering key objectives and which are not. The ultimate goal of SPMS is that teachers use SPMS assessments to focus on reviewing, revising, and improving daily instruction through rapid assessment and feedback. Through the daily/weekly review of student progress, districts will have a better way to track their expectations for meeting annual accountability requirements and for preparing students for national standardized tests. 3

District Use n n n Participation is a district and school decision. Districts registration for SPMS may include all schools or only targeted schools. For the 2005 -06 year, the state is funding the entire per pupil cost, and the MDE intends to fully fund SPMS for the 2006 -07 school year. Implementation depends heavily on local control; therefore, districts can structure the use of the system. Some districts have chosen to use SPMS more for “benchmark” testing, while others are using the system formative assessment data as well. 4

Application n n The item bank currently contains math, reading, and language arts items aligned to the “old” Mississippi Curriculum Frameworks. Coverage includes items for grades 1 -12 in these areas. Coverage for science and social studies includes items for grades 3 -12. SPMS also supports writing at grades 4, 7, and 10 (English II) with automated scoring. Tests can be built by teacher, school, or district. 5

Application n n Individual and summary reports enable teachers and administrators to track, monitor, and disaggregate data by various demographic categories at the classroom, school, and district level. Student portfolios enable teachers and students to track progress in real time. SPMS provides system-generated parent letters. A new resource unit will be unveiled at the district level at the end of January and will allow users to use lesson plans, models, and activities already in the system and to add personal resources for all SPMS users. 6

Application n Reporting features: n n n State test results are automatically loaded into the system. Summary performance reports provide results for curriculum competency and objective. Item analysis and roster reports show performance for each test and for each student. 7

Status Comparison n 2004 -2005 n Districts: 63 n Schools: 244 n Students: 133, 000+ n 2005 -2006 n Districts: 116 n Schools: 647 n Students: 398, 413 8

2005 -2006 State Users n n To date in 2005 -2006, 247 schools across 78 districts are recording test results in the Mississippi Student Progress Monitoring System. Over 70, 400 test sessions have been administered to students in elementary through high schools in the subject areas of reading, math, writing, science, and social studies. 9

2005 -2006 Users n n SPMS testing began in August 2005, with November and December being the heaviest testing months in the school year thus far. Testing continues as a brisk pace in 2006, with over 5, 000 tests already recorded in January. 10

2005 -2006 Test Sessions by Month 11

2005 -2006 Test Sessions by Month 12

Relationship Between SPMS Usage and State Test Scores n n n Little is known about the nature of the tests administered, the importance of the SPMS tests, and how the data from SPMS were used to drive instruction. Investigations into the relationship between usage of SPMS and performance on state assessments show that the usage across the state in the 2004 -05 year may not have been organized, integrated, or strategic enough in order to show major trends across the board. Case studies provide the most insight into the effectiveness of the SPMS. 13

Case Study: Forrest County 14

Forrest County Grade 3 MCT Math n Gains from 2003 -04 to 200405 for Grade 3 MCT Math mean scale score n n State gain: n 2. 4 scale score points n 499. 8 to 502. 2 Forrest County gain: n 10. 9 scale score points n 494. 6 to 505. 5 n Gains for MCT Math from 2003 -04 Grade 2 to 2004 -05 Grade 3 % Proficient and Above n n State gain: n 0. 2% n 91. 7% to 91. 9% Forrest County gain: n 6. 0% n 86. 7% to 92. 7% 15

16

Forrest County Grade 3 MCT Math Black-White Gap from 2003 -04 to 2004 -05 for Grade 3 MCT Math, % Proficient and Above n Forrest Co. State 200304 200405 Black 71 82 87 88 White 96 96 Gap 25 14 9 8 17

Forrest County Grade 3 MCT Math n Economically Disadvantaged Gap from 2003 -04 to 2004 -05 for Grade 3 MCT Math, % Proficient and Above Forrest Co. State 200304 200405 ED 81 88 89 89 Non-ED 95 96 96 96 Gap 14 8 7 7 18

Case Study: Kemper County 19

Kemper County Grade 8 MCT Reading n Gains from 2003 -04 to 2004 -05 for Grade 8 MCT Reading mean scale score n n State decrease: n 7. 0 scale score points n 563. 6 to 556. 6 Kemper County gain: n 9. 4 scale score points n 544. 5 to 553. 9 n Gains from 2003 -04 to 2004 -05 for Grade 8 MCT Reading % Proficient and Above n n State decrease: n -4. 8% n 61. 6% to 56. 8% Kemper County gain: n 14. 2% n 41. 1% to 55. 3% 20

21

Kemper County Grade 8 MCT Reading n Economically Disadvantaged Gap from 2003 -04 to 2004 -05 for Grade 8 MCT Reading, Percent Proficient and Above Kemper Co. State 200304 200405 ED 37 54 47 44 Non-ED 52 60 77 72 Gap 15 6 30 28 22

Kemper County and SPMS 2004 -05 n n n Vicki Parker is a retired Mississippi educator and certified Vantage trainer. Vicki worked with Kemper County during the 2004 -2005 school year. Vicki will also be a trainer for the SPMS Curriculum, Assessment, and Instruction (CAI) modules. 23

MS In-State Vantage Contact n n n David Timmons, who taught in the Bay. Waveland School District, is a certified Vantage Trainer. He is now serving as the MS Contact Person for SPMS. His e-mail is dtimmons@vantage. com. 24

New for 2006 n n n 3 -in-1 bubble sheets with significant cost reduction Version 2. 0 for district users MCT practice tests Science practice tests MS-teacher reviewed items for social studies and science SPMS Curriculum, Assessment, and Instruction (CAI) modules 25

Bubble Sheets for Off-line Testing 26

3 -in-1 Bubble Sheet 27

Version 2. 0 n n n Version 2. 0 will be ready for the 2006 -2007 school year. Vantage is making 2. 0 available for users at the district level only at the end of January for “trial use”; 2. 0 will not contain “live” files. Version 2. 0 will have a new resource site where districts users will be able to enter lesson plans or activities to the resource list already identified within the system. 28

Version 2. 0 n n Version 2. 0 will have new functions to allow users to nominate items to be included in the SPMS database for all users. Version 2. 0 will have functionality to allow external test data (such as DIBELS) to be entered into the system. 29

MCT and Science Practice Tests n n Release set for late January for first practice test for grades 3 and 7, all three content areas Second release set for February for first practice test for all grades, all three content areas Spring release for practice tests two and three for all grades and content areas Release in spring for practice science tests for grade 5, grade 8, and Biology I n (These are not the Riverside hard copy tests. ) 30

MS Teacher Reviewed Items n n New science and social studies items All items reviewed by Mississippi teacher committees Items to be identified to distinguish them from original SPMS items Mississippi courses new to SPMS n Science: Biology II, Physical Science, Anatomy and Physiology n Social Studies: MS Studies, US Government, and Economics 31

New Math and Language Arts Items n n Addition of math and language arts items to SPMS was delayed so that new items will match the revised frameworks. Teachers will meet in late spring and/or early summer to review items. Items will be added to SPMS by January 2007. New MCT 2 will not be administered until spring 2008. 32

SPMS CAI Modules 33

SPMS CAI modules n n n Districts selected for Series I or II were notified January 6. Ten teams were selected for Series I and for Series II based upon a rubric evaluation of the district’s application. Letters regarding Series III and IV will be mailed to superintendents and SPMS contacts next week. n Districts that applied but were not selected for Series I or II will be able to submit a “Reapplication” to be considered for the next series. n n n Blank boxes Non-users Series V and VI sites have not yet been selected, but one will be in south Mississippi, close to the coast. 34

CAI Modules n n n Each series consists of three days of training, each day separated by about two weeks. Districts are asked to send three-member teams. We suggest a central office person directly involved the district’s plan for student improvement, a school administrator, and a lead teacher/SPMS-user teacher. The same team should attend all sessions. These series are not initial training sessions for users. All team members who attend these sessions should be active SPMS users. 35

CAI Curriculum Objectives 1) Gain a clear understanding of the alignment between SPMS and the Mississippi Curriculum Frameworks 2) Develop a scope and sequence guide to assist with the integration of SPMS for instructional purposes 3) Develop a plan to use SPMS to ensure student growth in mastering Mississippi Curriculum Framework objectives as well as local district curriculum goals 36

CAI Assessment Objectives 1) Create appropriate SPMS assessments for targeted objectives 2) Understand SPMS reports available 3) Utilize SPMS assessment data to inform instruction 4) Establish district priority goals for increasing student achievement 5) Design a district/school strategic plan and program of action with measurable outcomes and an evaluative component 37

CAI Instruction Objectives 1) Use SPMS assessment results to review/critique instructional practices for targeted objectives 2) Explore SPMS reports to refocus instruction on targeted curriculum, including pre-requisite skills 3) Connect SPMS reports to the use of researchbased instructional strategies 4) Examine SPMS reports to create learning opportunities and revise instructional strategies based on individual or group needs 5) Access resources through SPMS to assist in providing quality instruction 38

Under Consideration n For the 2006 -2007 school year, the MDE and Vantage are working to have data available when school begins. The idea is to “promote” all SPMS students at the end of the 2005 -06 school year to the next grade and/or next school. This process would mean districts might have to “clean up” the data, but the process would allow districts access as soon as the 2006 -07 school year begins. 39

SPMS Advisory Committee n n The SPMS Advisory Committee meets quarterly during the school year. The committee makes recommendations and considers new functions. Currently, there are 21 members on this committee. Michael Hood, assistant superintendent from Montgomery County, is a “super” user of SPMS and a member of the Advisory Committee. 40