Milwaukee Mathematics Partnership Program Evaluation MTL Meeting November 6, 7 2006 November 6 -7, 2006
MMP Goals ¡ Comprehensive Math Framework ¡ Distributed Leadership ¡ Teacher Learning Continuum ¡ Student Learning Continuum November 6 -7, 2006 2
MMP Core Partners ¡ University of Wisconsin—Milwaukee ¡ Milwaukee Public Schools ¡ Milwaukee Area Technical College November 6 -7, 2006 3
Evaluation Goals ¡ ¡ Help the MMP better serve its constituents and improve its effectiveness Serve the broader mathematics education community through documentation and dissemination of MMP activities November 6 -7, 2006 4
Evaluation Logic Model Student Achievement Distal Outcomes Proximal Outcomes Teacher Content & Pedagogical Knowledge Classroom Practice Teacher Involvement Learning Team Effort MMP Activities New Courses Math Faculty Involvement MATC Buy-In School Buy-in District Buy-in UWM Buy-In MPA Ownership November 6 -7, 2006 5
2005 -06 Evaluation Activities MMP Online Survey ¡ MTS Survey ¡ Learning Team Observations ¡ Classroom Observations ¡ Assessment of Teacher MKT ¡ Social Network Analysis ¡ MPS Data Mining ¡ November 6 -7, 2006 6
Presentation Overview ¡ Part I: District Wide Analysis ¡ Part II: School Case Studies November 6 -7, 2006 7
Part I: District Wide Indicators Student Achievement Classroom Practice Teacher Involvement Learning Team Effort November 6 -7, 2006 School Buy-in 8
Part I Activities ¡ MMP Survey l ¡ MTS Survey l ¡ Designed to measure how well MTS perceived school to be, in terms of meeting the goals of the MMP MKT Assessment l ¡ Designed to measure differences in the quantity and quality of MMP related activities Designed to assess teachers’ mathematical knowledge for teaching WKCE Tests l November 6 -7, 2006 Designed to assess students’ mathematical content knowledge 9
Respondents Number of MPS Respondents by Role in the MMP Academic Year 2004 -05 2005 -06 Math Teacher Leader 124 140 Learning Team Member & Mathematics Teacher 167 284 LT Member (Administrative) 80 171 Math Teacher Only 676 1340 Literacy Coach 47 94 1094 2029 Total November 6 -7, 2006 10
Research Questions 1. 2. 3. 4. 5. 6. Validity of MMP Survey Change in MMP Impact Characteristics of math-focused schools MMP Impact on student achievement Characteristics of quality learning teams Characteristics of quality MTLs November 6 -7, 2006 11
Research Question #2 Has the perceived impact or focus of the MMP changed since last year? Analytical Approach ¡ Dependent t-tests conducted at the school level for all school level variables obtained in both administrations of the MMP Survey ¡ November 6 -7, 2006 12
Results – Trends in Impact of MMP ¡ This year statistically significant increases l l l November 6 -7, 2006 MTLs reported discussing mathematics with others at their school (t(90) = 12. 06, p <. 001) Teachers reported engaging in activities designed to align their curriculum to standards (t(111) = 8. 53, p <. 001) Teachers reported engaging in activities designed around CABS or student work samples (t(106) = 7. 04, p <. 001) 13
Research Question #3 ¡ What variables characterize schools that are more focused on increasing student achievement in mathematics? Analytical Approach ¡ Stepwise multiple regression November 6 -7, 2006 14
Results – Characteristics of Schools with Greater Math Focus ¡ 68% of variability in schools’ overall selfreported focus on mathematics could be explained by differences in: l l l November 6 -7, 2006 Teachers reported working together to improve content and pedagogical knowledge (b =. 46, t = 6. 7, p <. 001) Teachers reported consistent instructional practices used at their school (b =. 14, t = 2. 4, p =. 018) Teachers perceived the Learning Team to be supportive of efforts to improve math teaching and learning (b =. 38, t = 5. 6, p <. 001) 15
Research Question #4 What variables help to explain differences in the percentage of students classified as proficient in mathematics? Analytical Approach ¡ Stepwise multiple regressions controlling for previous achievement and SES ¡ November 6 -7, 2006 16
Results – Impact of MMP on Increasing Student Achievement ¡ ¡ Schools with a stronger focus on increasing student achievement in mathematics are have a higher percentage of students proficient in mathematics, after controlling for SES and previous achievement (b =. 26, t = 3. 7, p =. 001) An additional 7% of variability in student proficiency rates explained by the addition of this predictor November 6 -7, 2006 17
Research Question #5 What variables characterize Learning Teams that are perceived to be more helpful in terms of increasing student achievement in mathematics? Analytical Approach ¡ Stepwise multiple regression ¡ November 6 -7, 2006 18
Results – Characteristics of Supportive Learning Teams ¡ 64% of variability in schools’ overall perception of the level of support provided by the Learning Team could be explained by differences in: l l l November 6 -7, 2006 Teachers reported working together on improvement activities designed around CABS or student work samples (b =. 41, t = 5. 5, p <. 001) Teachers reported a greater alignment between their school’s adopted curriculum and standards/learning targets (b =. 18, t = 2. 4, p =. 021) Teachers perceived the MTL to be supportive of efforts to improve mathematics teaching and learning (b =. 46, t = 5. 9, p <. 001) 19
Research Question #6 What variables characterize Math Teacher Leaders that are perceived to be more helpful, in terms of increasing student achievement in mathematics? Analytical Approach ¡ Stepwise multiple regression ¡ November 6 -7, 2006 20
Results – Characteristics of Supportive Math Teacher Leaders ¡ 42% of variability in schools’ overall perception of the level of support provided by the MTL be explained by differences in: l l l November 6 -7, 2006 Teachers reported working together on improvement activities designed around CABS or student work samples (b =. 38, t = 4. 5, p <. 001) Teachers reported a greater alignment between their school’s adopted curriculum and standards/learning targets (b =. 26, t = 3. 0, p =. 004) MTLs perceived themselves as being supported by others at their school (b =. 27, t = 3. 2, p =. 002) 21
Conclusions ¡ MTSs in general have a strong sense of what is going on with school leadership, but less awareness about activity at the classroom level. ¡ MMP efforts are being felt beyond the learning team and MTL by school staff ¡ MMP activities are helping schools become more focused on increasing student achievement in mathematics ¡ Schools that are more focused have increased the proportion of students proficient in mathematics ¡ Adoption of MMP-related principles is reported to be related to supportive learning teams and to supportive Math Teacher Leaders November 6 -7, 2006 22
Part II: Case Study Schools Student Achievement Teacher Content & Pedagogical Knowledge Classroom Practice Teacher Involvement Learning Team Effort November 6 -7, 2006 Collaboration School Buy-in 23
Eleven Case Study Schools ¡ Schools were diverse in terms of l l l November 6 -7, 2006 Type Geography Student demographics 24
Case Study Data Collection ¡ 22 learning team observations— 2 in each school ¡ 44 classroom observations— 4 in each school; 2 teachers observed 2 times each ¡ MKT Assessment for math teachers ¡ SNA Survey for math education ‘stakeholders’ November 6 -7, 2006 25
Results of Learning Team Observations Team Functioning MMP Issues Leadership Participation Organization & Structure Results Overall Functioning Math Vision Integration Math Leadership MMP Work Overall MMP Overall, strongest areas were participation & mathematics leadership Biggest areas for improvement were math vision & results November 6 -7, 2006 26
Characteristics of Hi-Lo Scoring Learning Teams—Team Functioning Hi ¡ ¡ ¡ ¡ Lo Distributed leadership Positional authority is less important Multiple views are represented and heard Multiple segments of the school are represented Written agenda, note taker, facilitator Explicit action items Participants have hi knowledge and skill levels November 6 -7, 2006 ¡ ¡ ¡ Principal does all the talking A few individuals dominate the discussion No agenda or team is easily distracted from the agenda Little follow-through on assignments No clear action items 27
Characteristics of Hi-Lo Scoring Learning Teams—MMP Issues Hi ¡ ¡ ¡ Lo Consistent curriculum Math is addressed alongside and in combination with other subjects Coherent within grades and (at times) across grades MTL clearly in charge with respect to math Attention to CABS; reference to MMP courses; reviewing student work November 6 -7, 2006 ¡ ¡ ¡ Variation in curriculum Math not addressed at the meeting No clear math leader—i. e. , hard to tell who the MTL is Confusion about the MMP and CMF Too much non-academic housekeeping School climate is the priority 28
Results of Classroom Observations General Practice Articulating Math Task Formative Assessment Overall Comprehensive Math Framework Understanding Computing Application Reasoning Engagement Overall, strongest areas were articulating the math task & understanding Biggest areas for improvement were use of formative assessment & engagement November 6 -7, 2006 29
Characteristics of Hi-Lo Scoring Classroom Performance—General Hi ¡ ¡ ¡ Lo Math task within the lesson was easy to identify Math task was discrete and level-appropriate Encouraging selfassessment and peerassessment Establish criteria for proficiency Promoting problem solving and independent thinking November 6 -7, 2006 ¡ ¡ Math task was to complex or obscure Only feedback provided was if answer was correct Little teacher involvement in the lesson Feedback focuses on student behavior 30
Characteristics of Hi-Lo Scoring Classroom Performance—CMF Hi ¡ ¡ ¡ Lo Student explanations sought Computation is presented as a means to an end Problem solving was emphasized Students had to justify solutions Lessons are made relevant by using everyday things like money or time and seeking examples from students’ lives November 6 -7, 2006 ¡ ¡ ¡ Close ended questions are emphasized Only one way to solve problems presented Minimal time allowed to share solutions Students not accountable for responding to questions Problems not presented in context 31
Results of MKT Assessment 43 item assessment addressed 3 content areas: Number & Operations & Algebra & Geometry Overall Score November 6 -7, 2006 32
Results of MKT Assessment Average IRT Scores Number & n Operations 12 -0. 78 -0. 60 -0. 65 -0. 59 32 0. 12 0. 31 0. 20 0. 22 Mean -0. 36 -0. 13 -0. 32 -0. 31 Median -0. 38 -0. 09 -0. 39 -0. 44 0. 70 0. 77 0. 79 0. 75 Lo HI SD November 6 -7, 2006 Algebra Geometry Total 33
Social Network Analysis ¡ ¡ Math stakeholders in each school were asked to name individuals with whom the communicated about mathematics Statistical analysis focused on 1. 2. November 6 -7, 2006 Network and in-school density Importance of MTL and MTS 34
Overall SNA Results Metric n Total Named Network density Density in school MTL Role MTS Role --In Degree Lo 13 31 3. 9% 7. 6% 3. 33 1. 14 Hi 32 91 11. 4% 31. 1% 28. 24 18. 52 21. 1 54. 0 6. 7% 17. 6% 13. 81 5. 31 SD 6. 8 17. 6 2. 6% 9. 6% 7. 2 4. 9 Median 19 48 6. 2% 15. 4% 13. 07 3. 75 Mean Density—a perfect score is 100% where everyone names everyone else In-Degree scores are relative measures November 6 -7, 2006 35
Example Network November 6 -7, 2006 36
Example Network November 6 -7, 2006 37
Report Card Indicators ¡ 19 indicators in 7 domains based on inschool data collection, online surveys, and MPS data 1. 2. 3. 4. 5. 6. 7. November 6 -7, 2006 MTS Assessment Collaboration Learning Teams Classroom Practice Professional Development Teacher MKT Student Achievement 38
Report Card Results WKCE Mean % Proficient = 44% Teacher Content & Pedagogical Knowledge Overall IRT = -0. 34 Algebra IRT = -0. 18 Student Achievement November 6 -7, 2006 PD Hrs. = 17. 8 Facilitation Hrs. = 1. 0 PD Quality = 3. 1 Classroom Practice Teacher Involvement Learning Team Effort Team Functioning = 3. 5 MMP Principles = 3. 6 LT Quality = 3. 1 Overall rating = 3. 5 Gap MTL v. other teacher =. 2 Teacher Engagement = 3. 2 MTS Assessment = 38. 3 of 55 School Buy-in Network density = 6. 7% / School density = 17. 6% MTL Role = 13. 8 / MTS Role = 5. 3 SR MTL Engagement = 4. 4 / MTS Quality = 3. 0 39
Student Achievement & In-School Network Density November 6 -7, 2006 40
Student Achievement & Learning Team MMP November 6 -7, 2006 41
Student Achievement & Professional Development November 6 -7, 2006 42
Conclusions ¡ ¡ ¡ MMP is advancing concepts, ideas, & principles that can help schools improve student achievement results in math. Schools that score well with regards to MMP-related metrics have higher student achievement. Learning team adoption of MMP ideas and dense in-school communication networks are predictors of high student achievement November 6 -7, 2006 43
Conclusions ¡ At the same time… l l ¡ Some MPS schools are lagging behind in terms of adopting MMP ideas. These schools perform do not score as well on MMP metrics, which is consistent with student achievement results. We know that other factors—prior year student achievement and SES —are stronger predictors November 6 -7, 2006 44
Evaluation Next Steps District Wide Analysis ¡ ¡ Continue online survey & data mining Improve ability to link student and teacher data working with MPS Case Study Schools 1. 2. 3. 4. 5. 6. Recruit case study schools Nov Plan observations Nov-Dec Observations Round 1 Jan-Feb Observations Round 2 March-April MKT Assessment May SNA Survey May November 6 -7, 2006 45
Скачать презентацию Milwaukee Mathematics Partnership Program Evaluation MTL Meeting November
b5a7ddb4496d93c3292ee69533301719.ppt