State of Israel Ministry of Education Culture

State of Israel Ministry of Education Culture & Sport Office of the Director General Division of Evaluation & Measurement GEMS: Growth & Effectiveness Measures for Schools Formative Assessment for Schools

o o Gems is an aid to management for school principal and teachers Gems’ major purpose is to aid school staff in making data-based decisions, in order to use resources wisely and to improve school functioning Gems is a school oriented assessment elucidating the strengths and weaknesses of the school and thus enabling it to allocate resources where most needed Gems perceives the school as an holistic system 2

Learning environmen t Organization Teacher-Teacher relations Staff Teacher-student development relations Achie veme nts Educational plans Curriculum Teacher-Parents relations 3

o o Gems indicators were based on requests by 200 school principals and 200 supervisors when asked what reliable information they need in order to improve school functioning Gems includes the following indicators: n n n Pedagogical Environment o Action Plan o School priorities and staff consensus o Teaching resources & teaching methods o Weak students and the help they get Academic Achievements in 4 basic subjects (Language, Mathematics, Science, English) School Climate and Work Environment 4

o o o GEMS is administered each year in 50% of elementary and junior high schools, so that each school participates in GEMS once every two years GEMS sources of information: n Tests for 5 th and 8 th grades in Language, Math, Science and English n Questionnaires for all students in 5 th through 9 th grades dealing with school climate, pedagogical environment etc. n Interviews with all principals and teachers dealing with work environment, teaching methods, staff development etc. Gems is administered at the beginning of the school year so that schools may get the report during that same year, and shorten the possibly stressful period of anticipation of the assessment 5

o GEMS is school oriented. Its main goal is not to provide a national “state of the union message”, but rather to enable schools themselves to plan rationally and to follow up, over time, the fruits of their efforts o However, the data also serve the staff of the Ministry of Education and its units. Thus, over 20 different aggregate reports are prepared each year to help policy makers in different branches of the Ministry to make data-based decisions 6

o o The rationale of GEMS is that a necessary condition for improving the educational system is to improve the interaction between teacher and students on the class level Such an improvement is depends on schools basing their plans and practices on continuous, accurate, reliable, longitudinal data In order to plan and make data-based decisions schools have to take the following steps: n to set specific goals n to know were they are n to take actions directed to the goals GEMS is a tool facilitating these steps 7

Setting Specific Goals 8

o o Goals have to be specific in order to be measurable and to avoid contradiction between goals For example if the goal is “enhancing academic achievements” assessment must specify at least 4 indicators: n n o Average grade percent of failures percent of distinctions difference in grades between students from deprived SES and those from high SES As shown below these are not necessarily identical: 9

“enhancing academic achievements” -- 1 st measurement School A School B School C Average 60 60 60 S. D 9 9 9 % failure 33% 33% % excellent 0% 0% 0% Goal set for schools: raise average by 10% “enhancing academic achievements” -- 2 nd measurement School A School B School C Average 67 67 67 S. D 23 11 4 % failure 56% 11% 0% % excellent 33% 0% 0% 10

How schools operated to reach the goal? Does assessment distinguish between them? Year 1 3 schools School A Year later School B Year later School C Year later Student 1 70 100 80 75 Student 2 70 95 80 70 Student 3 70 95 80 70 Student 4 60 60 65 65 Student 5 60 50 65 65 Student 6 60 50 60 65 Student 7 50 50 60 65 Student 8 50 50 60 65 Student 9 50 50 50 60 11

“Without data you are just another person with an opinion” Knowing where we are A few examples of data for a specific school 12

School priorities: consensus? School Example o National data o Principal: Achievements Teachers: n language skills n school climate n violence prevention n Math n Arts n school climate n English n Respect for teachers n Keeping school clean n Independent study n Parental participation n “Closing the Gap” n language skills n Don’t know 13

Weak students weak students Teachers: reasons for learning difficulties 14

2004 2002 Students’ Achievements 15

Assessing Change in 2004 In order to evaluate the changes in students' achievements, the following analysis was undertaken: comparing the change in a given school (over the two years) to the change found in a Comparison Group, consisting of schools with same background characteristics and same starting point in 2002. The school may thus draw conclusions such as: o Size of change in CG Size of change in school 9+ Grades 2004 2002 3+ 74 71 Hebrew 8+ 9+ 74 65 Math 8+ 5 - 62 67 Science 6+ 13+ 76 63 English Achievements improved more in the school than in comparable schools o The change in the school is similar to the change in the comparison group o Achievements in the school show less improvement than that in other, similar, schools 16

2004 2002 Percent of students that didn't meet standards 20% to 25% of students have private tutor paid by parents 17

Achievements by SES -- 2004 The gap in achievements between the two groups is ½ to 2/3 standard deviation 18

Teaching methods and Homework -- 2004 o o o 10% of the teachers coordinate amount of homework Students are required to prepare 25 minutes homework for each hour in school Homework is mainly drill of material taught in class 10% of the teachers require learning new material or looking for sources 60% of teaching time is frontal teaching 19

2004 2002 Student perception of teachers’ attitudes 20

2004 2002 Teachers’ perception of the work environment 21

Taking actions to reach goals: Action Plan We are free to choose our actions, . . . but we are not free to choose the consequences of these actions. Stephen R. Covey (1996) 22

To define specific indicators and to diagnose school relative to goals ? indicators derived from previous data What to diagnose? ? indicators derived from school goals 23

To analyze GEMS data on two levels ? CLASS GEMS ? SCHOOL 24

To ask questions o Every school supervisor, principal, teacher should have a dialogue with available data (from GEMS and from school internal evaluations) and to ask himself: n Did I know… n Did I ask myself… n Did I check… 25

To plan according to data An example: what should be done in classes with different composition Class B Class A 26

Prepare school action plan and define (specifically) where am I and where I want to be… 27

Wo r ki n g ev al ua ti n g g n n in Pla 28

Feedback to assessment from 1, 036 principals: 2004 29