1 1 Organisation for Economic Cooperation and Development

1 1 Organisation for Economic Cooperation and Development (OECD) Seeing schools through the prism of PISA An international comparative perspective on quality and equity in education systems Tokyo 24 June 2005 Andreas Schleicher Head, Indicators and Analysis Division OECD Directorate for Education

2 2

3 3 In the dark, all education systems look the same… 暗がりのなかでは、どの学校も教育システムも同じように見える… だが、少し光を当てると…. But with a little light….

4 4 だが、少し光を当てると…. But with a little light…. …important differences become apparent… …重要な違いが明らかになってくる….

5 5 High mathematics performance Average performance of 15 -year-olds in mathematics Low mathematics performance

6 6 Overview 1. The PISA approach l Measuring the quality of learning outcomes internationally 2. Where we are today l What PISA shows students in different countries can do with what they have learned 3. Where we can be • Examples from the best performing countries 4. How we can get there l Some policy levers that emerge from international comparisons

7 The PISA approach Measuring the quality of learning outcomes

8 8 Three broad categories of key competencies Using “tools” interactively to engage with the world To analyse, compare, contrast, and e. g. evaluate language, symbols and texts Using To think imaginatively Interacting with information Capitalising PISA concept of on the potential literacy of technologies Acting Interacting in Accessing, managing, integrating autonomously diverse groups and evaluating written information e. g. in order to develop ones knowledge andwithin the bigger picture Acting potential, Relating well to others and to participate in, and contribute to, society Co-operating, working in teams Learning strategies Taking responsibility and Managing and resolving conflicts To apply knowledge in real-life situations understanding rights and limits To communicate thoughts and ideas effectively

9 9 Using “tools” interactively to engage with the world To analyse, compare, contrast, and e. g. evaluate language, symbols and texts Using To think imaginatively Interacting with information Capitalising on the Reading literacy potential of technologies Acting Interacting in Using, diverse groups and reflecting interpreting autonomously e. g. on written material e. g. Acting within the bigger picture Relating well to others Co-operating, working in Forming and conducting life plans teams Taking responsibility and Managing and resolving conflicts To apply knowledge in real-life situations understanding rights and limits To communicate thoughts and ideas effectively

10 10 Using “tools” interactively to engage with the world To analyse, compare, contrast, and e. g. evaluate language, symbols and texts Using To think imaginatively Interacting with information Capitalising on the potential Scientificof technologies literacy Acting Interacting in Using scientific knowledge, identifying scientific autonomously diverse groups questions, and drawing evidence-based conclusions to e. g. Acting within the bigger picture understand make well to others Relating decisions about the natural world Co-operating, working in Forming and conducting life plans teams Taking responsibility and Managing and resolving conflicts To apply knowledge in real-life situations understanding rights and limits To communicate thoughts and ideas effectively

11 11 Using “tools” interactively to engage with the world To analyse, compare, contrast, and e. g. evaluate language, symbols and texts Using To think imaginatively Interacting with information Capitalising on the potential Mathematical technologies literacy of Acting Interacting in Emphasis is on mathematical knowledge put into autonomously diverse groups functional use in a multitude of different e. g. situations e. g. Acting within the bigger in varied, Relating well to others reflective and insight-based ways picture Co-operating, working in Forming and conducting life plans teams Taking responsibility and Managing and resolving conflicts To apply knowledge in real-life situations understanding rights and limits To communicate thoughts and ideas effectively

12 12 Mathematical literacy in PISA The real world The mathematical World Making the problem amenable to mathematical treatment A model of reality Understanding, structuring and simplifying the situation A mathematical model Using relevant mathematical tools to solve the problem A real situation Validating the results Mathematical results Real results Interpreting the mathematical results

13 13 Deciding what to assess. . . looking back at what students were expected to have learned …or… looking ahead to what they can do with what they have learned. For PISA, the OECD countries chose the latter.

14 14 Development of assessments r r Frameworks by international experts Assessment materials l l l r submitted by countries developed by research consortium screened for cultural bias – by countries – by expert, international panel – items with prima facie cultural bias removed at this stage internationally validated translations trialled to check items working consistently in all countries Final tests l l items shown in trial to be culturally biased removed best items chosen for final tests – balanced to reflect framework – range of difficulties – range of item types (constructed response, multiple choice)

15 15 Key features of PISA 2003 r Information collected l volume of questions – 3½ hours of mathematics assessment – 1 hour for each of reading, science and problem solving l each student – 2 hours on paper-and-pencil tasks (subset of all questions) – ½ hour for questionnaire on background, learning habits, learning environment, engagement and motivation l school principals – questionnaire (school demography, learning environment quality) r Coverage l PISA covers roughly nine tens of the world economy

16 Where we are - and where we can be What PISA shows students can do Examples of the best performing countries

17 17 PISA provides five key benchmarks for the quality of education systems 1. Overall performance of education systems 2. Equity in the distribution of learning opportunities l Measured by the impact students’ and schools’ socio-economic background has on performance… … not merely by the distribution of learning outcomes 3. Consistency of performance standards across schools 4. Gender differences 5. Foundations for lifelong learning l Learning strategies, motivation and attitudes

18 18 High mathematics performance Average performance High average performance of 15 -year-olds in High social equity mathematics High average performance Top-performers Large socio-economic disparities l Finland remained first in reading and since 2000 moved further in math and science… … and is now on a par with the East Asian countries that were previously unmatched in math and science Strong sociol Also the Netherlands is economic impact on among the top-performers in student Progress performance l l math Other countriesreading and … though not in with improvements in at least two science. assessment areas were As is the Flemish Community Belgium, the of Belgium Czech Republic and Germany … In Belgium and Germany it was theaverage performance Low top performers who drove improvements. Large socio-economic disparities A widening gap Socially equitable More improvement at the top distribution of of the scale has widened the learning opportunities gap between the top and bottom performers in the OECD. l Progress l Poland raised it’s overall performance in all four assessment areas … thanks performance Low averageto big improvements among lower-performing High social equity students in the wake of a major reform in 1999. Low mathematics performance

19 19 High mathematics performance Durchschnittliche High average performance Schülerleistungen im Differences in socio-economic Large socio-economic disparities High social equity Bereich Mathematik background pose major challenges for education systems Students whose parents have better-paid jobs, are better educated or have more “cultural” possessions in their homes tend to perform better… Strong socio… But the performance advantage varies l Socially equitable distribution of learning opportunities economic impact on – Australia, Canada, student performance Finland, Iceland Japan provide examples showing that it is possible to combine quality and equity – In contrast, results for Belgium, Germany, Hungary and the Slovak Republic reveal large socio-economic inequalities in the distribution of learning opportunities. Low average performance Large socio-economic disparities High social equity Low mathematics performance

20 20 Is it all innate ability? Variation in student performance OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 4. 1 a, p. 383.

21 21 Is it all innate ability? Variation in student performance in mathematics In some countries, parents can rely on high and consistent standards across schools Variation of performance within l In Canada, schools Denmark, Finland, Iceland Sweden average student performance is high… … and largely unrelated to the individual schools in which students are enrolled. Variation of performance between schools 5 12 14 1 11 OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 4. 1 a, p. 383.

School performance and schools’ socioeconomic background - Germany 22 22 Student performance and student SES within schools Student performance School performance and school SES Student performance and student SES School proportional to size Disadvantage PISA Index of social background Advantage

23 23 学校の成績と学校の社会経済的背景（日本） 生徒の成績と生徒の社会経済的地位（SES） 生徒の成績と学校内における生徒のSES 学校の成績と学校のSES 生徒の成績 学校の規模 不利 社会的背景に関するPISA指数 有利

School performance and schools’ socioeconomic background - Norway 24 24 OECD Student performance and student SES within schools School performance and school SES Student performance and student SES School proportional to size Disadvantage PISA Index of social background Advantage

School performance and schools’ socioeconomic background - Finland 25 25 Student performance and student SES within schools School performance and school SES School proportional to size Disadvantage PISA Index of social background Advantage

School performance and schools’ socioeconomic background 27 27 Universal policies Increasing educational performance of all children through reforms applied equally across the school system, e. g. Student performance l School proportional to size – Altering content or pace of curriculum – Improving instructional techniques – Changing the learning environment in schools and classrooms – Standards and accountability – Teacher professional development Disadvantage PISA Index of social background Advantage

School performance and schools’ socioeconomic background 28 28 Socio-economically targeted policies Providing a specialised curriculum or additional educational resources to students from disadvantaged backgrounds Student performance l School proportional size – Students are often alsotoidentified through other risk factors, e. g. immigration, ethnicity, lowincome community Disadvantage PISA Index of social background Advantage

School performance and schools’ socioeconomic background 29 29 Compensatory policies Providing additional economic resources to students from disadvantaged backgrounds – Different to socio-economically targeted policies, efforts are directed toto size School proportional ameliorating economic circumstances, rather than providing specialised curriculum or additional educational resources Student performance l Disadvantage PISA Index of social background Advantage

School performance and schools’ socioeconomic background 30 30 Performance targeted policies Student performance l Providing additional economic resources to students based on their academic performance – Early intervention programmes – Remedial and recovery programmes School proportional to size – Performance-based tracking or streaming l l Countries with flat gradients In combination with SES-targeted policies for countries with steeper gradients Disadvantage PISA Index of social background Advantage

31 Gender differences r In reading, girls are far ahead l r In all countries, girls significantly outperform boys in reading l In most countries, boys outperform girls l Strong problem-solving performance for girls suggests… In mathematics, boys tend to be somewhat ahead l … but mostly by modest amounts… … and mainly because boys are overrepresented among topperformers while boys and girls tend to be equally represented in the “at risk” group – Within classrooms and schools, the gender gap is often larger … that it is not the cognitive processes underlying mathematics that give boys an advantage… … but the context in which mathematics appears in school Gender differences in interest and attitudes towards mathematics are significantly greater than the observed performance gap – Girls report much lower intrinsic (though not instrumental) motivation in mathematics, more negative attitudes and much greater anxiety with mathematics… … and this may well contribute to the significant gender difference in educational and occupational pathways in mathematics-related subjects

32 Interest in and enjoyment of mathematics OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 3. 4, p. 367 and Figure 3. 4, p. 126.

33 Instrumental motivation in mathematics OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 3. 2 a, p. 360 and Figure 3. 3 a, p. 122.

34 Anxiety in mathematics OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 3. 8, p. 374 and Figure 3. 8, p. 139.

35 Attitudes towards school OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 3. 4, p. 367 and Figure 3. 4, p. 126.

High Mathematics performance 36 High performance Low social equity Strong impact of social background on performance High performance High social equity Moderate impact of social background on performance Low performance Student anxiety in Low social equity mathematics High degree of anxiety High social equity Low degree of anxiety Low Performance

37 How can we get there? Levers for policy that emerge from international comparisons

38 38 OECD framework Domain 1 Domain 2 Domain 3 Outputs and Outcomes Policy Levers Antecedents impact of learning shape educational outcomes Quality and Individ attitudes, distribution of engagement and knowledge & skills behaviour contextualise or constrain ed policy Socio-economic background of learners Level A Individual learner Level B Instructional settings Quality of instructional delivery Level C Schools, other institutions Output and performance of institutions The learning environment at school Community and school characteristics Level D Country or system Social & economic outcomes of education Structures, resource alloc and policies National educ, social and economic context Teaching, learning Student learning, teacher working practices and conditions classroom climate

39 39 Money matters but other things do too Performance in mathematics Korea Czech republic Ireland Slovak republic Poland Finland Japan Netherlands Belgium Canada Australia Iceland Sweden Germany France Hungary Spain Portugal Switzerland Denmark Austria Norway United States Italy Spending per student is positively associated with average student performance… … but not a guarantee for high outcomes r Greece Mexico Australia, Belgium, Canada, the Czech Republic, Finland, Japan, Korea and the Netherlands do well R = 0. 28 in terms of “value for money”… … while some of the big spenders perform Cumulative expenditure (US$) below-average l 2

40 40 Sympathy doesn’t raise standards – aspiration does r In the focus countries l r National research teams report a strong “culture of performance” – Which drives students, parents, teachers and the educational administration to high performance standards PISA suggests… … that students and schools perform better in a climate characterised by high expectations and the readiness to invest effort, the enjoyment of learning, a strong disciplinary climate, and good teacher-student relations – Among these aspects, students’ perception of teacher-student relations and classroom disciplinary climate display the strongest relationships

41 41 Governance of the school system r Monitoring and equity-related goals r l Diverging views how evaluation and assessment can and In the focus countries r l should be used Decentralised decision-making is combined with Standard setting and equity-related goals – Some see them primarily as tools to reveal best practices and devices to ensure a fair distribution of l Key objectives: identify shared problems in order encourage substantive aspirations, establishto supportive teachers and educational opportunities and productive – Raise educational transparency schools to improve and develop more over learning environments reference framework educational objectives, l The provision of standards and curricula at for teachersextend their purpose to support contestability of – Others national/subnational levels is combined with public services or definition of broad l Approaches range from market-mechanisms in the allocation of advanced evaluation systems resources l l educational goals up to formulation of concise – That are implemented by professional agencies – e. g. by making comparative results of schools publicly available to performance expectations facilitate parental choice or by having funds following students l Some. Process-oriented assessments and/or countries go beyond establishing educational l Differences in type of performance benchmarks being used standards as mere yardsticks and use performance centralised final examinations are complimented and reported for the various stakeholders involved, benchmarks that students at particular age or with individual reports and feed-back including parents, teachers and schools grade levels should reach mechanisms on student learning progress Instruments – Minimum standards, targets defining excellence, normative performance benchmarks

42 42 High mathematics performance High average performance Large socio-economic disparities Durchschnittliche High average performance Schülerleistungen im High social equity Bereich Mathematik Strong socioeconomic impact on student performance Socially equitable distribution of learning opportunities Low average performance Large socio-economic disparities High social equity Low mathematics performance

High mathematics performance 43 43 Durchschnittliche Schülerleistungen im Bereich Mathematik Strong socioeconomic impact on student performance School with responsibility for deciding which courses are offered High degree of autonomy Low mathematics performance Socially equitable distribution of learning opportunities

44 44 Public and private schools Private schools perform better Public schools perform better

45 45 Organisation of instruction r In the focus countries l Schools and teachers have explicit strategies and approaches for teaching heterogeneous groups of learners – A high degree of individualised learning processes – Disparities related to socio-economic factors and migration are recognised as major challenges l l Students are offered a variety of extra-curricular activities Schools offer differentiated support structures for students – E. g. school psychologists or career counsellors l Institutional differentiation is introduced, if at all, at later stages – Integrated approaches also contributed to reducing the impact of students socio-economic background on outcomes

High Mathematics performance 46 46 High performance Low social equity High social equity Moderate impact of social background on performance Strong impact of social background on performance Low social equity High social equity Early selection and stratification High degree of stratification Low Performance

47 47 Support systems and professional teacher development r In the focus countries l l l Effective support systems are located at individual school level or in specialised support institutions Teacher training schemes are selective The training of pre-school personnel is closely integrated with the professional development of teachers Continuing professional development is a constitutive part of the system Special attention is paid to the professional development of school management personnel

48 48 Teacher support in mathematics Students’ views OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 5. 1 a, p. 403 and Figure 5. 1, p. 213.

49 49 Student-related factors affecting school climate Principals’ views OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 5. 2 a, p. 406 and Figure 5. 2, p. 216.

50 50 Teacher-related factors affecting school climate Principals’ views OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 5. 4 a, p. 410 and Figure 5. 4, p. 220.

51 51 Pre-school attendance and performance 38 score points is the average performance difference associated with one school year Percentage of students who attended pre-school Difference in performance between those who attended pre-school for more than one year and those with no pre-school

52 52 Creating a knowledge-rich profession in which schools and teachers have the authority to act, the necessary knowledge to do so wisely, and access to effective support systems The future of education systems needs to be “knowledge rich” Informed professional judgement, the teacher as a “knowledge worker” Informed prescription National prescription Professional judgement Uninformed prescription, teachers implement curricula Uninformed professional judgement The tradition of education systems has been “knowledge poor”

53 PISA – Participating Countries OECD countries participating from PISA 2000 OECD countries participating from PISA from 2003 OECD partner countries participating from PISA 2000 OECD partner countries participating from PISA 2003 OECD partner countries participating from PISA 2006

54 54 Further information l www. pisa. oecd. org – All national and international publications – The complete micro-level database l email: pisa@oecd. org l Andreas. Schleicher@OECD. org … and remember: Without data, you are just another person with an opinion