Andreas Schleicher American Diploma Project 11 September 2009

Andreas Schleicher American Diploma Project 11 September 2009 International Benchmarking 1 1 International Benchmarking What it means – what it takes Washington, September 11, 2009 Andreas Schleicher Head, Indicators and Analysis Division OECD Directorate for Education

Dimensions for educational benchmarking Domain 1 Domain 2 Domain 3 Outputs and Outcomes Policy Levers Antecedents impact of learning shape educational outcomes Quality and Individ attitudes, distribution of engagement and knowledge & skills behaviour contextualise or constrain ed policy Socio-economic background of learners Level A Individual learner Level B Instructional settings Quality of instructional delivery Level C Schools, other institutions Output and performance of institutions The learning environment at school Community and school characteristics Level D Country or system Social & economic outcomes of education Structures, resource alloc and policies National educ, social and economic context Teaching, learning Student learning, teacher working practices and conditions classroom climate

Cost per student Expenditure per student at tertiary level (USD) A world of change – higher education Graduate supply Tertiary-type A graduation rate

Cost per student Expenditure per student at tertiary level (USD) A world of change – higher education United States Finland Graduate supply Tertiary-type A graduation rate

Expenditure per student at tertiary level (USD) A world of change – higher education Australia United Kingdom Finland Tertiary-type A graduation rate

Expenditure per student at tertiary level (USD) A world of change – higher education Tertiary-type A graduation rate

Expenditure per student at tertiary level (USD) A world of change – higher education United States Australia Finland Tertiary-type A graduation rate

Dimensions for educational benchmarking Domain 1 Domain 2 Domain 3 Outputs and Outcomes Policy Levers Antecedents impact of learning shape educational outcomes Quality and Individ attitudes, distribution of engagement and knowledge & skills behaviour contextualise or constrain ed policy Socio-economic background of learners Level A Individual learner Level B Instructional settings Quality of instructional delivery Level C Schools, other institutions Output and performance of institutions The learning environment at school Community and school characteristics Level D Country or system Social & economic outcomes of education Structures, resource alloc and policies National educ, social and economic context Teaching, learning Student learning, teacher working practices and conditions classroom climate

14 14 Contribution of various factors to instructional cost per high school student Percentage points Education Indicators Programme 2009 edition of Education at a Glance as a percentage of GDP per capita (2006) B 7. 1

Dimensions for educational benchmarking Domain 1 Domain 2 Domain 3 Outputs and Outcomes Policy Levers Antecedents impact of learning shape educational outcomes Quality and Individ attitudes, distribution of engagement and knowledge & skills behaviour contextualise or constrain ed policy Socio-economic background of learners Level A Individual learner Level B Instructional settings Quality of instructional delivery Level C Schools, other institutions Output and performance of institutions The learning environment at school Community and school characteristics Level D Country or system Social & economic outcomes of education Structures, resource alloc and policies National educ, social and economic context Teaching, learning Student learning, teacher working practices and conditions classroom climate

Average performance of 15 -year-olds on PISA in science Andreas Schleicher International Assessment Seminar 11 September 2009 Lessons from PISA for developing assessments 17 17 High science performance … 18 countries perform below this line Low science performance

Andreas Schleicher International Assessment Seminar 11 September 2009 Lessons from PISA for developing assessments 19 19 Variation in the performance of 15 -year-olds in mathematics Consistency in quality standards

Andreas Schleicher International Assessment Seminar 11 September 2009 Lessons from PISA for developing assessments 20 20 Consistency in quality standards Variation in the performance of 15 -year-olds in mathematics Variation of performance within schools Variation of performance between schools OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 4. 1 a, p. 383.

Andreas Schleicher International Assessment Seminar 11 September 2009 Lessons from PISA for developing assessments 21 21 Strengths and weaknesses of countries in science relative to their overall performance France Science competencies Science knowledge OECD (2007), PISA 2006 – Science Competencies for Tomorrow’s World, Figure 2. 13

Andreas Schleicher International Assessment Seminar 11 September 2009 Lessons from PISA for developing assessments 22 22 Strengths and weaknesses of countries in science relative to their overall performance Czech Republic Scientific competencies Scientific knowledge OECD (2007), PISA 2006 – Science Competencies for Tomorrow’s World, Figure 2. 13

Dimensions for educational benchmarking Domain 1 Domain 2 Domain 3 Outputs and Outcomes Policy Levers Antecedents impact of learning shape educational outcomes Quality and Individ attitudes, distribution of engagement and knowledge & skills behaviour contextualise or constrain ed policy Socio-economic background of learners Level A Individual learner Level B Instructional settings Quality of instructional delivery Level C Schools, other institutions Output and performance of institutions The learning environment at school Community and school characteristics Level D Country or system Social & economic outcomes of education Structures, resource alloc and policies National educ, social and economic context Teaching, learning Student learning, teacher working practices and conditions classroom climate

Lessons from PISA for developing assessments 24 A real-time assessment 24 High policy value Quick wins Andreas Schleicher International Assessment Seminar 11 September 2009 Must haves environment that bridges the gap between formative and Examine individual, institutional summative assessment. and systemic factors associated Extending the range of with high performance competencies through which outcomes are assessed Monitor educational progress Measuring growth in learning Low feasibility High feasibility Assuming that every new skill domain is orthogonal to all others Establish the relative standing of states on international standards Money pits Low-hanging fruits Low policy value

Andreas Schleicher American Diploma Project 11 September 2009 International Benchmarking 25 25 What it takes…

Andreas Schleicher American Diploma Project 11 September 2009 International Benchmarking 26 26 OECD’s PISA assessment of the knowledge and skills of 15 -year-olds Coverage of world economy 83% 87% 86% 85% 81% 77% Subnational/regional PISA assessments in l l l l l Country Australia Belgium Brazil Canada Germany Italy Mexico Spain Switzerland United Kingdom Coordinated by National authorities Regions National authorities Regions Reported at National level OECD level National level OECD level

Andreas Schleicher American Diploma Project 11 September 2009 International Benchmarking 27 27 What it takes r Implementing PISA volume of the tests, e. g. – 3½ hours of main assessment area – 1 hour for each of the minor assessment areas each student – 2 hours on paper-and-pencil tasks (subset of all questions) – ½ hour for questionnaire on background, learning habits, learning environment, engagement and motivation school principals – questionnaire (school demography, learning environment) r Alternatives l Suitable if state performance is main interest – Curriculum match / assessment match – Embedding PISA items in state tests – Embedding state items in PISA tests l Requires coherence between assessment frameworks.

Andreas Schleicher American Diploma Project 11 September 2009 International Benchmarking 28 28 Test Items r A unit structure r Authentic stimuli/contexts r High proportion of constructed response items l Multiple-choice l Short constructed responses l Open constructed responses.

Andreas Schleicher American Diploma Project 11 September 2009 International Benchmarking 29 29 Main products r r A set of basic indicators that provide policy makers with a baseline profile of the knowledge, skills and competencies of students in their state relative to those in other countries A set of contextual indicators that provide insight into how such skills relate to important demographic, social, economic and educational variables Trend indicators that become available because of the on-going, cyclical nature of the data collections A knowledge base for further focused policy analysis.

Andreas Schleicher American Diploma Project 11 September 2009 International Benchmarking 30 30 l www. oecd. org; www. pisa. oecd. org – All national and international publications – The complete micro-level database l email: pisa@oecd. org l Andreas. Schleicher@OECD. org Thank you ! … and remember: Without data, you are just another person with an opinion