Benő Csapó http www staff u-szeged hu csapo Current

Benő Csapó http: //www. staff. u-szeged. hu/~csapo/ Current Trends in Educational Assessment in Hungary: Research and Development Prague, 26 November 2014 UNIVERSITY OF SZEGED INSTITUTE OF EDUCATION

Outline • Current trends in educational assessment • Educational assessment in Hungary • Developing an online diagnostic assessments system – a large-scale project at the University of Szeged • Further plans to develop the educational assessment in Hungary

Function of assessment: providing feedback “Measure what is measurable, and make measurable what is not so. ” Galileo Galilei “If you can not measure it, you can not improve it. ” Kelvin

Current trends in educational assessment

What is new? (1) New constructs (2) New contexts (3) New instruments

(1) Assessment of new constructs “ 21 st Century Skills”

(2) Assessments in new context Assessment for learning • Formative and diagnostic assessment • Embedded in learning and teaching • Based on a better understanding of learning and development • Detailed, frequent • Provides immediate feedback • Helps to determine the directions of intervention

(3) New Instruments Technology-Based Assessment • • • Innovative item formats, improved validity Improved objectivity and reliability Improved precision Automated scoring Immediate feedback

TBA in PISA • 2006, major domain: science – Technology-Based Assessment of Science – international option • 2009, major domain: reading – Electronic Reading Assessment – international option • 2012, major domain: mathematics – Computer-Based Problem Solving – innovative domain • 2015, major domain: science – Computer-based assessment at all domain – Collaborative problem solving as innovative domain

USA: Assessment in the “Race to the Top” • “Adopting standards and assessments that prepare students to succeed in college and the workplace and to compete in the global economy” • “Building data systems that measure student growth and success, and inform teachers and principals about how they can improve instruction”

Characteristics of the assessment projects in “Race to the Top” • Two grant categories – Comprehensive Assessment Systems – High School Course Assessment • Coalition of states • Two large coalitions received approximately $330 million • Partnership for Assessment of Readiness for College and Careers (PARCC) $170 million • SMARTER Balanced Assessment Consortium (SBAC) $160 million

Educational assessment in Hungary

History and context: Two research and development communities • Assessment centers at Budapest – National Educational Institute (-1990) • Hungary participated in the first IEA project in the 1970 s and 1980 s – After 1990: Several reorganizations – Today: Educational Authority • International and national assessment programs • Research and development centers at the University of Szeged – Test development projects from the 1970 s – Several research groups established in the 1980 s and 1990 s – Today • Research Group on the Development of Competencies (1996 -) – Hungarian Educational Longitudinal Program • Center for Research on Learning and Instruction – Developing Diagnostic Assessments – Training of educational assessment experts (1991 -)

Educational Assessments in Hungary PIRLS Grade TIMSS PISA Diagnostic Assessment System Diagnostic Assessments Assessment of Basic Competencies Matura (Abitur)

Characteristics of the Assessment of Basic Competences • Reading and mathematics are the assessment domains • Every student is measured at grade 6, 8 and 10 • The data are longitudinally connected • Sophisticated value-added models are used • School-level results are published (‘soft accountability’)

Developing Diagnostic Assessments A project implemented by the Center for Research on Learning and Instruction University of Szeged Two phases: 2009 -2011, 2012 -2014 edia. hu

http: //edia. hu English

http: //edia. hu/? q=en Publications / Books Demo test

Online Diagnostic Assessment System • A complex system for supporting learning and teaching • Based on the analysis of learning and developmental processes, and the outcomes of learning See -> framework development • An online accessible assessment platform See -> e. Dia • An item bank containing thousands of items in innovative formats See -> demo test for item formats • Teacher training preparing the teachers for the utilization of the system • Developing intervention materials (e. g. computer games) for enhancing students’ skills if needed

Devising assessment frameworks for reading, mathematics and science for the first six grades

How an assessment framework looks like? PISA TIMSS - application (literacy) - content - reasoning - application Diagnostic - content - reasoning - application

A three-dimensional model for framework development GENERAL ABILITIES Internal, psychological There may be unexploited potentials here LITERACY External, social expectations Disciplinary content CONTENT KNOWLEDGE – EXPERTISE Curriculum-based assessment, 1 st and 2 nd IEA Science and Mathematics Study PISA, three main literacy domains

Devising assessment frameworks: working methods and products • Broad national and international cooperation • Results published in 2 x 3 volumes (in English and in Hungarian) • Opening theoretical chapters (international co-authors) • Detailed standards by Hungarian expert groups

Reading Editors Benő Csapó and Valéria Csépe Pyschology Literacy Discipline Leo Blomert Wolfgang Schnotz Krisztián Józsa Valéria Csépe Edit Katalin Molnár János Steklács

Mathematics Editors Benő Csapó and Mária B. Szendrei Reasoning Literacy Discipline Terezinha Nunes Lieven Verschaffel Mária B. Szendrei Benő Csapó Csaba Csíkos Júlia Szendrei

Science Editors Benő Csapó and Gábor Szabó Reasoning Literacy Discipline Philip Adey Erzsébet Korom Gábor Szabó Benő Csapó Mária B. Németh Erzsébet Korom

Result of the Framework Development Three volumes, published both in Hungarian and in English May be downloaded in. pdf

The e. Dia platform • The system runs at the servers of the University of Szeged • Item builder module • Item bank unit • Test delivery module • Data bank • Data analysis module • Feedback module

The item development process • Training the item writers (350+) – assessment theory and item writing – content of the assessment (frameworks) – ICT and the e. Dia platform • • • Item writing Reviewing by the group leader Supervising by the „dimension chief” Piloting Determining the item parameters

Innovative item formats • Text and voice for the grades 1 -3 • Stimulus – text, voice – picture, animation – video – simulation • Response capture – everything what is possible on paper, keyboard, mouse – manipulating and moving objects on the screen, drag and drop – touchscreen solutions

Characteristics of the Online Diagnostic Assessment System • Serving ca. 600 000 students (grades 1 -6) • Communicating with ca. 20 000 teachers • Making the service available in ca. 3 000 schools • Managing ca. 10 000 test items • Maintaining and analyzing students’ data • Providing both students and teachers with sophisticated feedback in user friendly format

Assessments of the major domains • Reading, mathematics and science are the major domains • Each major domain is measured in 3 dimensions in different scales • Large item banks are used to measure the major domains

Assessment of minor domains • There are 16 minor domains in the system – school readiness – writing skills, musical abilities, English as a second language – health literacy, financial literacy, visual skills, civic competencies – combinatorial reasoning, inductive reasoning – problem solving, learning to learn, ICT literacy, creativity – social skills, motivation • Test batteries are prepared for the minor domains

Plans for the next phase: 2015 • Online developmental exercises for helping slow students to catch up • Gamification • Visualization • Integration with the regular instructional processes • Extending to the kindergarten population

The significance of early education The economists’ view (The Heckman curve) The Basic skills developmental psychologists’ view: Interest Solid foundation for later Attitudes Motivation learning

Can online assessments be used in kindergarten?

YES!

Yes!

Further plans to develop educational assessment in Hungary

Digitizing the Assessments of Basic Competencies • Present problems – only reading and mathematics is assessed – expensive – slow feedback – difficult to interpret • Plans for TBA – adding science – developing item banks – immediate feedback – visualizing the data to help the interpretation

Technology-based reform of the ‘matura’ • Present problems – too many elective subjects – low objectivity – no comparable scales – etc. • Plans for the reform – technology based assessments only – 5 compulsory subjects: Hungarian language and literature, history, mathematics, science, English (no electives)

Conclusions • Technology-based assessments work well in kindergarten and primary schools • Technology can be extended to other domains and age groups, the precision, objectivity and reliability can be improved • Further work is needed to integrate the assessments into the regular classroom activities and educational processes

Thanks for your attention! www. staff. u-szeged. hu/~csapo