Understanding the work of the HKEAA Quality Assurance

Understanding the work of the HKEAA: Quality Assurance C. S. Tong Secretary General 7 June 2013 1

About HKEAA l l l Established in 1977 (independent statutory body) 400 staff in 7 office locations 8, 000 exam personnel (seasonal) Mission: To provide valid, reliable and equitable examinations and assessment services in a professional, innovative, efficient and effective manner.

Quality Assurance

How are examination papers set? l l Curriculum and Assessment (C&A) Guides Subject Committees to oversee the continual development of each subject l l Moderation committees to set papers with rigorous security and quality control mechanism Moderation committee members are appointed on an annual basis subject to various security requirements

Who are Moderation Committee members? Subject Expert/ University Academics HKEAA Secondary Subject School Managers Teachers Open nomination for membership from schools as well as from the relevant Subject Committee (current S 5 or S 6 teachers not eligible)

How are scripts handled? l All scripts are scanned and then archived for safe-keeping: over 10, 000 pages scanned for Onscreen Marking 6

How are Written Papers marked? Appointment of Markers Sample scripts selection Allocation of scripts to markers Markers’ meeting Standardisation meeting Marking Checkmarking Marking scheme Statistical analysis of marks

How are Written Papers marked? OSM facilitates real time, continuous monitoring of the quality of the marking: l l Marking by questions to enhance consistency in applying the marking criteria Double marking for open-ended questions Use of control scripts to validate marking consistency Scripts are selected for checkmarking by experienced Chief/Assistant Examiners 8

Where are the Assessment Centres? 0 About 1, 40 ns workstatio Sheung Shui: 185 Sha Tin Wai: 92 Tai Hing: 100 San Po Kong: 73 Tsuen Wan: 163 Lai King: 376 Tseung Kwan O: 21 Fortress Hill: 382

Facilities at the Assessment Centres Reception Area Workstations Lounge/Pantry Control Room Conference Room 10

OSM Procedures Markers’ meeting 1 st checkmarking satisfactory marking 2 nd checkmarking satisfactory marking Complete Training scripts Qualifying scripts Reference scripts Normal scripts Control scripts

What are the advantages of OSM? Security l Data availability l Efficiency l Accuracy l Marking quality l

How does the Authority monitor Quality of Marking? l l l Markers’ meeting to clarify and finalize marking scheme Provision of Reference Scripts to illustrate the marking requirements and standard Training Scripts & Qualifying Scripts to ensure quality of marking before markers start marking ‘real’ scripts

Monitoring Quality of Marking l l Selected scripts (including control scripts etc) have been discussed by a team including the Chief Examiner and awarded standardized scores Control scripts are randomly distributed to markers to ensure consistency of marking standards

Monitoring Quality of Marking l l Marker’s score can deviate from the standardized score to within a specified tolerance When the deviation exceeds the tolerance level, the marker will be contacted for clarification of marking scheme/ further training and qualifying/ or stopped from further marking and problematic scripts can be voided and redistributed to other markers for marking as necessary

Illustration Marker’s score agrees with Standardized Score Marker’s Score Tolerance level Standardized Score

Illustration Marker’s score agrees with Standardized Score Marker’s score much lower than standardized score Marker’s score much higher than standardized score Tolerance level Standardized Score

Illustration Marker’s score agrees with Standardized Score Marker’s score much lower than standardized score Marker’s score much higher than standardized score Tolerance level Standardized Score

Monitoring Quality of Marking Chief / Assistant Examiners will l check each marker’s marking statistics (including marking speed, mark distribution, score deviation from control scripts etc) l scrutinize markers’ marking records and select scripts for checkmarking l contact markers and further clarify the marking scheme if necessary 19

Why Double Marking? l l To address the issue of inter-rater variability for open ended questions, double marking may be adopted Feasibility study carried out to assess the cost benefit between such factors as l l Marking reliability Markers’ standardization Increased marking time Marking cost

Double Marking: how? Big discrepancy between the two marks The discrepancy among marks is still out of range Marker 3 Marker 1 Marker 2 Chief Examiner or Assistant Examiner Candidate’s mark Discrepancy within range Candidate’s mark The Candidate’s final mark will be the sum of the two highest closest marks

How to ensure overall Quality? Post examination review HKDSE Recognition Compliance & Quality of Delivery }External vetting of examination papers }Post examination review }Internal audit & risk monitoring }Examination Paper Quality Audit (EPQA) }Internal post-mortem meetings Processing & reporting of results }SBA }Marking process }Mechanisms for handling irregularities, complaints and queries }Standards setting and maintenance }Grading procedures }Quality check of examination data }Feedback to schools Setting of examination papers }Curriculum and Assessment Guide & Assessment Framework }Examination paper development mechanism }Confidentiality and security measures Continuous Improvement Conduct of examinations }Examination regulations }Public Examinations Information Centre (PEIC) }Public Examinations Communication and Support System (PECSS) }Oral Recording System (ORS) }On-site checking & audits }Examination logistics & security measures Safeguarding of Standards

Vetting of Examination papers l l l Different subjects are selected annually for vetting and peer review Vetting reports provide useful feedback for subject managers and Subject Committees Reports and responses from the vetting exercise as well as reports on examination irregularities are submitted to the Public Examinations Board for discussion

Discrepancy Checking Scripts are sampled for cross checking at various stages of marking and grading as part of our quality assurance framework: l l l Randomly selected candidates and all their scripts extracted for manual checking Scripts of candidates with large discrepancies in the levels obtained in different subjects Scripts of candidates with large discrepancies in component levels (eg over 1000 scripts from Chinese Language checked in 2012) 24

Discrepancy Checking Performance at the school level are crosschecked across different core subjects: eg LS vs Chin Outlier schools to be manually checked 25

Discrepancy Checking Math vs English 26

Assessment Quality-Assurance Platform l l Based on Rasch Model Item-level and paper-level analysis to ensure quality: l l Item difficulties Discrimination power Item reliability Paper reliability (internal consistency)

AQP 評核素質保證平台 l Item Level

AQP 評核素質保證平台 l Paper Level

Supporting Quality Assessment at Schools

l User-friendly version of AQP is available for schools to support Assessment for Learning

AQP 評核素質保證平台 l Student-Level

l Quality Assessment Management

International Recognition

International Recognition l l l Over 150 overseas institutions responded to our survey proforma 70 institutions from the Mainland accept students from HK on the basis of HKDSE 149 institutions from Taiwan accept students from HK on the basis of HKDSE 36

International Recognition Comparison UCAS HKDSE Tariff 5** 145 140 130 5* 120 5 110 100 90 80 4 70 60 50 40 3 30 20 Advanced Placement (AP) 5 4 3 GCE Advanced Level International (GCE AL) Baccalaureate (IB) A* A B C D E 7 6 5 4 3 37

香 港 中 學 文 憑 考 試 一紙文憑 世界通行

Thank You! Delivering HKDSE with Pride 香港中學文憑 眾志昂首同行