Blueprinting Brian Jolly Centre for Medical and Health Sciences Education Monash University Train the Trainer Workshop October 31, 2003 Hong Kong International Consortium
Objectives • Describe basic ideas behind blueprinting • Develop 2 examples • Show this does/does not articulate with IDEAL database Hong Kong International Consortium
Principle • The PURPOSE of the assessment will define the content, strategy, & technique used. Hong Kong International Consortium
Curriculum and Assessment: ‘Ideal’ Hong Kong International Consortium
Curriculum and Assessment: Real Hong Kong International Consortium
Curriculum and Assessment: Real H. C. Hong Kong International Consortium
Curriculum and Assessment: Blueprinting Hong Kong International Consortium
Aspects of Validity - Constructs § § § Face validity Content Construct Criterion Effects on learning Hong Kong International Consortium
Aspects of Validity - Constructs § § § Face validity Content Construct Criterion Effects on learning Hong Kong International Consortium
Aspects of Validity - Constructs § § § Face validity Content Construct Criterion Effects on learning Hong Kong International Consortium
Content Steps § Define what is to be tested § Clinical problems at some resolution § Tasks for each problem § Blueprint § Select test methods § Authenticity appropriate to clinical tasks § Clinical tasks determine the method § Titrate against practical constraints Hong Kong International Consortium
Simple Blueprint Competence Categories CVS RS … GI …. . MSS History Taking Physical Exam Tests and Procedures …. . Management Communication and Patient Education Hong Kong International Consortium Endo/ Metab Haem / Oncology ….
Other Parameters - Hierarchical The Hospital Setting Competence Categories CVS RS … GI The Community Setting …. . CVS RS History Taking Physical Exam Tests and Procedures x x …. . Hong Kong International Consortium … GI
Hierarchical - IDEAL® Related Inpatient Competence Categories CVS RS … Outpatient/Ambulatory GI …. . CVS RS History Taking Physical Exam Tests and Procedures x x …. . Hong Kong International Consortium … GI
Other Dimensions The Child Competence Categories History Taking CVS RS … The Elderly GI …. . CVS x Physical Exam Tests and Procedures …. . Hong Kong International Consortium RS x … GI
Other Dimensions - Response The Child Cognitive /Affective Categories Recall CVS RS … The Elderly GI …. . CVS RS x x Reasoning Affect Skill …. . Hong Kong International Consortium … GI
Test – Response Matrix The Child Cognitive /Affective Categories Recall CVS RS … The Elderly GI …. . CVS RS KFI CI SP Reasoning Affect Skill …. . Hong Kong International Consortium … GI
Steps: A plausible test of Content Validity § Chart distribution of questions by level & topic § Compare this distribution with weighting of topics in curriculum or course documents § Note to establish content validity we need a demonstrated relationship Jolly 2000, Medical Education, Hong Kong International Consortium
Steps: A plausible test of Content Validity § Some researchers use judgments of clinicians or successful test-takers § Dimensions of blueprint not always easy to define c. f. the items § How clinical problems, tasks or other 'concrete examples' relate to content often taken for granted in blueprinting Jolly 2000, Medical Education, Hong Kong International Consortium
Summary • Before selecting evaluation tools, need to create an evaluation “blueprint” • The blueprint = matrix that outlines parameters for the assessment in 3 major areas: domains to be assessed, content & assessment instruments Hong Kong International Consortium
Summary • Blueprint matrix ensures balance across assessment formats - conversely • Multiple methods usually required to achieve blueprint objectives • Different assessment instruments allow balance of strengths and weaknesses of each method Hong Kong International Consortium
Blueprinting Issues: e. g. , RACS Assessment § Content § In Australia RACS has sole responsibility for goals, training delivery & standards § See CANMEDS, Duties of a Doctor, § Investigate ways of addressing these issues § Get profession & public to assist § Responsibility for the ‘hard to measure’ § RACS blueprint – are we a surgeon first & a specialist second? Hong Kong International Consortium
RACS Assessment - Content § Judgement (80%) § Literature on judgement § Measures of judgement (as yet undeveloped) § Manual dexterity (20%) § Simulation and testing in training § Reznick, Paisley vs. Beard et al Hong Kong International Consortium
Скачать презентацию Blueprinting Brian Jolly Centre for Medical and Health
8f5fd943866ac07449d4973ee51a0947.ppt