Скачать презентацию Blueprinting Brian Jolly Centre for Medical and Health Скачать презентацию Blueprinting Brian Jolly Centre for Medical and Health

8f5fd943866ac07449d4973ee51a0947.ppt

  • Количество слайдов: 23

Blueprinting Brian Jolly Centre for Medical and Health Sciences Education Monash University Train the Blueprinting Brian Jolly Centre for Medical and Health Sciences Education Monash University Train the Trainer Workshop October 31, 2003 Hong Kong International Consortium

Objectives • Describe basic ideas behind blueprinting • Develop 2 examples • Show this Objectives • Describe basic ideas behind blueprinting • Develop 2 examples • Show this does/does not articulate with IDEAL database Hong Kong International Consortium

Principle • The PURPOSE of the assessment will define the content, strategy, & technique Principle • The PURPOSE of the assessment will define the content, strategy, & technique used. Hong Kong International Consortium

Curriculum and Assessment: ‘Ideal’ Hong Kong International Consortium Curriculum and Assessment: ‘Ideal’ Hong Kong International Consortium

Curriculum and Assessment: Real Hong Kong International Consortium Curriculum and Assessment: Real Hong Kong International Consortium

Curriculum and Assessment: Real H. C. Hong Kong International Consortium Curriculum and Assessment: Real H. C. Hong Kong International Consortium

Curriculum and Assessment: Blueprinting Hong Kong International Consortium Curriculum and Assessment: Blueprinting Hong Kong International Consortium

Aspects of Validity - Constructs § § § Face validity Content Construct Criterion Effects Aspects of Validity - Constructs § § § Face validity Content Construct Criterion Effects on learning Hong Kong International Consortium

Aspects of Validity - Constructs § § § Face validity Content Construct Criterion Effects Aspects of Validity - Constructs § § § Face validity Content Construct Criterion Effects on learning Hong Kong International Consortium

Aspects of Validity - Constructs § § § Face validity Content Construct Criterion Effects Aspects of Validity - Constructs § § § Face validity Content Construct Criterion Effects on learning Hong Kong International Consortium

Content Steps § Define what is to be tested § Clinical problems at some Content Steps § Define what is to be tested § Clinical problems at some resolution § Tasks for each problem § Blueprint § Select test methods § Authenticity appropriate to clinical tasks § Clinical tasks determine the method § Titrate against practical constraints Hong Kong International Consortium

Simple Blueprint Competence Categories CVS RS … GI …. . MSS History Taking Physical Simple Blueprint Competence Categories CVS RS … GI …. . MSS History Taking Physical Exam Tests and Procedures …. . Management Communication and Patient Education Hong Kong International Consortium Endo/ Metab Haem / Oncology ….

Other Parameters - Hierarchical The Hospital Setting Competence Categories CVS RS … GI The Other Parameters - Hierarchical The Hospital Setting Competence Categories CVS RS … GI The Community Setting …. . CVS RS History Taking Physical Exam Tests and Procedures x x …. . Hong Kong International Consortium … GI

Hierarchical - IDEAL® Related Inpatient Competence Categories CVS RS … Outpatient/Ambulatory GI …. . Hierarchical - IDEAL® Related Inpatient Competence Categories CVS RS … Outpatient/Ambulatory GI …. . CVS RS History Taking Physical Exam Tests and Procedures x x …. . Hong Kong International Consortium … GI

Other Dimensions The Child Competence Categories History Taking CVS RS … The Elderly GI Other Dimensions The Child Competence Categories History Taking CVS RS … The Elderly GI …. . CVS x Physical Exam Tests and Procedures …. . Hong Kong International Consortium RS x … GI

Other Dimensions - Response The Child Cognitive /Affective Categories Recall CVS RS … The Other Dimensions - Response The Child Cognitive /Affective Categories Recall CVS RS … The Elderly GI …. . CVS RS x x Reasoning Affect Skill …. . Hong Kong International Consortium … GI

Test – Response Matrix The Child Cognitive /Affective Categories Recall CVS RS … The Test – Response Matrix The Child Cognitive /Affective Categories Recall CVS RS … The Elderly GI …. . CVS RS KFI CI SP Reasoning Affect Skill …. . Hong Kong International Consortium … GI

Steps: A plausible test of Content Validity § Chart distribution of questions by level Steps: A plausible test of Content Validity § Chart distribution of questions by level & topic § Compare this distribution with weighting of topics in curriculum or course documents § Note to establish content validity we need a demonstrated relationship Jolly 2000, Medical Education, Hong Kong International Consortium

Steps: A plausible test of Content Validity § Some researchers use judgments of clinicians Steps: A plausible test of Content Validity § Some researchers use judgments of clinicians or successful test-takers § Dimensions of blueprint not always easy to define c. f. the items § How clinical problems, tasks or other 'concrete examples' relate to content often taken for granted in blueprinting Jolly 2000, Medical Education, Hong Kong International Consortium

Summary • Before selecting evaluation tools, need to create an evaluation “blueprint” • The Summary • Before selecting evaluation tools, need to create an evaluation “blueprint” • The blueprint = matrix that outlines parameters for the assessment in 3 major areas: domains to be assessed, content & assessment instruments Hong Kong International Consortium

Summary • Blueprint matrix ensures balance across assessment formats - conversely • Multiple methods Summary • Blueprint matrix ensures balance across assessment formats - conversely • Multiple methods usually required to achieve blueprint objectives • Different assessment instruments allow balance of strengths and weaknesses of each method Hong Kong International Consortium

Blueprinting Issues: e. g. , RACS Assessment § Content § In Australia RACS has Blueprinting Issues: e. g. , RACS Assessment § Content § In Australia RACS has sole responsibility for goals, training delivery & standards § See CANMEDS, Duties of a Doctor, § Investigate ways of addressing these issues § Get profession & public to assist § Responsibility for the ‘hard to measure’ § RACS blueprint – are we a surgeon first & a specialist second? Hong Kong International Consortium

RACS Assessment - Content § Judgement (80%) § Literature on judgement § Measures of RACS Assessment - Content § Judgement (80%) § Literature on judgement § Measures of judgement (as yet undeveloped) § Manual dexterity (20%) § Simulation and testing in training § Reznick, Paisley vs. Beard et al Hong Kong International Consortium