Teaching and Assessing in Competency Based Education Faculty

Teaching and Assessing in Competency Based Education Faculty Development Workshop UCSF Current Development Team: Patricia O’Sullivan, Ed. D; Erick Hung, MD; Lindsay Mazotti, MD Contributors: Lee Learman MD, Ph. D; Susan Promes, MD

Creative Commons License Attribution-Non. Commercial-Share Alike 3. 0 Unported You are free: • to Share — to copy, distribute and transmit the work • to Remix — to adapt the work Under the following conditions: • Attribution. You must give the original authors credit (but not in any way that suggests that they endorse you or your use of the work). • Noncommercial. You may not use this work for commercial purposes. • Share Alike. If you alter, transform, or build upon this work, you may distribute the resulting work only under a license identical to this one. See http: //creativecommons. org/licenses/by-nc-sa/3. 0/ for full license.

Objectives · Describe competency based education · Identify promising teaching and assessment practices and strategies for successful implementation · Describe the relationship between teaching and assessing competencies and overall program evaluation.

Schedule 00 – 0: 15 Welcome, Overview and Introductions 0: 15 – 0: 45 Introduction of Competency 0: 45 - 1: 45 Teaching Strategies 1: 45 - 2: 00 Break 2: 00 – 3: 30 Assessment Tools 3: 30 – 3: 45 Implementation and summary 3: 45 – 4: 00 Performance Assessment & Evaluation

Competency Based Education An outcomes based approach to the design, implementation, assessment and evaluation of a medical program using an organizing framework of competencies The unit of progression is mastery of specific knowledge, skills and attitudes

Shifting the Paradigm A competency-based system of education requires four components: – identifying the outcomes – defining performance levels for each competency – developing a framework for assessing competencies – continuous evaluation of the program to see if it is indeed producing the desired outcomes

Step 1: Identifying the Outcomes • Determine a competency framework comprised of broad “domains of competence” that, in aggregate, define the desired outcome.

Desired Outcome: Driving A Car Competency Domains 1. Knowledge – traffic laws – controls of a car – dynamics of the car 2. Skill – Physical skill at controlling car – Computational skill/process of adaptation 3. Attitude – “Professionalism”- Etiquette of driving – Self-control, anger management

Step 2: Define Performance Levels (Milestones) • Markers of achievement of levels of performance (ability) in a developmental continuum. • Can roughly corresponding to stages of progress: novice through expert.

Desired Outcome: Driving A Car Domain of Competence: Knowledge Competency: Traffic laws – Milestones: » Novice: » Proficient: » Expert:

Dreyfus Model

Desired Outcome: Driving A Car Domain of Competence: Knowledge Sub competency: Traffic laws – Milestones: » Novice: Describe common traffic laws in the US » Proficient: Identify three main traffic citations in US » Expert: Contrast US traffic laws and laws in UK

Step 3: Develop a Framework for Assessment How do I know if my student can drive a car? Imagine an observable, measurable activity that a driver must be trusted to perform that crosses domains of competence.

Step 3: Develop a Framework for Assessment How do I know if student can drive a car? • Responds to unexpected changes on the road (Yields appropriately to a biker in the crosswalk) • Manages car on limited access roads (Merges onto a freeway) • Parallel parks EPAs

Competency based education means that a. Students must complete numerous assessments to demonstrate skills. b. Students are ranked by proficiency level. c. Individuals can complete whenever they have demonstrated competency. d. Students work to achieve at least a minimal level of skill.

Which of the following is a competency for US residents? a. b. c. d. Interprofessionalism Quality Improvement Professionalism Procedural skills

UCSF’s (and ACGME’s) Competency Framework • • • Medical Knowledge Patient Care Practice Based Learning and Improvement Systems Based Practice Professionalism Interpersonal and Communication Skills • http: //meded. ucsf. edu/ume/md-competencies

Other Competency Frameworks • CANMEDS • Physician Competency Reference Set (also includes interprofessional collaboration and personal & professional development)

Which statement best describes the relationship among competencies, milestones and EPAs? a. EPAs>competencies>milestones b. Competencies>milestones>EPAs c. Milestones>Competencies>EPAs

Milestone

EPA defined • A core unit of work reflecting a responsibility that should only be entrusted upon someone with adequate competencies Olle ten Cate Medical Teacher 2010; 32: 669 -675 CONTEXT is key!

Entrustable Professional Activities (EPAs)

Viewpoint: Competency-Based Postgraduate Training: Can We Bridge the Gap between Theory and Clinical Practice? ten Cate, Olle; Scheele, Fedde Academic Medicine. 82(6): 542 -547, June 2007. 2

Entrustable Professional Activities (EPAs)

Programs should document and demonstrate • Learning opportunities in each competency domain • Evidence of multiple assessment methods • Use of aggregate data to improve the educational program

Teaching Strategies 0: 45– 1: 15 Group Activity – Madness to Methods 1: 15 – 1: 45 Share Best Method to Objective Match _________________________ Each group nominate one or two “best method to objective match”

Playing Method to Madness • Each player receives methods cards equal to the number of players in the game, + one extra (5 players = 6 cards per player). • Dealer turns over the first objective card leaving it face up in the table. Each player chooses one methods card from his/her hand that will achieve the teaching objective. A player holding a “wild card” can create a new or use a previously discussed method. • Beginning clockwise from the dealer, all players place their chosen methods card face up near the objective card in the center of the table. • Each player has one minute to persuade the other players that their teaching method will effectively, efficiently and appropriately achieve the objective. • Once all players have presented their “method”, all players vote for the best method for achieving the teaching objective, but cannot vote for themselves. • Players receive a point for each vote they receive. • The dealer tallies the votes by player and records on a score sheet. • The winner is the player with the most votes. Simpson D, Fenzel J, Rehm J, Marcdante K. Enriching Educators' Repertoire of Appropriate Instructional Methods. Med. Ed. PORTAL; 2010. Available from: www. mededportal. org/publication/7968

Debrief the Game • What were examples of good matches between objective and method? • What did you learn from the game? – Large variety of teaching methods – Some work better than others for a specific objective – Match objective and teaching method

BREAK

Why assess? Whatever we measure, we tend to improve David Leach, former CEO of ACGME

What makes a good assessment tool? • Appropriate for what you are measuring • User must know how to use it – Assessors trained – Learners prepared • Monitored results to make sure the tool is working

What makes a good assessment of a learner? • Sufficient data • Sufficient contact with learner • Sufficient sources of information • Ability to render a judgment based on the information

Miller’s Assessment Triangle Undercover SPs Observation Video logs Record Review Patient Survey 360° Ratings Does Shows How Knows Checklists OSCE’s SP’s Clinical context-based tests MCQ’s Oral exams Checklists Essays Procedure/Case Logs Factual tests Oral exams MCQ’s Essays

Assessment Tools 2: 00– 2: 30 Group Activity – Matching Assessment and Objectives 2: 30 – 3: 00 Share Best Practices _________________________ Play the game again, but this time with assessment cards. Deal each person 4 assessment cards Select one objective-assessment method pair: Why is it a good match? How will assessors be trained? How will data be collected? How frequently will data be collected?

Learner Evaluation Expectations • Formative – An assessment should occur for each learning experience – Assessments of competency/milestones reflective of the 6 areas – Multiple formats & evaluators (faculty, peers, patients, self, others) – Progressive improvement by learner across the continuum – Documented • At least semi-annual evaluation for residents • At end of course/clerkship/rotation • Summative – Screening Committee review – Competency Committee review – In other health professions schools?

Clinical Competency Committees • Must have a written description of committee responsibilities – Review all resident evaluations by all evaluators semi-annually – Prepare and assure the reporting of Milestones evaluations of each resident semiannually to ACGME – Make recommendations to program director for resident progress (promotion, remediation, dismissal)

Reflect on Teaching and Assessing to Evaluate the Program

Program Improvement • Formal systematic evaluation annually • Must monitor and track: – Learner performance – Faculty development – Graduate performance – Program evaluations by learners and faculty • Must use results to improve the program – Written plan of action – Reviewed and approved by teaching faculty

Review of Objectives · Describe competency based education · Identify promising teaching and assessment practices and strategies for successful implementation · Describe the relationship between teaching and assessing competencies and overall program evaluation.

Performance Assessment Teach for UCSF Certificate • Please complete skills assessment and evaluation to fulfill requirements of the workshop • http: //tiny. ucsf. edu/Competencies

UCSF Resources • Glossary of Competencies and Assessments http: //meded. ucsf. edu/gme/competencies-assessment • Evaluation and Assessment for GME Programs http: //meded. ucsf. edu/gme-evaluation-assessment-tools • UCSF UME Competencies http: //meded. ucsf. edu/ume/md-competencies