Evaluating procedural skills competenc Stephen Ritz DO

Evaluating procedural skills competenc Stephen Ritz , DO Gregory Smith, MD UPMC St. Margaret Family Medicine Residency Pittsburgh, PA

Learning objectives: • • Analyze the challenges to defining competence in the performance of procedural skills. Identify the major elements in the development of a procedural skills credentialing process. Compare the components of several procedural assessment tools. Analyze the relationship between instruction, evaluation, assessment.

Workshop agenda: Part I. • • • Why is this necessary? RRC requirements. Challenges. What is “competence”? How do you credential procedural competence? Evaluation methods.

Part II. • Small group breakout: - development of an assessment tool - develop a brief curriculum, including teaching methods / resources for your procedure.

Part III. Discussion /presentation of assessment tools and curricula • • • Curriculum design Challenges Resources / methods used Arriving at consensus User friendly / “real world” applicability

I. RRC requirements: • 1. Family Medicine programs should provide opportunity to learn in multiple settings those procedures that are within the scope of Family Medicine.

RRC requirements: • 2. Residents must receive training to perform those clinical procedures required for their future practices in the ambulatory and hospital environments.

RRC requirements: • The program director/faculty should develop a list of procedural competencies required for completion by all residents in the program prior to their graduation. The list must be based upon the anticipated practice needs of all FM residents. The faculty should consider the current practices of the program graduates, national data, and the needs of the community served.

RRC requirements: • Procedural skill documentation should indicate when the resident is capable of independent performance of the procedure.

RRC requirements: • Residency programs should devise a method by which all procedures are supervised and evaluated. They must devise a credentialing process to establish whether or not a resident is competent to perform a procedure.

Overall: • • • No specific list of “required” procedures yet. ( Except OB ) Programs must devise a process to credential residents in the performance of procedures. Documentation / credentialing process should indicate when a resident is capable of independent performance of a procedure.

AAFP – Process framework for procedural skills training: 1. 2. 3. 4. 5. 6. 7. 8. 9. Background Indications Contraindications Alternatives Complications Informed consent patient counseling Patient preparation Anesthesia /analgesia Equipment selection 10. Equipment sterility 11. Patient positioning 12. technique: - steps - observation - supervised performance - practice of skills 13. Pathology recognition 14. Complication mgmt. 15. Practice mgmt aspects 16. Pt monitoring/resusc. 17. Outcome evaluation

Procedural skills teaching principles: ü ü ü ü Plan ahead Demonstrate procedures Observe learner in action Provide feedback Engage self –assessment Allow for practice under less than ideal conditions Prepare to modify approach: unprepared learner, diffe

AAFP Residency criteria for procedural skills training • The training must include some form of evaluation including : 1. Cognitive knowledge 2. Psychomotor skills / technique 3. Visual recognition of pathology ( if applicable)

ACOFP – ( Osteopathic) • • • List of 25 procedures ( handout ) Program director signs off when “competent” Credentialing process up to individual program

Components of competence for procedural skills. • “Clinical competence exists when a practitioner has sufficient knowledge and skills such that a procedure can be performed to obtain intended outcomes without harm to the patient. ” Miller, MD. Office procedures. Education, training and proficiency of procedural skills. Primary Care; clinics in office practice. 24 (2): 231 -40, 1997 June

Components of competence for procedural skills 1. 2. 3. 4. 5. 6. 7. 8. Knowledge Clinical decision making Judgment Technical Skills Attitudes Professional habits Interpersonal skills Consistency * Adapted from Miller, MD. Office procedures. Education, training and proficiency of procedural skills. Primary Care; clinics in office practice. 24 (2): 231 -40, 1997 June

Implementation – Applying the competency concepts 1. Knowledge : Learner needs assessment A. self assessment : formal and informal inquiries: group e-mailings, surveys of current and former residents, focus groups B. Faculty assessment - review of procedure logs (by individual and class), query of affiliated specialists ( e. g. surgery, derm ) survey of practicing physicians, research data

Implementation 2. Clinical decision making / judgement: - Direct observation - group learning activities - Simulated clinical scenarios ( OSCE ) - input from faculty / preceptors

Implementation: Technical Skills A. Knowledge – knowledge set necessary for successful procedural performance - Resources: Books ( e. g Pfenninger ) CD ROM / DVD Web based ( AAFP, ACOFP ) Articles

Implementation: Technical skills • B. Hands – On: (psychomotor) - workshops - 1: 1 precepting with selected faculty - Office experiences w/ community specialists ( derm ) - Virtual reality ( future )

Implementation: Attitudes/Habits/Interpersonal skills • • • Direct observation Review of assessment tools Review of evaluations – ACGME/AOA competency based Self assessment – what are they comfortable with currently 360 evaluations / Nursing staff assessments

5. Consistency • • • Serial evaluation Multiple rater evaluation Standardized, objective assessment tools - assists in identifying the gifted or problem learner

Competency assessment: Process model 1. Learner pre - assessment - Self - assessment inventory (handout) - knowledge/skills assessment 2. Knowledge assessment 3. Psychomotor skills assessment 4. Performance debriefing /feedback 5. Documentation of encounter (assessment tool) 6. Review and summary of encounters 7. Sign -off by faculty /program director

Assessment tools: Purposes / benefits • • Standardized Promotes objectivity Teaching tool ( prompts) Facilitates feedback process Facilitates objective, specific feedback Documentation of experiences by learner / class Objective information for procedural curriculum needs

Challenges / Barriers – Assessment tools • • No national standards Time Yet another paperwork documentation burden Faculty / resident buy –in Paper or electronic? Procedure specific vs. generic Redundancy – e. g procedure logs Clerical resources

Assessment tools - components • • Identify resident , patient , preceptor, date Knowledge components: 1. Pre –procedural: - indications - contraindications 2. procedural: - relevant anatomy - medications / dosages - instruments - site prep Psychomotor components: - instrument handling - motor skills Post procedural - overall assessment

VI. Credentialing process • • For “independent” performance of the procedure. Rationale: * Fulfills RRC requirements * More objective endorsement of procedural skills competence * Help PD verify resident competence for outside entities ( e. g. hospital credentialing , legal system ) * Provides data for curriculum design and review

Our system • • • Resident and preceptor completes assessment tool together following each procedure performed Tool is submitted to clerical staff Evaluation result is recorded on compilation sheet for each resident; paper copy of tool is filed. Upon receiving 3 “ competent “ ratings for a given procedure, resident is credentialed for that procedure if advisor and PD agree. Compilation sheet data are reviewed by faculty, advisor and PD by resident and by class. Results are discussed at annual resident promotional review and at advisee meetings.

Part II : Small Group Session Appoint a leader / scribe for your group • Devise the following for your procedure: 1. Curriculum : elements 2. Learner knowledge / skills assessment – * Make an assessment tool 3. Competency credentialing *Write out on poster paper * Prepare to present to larger group •

Discussion Points – • • • What was hardest part of the exercise? Was consensus hard to achieve for curriculum? For the assessment tool ? Was it difficult to devise a method to “certify” competence? What procedural teaching methods have worked best for you? What methods are you currently using to credential your residents for independent performance of procedures? Identify barriers / advantages of implementing your curriculum and assessment tool in the “real world”