Portfolio Smorgasbord Members of the STFM Group On

Portfolio Smorgasbord Members of the STFM Group On Learner Portfolios STFM, May 2009

Brief Background/Rationale • We are members of STFM’s Group On Learner Portfolios • We have all grappled with issues surrounding learner portfolios • We would like to support others working on portfolios – sharing successes and challenges! • We would like to make today useful for you

Objectives • Describe main types of portfolios and discuss the difference between formative and summative use of portfolios. • Explore the practicality of using portfolios at your home institution. • Network with colleagues around the country who are leaders in the implementation and use of portfolios.

Introductions • Teresa Kulie, MD – University of Wisconsin – Madison – Paper, reflective portfolio – Residents – Middle/advanced

Introductions • Nancy Barrett, Ed. D – Carle Foundation Hosp FPRP Urbana, IL – Residents – Paper-based developmental portfolio – Enthusiastic advanced beginner!

Introductions • Lynda Alper, LCPC, MEd – Carle Foundation Hosp FPRP Urbana, IL

Introductions • Craig Gjerde, Ph. D – University of Wisconsin – Madison – Paper, reflective portfolio – Residents – Middle/advanced

Introductions • Stacy Potts, MD • University of Massachusetts Worcester Family Medicine Residency

Introductions • Susanne Krasovich, MD – Waukesha Family Practice Residency Waukesha, WI – Electronic

Introductions • Joseph Brocato, Ph. D – University of Minnesota Minneapolis, MN – EPortfolio

Introductions • Ronald Williams, MEd, MS – University of Texas Medical Branch Galveston, TX – Electronic Portfolio

Introductions • Carrie Anderson, MD – Indianapolis St Francis FPR Indianapolis, IN – Paper reflective portfolio – Middle

Introductions • Andrea Pfeifle, Ed. D – University of Kentucky, Lexington KY – andrea. pfeifle@uky. edu – 859 -323 -6581

Introductions • Netra Thakur, MD – Franklin Square Baltimore, MD

Agenda • • • Portfolio Basics 101 Successes and Challenges Portfolio examples Breakout with session leaders Future of Portfolios Wrap - Up

Not That “P” Word Again ! Nancy F. Barrett, Ed. D STFM April 2009

Goals 1. 2. 3. Define a portfolio Understand the different purposes of portfolios including evaluation Describe potential uses as defined by ACGME

ACGME Definition “The portfolio is an interactive web-based professional development tool that residents can use throughout their residencies to record and organize their learning and to reflect and receive feedback on their skills as physicians, building evidence that allows them to chart their own progress over time. ” http: //www. acgme. org/ac. Website/portfolio/cbpac_faq. pdf

Purposes of Portfolios 1. Documenting what is 2. Showcasing achievements 3. Developing expertise 4. Reflecting on the experience 5. Empowering the learner

Four Types of Portfolios Develop. Reflective ment Documentation Showcase Past Present Future Present Key Word Justify Demonstrate Improve Consider Audience Public Private* What SHOULD happen? WHY? Orientation Question What DID What IS happen? happening *Includes relevant program staff

Documentation Portfolio Demographics Training (med school, residency, fellowships) Awards Work history What you would show an employer Maintained by program AND RESIDENT

Showcase Portfolio Demonstrates competence Identified by faculty OR RESIDENT

Showcase Example from ACGME http: //www. acgme. org/ac. Website/portfolio/Alpha. Demonstration. wmv

Developmental Portfolio ü To improve performance ü Identified by faculty OR RESIDENT Example: Improve ability to formulate an appropriate differential diagnosis

Reflective Portfolio Competency based reflections IDENTIFIED BY THE RESIDENT Resident gives permissions Faculty Other residents

Reflective Portfolio ACGME questions: 1. What did you learn? 2. Who helped you with the patient? 3. What might you do differently next time? 4. Were your values challenged? 5. What made you feel good? 6. How to share with others?

Another Approach. . . Three questions: 1. What? 2. So what? 3. Now what? Source: Kulie, T. 2006. STFM Conference. Lessons Learned-Portfolios. http: //fmdrl. org/index. cfm? event=c. get. Attachment&riid=651

Getting Started 1. Decide on a process 2. Start slowly and build up 3. Get faculty buy-in 4. Get resident buy-in 5. Provide faculty development 6. Think about faculty models for the process 7. Provide faculty and resident time

Benefits of Portfolios Documentation Showcase Developmental Reflection Bonus: Access Achievement Show improvement Introspection Resident involvement in learning

Evidence Based Portfolio Recommendations presented by Lynda Alper, MEd, LCPC

1) A. Introduce faculty AND students to the portfolio. B. Assure that training is given in HOW to use it. C. Clearly communicate purpose of utilizing.

2) Ensure that the portfolio is supported by mentoring. Students appreciate and learn from conversations with mentors who give feedback on the portfolio and help them set new, realistic goals and find ways to achieve them.

3) Ensure that the portfolio is firmly embedded within the context and procedures of the curriculum/competencies of which they are a part.

4) Do not make them do it if you aren’t going to use it!! Unevaluated portfolios tend to fall victim to competition from educational activities that are evaluated.

5) Support students by giving clear guidelines about what they are expected to do but do not curtail their freedom by over-regulating the portfolio.

6) a. User friendly for students and faculty b. Be sensitive to time involved. c. Do not inundate with paper d. Whether using paper or electronic, ensure that the system is designed for ease of use by all

IMPORTANT: Good mentoring seems to be the key ingredient to the successful use of portfolios! presented by Lynda Alper, MEd, LCPC

References Driessen, E. , van Tartwijk, J. , van der Vleuten, C. , & Wass, V. (2007, November 29). Portfolios in medical education: why do they meet with mixed success? A systematic review. [Review of the research studies on portfolios Portfolios in medical education: why do they meet with mixed success? A systematic review. ]. Medical education (Med Educ), published in England. (Language: eng), 41(12), 1224 -33. Retrieved April 9, 2009, from http: //www. find -health-articles. com/Web site: http: //www. find-healtharticles. com/_pub_18045373 -portfolios-medical-education-meet-mixed-success -systematic-review. htm Driessen, E. W. , Muijtjens, A. , van Tartwijk, J. , & van der Vleuten, C. (2007, October 25). Web- or paper-based portfolios: is there a difference? Medical Education, 41(11), 1067 -1073. Abstract retrieved April 9, 2009, from http: //www 3. interscience. wiley. com///? CRETRY=1&SRETRY=0

Agenda • • • Portfolio Basics 101 Successes and Challenges Portfolio examples Breakout with session leaders Future of Portfolios Wrap - Up

Successes & Challenges • Break into Small Groups • Groups to discuss: – At what stage of implementation are you? – What successes have you had so far? – What challenges have you had so far? – What do you hope to get out of today’s session?

Agenda • • • Portfolio Basics 101 Successes and Challenges Portfolio examples Breakout with session leaders Future of Portfolios Wrap - Up

• Joseph Brocato, Ph. D – University of Minnesota Minneapolis, MN – EPortfolio

Overview of Portfolios at the University of Minnesota Medical School Joseph Brocato, Ph. D Department of Family Medicine and Community Health

Graduate Medical Education at The University of Minnesota • 67 accredited & 40 non-accredited training programs • 850 residents and fellows Louis Ling, M. D, Associate Dean & DIO, UMN GME

The Department of Family Medicine & Community Health at The University of Minnesota • Department: Located on TC campus, 73 fulltime faculty and 994 non salaried community faculty • Eight Residencies (five TC urban & suburban sites, plus two rural and small town site • 150+ Residents • Two fellowships: Sports Medicine and Palliative Care • Founded in 1971 with over 1, 500 residency alumni

UMN Electronic Portfolio E-Portfolio is a secure web site at the University of Minnesota for entering, saving, organizing, viewing, and selectively sharing personal educational records.

UMN E-Portfolio Homepage https: //portfolio. umn. edu/portfolio/index. jsp

UMN Medical School E-Portfolios • • UME GME CME Faculty Development

Uses at UMN: UME • Used as part of physician and society course (PAS) for M 1 and M 2 students. • Students are asked to reflect upon experiences with standardized patients. • Reflections are shared with small group leader.

Uses at UMN: GME Three programs are currently using e-Portfolio at UMM. • • Medicine-Pediatrics Residency Orthopedic Surgery Plastic Surgery With other like FM in development Each organizes the information in the portfolio by ACGME competency heading.

Uses at UMN: Faculty Development • Faculty from multiple departments are using for promotion and tenure • There are common templates as well as department/school/university templates • There also customizable templates

UMN E-Portfolio Homepage https: //portfolio. umn. edu/portfolio/index. jsp

Open Source Portfolio (OSP) https: //sourceforge. net/projects/osep/ • Open Source Portfolio (OSEP) is a robust, nonproprietary, open-source electronic portfolio application, developed by a community of individuals and organizations from around the world. • Background video on OSP: http: //bugs. sakaiproject. org/confluence/download/atta chments/14752/OSP. mp 4? version=1

Global Next Steps at UMN Medical School • Waiting for further national mandate for development and usage • UME: Need continued medical school curriculum reform to place portfolio • GME: Need early adopters (faculty and residents) plus clear national mandate • Faculty: Compromise usage for promotion and tenure as a documentation collection aid. Future usage for evaluation?

• Carrie Anderson, MD – Indianapolis St Francis FPR Beech Grove, IN

Journaling to Capture the Competencies Carrie Anderson, MD St. Francis FM Residency Indianapolis/Beech Grove, IN STFM Annual Conference, 2009

St. Francis FMRP 20 Residents, 6 -7 -7 n 5 full time faculty n Community-based n Using electronic evaluations (E*Value™) for most evaluations. n “Moments Journal” is paper-based currently, in use for 3 years n

“Moments Journal”: What is it? Reflective Portfolio n A way to capture performance measures – “competencies” n A way to determine if the resident recognizes his/her own competency achievements n Areas for discussion with advisor n Also includes samples of journal reading the resident does, with mini-analyses n

“Moments Journal” Includes: Instructions n Competency-based Goals and Objectives n Core Values of Family Medicine (from Future of Family Medicine publication) n Moments of Brilliance n Moments of Frustration n EBM entries n Evaluation checklist n

Moments of Brilliance and Moments of Frustration “Describe an encounter during which you were able to make a significant difference in patient care” “… you felt particularly helpless, ineffective, of frustrated…” n “Describe how the incident changed or impacted the way you practice…” n “Any future areas of learning? ” n “What competencies are demonstrated? ” n “What competencies can you improve upon? ” n

Evidence-Based Medicine Entry Title, description of article n Type of studies mentioned or type of study of article n “What prompted you to search out this info? ” n “Did you share your findings? ”…”With whom? ” n “Will you apply what you’ve learned to your practice…why/why not? ” n

“Capturing” the Competencies n n n Gives greater insight into our trainees’ thought processes They get better at knowing the competencies Examples of Professionalism, Interpersonal and Communication Skills, and Systems-Based Practice especially Make them do it because “it’s good for them”? ? Currently not worked into our “competency average” by electronic evaluation

For Discussion Organization of reflective portfolio n How residents feel about the exercise n Resident compliance n Faculty perceptions n Areas for improvement n

• Susanne Krasovich, MD – Waukesha Family Practice Residency Waukesha, WI – Electronic

Building A Custom Electronic Portfolio Susanne Krasovich, MD Waukesha FM Residency Program Waukesha, WI STFM Spring Conference, 2009

Waukesha FM Residency n n 6 -6 -6 -? 6 full time faculty Community based program P 4 program= radical curriculum change

Why Use A Portfolio? n Complexity of schedules – Majors and Masteries – Individual Plan of Learning n Evaluation – Document competence and completion n Promote lifelong learning – Cycle of Goals Experience Evaluation Reflection New Goals n Independent learning tasks

Development Process Reviewed literature and available products n Stakeholder discussions n – Identify goals, needs, wants Start building n Use, feedback, revision concurrent n

Considerations n General – Easy, useful, resident/physician focused, comprehensive, accessible n Learner driven – Learning, practicalities, tracking, future job n Program/Faculty driven – Teaching, evaluation, advising, practicalities, administrative

The Electronic Learner’s File (ELF)

Pros And Cons n Pros – Electronic format accessible from anywhere, allows multiple contributors, allows multiple views of same info, automated processes, customizable n Cons – Set up time, cost, safety of data issues n Building + using: good buy in, but frustration inevitable, input better than output

• Ronald Williams, MEd, MS – University of Texas Medical Branch Galveston, TX – Electronic Portfolio

UTMB Department of Family Medicine Electronic Portfolio

Portfolios It is NOT part of performance evaluation, merely an instrument for tracking the achievement of personal goals and ensuring a resident’s professional growth. Our portfolio tracks ACGME competencies in both snapshot and longitudinal methods to track trends.

Why use one? • Residents become proactive in their own learning and development. • They identify their own educational needs. • They are responsible for developing strategies to meet their individual educational needs.

Why use one? • Empowered to apply new learning to practice. • Objectives and learning plans are modifiable based on personal experiences. • The structure is based on reflection on their own experience and practice.

Electronic Portfolios • Easy to create back-up files • • Portability Long shelf life Person-centered Accessibility

Administrative Functionality • The portfolio also has built into it an administrative work flow that allows residents, advisors and program directors to communicate with each other in a guided efficient structure.

Resident’s Dashboard

ILP Work Sheet

Resident’s Dashboard

Resident’s Schedule and Progress

Resident’s Rotation Journal

Cons • Reliability Issues • More time intensive for administrators • Ability of individuals to self evaluate lags behind their ability to perform the specific skill

Development Team Lisa R. Nash, DO, FAAFP Darren Burns, B. S Becky Hamilton, BA Linda S. Hubbell, Ron Williams, Med, MS

Resources • http: //www 1. umn. edu/tel/TELat. UM/Portfolio. html • http: //www. itdean. umn. edu/career/portfolio. html • http: //www. eport. iu. edu/documents_frameset. ht m • http: //eportfolio. d. umn. edu/about. html • http: //www. electronicportfolios. com/portfolios. ht ml#pubs • http: //www. osportfolio. org/ • http: //www. educause. edu/content. asp? page_id= 5524&bhcp=1

• Andrea Pfeifle, Ed. D • University of Kentucky, Lexington KY • andrea. pfeifle@uky. edu • 859 -323 -6581

University of Kentucky • Paper • Formative and reflective • Exhibits: – – Credentials Personal/professional goals Portrait of a Family Medicine physician Evidence of growth in longitudinal curricular elements – Competency exhibits for all 3 years (can combine up to 3 competencies)

University of Kentucky • Portfolio Assessment – Developmental feedback provided by faculty mentor – Formative review prior by 2 faculty prior to each Progress and Promotions meeting • PGY-1: 4/yr • PGY-2: 3/yr • PGY-3: 2/yr – Summative review at end of each year by Director and 1 faculty

University of Kentucky Lessons Learned • Less is definitely more • Formative feedback is absolutely essential • Requires more than one or two faculty champions • Faculty development is key and must be ongoing

Agenda • • • Portfolio Basics 101 Successes and Challenges Portfolio examples Breakout with session leaders Future of Portfolios Wrap - Up

Breakout with leaders

Agenda • • • Portfolio Basics 101 Successes and Challenges Portfolio examples Breakout with session leaders Future of Portfolios Wrap - Up

The Future: Other E- Portfolio-based Learning Systems in Development • The ACGME • The American Board of Family Medicine • New Innovations (Residency Management Suite)

ABFM E-Portfolios • The resident portfolio systems (RPS): system for storing and maintaining resident information and includes two components – Resident training management (RTM) – Resident performance assessment (RPA) • https: //www. theabfm. org/residency/tracking. a spx

ABFM: Resident Training Management (RTM) • CME Modules that involve web-based training self assessment modules • Topics: asthma, coronary artery disease, depression, diabetes, heart failure, and hypertension. Planned for 2007: Sports Medicine, plus 3 -4 other modules • https: //www. theabfm. org/MOC/sam. a spx

ABFM: Resident Performance Assessment (RPA) • Stores and maintains resident performance information • Free access for residents to modules in the ABFM's maintenance of certification program. This access may be controlled by programs for evaluative purposes and/or opened directly to residents for self-assessment purposes. Residents can elect to have their performance on the modules maintained as part of their portfolio. • A resident transcript report for any resident within the program that documents his/her residency experience. This data can be accessed by residents even after they graduate from a program.

ABFM: Resident Performance Assessment (RPA) (continued) • Program reports regarding data elements such as rotations, clinical procedures, competency assessment, and evaluations. • Board input into resident transfers which can be used to ensure the resident's eligibility to sit for the certification examination. • The development of reports in preparation for Residency Review Committee visits.

New Innovations Portfolio

Next Steps/Summarization • Continued national mandate clarification • Continued development of platform technology • Development of value added technology

Contact Info • Joseph J. Brocato, Ph. D. • broca 003@umn. edu • (612) 624 -4464

• Insert Netra Thakur slides (ACGME national movement) here

Agenda • • • Portfolio Basics 101 Successes and Challenges Portfolio examples Breakout with session leaders Future of Portfolios Wrap - Up

Wrap - Up • Take home thoughts • Did we meet your needs? • Questions for panel • Generate “Best Practices” list

Objectives (revisited) • Describe main types of portfolios and discuss the difference between formative and summative use of portfolios. • Explore the practicality of using portfolios at your home institution. • Network with colleagues around the country who are leaders in the implementation and use of portfolios.

Contact Information • • • teresa. kulie@fammed. wisc. edu broca 003@umn. edu susanne. krasovich@phci. org netra. thakur@medstar. net carrie. anderson@ssfhs. org Or look up contact info in the list of presenters!