V i r g i n i a C o m m o n w e a l t h U n i v e ACGME Competencies: Web-based Evaluation and Documentation Charles O. Frazier, M. D. September 22, 2003 STFM Theme Day - Competencies r s i t y
V i r g i n i a C o m m o n w e a l t h U n i v Objectives • ACGME Toolbox • Using a web-based, databasedriven application • Demonstration • Wrap-up September 22, 2003 STFM Theme Day - Competencies e r s i t y
V i r g i n i a C o m m o n w e a l t h U ACGME Competencies September 22, 2003 STFM Theme Day - Competencies n i v e r s i t y
V i r g i n i a C o m m o n w e a l t h U n i ACGME Competencies • Clinical effectiveness? (clinical outcomes) • Competence (knows how) vs. performance (shows how) September 22, 2003 STFM Theme Day - Competencies v e r s i t y
V i r g i n i a C o m m o n w e a l t h U n i v e r s ACGME “Toolbox” • Chart stimulated recall oral exam • Checklist evaluation • OSCE • Patient surveys • Simulations and models • Standardized oral exam • Standardized patients September 22, 2003 • 360 ° Evaluation Instrument • Global rating of recorded performance • Procedure/Case logs • Portfolios • Record Review • Written exam STFM Theme Day - Competencies i t y
V i r g i n i a C o m m o n w e a l t h U n i v e r s ACGME “Toolbox” • What did I see when I saw the toolbox? Reams of nearly useless paper… September 22, 2003 STFM Theme Day - Competencies i t y
V i r g i n i a C o m m o n w e a l t h U n i v e r s i ACGME “Toolbox” • 360° evaluation instrument – “Implementing an electronic system should make the 360 -degree-evaluation feasible” • Procedure, case logs – “Electronic recording devices and systems facilitate the collection and summarization of patient cases or procedures performed” September 22, 2003 STFM Theme Day - Competencies t y
V i r g i n i a C o m m o n w e a l t h U n i v e r s i ACGME “Toolbox” • Patient Surveys – “It may be possible to improve feasibility by … using computers to collect and summarize survey data” • Simulations and models – “computer-automated scoring rules to record the examinees’ actions” September 22, 2003 STFM Theme Day - Competencies t y
V i r g i n i a C o m m o n w e a l t h U n i v e r s Riverside Family Practice Residency September 22, 2003 STFM Theme Day - Competencies i t y
V i r g i n i a C o m m o n w e a l t h U n i v e r s i t Advising / Evaluation System – Faculty Advisor / Educational Partner – Quarterly advisor meetings with resulting Educational Prescriptions – Semiannual director meetings – Evaluation forms reflect competencies; criteria clear and widely distributed – Expectations defined for residents and faculty – Comp-E-Val™ pulls it all together September 22, 2003 STFM Theme Day - Competencies y
V i r g i n i a C o m m o n w e a l t h U n i v e r Web Based Competency Evaluation Comp-E-Val™ September 22, 2003 STFM Theme Day - Competencies s i t y
V i r g i n i a C o m m o n w e a l t h Introduction – Why database-driven? – Why web-based? – Our methods – Brief technical description – Demonstration September 22, 2003 STFM Theme Day - Competencies U n i v e r s i t y
V i r g i n i a C o m m o n w e a l t h U n i v e r Why database-driven? – To get rid of a lot of the paper – To allow collation, computation, and comparison September 22, 2003 STFM Theme Day - Competencies s i t y
V i r g i n i a C o m m o n w e a l t h U n i Why web-based? • • Universal access Real-time data acquisition To facilitate PDA input To get rid of a lot of the paper September 22, 2003 STFM Theme Day - Competencies v e r s i t y
V i r g i n i a C o m m o n w e a l t h U n Methods – Aligning competencies / tools – Dissemination among faculty, residents, and staff – Choosing when data collected – Data analysis and review September 22, 2003 STFM Theme Day - Competencies i v e r s i t y
V i r g i n i a September 22, 2003 C o m m o n w e a l t STFM Theme Day - Competencies h U n i v e r s i t y
X September 22, 2003 X X STFM Theme Day - Competencies X X X
V i r g i n i a C o m m o n w e a l t h U n Methods – Aligning competencies / tools – Dissemination among faculty, residents, and staff – Choosing when data collected – Data analysis and review September 22, 2003 STFM Theme Day - Competencies i v e r s i t y
V i r g i n i a C o m m o n w e a l t h U n i v e r s i Schedule Jul Aug Sep Oct Nur Nov Dec Jan Nur Fac Adv Feb Mar Apr May Jun Nur Fac Adv Dir Pat Sur Precepting Care Man Care Man Care Man Rot Rot Rot September 22, 2003 STFM Theme Day - Competencies t y
V i r g i n i a September 22, 2003 C o m m o n w e a l t STFM Theme Day - Competencies h U n i v e r s i t y
V i r g i n i a C o m m o n w e a l t h U n Methods – Aligning competencies / tools – Dissemination among faculty, residents, and staff – Choosing when data collected – Data analysis and review September 22, 2003 STFM Theme Day - Competencies i v e r s i t y
V i r g i n i a C o m m o n w e a l t h U n i v e r s i Technical description - Intranet • Compaq Proliant Server • Microsoft Windows NT 4. 0 • Microsoft Internet Information Services 5. 0 • Microsoft Front. Page 2000 – Active Server Pages • Microsoft Internet Explorer September 22, 2003 STFM Theme Day - Competencies t y
V i r g i n i a C o m m o n w e a l t h U n i v e r s i Active Server Pages • Script (programming) is executed on the server, and resulting HTML is sent back to and displayed on browser – Connects to database – Underlying script secure September 22, 2003 STFM Theme Day - Competencies t y
V i r g i n i a C o m m o n w e a l t h U n i v e r s i Technical Description - Intranet • Security – NT Server Authentication • Cost – Server – Software – Labor (estimate) 450 hours September 22, 2003 STFM Theme Day - Competencies t y
V i r g i n i a C o m m o n w e a l t h U n i v e r s i Technical Description - Internet • VCU IIS Public Information Server June 2002 • SSL encryption security with ASP database passwords • No direct cost to residency • Domain name registration September 22, 2003 STFM Theme Day - Competencies t y
V i r g i n i a C o m m o n w e a l t h U Demonstration September 22, 2003 STFM Theme Day - Competencies n i v e r s i t y
V i r g i n i a C o m m o n w e a l t h U n i v e r s Current Status • 40 residencies all across the USA – Not all are using yet – 1 neurology; 1 radiology • Full toolbox, with custom tools available • Conference and procedure surveillance; faculty evaluation • ERAS Import • No scheduling September 22, 2003 STFM Theme Day - Competencies i t y
V i r g i n i a C o m m o n w e a l t h U n i v e r s i Residencies • • • All 6 VCU FP programs Riverside Transitional UCSD FMR WVU FMR Loma Linda U. FPR Indiana U. FPR September 22, 2003 • • • Carilion FPR Fort Wayne Med Ed Prg East Jefferson GH FPR Mountainside FPR Med. Coll. Ohio FPR Crozer-Keystone FPR STFM Theme Day - Competencies t y
V i r g i n i a C o m m o n w e a l t h U n i v e r s i Comp-E-Val™ • Acquired by the American Board of Family Practice • Combine with resident tracking program • Pilots early 2004 • Production 2005 • Free to all FP residencies September 22, 2003 STFM Theme Day - Competencies t y
V i r g i n i a C o m m o n w e a l t h U n i v e r s i t Comp-E-Val™ • https: //www. pubinfo. vcu. edu/webeval Examalyzer • https: //www. pubinfo. vcu. edu/webeval/examalyzer Riverside Family Practice Residency • • http: //www. riverside. vcu. edu 757 -594 -3878 (FAX) 757 -591 -9021 charles. frazier@rivhs. com September 22, 2003 STFM Theme Day - Competencies y
V i r g i n i a C o m m o n w e a l t h U n i v e r s The future – Ability to study The Outcomes Project – Evaluate Program Curriculum – Ability to describe expected “Natural History” or “Growth Curve” for resident – Add clinical effectiveness September 22, 2003 STFM Theme Day - Competencies i t y
V i r g i n i a C o m m o n w e a l t h U n i v e r s Summary • ACGME outcomes-based education/evaluation may improve our educational outcomes • A web-based, database-driven, competency-based evaluation system can facilitate those efforts, and may aid evaluation of the ACGME Outcomes Project September 22, 2003 STFM Theme Day - Competencies i t y
V i r g i n i a C o m m o n w e a l t h Questions? September 22, 2003 STFM Theme Day - Competencies U n i v e r s i t y