Use Usability Usefulness and Impact of an Infobutton

Use, Usability, Usefulness and Impact of an Infobutton Manager James J. Cimino, M. D. Department of Biomedical Informatics Columbia University College of Physicians and Surgeons 2006 AMIA Fall Symposium

Evaluating the Infobutton Manager • What are Infobuttons? • What is the Infobutton Manager? • How do we study it (modalities)? • What did we learn? – How is it used? – How usable is it? – How useful is it? – What is its impact?

Infobuttons vs. Infobutton Manager Resource s Clinical System Infobutton Context Infobutton Manager Page of Hyperlinks

CPMC Infobutton Managers Deployed: • New York Presbyterian Hospital’s Web. CIS • New York Presbyterian Hospital’s Eclipsys • New York State Psychiatric Institute’s PSYCKES • Regenstrief Medical Record System • Crystal Run Healthcare’s Next. Gen

Evaluation of Infobuttons Rosenbloom ST, Geissbuhler AJ, Dupont WD, et al. Effect of CPOE user interface design on user-initiated access to educational and patient information during clinical care. JAMIA. 2005; 12(4): 458 -73. • 105 house officers studied over 11 months • 278 uses • Findings: 15 -fold increased use of resources Maviglia SM, Yoon CS, Bates DW, Kuperman G. Knowledge. Link: impact of context-sensitive information retrieval on clinicians' information needs. JAMIA. 2006; 13(1): 67 -73. • 359 users over 12 months • 7, 972 uses • Findings: 84% found answers to their question, positive impact on patient care 15% of the time

Evaluation Modalities • System log files

Mar 5 06: 46: 03|xxx 9017|156. 145. 18. 47|lab^524732|view Mar 5 06: 46: 33|xxx 9017|156. 145. 18. 47|healthresource|view Mar 5 06: 46: 35|xxx 9017|156. 145. 18. 47|url^http: //www. crlonline. com Mar 5 06: 47: 21|xxx 9017|156. 111. 245. 19|Lab. Detail|ibutt^35776 Mar 5 06: 47: 24|xxx 9017|156. 111. 245. 19|Lab. Detail|Infomgr^70^35776 What does the CPMC Lab Manual say about the Serum Ferritin test?

Evaluation Modalities • System log files • Pop-up surveys

Evaluation Modalities • System log files • Pop-up surveys • On-line feedback

Evaluation Modalities • System log files • Pop-up surveys • On-line feedback • E-mail surveys

1) How often do you use Infobuttons 5) When I click on a question on the question list __ Once a month or less __ I almost always get a helpful answer __ A few times a month __ I usually get a helpful answer __ A few times a week __ I get a helpful answer about half the time __ A few times a day __ I usually do not get a helpful answer -----------------------------__ I almost never get a helpful answer 2) Infobuttons are easy to use: ------------------------------__ Strongly Agree 6) Which statement most accurately describes your __ Agree Somewhat experience with Infobuttons? __ Neither Agree nor Disagree __ Infobuttons almost always have a negative effect on __ Disagree Somewhat my patient care decisions __ Strongly Disagree __ Infobuttons sometimes have a negative effect on ------------------------------patient care 3) When I click on an Infobutton in Web. CIS, the __ Infobuttons have no effect (positive or negative) on question I am interested in is: my patient care decisions __ Almost never on the list __ Infobuttons sometimes have a positive effect on my __ Usually not on the list patient care decisions __ Is only on the list about half the time __ Infobuttons almost always have a positive effect on __ Usually on the list my patient care decisions __ Almost always on the list ------------------------------------------------------------ 7) Please tell us about a time that Infobuttons 4) Compared to the way I usually get answers to improved a patient care decision (please do my questions, the Infobuttons are: not include any information that would __ Much slower identify a patient) __ Somewhat slower ------------------------------__ About the same speed 8) Please tell us about a time that Infobuttons hindered __ Somewhat faster a patient care decision (please do not include __ Much faster any information that would identify a patient)

Evaluation • Log file analysis 32 months of data analysis: • On-line survey • Popup questionnaires • E-mail survey

Results: System Log Files • March 2004 -October 2006: – IM: 3, 009 users, 29, 541 times

Infobutton Manager (IM) Usage

Resource Use by Context Laboratory Results Review Inpatient Drug Order Review Microbiology Sensitivity Results Diagnosis List Inpatient Lab Order Entry Inpatient Drug Order Entry Health Resources Infobutton Manager

Results: System Log Files • March 2004 - October 2006: – IM: 3, 009 users, 29, 541 times – HR: 3, 609 users, 155, 718 times • October 2006: – IM: 281 users, 1, 022 times (33/day) – HR: 708 users, 5, 744 times (185/day)

Resource Use by Context - October

Finding a Resource or Question • Users chose questions 48. 7% of the time Diagnosis list: 10. 5 -24. 3% Outpatient drug order review: 23. 0 -63. 3% Inpatient drug order review: 63. 3 -78. 7% • Users chose at least one resource from HR page 92. 4% of the time (82. 6 -93. 7%, depending on user type and context)

Evaluation • Log file analysis 32 months of data analysis: • On-line survey 108 comments survey: • Popup questionnaires • E-mail survey

Results: On-line Feedback • 108 questions or topics submitted by users – 54 (50%) in Laboratory Results Review context • 30 were about test interpretation – 22 (20%) in Inpatient Drug Order Review • 15 were about side effects, dosage, cost, trade names, and interactions

Evaluation • Log file analysis: 32 months of data • On-line survey: 108 comments • Popup questionnaires: 195/~3, 642 (5. 4%) • E-mail survey

Pop-up Survey Responses Q 1: Easy to use Q 2: Got answer Q 3: Helpful

Evaluation • Log file analysis: 32 months of data • On-line survey: 108 comments • Popup questionnaires: 195/~3, 642 (5. 4%) • E-mail survey 73/1, 228 (5. 8%) survey:

E-mail Survey Responses Q 2: Infobuttons are easy to use Q 3: Presence of question on the list Q 4: Speed of answer Q 5: Helpful answer Q 6: Positive effect on patient care decisions 1 (strongly positive) to 5 (strongly negative)

Results: Pop-Up and E-Mail Surveys Pop-Up 83% E-Mail 92% 89% Easy to use Question on list >50% of time Answered question 69% Useful 77% Helpful >50% of time 90% Positive effect on care 74% Specific cases of improved care 14 (40% of negative responses due to Lab Manual) (Students were the least positive)

Summary of Impact • Over 29, 000 additional accesses to resources (19%) • Positive impact at least half the time: 74% • 14 respondents identified one or more specific situations in which patient care improved

Lessons Learned • Use of the IM has been more gradual than desired • User fails to select a question half the time – Is the IM failing to anticipate the information need? – Example - questions sugested during lab review: Housestaff did not select a question 57. 8% of time Online survey: 4/11 dealt with reference ranges IM almost always provides a link to lab manual – Is the user unable to find questions on the IM page? • Solutions: – Education – Add questions – User interface

Conclusions 1. Context-specific access to health knowledge resources has been successfully accomplished 2. Impact on patient care decisions has been positive 3. Need to improve user interface for navigation 4. Increased resource use should result in clinicians making better informed patient care decisions

Acknowledgments • This work is supported by NLM grant R 01 LM 07593 • Evlauation plan: – Vimla Patel – Sue Bakken – Leanne Currie – Beth Friedman • Programming: Jianhua Li • Log files: Rick Gallagher

Questions? www. dbmi. columbia. edu/cimino/amia 06 -app. pdf