UWM CIO Office Building Intelligent Dialog Systems to

UWM CIO Office Building Intelligent Dialog Systems to Promote Health Susan Mc. Roy University of Wisconsin-Milwaukee May 2016

UWM CIO Office Outline Benefits of Intelligent Dialog Systems Our Recent dialog-related research – – Question answering Question & discourse classification Other recent research – Frameworks and formative studies for improved software designs for informal caregiving

UWM CIO Office Benefits of Intelligent Dialog Systems • Health providers and consumers often need to obtain complex new information to make the best choices. • Such information may be available from many sources and finding the most helpful information can be difficult. • It is also difficult for providers of health information to create documents that serve a range of information needs,

UWM CIO Office An example where an IDS can help • Women who are pregnant may wonder what they should eat or not eat, what sorts of activities are safe (or unsafe) and when they should worry about various pains, swelling, or other physical conditions that are unusual for them. • They may have these information needs 24/7. • For many, searching the internet is not

UWM CIO Office Other examples • Giving users access to information from their electronic health record or a personally controlled health record. – E. g. to help people with chronic illnesses remember tasks from their discharge instructions and be able to track possible changes in their health status to catch serious changes early.

UWM CIO Office Our recent dialog-related research • We have used computational approaches to natural language dialog to support the practice of Public Health in the city of Milwaukee • Public health is – "the science and art of preventing disease, prolonging life and promoting health through the organized efforts and informed choices of society, organizations, public and private, communities and individuals" (Winslow, 1920)

UWM CIO Office Two target QA tasks • Provide information to improve birth outcomes – Milwaukee has a very high rate of fetal and infant mortality. – Many such deaths are considered preventable – e. g. by changes in moms' health behavior (smoking, nutrition, seeking help). • Provide information to explain complex choices related to prostate cancer screening. – Each year > 200, 000 men are diagnosed and about 32, 000 will die. – However, routine screening does not improved outcomes. CDC says screening decisions should be shared & consider individual risk – Both these tasks create a diverse, but highly individualized sets of information needs – QA might help, by focusing on needed information.

UWM CIO Office The approach: 2 way automated textmessaging • Text-messaging can be a good way to reach many people: – Many people across all ages and ethnic groups now use messaging (For the latest numbers, see Pew research) – We found typically used WWW sources often have > 11 th grade reading level, while MHD clients typically have a 5 th grade level – In our surveys: – Very few subjects reported using the web for health info (either pregnancy or prostate cancer) – Among the older men from low-income areas of Milwaukee, slightly more reported using text messaging (56. 4%) compared the Internet (47%) and only 37% had internet at home

UWM CIO Office A Reusable Architecture for NL Dialog on low-cost devices Content G A T E W A Y Dialog Strategy Dialog Manager Language Manager Authoring Tool Dynamic User Model Dialog History Manager

UWM CIO Office Dialogs to Improve Birth Outcomes • This project used automated question-answering on low-cost cell phones to increase Milwaukee mom’s awareness of factors related to healthy pregnancy and childcare. • It was conducted as part of the Public Health Impact Initiative, at the UWM Zilber School of Public Health.

UWM CIO Office User Study • We built a text database of answers targetted to the needs of low-income moms with the help of City of Milwaukee Health Department (MHD). • Surveys and focus groups were used to select topics. • We created software to map the stored content onto a logic for controlling a question and answer system that can be accessed either via email or SMS messaging. • For the assessment, we recruited women who were MHD clients to use the system for a period of one month each; 20 women completed the study.

UWM CIO Office The NL Dialog Engine At runtime, the dialog engine uses a combination of NL processing techniques (phrases, semantics, etc and not just keywords) to find the best match between stored questions and the user's information need Answers can also be customized, if desired, using patterns that whose values are retrieved dynamically from a separate database.

UWM CIO Office Results of the study • The 20 women who completed the study, generated a total of 291 unique questions, corresponding to 161 different topics. • About 46 % of the questions were covered by topics in the database – 26 % of these were matched correctly within the allowed time; – 58% of these were unmatched; – 1 % were mismatched and – 16% attempted matching exceeded the time allowed.

UWM CIO Office SMS-Expressed Topics • • • 1. Abdominal pains/ cramping (4. 1%) 2. Contractions/labor (3. 4%) 3. Baby movement (3%) 4. Breast feeding (2. 7%) 5. Heartburn (2. 4%) 6. Pelvic pressure/pain (2. 4%) 7. Eating enough (2%) 8. Morning sickness (2%) 9. Prenatal vitamins (2%) 10. Swelling (1. 7%)

UWM CIO Office Key Lessons The information needs of subjects and the forms they used for expressing them were very diverse. Thus, a dialog-based approach, which can provide a tailored response to each person’s needs would be beneficial. Methods for matching must be fast and precise, with extra care given to focusing on the main part of the sentence and the

UWM CIO Office Feasibility of using IDS to support prostate cancer screening decisions To support cancer education, we explored approaches to assessing information needs and performing language recognition which would be needed to create an IDS. • One approach involved collecting consumer's questions posted to Community-based Question Answering web sites like Ask. MD. • We also conducted observational studies, including one where men could send questions via SMS and a MHD nurse provided the answers at the backend. – This followed surveys and lab-based observations to assess whether the modality would be likely acceptable.

UWM CIO Office Our Taxonomy of Expected Answer Types • Our taxonomy for cancer questions has 3 levels. • The top level includes 3 types: – Factoid – Patient Specific – Non-medical • These types correspond to the three possible ways that an automated system should respond: – Provide a direct answer – Explain why the answer must come from a health professional – Explain that the answer is outside the domain of the system

UWM CIO Office Lower levels of EAT Taxonomy Factual (F) Patient Specific (PS) Definition (D) Patient Diagnosis (PD) Entity (E) Patient Outcome (PO) Entity Explanation (EE) Patient Recommendation (PR) Patient Explanation (PE) Numeric Property Value (NPV) Non-Clinical (NC) Reference (R)

UWM CIO Office Phase 1 Results • Our corpus contains about 1200 cleaned and coded questions (about 3% of those that we collected). • However, the distribution among the various categories was found to be very uneven. • Currently the intercoder agreement is best for the toplevel classification (factual, patient-specific, noncliical), which was around. 7 • To improve the distribution, we merged some categories in the taxonomy. • We also explored methods for dealing with unevenness.

UWM CIO Office Filtered Class Distributions

UWM CIO Office Classifier Testing 4 Classification Algorithms were tested – Multinomial Naïve Bayes (MNB) – Naïve Bayes (NB) – J 48 Decision Trees (J 48) – Sequential Minimal Optimization (SMO) 12 Corpora Modifications were tested (6 shown) • Unmodified (U), Spelling Correction (SC) Threshold Term Reduction (T 5, T 25), Range Term Reduction (R 30, R 50) But, best performance was achieved only by using weighted classification and resampling, which yielded accuracy of F 1= 0. 963

UWM CIO Office Human Subjects Studies related to Men's QA • We did 3 types of studies: – – – Written surveys A lab-based study where men were asked to obtain information on prostate cancer either using a search engine or by forming and sending a text message with their questions. A field study where participants could text to an email address monitored by a health department nurse.

UWM CIO Office Results of 2 studies • In surveys, lab and field studies, men reported preference for using text messaging to web-based search. • Subjects in the lab study were as successful as those using search (in terms of number of tasks completed) even though they were using an unfamiliar cell phone and the cell service was often slow (we used the lowest-cost prepaid svc). • In the field study, subjects’ questions were factual, but distinct. One subject sent an average of 3 questions per day – but most averaged 1 or 2. A few men sent no questions, but said “ok” or “thank you”.

UWM CIO Office Some other projects • We have been investigating new health management systems for informal caregivers in the home. We are interested in how targetted feedback and communication can help improve user acceptance. – Completed work includes studies of physical ability and attitudes among older caregivers. – We have also created a framework for predicting the usability and long term use of software and have been conducting user studies to see how well it aligns with people's assessments.

UWM CIO Office Summary • My research spans a broad range of areas, including natural language processing and intelligent user interfaces, and usability studies. • Recent applications involve both public health and health care. • Joining me in this work have been faculty from Communication, Nursing, and Health Sciences as well as students from computer science, biomedical and health informatics, and biomedical engineering.

UWM CIO Office References Mc. Roy, S. Jones, S. and Kurmally , A. “Toward automated classification of consumers' cancer-related questions with a new taxonomy of expected answer types”, Health Informatics Journal, 2015. Mc. Roy, S. , Cramer, E. M. , and Hayeon Song, “Assessing Technologies for Information-Seeking on Prostate Cancer Screening by Low. Income Men”, Journal of Patient-Centered Research and Reviews, 2014 1(4): 188 -196. Song, H. , May, A. , Vaidhyanathan, V. , Cramer, E. M. , Owais, R. W. , and Mc. Roy, S. “A two-way text-messaging system answering health questions for low-income pregnant women”, Patient Education and Counseling, 2013 [PMID: 23711635] Mc. Roy, S. , Vaidyanathan, V. , May, S. and Song, H. “An Open Architecture for Messaging-Based Consumer-Health Question