Knowledge-Based Interpretation Visualization and Exploration of Time-Oriented Medical

Knowledge-Based Interpretation, Visualization, and Exploration of Time-Oriented Medical Data Yuval Shahar, M. D. , Ph. D. Medical Informatics Center Information Systems Engineering Ben Gurion University Beer Sheva, Israel And Departments of Medicine and Computer Science Stanford Medical Informatics Stanford University Stanford, CA, USA

Time in Natural Language From— “Mr. Jones was alive after Dr. Smith operated on him” Does it follow that— “Dr. Smith operated on Mr. Jones before Mr. Jones was alive? ” Is Before the inverse of After?

Timing is Everything : Applications of Temporal Reasoning • Natural-language processing (e. g. , medical record understanding) • Planning (e. g. , robot planning, therapy planning) • Causal reasoning (e. g. , diagnosis) • Archeology (e. g. , seriation) • Psychology (e. g. , developmental beahvioral psychology) • Scheduling (e. g. , optimal ordering) • Circuit design (e. g. , sequential circuits) • Software design (e. g. , parallel processing, communication, verification) • Other, not necessarily time-oriented, domains where interval algebra is useful, such as molecular biology (e. g. , arrangement of DNA segments along a linear DNA chain) and evaluation of spatiotemporal traffic-control patterns

Allen's Temporal Logic (1981– 1984) • Only temporal intervals - no instantaneous events • 13 basic (binary) interval relations (b, a, eq, o, oi, s, si, f, fi, d, di, m, mi) and transitivity relations between them • Properties hold over every subinterval of an interval —> Holds(p, T) e. g. , "Patient 1's skin was blue throughout sunday" • Events hold only over an interval and not over any subinterval of it —> Occurs(e, T) e. g. , "patient 2 broke a leg at 5 pm" • Processes hold over some subintervals of the interval they occur in —> Occuring(p, T) e. g. , "patient 3 is chasing the nurse"

Allen’s 13 Temporal Relations

The Temporal-Abstraction Task • Input: time-stamped clinical data and relevant events (interventions) • Output: interval-based abstractions • Identifies past and present trends and states • Supports decisions based on temporal patterns, such as: “modify therapy if the patient has a second episode of Grade II bone-marrow toxicity lasting more than 3 weeks” • Focuses on interpretation, rather than on forecasting

Temporal Abstraction: The Bone-Marrow Transplantation Domain PAZ protocol BMT Expected CGVHD M[0] Platelet counts ² ² • • ( • ) 150 K 100 K 0 50 100 M[1] M[2] M[3] . M[1] ²² ² ² ² • • • 200 Time (days) ² • M[0] Granulocyte counts ² ²² • • • (²) 2000 1000 400

Temporal Abstractions: A Graphical View

Uses of Temporal Abstractions • Therapy planning and patient monitoring. E. g. , the EON project (a modular architecture to support guideline-based care) • Creating high-level summaries of time-oriented medical records • Supporting an explanation module for a medical DSS • Representing goals and policies of therapy plans and guidelines for quality assessment purposes (at runtime and retrospectively). E. g. , the Asgaard project: Intentions of guideline designers with respect to both process and outcomes are captured as temporal patterns to be achieved or avoided. • Visualization of time-oriented clinical data: the KNAVE project

The Temporal-Abstraction Ontology • Events (insulin therapy) - part-of, is-a relations • Parameters (hemoglobin values and abstractions) - abstracted-into, is-a relations • Abstraction goals (therapy of diabetes patients) - is-a relations • Interpretation contexts (effect of regular insulin) - subcontext, is-a relations • Interpretation contexts are induced by other entities and can have any temporal relationship to the inducing entity

Temporal-Abstraction Output Types • • State abstractions (LOW, HIGH) Gradient abstractions (INC, DEC) Rate Abstractions (SLOW, FAST) Pattern Abstractions (CRESCENDO) - Linear patterns - Periodic patterns

Abstraction of Periodic Temporal Patterns Temperature Hemoglobin Level Periodic Pattern Linear Component Fever Anemia Week 1 Fever Linear Component Fever Anemia Week 2 Anemia Linear Component Fever Anemia Week 3

Temporal-Abstraction Knowledge Types • Structural (e. g. , part-of, is-a relations) - mainly declarative/relational • Classification (e. g. , value ranges; patterns) - mainly functional • Temporal-semantic (e. g. , “concatenable” property) - mainly logical • Temporal-dynamic (e. g. , interpolation functions) - mainly probabilistic

The RÉSUMÉ System Architecture. Temporal-abstraction mechanisms Domain knowledge base Temporal fact base E v e n ts Event ontology C o n te x ts Context ontology A b s tr a c te d in te r v a ls Parameter ontology P r im itiv e d a ta External patient database Events Primitive data • + + • • +

Test Domains for the RÉSUMÉ System • Medical domains: – Guideline-based care • AIDS therapy • Oncology – Monitoring of children’s growth – Therapy of insulin-dependent diabetes patients • Non-medical domains: – Evaluation of traffic-controllers actions – summarization of meteorological data

Acquisition of Temporal-Abstraction Knowledge

Temporal Reasoning and Temporal Maintenance • Temporal reasoning supports inference tasks involving time-oriented data; often connected with artificial-intelligence methods • Temporal data maintenance deals with storage and retrieval of data that has multiple temporal dimensions; often connected with database systems • Both require temporal data modelling

Tzolkin: A Temporal-Mediation Architecture or: Combining Temporal Reasoning and Temporal Maintenance

Knowledge-Based Visualization and Exploration of Time-Oriented Medical Data: Desiderata • • Interactive composition of (temporal-abstraction) queries Visualization of query results Exploration of multiple levels of temporal abstractions The semantics of the query, visualization and exploration operators should be domain independent, but should use the terms and relations specific to each (e. g. , medical) domain

The KNAVE Project (Knowledge-Based Navigation of Abstractions for Visualization and Explanation) • A conceptual and computational framework for temporal abstraction, visualization, and exploration • Capitalizes on existing components (RÉSUMÉ, temporal database mediator, KA tool, domain-knowledge server) • The exploration operators reuse (and are defined by) the domain’s temporal-abstraction ontology • Introduces new graphical and computational modules

The KNAVE Architecture Temporal Mediator Résumé Expert physician Chronus KA Tool Controller Domain-knowledge Server Ontology server DB Visualization and exploration module End user Graphical Interface Computational Module KB

Beginning a Visualization Session: A Temporal-Abstraction Query

The Browsing and Exploration Interface

Semantic Exploration Operators • Motion across semantic links in the domain’s knowledge base; in particular, relations (and their inverse) such as: - part-of - is-a - abstracted-from - subcontext • Motion across abstraction types: state, gradient, rate, pattern • Application of aggregation operators such as mean and distribution • Dynamic change of temporal-granularity (e. g. , from days to months) changes the display, using domain-specific aggregation knowledge • Explanation by display of relevant knowledge, or through “What-if” queries, which allow hypothetical assertion or retraction of data or knowledge and examination of resultant patterns

An Abstracted-From Exploration Result

A Statistical-Query Example

Responding to an Explanation Query (“How”): A Bone-Marrow–toxicity Classification Table

The Preliminary Evaluation Study • • Developmental assessment of the prototype Seven users with varying medical/computer use backgrounds Each user given a 10 minute introduction to the KNAVE system A single electronic patient file constructed from several cases in the domains of AIDS and bone-marrow transplantation • Each user asked to perform three tasks (a complex temporal query, a context-sensitive abstraction, and a statistical query) • Qualitative impression and quantitative (time) measures noted

The Preliminary-Evaluation Results • All users answered all queries within 3 minutes; 6 of 7 users completed all three tasks within 90 seconds • All users expressed enthusiasm and found the interface useful • Striking redundancy noted in use of interface: At least four different paths were found to the same answers, and five different patterns of use of the exploration operators • Difficult to compare to manual tools, since these do not support any automated abstraction or explanation of such

KNAVE: Current State and Future Directions • Basic prototype in Visual Basic; Java implementation under way • Collaboration with an industrial company to create a web-based version • Current research issues: – Implementation of temporal-granularity semantic zoom – Runtime linear and periodic pattern queries – Semantics and implementation of distributed What-If queries, which modify either the knowledge or the data at runtime and examine the effect of the result on the displayed patterns – Enhancement of RÉSUMÉ and the KA tool as needed, including integration with statistical tools – Future link to a text summarization module

Temporal-Abstraction and Visualization: Conclusions • Temporal abstraction of time-oriented data can employ reusable domain-independent computational mechanisms that rely on access to a domain-specific temporal-abstraction ontology • Temporal abstraction is useful for planning, monitoring, data summarization and visualization, explanation and critiquing • Interactive query, visualization of, and exploration requires runtime access to the domain’s temporal-abstraction ontology • The visualization and exploration semantics can be specific to the temporal-abstraction task, but not to the domain