Knowledge and Usability Franz J Kurfess Computer Science

Knowledge and Usability Franz J. Kurfess Computer Science Department California Polytechnic State University San Luis Obispo, CA, U. S. A. Franz Kurfess: Knowledge Retrieval

Acknowledgements Some of the material in these slides was developed for a lecture series sponsored by the European Community under the BPD program with Vilnius University as host institution Franz Kurfess: Knowledge Retrieval

Use and Distribution of these Slides ❖ These slides are primarily intended for the students in classes I teach. In some cases, I only make PDF versions publicly available. If you would like to get a copy of the originals (Apple Key. Note or Microsoft Power. Point), please contact me via email at fkurfess@calpoly. edu. I hereby grant permission to use them in educational settings. If you do so, it would be nice to send me an email about it. If you’re considering using them in a commercial environment, please contact me first. © Franz J. Kurfess 5

Structure ❖ Introduction ❖ Dimensions of Knowledge Usability ❖ Metrics for Usability of Knowledge ❖ Knowledge-Intensive Activities ❖ Usability Evaluation Methods ❖ Problem Sources © Franz J. Kurfess 6

Introduction Terminology Data, Information, and Knowledge Using Knowledge Search Dimensions of Knowledge Usability 7

Using Knowledge ❖ search v ❖ augmentation v ❖ answers, not documents (that may or may not contain the answer)! knowledge and information relevant to the current task and context actionable knowledge v presented in a way that allows users to apply it immediately © Franz J. Kurfess 14

Example: Tables ❖ knowledge and information is often arranged in tables v columns identify categories v rows represent instances v cells are values in the respective category for an instance © Franz J. Kurfess 15

Sample 1 © Franz J. Kurfess 16

Sample 2 © Franz J. Kurfess 17

Sample 3 4 4/19 Usability and Knowledge Reviewer feedback to paper topic proposal MS Week 4: Prototype 1 (alpha) 4/21 Knowledge Organization A 1 due 5 4/26 Knowledge Search and Retrieval 4/28 Exploratory Search and Retrieval Draft Paper 6 5/3 Knowledge Presentation A 3: Validation MS Week 6: Prototype 2 (beta) 5/5 A 2 due 7 5/10 Knowledge Exchange Reviewer feedback to draft version 5/12 Knowledge Interaction A 2 due 8 5/17 Constrained Access © Franz J. Kurfess 18

Sample 4 ❖ <? xml version="1. 0" ? >   <? xml-stylesheet type="text/css" href="css/Course-Schedule. css"? >    <? xml-stylesheet type="text/css" href="css/581 -Schedule. css"? > <link rel="stylesheet" href="css/Course-Schedule. css" type="text/css"> <link rel="stylesheet" href="css/581 -Schedule. css" TYPE="text/css"> © Franz J. Kurfess 19

Sample 5 User Modeling A 2: Conceptual Model: System/User Paper topic proposal 4/15&16 Open House Activities 4 4/19 Usability and Knowledge Reviewer feedback to paper topic proposal MS Week 4: Prototype 1 (alpha) 21 -Apr Knowledge Organization A 1 due 5 4/26 Knowledge Search and Retrieval 28 -Apr Exploratory Search and Retrieval Draft Paper 6 5/3 Knowledge Presentation A 3: Validation MS Week 6: Prototype 2 (beta) 5 -May A 2 due 7 10 -May Knowledge Exchange Reviewer feedback to draft version 12 -May Knowledge Interaction A 2 due 8 17 -May Constrained Access MS Week 8: Final Version © Franz J. Kurfess 19 -May 20

Usability of Tables ❖ humans are good at interpreting the contents of tables even in the absence of headers v ❖ easy cases for computers v ❖ interpretation based on context, contents of cells, relationships between cells, likelihood of co-occurence of categories data bases, spread sheets, XML schemata, difficulties for computers v images v visually formatted text (e. g. PDF files) v tables in word processors v tables in Web pages © Franz J. Kurfess 21

Current Usage of Retrieval Systems ❖ tools to identify potentially relevant documents ❖ formulation of questions as unnatural queries v ❖ ranking of retrieved documents according to obscure criteria v ❖ either simplistic sets of keywords, or complex expressions re-formulation of queries to influence ranking mostly batch processing v v wait v ❖ submit query view result inconsistent, wrong context, … © Franz J. Kurfess 22

Better Usage of Retrieval Systems ❖ provide answers to questions ❖ find the right information fast ❖ analyze information, combine it into easily digestible formats ❖ summarize longer documents, sets of related documents ❖ relate it to decisions to be made © Franz J. Kurfess 23

Utilization and Usability ❖ utilization v v ❖ dissemination, distribution application of relevant knowledge usability v ease of use v convenience v effectiveness v efficiency © Franz J. Kurfess 24

Usage Aspects ❖ collection and evaluation of usage information v v individuals vs. groups v temporal relationships v ❖ single items vs. sets of items info-space relationships examples v ❖ relevance feedback, user profiles, citation analysis, hypertext links, collaborative filtering problems v technical aspects, quantity of data v privacy © Franz J. Kurfess 25

Knowledge Usage ❖ conceptual use v ❖ instrumental use v ❖ changes in levels of knowledge, understanding, or attitude changes in behavior and practice strategic use v manipulation of knowledge to attain specific goals v power, profit, political gain © Franz J. 1996] [NCDDRKurfess 26

Knowledge Usage Metaphors ❖ “tabula rasa” v ❖ learner as a sponge v ❖ the learner’s mind is an empty slate upon which people “in the know” impress knowledge soaking up knowledge, largely without filtering or processing brain as a computer v processes information in a systematic fashion as it is received from outside sources © Franz J. Kurfess 27

Knowledge Use as Learning Process ❖ role of knowledge v ❖ dynamic set of understandings influenced by its originators and its users role of the learner v actively filters and shapes knowledge v v constructs models of the environment v v integration into existing knowledge explanations to make sense of the world pre-existing (mis-)understandings may have to be changed v they result in discrepancies of the mental model © Franz J. 1996] [NCDDRKurfess 28

Dimensions of Knowledge Usability Functionality Task Completion User Satisfaction Consistency 39

Functionality ❖ feature-complete v v ❖ provides all necessary features to complete the task under investigation provides additional features that make it more convenient for the user feature overload v ratio of frequently used/unused features v ratio of relevant/irrelevant features for a given set of tasks © Franz J. Kurfess 40

Task Completion ❖ time v v milestones v ❖ completion critical activities quality v final product v aspects of the final product v intermediate products © Franz J. Kurfess 41

User Satisfaction ❖ satisfied with the outcome ❖ easy, pleasant to use ❖ need for training ❖ rate of abandon © Franz J. Kurfess 42

Consistency ❖ internal ❖ in-family ❖ external © Franz J. Kurfess 43

Metrics for Usability of Knowledge based on the dimensions of usability identified above 44

Direct Measurements ❖ registration of user interaction activities v mouse movements v mouse clicks v keys pressed v other activities can be registered, but may be problematic v use of Web cams or built-in cameras v may pose privacy problems v may influence the behavior of users © Franz J. Kurfess 45

Practicality of Direct Measurements ❖ aspects like user satisfaction are highly subjective ❖ conditions under which measurements are performed are unrealistic v lab vs. work place ❖ participants in usability evaluations may not be representative for the intended user group ❖ not enough participants for statistically valid results ❖ voluntary participation skews the results v v positive - mostly satisfied users negative - mostly annoyed, unhappy users © Franz J. Kurfess 46

Methods for Direct Measurements ❖ computer-supported event capturing v ❖ keystrokes, mouse-clicks, . . . user observation v v eye tracking, movements, . . . some observed activities may only be indirect measurements for something else v e. g. eye tracking for attention focus ❖ stream of consciousness protocols ❖ user feedback v ❖ typically yields subjective data often supported by video or audio taping © Franz J. Kurfess 47

Indirect Measurements ❖ in principle less desirable v correlation between the observed behavior and the intended measurement can be questionable v ❖ other factors may influence the observed results often the only practical alternative © Franz J. Kurfess 48

Methods for Indirect Usability Measurements ❖ questionnaires, interviews, focus groups, . . . v ❖ stream of consciousness protocols v ❖ usually done after the experiment emphasis on implicit statements user observation v concentration, attention, comfort © Franz J. Kurfess 49

Knowledge Usability Measurement Examples ❖ Internet advertising v significant efforts in measurements v knowledge aspects unclear © Franz J. Kurfess 50

Internet advertising ❖ number of page views ❖ time spent on page ❖ click-through rates v emphasizes the last ad the viewer selected v neglects prior activities and interactions ❖ link trail analysis ❖ engagement mapping v ❖ social interactions v ❖ attempts to take into account all the interactions with a company's marketing message and brand that may have lead up to a purchase or other user action influence of other people close to the participant customer feedback v questionnaire about factors influencing purchasing decisions ❖ very important due to significant monetary interests ❖ soundness of the methods used may be questionable © Franz J. Kurfess 51

Knowledge-Intensive Activities presentation retrieval identification manipulation acquisition creation of knowledge 52

Presentation of Knowledge ❖ How is existing, identified, and retrieved knowledge presented to the user? v Methods v Visual v v graphic v v textual visual metaphors Non-Visual v sound, touch, smell, taste © Franz J. Kurfess 53

Textual Knowledge Presentation ❖ natural language v ❖ complete sentences (prose) phrases v e. g. bullet points ❖ words ❖ numbers © Franz J. Kurfess 54

Graphic Knowledge Presentation ❖ the main emphasis lies on visual primitives v ❖ shape, color, proximity, cohesion examples v v sketch v diagram v graph v ❖ image tree see http: //www. many-eyes. com/ v service for the general public to visualize data sets © Franz J. Kurfess 55

Visual Metaphors ❖ visual displays of familiar objects or situations are used to present knowledge v "house" or "building" as a metaphor from architecture for the structure of computer-based systems v "funnel" as a selection process, filter v emphasis on the reduction of the input quantity v "scale" to "weigh" entities or properties v often used to point out advantages, disadvantages © Franz J. Kurfess 56

Non-Visual ❖ sound v spoken language v music v soundscapes ❖ touch ❖ taste ❖ smell © Franz J. Kurfess 57

Critical Aspects Knowledge Presentation ❖ user background v How familiar is the user with the domain, task, system, . . . ? v How wide is the range of users? v ❖ abilities, motivation, task v v Will the same user perform the same task rarely, repeatedly, regularly? v ❖ What is the (set of) task(s) to be performed by the user? Complexity of the task? context v What is the environment in which the task is to be performed? v multi-tasking v noise © Franz J. Kurfess 58

Examples ❖ search results in search engines © Franz J. Kurfess 59

Retrieval of Knowledge ❖ How is existing and identified knowledge accessed, encapsulated, and transported to the user? © Franz J. Kurfess 60

Retrieval Methods ❖ location (address) ❖ identity (unique name, identifier) ❖ properties ❖ context v v ❖ vicinity membership in a set association v ❖ an item is accessed by following (a chain of) associations with other items similarity v requires a metric or at least an operational characterization of similarity v does the user have to know this? © Franz J. Kurfess 61

Critical Aspects of Knowledge Retrieval ❖ access time ❖ transmission time ❖ encoding and encapsulation v ❖ e. g. encryption, serialization, conversion between different representations Examples v search with search engines v v v text (keywords) images data base query © Franz J. Kurfess 62

Identification of Knowledge ❖ How does the user distinguish wanted from unwanted knowledge entities? © Franz J. Kurfess 63

Methods of Knowledge Identification ❖ browsing ❖ query ❖ similarity ❖ relevance feedback © Franz J. Kurfess 64

Browsing ❖ the user follows an organizational structure of the collection v v v e. g. walking through the stacks of a library, scanning the table of contents in a magazine, following links on a Web page the organizational structure often is related to the content of the documents v e. g. topics can also be arranged according to other criteria v size v date of acquisition v external scheme v e. g. books by ISBN numbers © Franz J. Kurfess 65

Query ❖ the user tries to describe relevant aspects of the items to be identified v presence of keywords, phrases, sentences v sometimes also more complex queries v v e. g. Boolean operators description of properties v ranges or values of important features © Franz J. Kurfess 66

Relevance Feedback ❖ the user provides feedback on the suitability of the items presented ❖ very good basis for usability measurements © Franz J. Kurfess 67

Similarity ❖ the user identifies entities that share important aspects of the one to be identified ❖ there also have to be some undetermined properties v ❖ otherwise the entity is already identified can be combined with v query - identification of distinguishing features v browsing - looking at a set of similar items v relevance feedback - selecting a better instance © Franz J. Kurfess 68

Critical Aspects Knowledge Identification ❖ search for v specific instance v v v unique identification distinguishing features of similar entities must be available “exemplar” v v ❖ entity that satisfies certain requirements other features may be irrelevant identification and retrieval are often combined v “search” © Franz J. Kurfess 69

Examples Knowledge Identification ❖ core activities of search engines v ❖ especially for search in “closed” systems data base queries © Franz J. Kurfess 70

Modification of Knowledge ❖ Methods ❖ Critical Aspects ❖ Examples © Franz J. Kurfess

Knowledge Modification Methods ❖ update v v ❖ overall structure remains unchanged values of some properties of entities are changed structural changes v ❖ relationships between entities are modified insertion/deletion v entities are added or removed v typcially results in structural changes © Franz J. Kurfess 72

Critical Aspects Knowledge Modification © Franz J. Kurfess 73

Examples Knowledge Modification © Franz J. Kurfess 74

Acquisition of Knowledge ❖ explicit knowledge is converted into a representation suitable for storage on computers v often done by knowledge engineers to capture expertise v tacit knowledge has to be converted into explicit knowledge first ❖ Methods ❖ Critical Aspects ❖ Examples © Franz J. Kurfess 75

Knowledge Acquisition Methods ❖ interview ❖ refinement ❖ observation © Franz J. Kurfess 76

Creation of Knowledge ❖ knowledge is made explicit v conversion from the internal representation of the knowledge owner into one that can be shared with others ❖ Methods ❖ Critical Aspects ❖ Examples © Franz J. Kurfess 79

Problems Knowledge Usability ❖ Semantic Gap ❖ Observability and Controllability ❖ Task Specificity ❖ Context and Environment © Franz J. Kurfess 84

Semantic Gap ❖ in the knowledge-based systems domain, this refers to the difference between knowledge processing by humans and by computers v v ❖ humans: emphasis on understanding (semantics) computers: emphasis on symbol manipulation (syntax) here: gap between measurable and observable activities, and the achievement of the goal for a task v e. g. measuring that a user has "understood" the instructions for solving a computer setup problem © Franz J. Kurfess 85

Observability ❖ technical term from control theory v counterpart of controllability ❖ internal states of a system can be inferred by knowledge of its external outputs ❖ it is possible to determine the behavior of a system from its outputs © Franz J. Kurfess 86

Controllability ❖ technical term from control theory v ❖ counterpart of observability to move a system around in its entire configuration space using only certain admissible manipulations v typically inputs are manipulated v does not necessarily mean that the system can be kept in a certain state, only that it is possible to get to that state © Franz J. Kurfess 87

Usability and Observability ❖ internal states of users are typically unknown v v ❖ lack of a cognitive model insufficient information about the user mental activities are difficult to observe and measure v technology such as f. MRI exists, but is impractical for most situations v self-observation can be used, but is also problematic v subjective v probably has an impact on performance since it requires mental resources © Franz J. Kurfess 88

Usability and Controllability ❖ usually it is impractical (and unnecessary) to cover the entire configuration space of a system v ❖ with exceptions, e. g. safety-critical systems, legal requirements user may have to be considered part of the entire system © Franz J. Kurfess 89

Task Specificity ❖ goal criteria may be highly task-specific v difficult to generalize v usability metrics may depend on goal criteria © Franz J. Kurfess 90

Context and Environment ❖ knowledge can be highly context-dependent v ❖ environment can have a significant influence on user performance v ❖ makes measurements difficult lab vs. work environment observation effect v the behavior of the user may be different when observed © Franz J. Kurfess 91

Sources of Problems ❖ Conceptual Mismatch ❖ Labeling Mismatch ❖ Descriptive Mismatch ❖ Representational Mismatch © Franz J. Kurfess 92

Conceptual Mismatch ❖ The concepts the user has in mind do not match those utilized by the developers of the knowledge representation, retrieval, or presentation system. © Franz J. Kurfess 93

Labeling Mismatch ❖ The concepts used by the user and by the developer match reasonably well, but they use different labels (terms) for them. This may include ambiguities, homonyms, context (domains, everyday vs. technical terms) problems across languages © Franz J. Kurfess 94

Descriptive Mismatch ❖ User and developer describe relevant aspects of entities in different ways. v A user saw a pretty red skirt with polka dots and ruffles in a fashion magazine, and has problems finding it on the Internet using the above keywords. © Franz J. Kurfess 95

Representational Mismatch ❖ The internal structure of the representation of the knowledge is different from the one used by the user. v Example: A tall basketball player wants to buy a car, and needs one that can accommodate drivers larger than 2. 00 meters. Most knowledge sources for cars (data bases, reviews, manufacturer information) will only contain related data that may give hints, but not the full answer to such a requirement. © Franz J. Kurfess 96

References ❖ [Gil 2000] Yolanda Gil, Knowledge Mobility. Dagstuhl Workshop “Semantics for the Web, ” March 2000. ❖ [NEEDS] National Engineering Digital Library, www. needs. org ❖ [Russell & Norvig 1995] Stuart Russell and Peter Norvig, Artificial Intelligence - A Modern Approach. Prentice Hall, 1995. © Franz J. Kurfess 99