DAFFODIL Strategic Support Evaluated Claus-Peter Klas Norbert Fuhr

DAFFODIL Strategic Support Evaluated Claus-Peter Klas Norbert Fuhr Andre Schaefer University of Duisburg-Essen

Outline n n Introduction to DAFFODIL Evaluation n n Procedure Hypotheses Main Results Outlook 15. 09. 2004 University of Duisburg-Essen

DAFFOFIL is n n n Virtual Digital library with strategic support given by high-level search functions an agent-based system with currently over 35 different agents providing the low-level and high-level functions connects ~15 data sources for searching and browsing from the computer science area 15. 09. 2004 University of Duisburg-Essen

15. 09. 2004 University of Duisburg-Essen

DAFFODIL: Concepts n n n Strategic Support Scientific Workflow Personalization n n Personal library Collaboration Awareness & Annotation Proactivity 15. 09. 2004 University of Duisburg-Essen

Strategic Support n is provided by high-level functions like n metadata search n n n duplication elemination citation search author network based strategems journal and conference run in a integrated way 15. 09. 2004 University of Duisburg-Essen

Scientific workflow create new knowledge Choose data source retrieve knowledge interpret information structured storing 15. 09. 2004 University of Duisburg-Essen

Personalization: Personal Library n Structured folders n n n n document metadata fulltext authors conferences journals websites queries Collaboration through group folders 15. 09. 2004 University of Duisburg-Essen

Personalization: Awareness & Annotation n Long term queries: n n Query-Profiles: interval processing Authors: new publications Journals, conferences: new issues, proceedings See whats new: n n n Message via Email, Fax, SMS, . . . Annotation for scientific discurs 15. 09. 2004 University of Duisburg-Essen

Proactivity n n Contextual monitoring of users & system Direct action n n Spellchecking Authorname completion Related terms Recommend next step n n based on current stage based on collaborative usage 15. 09. 2004 University of Duisburg-Essen

Evaluation framework n Formal bases given by [Saracevic&Covi] Why to evaluate ? n What to evaluate ? n How to evaluate ? n 15. 09. 2004 University of Duisburg-Essen

DAFFODIL Evaluation Scheme n n Hypo I: More effective IR through strategic support Hypo II: More effective IR through (pro)-active system Hypo III: More effective IR through personalisation and collaboration Criteria n n n Effectiveness Efficiency Measure n n n Time Relevance User Satisfaction 15. 09. 2004 University of Duisburg-Essen

Evaluation Phases n 1 st Phase n n n 2 nd Phase n n n 14 students in parallel sessions System efficiency & bug hunting 8 researchers Focus on user: Efficiency and Effectiveness 3 th Phase (expert phase) n 3 librarians, 3 professors Comments, critics, hints 15. 09. 2004 University of Duisburg-Essen n

Gathering Evaluation Data n n n Tasks on questionnaire 1 st phase: protocolls Other phases: video capturing of monitor and face n n store usage see reaction store comments 11 h video data 15. 09. 2004 University of Duisburg-Essen

Real world task classified n 5 Tasks (4 presented) on 4 levels, classified by a „information need typology matrix“ n n work task information need no knowledge 4 L 2 L full knowledge 1 L one object 15. 09. 2004 3 L heterogenous set of objects University of Duisburg-Essen

Result (1 st Phase) Phase 1: 14 students n n Error hunting, system efficiency Outcome: n n n speed up 5 -7 times Find a lot of bugs Comment: „Even so slow when searching, i want to use it, since it saves much time, handling the found knowledge“ 15. 09. 2004 University of Duisburg-Essen

Main Hypotheses n n n more effective and efficient IR through strategic support given with DAFFODIL compared to other (DL) information system more effective, because user can cognitivly process and interpret information more easily more efficient, because user needs to invest less time 15. 09. 2004 University of Duisburg-Essen

Result (2 nd phase) Task 1: Known item instantiation (L 1) n Find document by given title (precise) n 15. 09. 2004 „Search strategies in content-based image retrieval“ University of Duisburg-Essen

Task 2: Known item instantiation (L 2) n Find document by given title (not precise) n 15. 09. 2004 „Methods of automated reasoning“ University of Duisburg-Essen

Result Task 3: A real search (L 3) n n Find articles by author X in the area of digital libraries within the last 4 years. Results n n n 15. 09. 2004 All participants found with Daffodil at least 12 -19 relevant articles. With Google only two persons found articles any articles, the other gave up. Actually, user said, it will be hard with google prior to search. University of Duisburg-Essen

Result Task 5: New area (L 4) n find literature about a specific subject: n not known by any participant prior to the evaluation Explorative Search n n interactive information retrieval; information searching; information seeking Scanning result lists; related terms feature Results: n n 15. 09. 2004 Participants at least found a couple of entry points, to further search Got very soon an idea, what they don‘t want. University of Duisburg-Essen

Result (3 th phase) Experts: Free Search n n n Three librarians and three professors Free Search with own theme Results: n n n input for improving satisfied Comments: n librarian n professor n n 15. 09. 2004 „rich functionality“ „D. mediates the knowledge about the tactics to the user“ „D. seems in a state that it can be used“ „I will propose it to my assistents“ University of Duisburg-Essen

Summary of evaluation n n Evaluation with 28 participants showed, that a user is more effective when searching for scientific literature with the strategic support, here given by the DAFFODIL system, but not always more efficient (task 1). But, as the tasks become more complex, the hypotheses hold. 15. 09. 2004 University of Duisburg-Essen

Outlook n Continue Evaluation n n other systems Long-Term Evaluation n One task from start to end n n new paper for ECDL 2005 Participants are welcome Try it out http: //www. daffodil. de 15. 09. 2004 University of Duisburg-Essen