ITS Data Collection Framework Capturing data based on agent communication standard Olga Medvedeva, Center for Pathology Informatics, University of Pittsburgh July 10, 2005 Educational Data Mining Workshop 20 th AAAI-05 Conference
Outline • Need for communication standard for Intelligent Tutoring Systems • Existing standard for multi-agent communication • Implementation in Slide. Tutor – Communication protocol – Data collection – Database query tool – Lessons learned • Comparison with recent standardization effort • Advantages of using the existing standard July 10, 2005 Educational Data Mining Workshop 20 th AAAI-05 Conference
Intelligent Learning Environment Common Base • Underlying theory – Cognitive tutors (Anderson et al. 1995) – Adaptive hypermedia (Brusilovsky et al. 1996) – Constraint-based (Mitrovic et al. 2001) • Modules – Expert, Student, Interface, Pedagogic • “Single-purpose” development approach July 10, 2005 Educational Data Mining Workshop 20 th AAAI-05 Conference
Keystone – communication standard • Previous efforts: – Inter-tutor communication (Ritter, Koedinger 1996; Brusilovsky et al. 1997) one-to-one translators, strict channel, no real protocol – Shared resources (Koedinger et al. 1999) – limited use: lack of standard – DORMIN protocol (developed at CMU) – used in commercial product • Our approach – Multi-agent technology – Use existing inter-agent communication standard July 10, 2005 Educational Data Mining Workshop 20 th AAAI-05 Conference
Foundation for Intelligent Physical Agents (FIPA) FIPA (www. fipa. org) - collection of standards for inter-agent communication: • Agent Management System – manages an agent life-cycle, maintains a registry with unique Agent Identifier (AID) • Transport – describes message exchange protocol: transport type and specific address for an agent • Agent Communication Language (ACL) – communication specifications FIPA was officially accepted by the IEEE as one of its standards committees on 8 June 2005 July 10, 2005 Educational Data Mining Workshop 20 th AAAI-05 Conference
FIPA Design Principals Envelope: Sender (locator) Receiver (locator) Timestamp Message (ACL): Sender (AID) (ACL): Message Receiver (AID) Performative (String) Sender (AID) Content: ( ACR) Receiver (AID) Performative (String) Reply-to(Message ID) Content: ( ACR) Reply-to (Message ID) July 10, 2005 • Forms abstract basis for concrete architecture • Sets minimum required elements • Permits introduction of new elements • Permits arbitrary content language, uses Abstract Content Representation (ACR) for ACL as key-value pairs Educational Data Mining Workshop 20 th AAAI-05 Conference
FIPA ACL Message Structure : sender – identity of the sender : receiver – identity of the recipient : content – the object of the action : performative – the type of the communicative act Optional: : reply-with : replay-to : in-replay-to : replay-by– replay constraints : language – encoding schema of the content of the message : encoding – encoding identifier : ontology – is used to give a meaning to symbols/concepts in the content : protocol – gives additional context for the interpretation of the message : conversation-id – identifies the ongoing sequence of communicative act, manages the conversation strategies July 10, 2005 Educational Data Mining Workshop 20 th AAAI-05 Conference
FIPA Performatives • • • Accept-proposal Agree Cancel Call-for-proposal Confirm Disconfirm Failure Inform-if Inform-ref Not-understood July 10, 2005 • • • Propagate Propose Proxy Query-if Query-ref Refuse Reject-proposal Request-whenever Subscribe Educational Data Mining Workshop 20 th AAAI-05 Conference
FIPA Implementation in Java • Java Agent Services (JAS) (www. jcp. org) defines a set of objects and service interfaces to support the deployment and operation of the agents. • Contains interfaces for building messages, directory services and a factory for message transfer services. • JAS is a base for multi-agent communication in our system July 10, 2005 Educational Data Mining Workshop 20 th AAAI-05 Conference
Slide. Tutor Architecture http: //slidetutor. upmc. edu Slide. Tutor - an agent-based model tracing ITS for visual classification problem solving in surgical pathology July 10, 2005 Educational Data Mining Workshop 20 th AAAI-05 Conference
July 10, 2005 Educational Data Mining Workshop 20 th AAAI-05 Conference
Generic Representation of Problem-Solving Space July 10, 2005 Educational Data Mining Workshop 20 th AAAI-05 Conference
Collected Data • Interface. Event – lowlevel human-computer interactions • Client. Event – collection of Interface. Events that represents an elementary subgoal, understood by tutor • Tutor. Response – system response to a Client. Event July 10, 2005 Educational Data Mining Workshop 20 th AAAI-05 Conference
Message Example Client. Event Envelope: Sender: Client_1 Receiver: PROTOCOL Time. Stamp = 1114444377783 Message: Sender: Concept 2 Receiver: PROTOCOL Performative: X-Created In-reply-with: 1114444378242 Content: Type = Finding Label = blister Id = Concept 2 Object. Description = Finding. blister. Concept 2 Parent = null Input: name = text value = blister name = y value = 11808 name = x value = 38048 name = z value = 0. 03 Interface. Event. IDS = [1114444374333, 1114444375546, 1114444376304, 1114444376798, 1114444377444] July 10, 2005 • Envelope indicates the locators of client and protocol agents • 4 required key-value pairs for a message • Performative defines a type of communicative act • List of preceding Interface. Event Ids: – click on Finding button – Click on image – Selecting 3 times down a tree of findings Educational Data Mining Workshop 20 th AAAI-05 Conference
Message in Depth Client. Event • Envelope: Sender: Client_1 Receiver: PROTOCOL Time. Stamp = 1114444377783 Message: Sender: Concept 2 Receiver: PROTOCOL Performative: X-Created In-reply-with: 1114444378242 Content: Type = Finding Label = blister Id = Concept 2 Object. Description = Finding. blister. Concept 2 Parent = null Input: name = text value = blister name = y value = 11808 name = x value = 38048 name = z value = 0. 03 Interface. Event. IDS = [1114444374333, 1114444375546, 1114444376304, 1114444376798, 1114444377444] July 10, 2005 • • Widget object (agent) description parameters – Type (“Button”, “Finding”) – Label (“Next”, “Blister”) – Id – unique within a session – Object. Description – combination of Type+Label+Id (“Finding. blister. Concept 2) – Parent – list of all parent Object. Descriptions for hierarchical structures Common for ITS user action triplet – Action = Performative – Selection = Object. Description+Parent – Input = list form Content Input Message encoded in XML is easy to translate into other languages (RDF, KIF, SL, etc. ) Educational Data Mining Workshop 20 th AAAI-05 Conference
Tutor. Response Example • Student performance data Envelope Sender: Tutor. Engine 0 Receiver: PROTOCOL Time. Stamp: 1114444379378 Message: Sender: Tutor. Engine 0 Receiver: PROTOCOL Performative: FAILURE Conversation_ID: 1114444378242 Content: Error. Code = 15 Next. Step. Type = Evidence Next. Step. Label = blister Next. Step. ID = Concept 2 Next. Step. Parent = null Next. Step. Action = DELETE Input: name = Messages value = "[TEXT: There is BLISTER present, but not where you have pointed in the image. See if you can find where. POINTERS: [Point. To: Concept 2 Is. Permanent: false Method: set. Flash Args: [true]]]“ name= Tutor. Action value = "Point. To: Concept 2 Is. Permanent: false Method: set. Background. Color Args: [RED]" – Performative: FAILURE – user took incorrect step – Error. Code = 15 – user incorrectly located existing finding – Input: - contains a description of an error message to be presented to user • Tutor performance data July 10, 2005 – Best possible next step – action expert model would take in this problem state Educational Data Mining Workshop 20 th AAAI-05 Conference
Database Schema • High-level static tables similar to Mostow et al. 2002 contains Experiment, Case. List, Student, etc. • Low-level tables for captured events, including start/end of problem and session closely follow the FIPA standard, generic with any number of event parameters stored in corresponding Input tables July 10, 2005 Educational Data Mining Workshop 20 th AAAI-05 Conference
Web-Based Protocol Query Tool • Allows the user to obtain data sets specific to a wide range of constraints • Outputs to HTML file that can be transferred to Excel • Query can be saved and viewed in SQL • Interface, Client and Tutor events data can be joined in different ways July 10, 2005 Educational Data Mining Workshop 20 th AAAI-05 Conference
Interface. Events Query Tool Results for Identifying Blister Client. Events Tutor. Responses July 10, 2005 Educational Data Mining Workshop 20 th AAAI-05 Conference
Advantages of Event-Based Data Representation • Usability Perspective: Interface. Events linked to Client. Events (Saadawi et al. 2005) – How many actions were performed – How much time was required to achieve a particular subgoal, such as identification of Blister – How many Interface. Events were unrelated to any Client. Event • Student Performance over time: Client. Events linked to Tutor. Responses – Number of hints requested – Depth of hints – Error frequency and distribution • Tutor Performance: Next. Step fields in Tutor. Responses – Compare next student actions to those predicted by tutor July 10, 2005 Educational Data Mining Workshop 20 th AAAI-05 Conference
Slide. Tutor Data Sharing Limitation • This paper and presentation have been approved by Institutional Review Board (IRB) • Researcher needs to sign a Limited Use Agreement • There might be one agreement with consortiums July 10, 2005 Educational Data Mining Workshop 20 th AAAI-05 Conference
Lessons Learned For the past year our data collection framework was used in 4 small HCI studies and one large experiment with a total of 50 students. • Keep data clean: ended up maintaining ‘raw’ and ‘clean’ copies of database • Granularity of captured data: capturing of detailed data slows the system • Separate database for assessment: no explicit mapping of performance on tests and in the tutoring system July 10, 2005 Educational Data Mining Workshop 20 th AAAI-05 Conference
Data Collection Framework Advantages • Advantages of relational database (Mostow et al. 2002) – Eases the analysis of the enormous volume of complex data • Generic framework that might be adapted to other model-tracing ITS – Adapted in the extension of Slide. Tutor – Report. Tutor that teaches how to write the pathology reports • Flexibility of FIPA-based communication protocol – Flexibility to describe interaction events – Extendable set of performatives – Multiple messages in one envelope, unrestricted number of input parameters – Potential to reference ontologies within the message – Can be easily reused in the Data Shop July 10, 2005 Educational Data Mining Workshop 20 th AAAI-05 Conference
Data Shop Project, Pittsburgh Science of Learning Center (http: //www. learnlab. org ) • Logging and Analysis: Tools and reports to aid PSLC researchers and course developers – Log the activities of the experiments to a database – Provide the reports and queries on that experiments • Goal: Standardize the messaging format among tools, tutoring translators and agents – Message types: tool_message, tutor_message, curriculum_message, message • Data Shop Tutor Logging v 3 released in June 2005 July 10, 2005 Educational Data Mining Workshop 20 th AAAI-05 Conference
Data Shop Tool Message and Slide. Tutor Interface/Client events tool_message attempt_id meta (0 or 1) user_id session_id time_zone problem_name (0 or 1) semantic_event id semantic_event_id name trigger event_descriptor (0+) event_id selection (0+) id type action (0+) id input (0+) id step (0+) probability ui_event id July 10, 2005 (1+) Educational Data Mining Workshop 20 th AAAI-05 Conference
Data Shop Tutor Message and Slide. Tutor. Response meta (0 or 1) user_id session_id time_zone event_descriptor (0+) event_id problem_name (0 or 1) semantic_event id semantic_event_id name trigger selection (0+) id type action (0+) id input (0+) id step (0+) probability ui_event id action_evaluation (0+) current_hint_number total_hints_available classification tutor_advice (0+) skill (0+) probability production (0+) July 10, 2005 step_interpretation (0+) name (1) value (1) custom_field (0+) name (1) value (1) Educational Data Mining Workshop 20 th AAAI-05 Conference
FIPA Advantages • FIPA as a information exchange underlying standard – Develop a set of performatives – a controlled vocabulary for ITS communication – Create sharable ontologies for domain knowledge, hint content, error categories and use ‘: ontology’ FIPA parameter to give a meaning to the message content – Use ‘: protocol’ parameter to identify the translator and to preserve the internal component structure • Syntactically aligned systems – Ease meta-analysis for tutors with the identical performatives – Reuse data for simulations – Shared services for real-time interoperability • Identifying particular help-seeking behavior • Calculating knowledge tracing probabilities July 10, 2005 Educational Data Mining Workshop 20 th AAAI-05 Conference
Acknowledgements Grants: • National Library of Medicine • National Cancer Institute People: • • • Rebecca Crowley Girish Chavan Eugene Tseytlin Elizabeth Legowski Katsura Fujita Maria Bond July 10, 2005 Educational Data Mining Workshop 20 th AAAI-05 Conference
References • • Anderson JR, Corbett AT, Koedinger KR, and Pelletier R. Cognitive Tutors: Lessons learned. Journal of the Learning Sciences 4(2): 167 -207, 1995 Brusilovsky, P. , Kommers, P. & Streitz, N. (Eds. ) (1996) Multimedia, Hypermedia, and Virtual Reality (LNCS Vol. 1077). Berlin: Springer-Verlag, 1996 Mitrovic A, Mayo M, Suraweera, P and Martin, B. Constraint-Based Tutors: A Success Story. In Monostori, L. and Vancza, J. (Eds). Proceedings of the 14 th International Conference on Industrial & Engineering Applications of Artificial Intelligence and Expert Systems, Budapest, Hungary, Springer, pp. 931 -940, 2001 Ritter, S. and Koedinger, K. R. (1996). An architecture for plug-in tutor agents. Journal of Artificial Intelligence in Education, 7, 315 -347 Brusilovsky, P. , Ritter, S. , & Schwarz, E. Distributed intelligent tutoring on the Web, Proceedings of AIEDâ 97, the Eighth World Conference on Artificial Intelligence in Education. 1997 Koedinger KR, Suthers DD, & Forbus KD. Component-based construction of a science learning space: A model and feasibility demonstration. International Journal of Artificial Intelligence in Education: 10, 392 -31, 1999 Mostow J, Beck J, Chalasani R, Cuneo A, and Jia P. Viewing and Analyzing Multimodal Human-computer Tutorial Dialogue: A Database Approach. Proceedings of the ITS 2002 Workshop on Empirical Methods for Tutorial Dialogue Systems, 75 -84 Saadawi G, Legowski E, Medvedeva O, Chavan G, and Crowley RS. A method for automated detection of usability problems from client user interface events. Accepted to Proceedings of the American Medical Informatics Association Symposium 2005 July 10, 2005 Educational Data Mining Workshop 20 th AAAI-05 Conference
Скачать презентацию ITS Data Collection Framework Capturing data based on
f94f4da16918c4d900ab7c78f7a8a37f.ppt