Using Collaborative Filtering in an Intelligent Tutoring System

Using Collaborative Filtering in an Intelligent Tutoring System for Legal Argumentation Niels Pinkwart, Vincent Aleven, Kevin Ashley, and Collin Lynch Carnegie Mellon University of Pittsburgh

An ITS for legal argumentation o o o Problem: legal argumentation is an illdefined domain ITS approach: Engage students in analyzing & reflecting about examples of expert Socratic reasoning Application of collaborative filtering and social navigation principles: n n “Standard” activities: markup, “tagging” resources, recommending objects created by peers Novel function: indirect, results employed as tools to generate better feedback in ITS Niels Pinkwart AH 2006 Using Collaborative Filtering in an Intelligent Tutoring System for Legal Argumentation 2

US Supreme Court Oral Arguments o o o Important part of decision process Attorneys propose a decision rule (“test”) to determine how to decide a case Justices challenge these tests, often by posing hypothetical scenarios Niels Pinkwart AH 2006 Using Collaborative Filtering in an Intelligent Tutoring System for Legal Argumentation 3

An Example Case o o o Example: Lynch v. Donnelly 465 U. S. 668 (1984) Facts: The city of Pawtucket annually erected a Christmas display located in the city's shopping district. The display included such objects as a Santa Claus house, a Christmas tree, a banner reading "Seasons Greetings, " and a nativity scene. Question: Did this violate the constitutional separation of Church and State? Niels Pinkwart AH 2006 Using Collaborative Filtering in an Intelligent Tutoring System for Legal Argumentation 4

Example: Tests and Hypotheticals MR. DE LUCA: With the possible exception of the cross, the nativity scene is one of the most powerful religious symbols in this country, and most certainly one of the most powerful Christian religious symbols in this country. (…) Pawtucket's purchase, the maintenance, and the erection of the fundamental Christian symbol involves government in religion to a profound and substantial degree. (…) JUSTICE: Now, if the city did not own the crèche itself, so that everything that was contributed to the display, including the crèche, were privately owned, it wouldn't violate the First Amendment, the fact that it was right next door to the City Hall, would it? MR. DE LUCA: I think that in understanding that the city owns all of the symbols and all of the artifacts that are contained in this display, and assuming that the crèche were purchased and paid for privately without any other explanation that it is private, then I think it would still violate the establishment clause for the First Amendment, because there is no indication to anyone looking at that the display or the crèche is not part of the broader display which is put up and sponsored by the city. (…) JUSTICE: Would you regard the prayer that I spoke of to your friend in the House or the Senate or in any state legislature as purely symbolic, or is it a matter of Niels Pinkwart Using Collaborative Filtering in an Intelligent substance? AH 2006 Tutoring System for Legal Argumentation Test Hypo Test Modif. Hypo

A Tool For Graphical Argument Visualization Niels Pinkwart AH 2006 Using Collaborative Filtering in an Intelligent Tutoring System for Legal Argumentation

An Example Diagram (Result of Pilot Study) Niels Pinkwart AH 2006 Using Collaborative Filtering in an Intelligent Tutoring System for Legal Argumentation 7

Intelligent Support o How help students n n o Automated diagram analysis allow for: n n o analyze the argument transcript? navigate the interlinked information spaces? Graph structure inspection general argumentation principles, e. g. “there should be at least one test” Checks of links between graph and transcript case specific “important passages” Not subject of this talk ( ITS 2006) Niels Pinkwart AH 2006 Using Collaborative Filtering in an Intelligent Tutoring System for Legal Argumentation 8

Content Analysis? Niels Pinkwart AH 2006 Using Collaborative Filtering in an Intelligent Tutoring System for Legal Argumentation 9

Content Weaknesses Idea: o o Make use of peer students working on the same task Have students rate peer solutions as part of their working with the system Niels Pinkwart AH 2006 Using Collaborative Filtering in an Intelligent Tutoring System for Legal Argumentation 10

Content Weaknesses o Student A Exploit that the system knows what part of the graph refers to certain important parts of text… Student C Student B Niels Pinkwart AH 2006 Using Collaborative Filtering in an Intelligent Tutoring System for Legal Argumentation 11

Content Weaknesses o Student A … and use these relations for generating the dialogs. Student C Student B Niels Pinkwart AH 2006 Using Collaborative Filtering in an Intelligent Tutoring System for Legal Argumentation 12

Content Weaknesses Principle for quality rating q: weighted average of base rating and evaluation rating (0=poor, 1=excellent) o Base rating n Based on how student rates other solutions n Serves as initial score heuristic, immediately available n Assumption: having good solution correlates to recognizing good solutions Recommended items Niels Pinkwart AH 2006 Non-recommended items Using Collaborative Filtering in an Intelligent Tutoring System for Legal Argumentation 13

Content Weaknesses o Evaluation rating n Based on recommendations a student’s answer receives (or not), and by whom n Develops over time n Takes peer opinions into account n Assumption: measures actual quality Actual recommenders All possible recommenders Niels Pinkwart AH 2006 Using Collaborative Filtering in an Intelligent Tutoring System for Legal Argumentation 14

Feedback: Self Explanation Prompts o o o Use detected weaknesses as tailored self explanation prompts Offer opportunities for reflection about specific parts of Socratic reasoning examples Present if quality rating below specific threshold (e. g. 0. 3) Niels Pinkwart AH 2006 Using Collaborative Filtering in an Intelligent Tutoring System for Legal Argumentation 15

Does the System provide Feedback when appropriate? Niels Pinkwart AH 2006 Using Collaborative Filtering in an Intelligent Tutoring System for Legal Argumentation

Conclusion and Outlook Use collaborative filtering methods to generate a quality heuristic for important parts of argument diagrams o o o Filter for quality Aim is not to n show only “best” or “most matching” argument descriptions n get to very precise rating Instead: use implicitly to generate adaptive feedback in the ITS Pilot studies with single users successful, studies with small groups to come Larger lab studies to evaluate the ITS: Fall 2006 Niels Pinkwart AH 2006 Using Collaborative Filtering in an Intelligent Tutoring System for Legal Argumentation 17

o Please visit our project website: http: //www. cs. cmu. edu/~hypoform o Email: nielsp@cs. cmu. edu Niels Pinkwart AH 2006 Using Collaborative Filtering in an Intelligent Tutoring System for Legal Argumentation 18