PSLC Data Shop Introduction http pslcdatashop org Slides

PSLC Data. Shop Introduction http: //pslcdatashop. org Slides current to Data. Shop version 4. 1. 8 John Stamper Data. Shop Technical Director

The Data. Shop Team • John Stamper – Data. Shop Technical Director • Alida Skogsholm – Data. Shop Manager, Developer • Brett Leber – Interaction Designer • Duncan Spencer – Data. Shop Developer • Shanwen Yu – Data. Shop Developer • Sandy Demi – QA (Quality Assurance – Testing) 2

What is Data. Shop? • Central Repository – Secure place to store & access research data • Every Learn. Lab and every study – Supports various kinds of research • Primary analysis of study data • Exploratory analysis of course data • Secondary analysis of any data set • Analysis & Reporting Tools – Focus on student-tutor interaction data – Learning curves & error reports provide summary and low-level views of student performance – Performance Profiler aggregates across various levels of granularity (problem, dataset levels, knowledge components, etc. ) – Data Export • Tab delimited tables you can open with your favorite spreadsheet program or statistical package – New tools created to meet highest demands 3

Repository

Web Application • Knowledge component model analysis with learning curves • Learning curve point decomposition

Web Application ◄ Performance Profiler tool for exploring the data ► Easy knowledge component model creation

What does the data look like? • Transaction – A transaction is an interaction between the student and the tutoring system. – Students may make incorrect entries or ask for hints before getting a step correct. Each hint request, incorrect attempt, or correct attempt is a transaction; and a step can involve one or more transactions. • Step – A step is an observable part of the solution to a problem. Because steps are observable, they are partly determined by the user interface available to the student for solving the problem.

How do I get data in? • Directly – Some tutors are logging directly to the PSLC logging database – CTAT-based tutors (when configured correctly) • Indirectly – Other tutors are logging to their own file formats or their own databases – These data require a conversion process – Many studies are in this category 8

Improving learning by improving the cognitive model: A data-driven approach Cen, H. , Koedinger, K. , Junker, B. Learning Factors Analysis - A General Method for Cognitive Model Evaluation and Improvement. 8 th International Conference on Intelligent Tutoring Systems. 2006. Cen, H. , Koedinger, K. , Junker, B. Is Over Practice Necessary? Improving Learning Efficiency with the Cognitive Tutor. 13 th International Conference on Artificial Intelligence in Education. 2007. Koedinger, K. Stamper, J. A Data Driven Approach to the Discovery of Better Cognitive Models. 3 rd International Conference on Educational Data Mining. 2010. Koedinger, K. R. , Baker, R. S. J. d. , Cunningham, K. , Skogsholm, A. , Leber, B. , Stamper, J. (in press) A Data Repository for the EDM commuity: The PSLC Data. Shop. To appear in Romero, C. , Ventura, S. , Pechenizkiy, M. , Baker, R. S. J. d. (Eds. ) Handbook of Educational Data Mining. Boca Raton, FL: CRC Press.

Why we need better expert & student models in ITS Two key premises • Expert & student model drives instruction – Cognitive model in Cognitive Tutors determine much of ITS behavior; Same for constraints… • These models are sometimes wrong & almost always imperfect – ITS developers often build models rationally – But such models may not be empirically accurate • A correct cognitive model should predict task difficulty and transfer => generate smooth learning curves => Huge opportunity for ITS researchers to improve their tutors

Cognitive Model Determines Instruction

Cognitive Tutor Technology • Cognitive Model: A system that can solve problems in the various ways students can If goal is solve a(bx+c) = d Then rewrite as abx + ac = d 3(2 x - 5) = 9 If goal is solve a(bx+c) = d Then rewrite as abx + c = d If goal is solve a(bx+c) = d Then rewrite as bx+c = d/a 6 x - 15 = 9 2 x - 5 = 3 6 x - 5 = 9 • Model Tracing: Follows student through their individual approach to a problem -> context-sensitive instruction

Cognitive Tutor Technology • Cognitive Model: A system that can solve problems in the various ways students can If goal is solve a(bx+c) = d Then rewrite as abx + ac = d 3(2 x - 5) = 9 If goal is solve a(bx+c) = d Then rewrite as abx + c = d Hint message: “Distribute a across the parentheses. ” Known? = 85% chance 6 x - 15 = 9 Bug message: “You need to multiply c by a also. ” Known? = 45% 2 x - 5 = 3 6 x - 5 = 9 • Model Tracing: Follows student through their individual approach to a problem -> context-sensitive instruction • Knowledge Tracing: Assesses student's knowledge growth -> individualized activity selection and pacing

If you change cognitive model you change instruction • Problem creation, selection, & sequencing – New skills or concepts (= “knowledge components” or “KCs”) require: • New kinds problems & instructional activities • Changes to student modeling – skillometer, knowledge tracing • Feedback and hint message content – One skill becomes two => need new hint messages for new skill – New bug rules may be needed • Even interface design – “make thinking visible” – If multiple skills per step => break down by adding new intermediate steps to interface

Expert & student models are imperfect in most ITS • How can we tell? • Don’t get learning curves – If we know tutor works (get pre to post gains), but “learning curves don’t curve”, then the model is wrong • Don’t get smooth learning curves – Even when every KC has a good learning curve (error rate goes down as student gets more opportunities to practice), model still may be imperfect when it has significant deviations from student data

PSLC Data. Shop Tools http: //pslcdatashop. org Slides current to Data. Shop version 4. 1. 8 Koedinger, K. R. , Baker, R. S. J. d. , Cunningham, K. , Skogsholm, A. , Leber, B. , Stamper, J. (in press) A Data Repository for the EDM commuity: The PSLC Data. Shop. To appear in Romero, C. , Ventura, S. , Pechenizkiy, M. , Baker, R. S. J. d. (Eds. ) Handbook of Educational Data Mining. Boca Raton, FL: CRC Press.

Analysis Tools • • • Dataset Info Performance Profiler Error Report Learning Curve KC Model Export/Import

Getting to Data. Shop • Explore data through the Data. Shop tools • Where is Data. Shop? – http: //pslcdatashop. org – Linked from Data. Shop homepage and learnlab. org • http: //pslcdatashop. web. cmu. edu/about/ • http: //learnlab. org/technologies/datashop/index. php 22

Creating an account • On Data. Shop's home page, click "Sign up now". Complete the form to create your Data. Shop account. • If you’re a CMU student/staff/faculty, click “Log in with Web. ISO” to create your account. 23

Getting access to datasets • By default, you will have access to the public datasets. • Of these, we recommend three for getting started: – Geometry Area (1996 -1997) – Joint Explanation - Electric Fields - Pitt - Spring 2007 – Chinese Vocabulary Fall 2006 • For access to other datasets, contact us: datashop-help@lists. andrew. cmu. edu 24

Data. Shop – Dataset selection Datasets you can view or edit. You have to be a project member or PI for the dataset to appear here. Private datasets you can’t view. Email us and the PI to get access. Public datasets that you can view only. 25

Dataset Info • • Papers and Files storage Dataset Metrics Meta data for given dataset PI’s get ‘edit’ privilege, others must request it Problem Breakdown table 26

Performance Profiler Multipurpose tool to help identify areas that are too hard or easy View measures of • • • Error Rate Assistance Score Avg # Hints Avg # Incorrect Residual Error Rate View multiple samples side by side Aggregate by • • • Step Problem Student KC Dataset Level Mouse over a row to reveal uniqueness

Error Report • • View by Problem or KC Provides a breakdown of problem information (by step) for finegrained analysis of problem-solving behavior Attempts are categorized by evaluation

Learning Curves Visualizes changes in student performance over time Hover the y-axis to change the type of Learning Curve. Types include: • Error Rate • Assistance Score • Number of Incorrects • Number of Hints • Step Duration • Correct Step Duration • Error Step Duration Time is represented on the xaxis as ‘opportunity’, or the # of times a student (or students) had an opportunity to demonstrate a KC 29

Learning Curves: Drill Down Click on a data point to view point information Click on the number link to view details of a particular drill down information. Four types of information for a data point: • KCs • Problems • Steps • Students Details include: • Name • Value • Number of Observations 30

Learning Curve: Latency Curves For latency curves, a standard deviation cutoff of 2. 5 is applied by default. The number of included and dropped observations due to the cutoff is shown in the observation table. Step Duration = the total length of time spent on a step. It is calculated by adding all of the durations for transactions that were attributed to a given step. Error Step Duration = step duration when first attempt is an error Correct Step Duration = step duration when the first attempt is correct 31

Dataset Info: KC Models Toolbox allows you to export one or more KC models, work with them, then reimport into the Dataset. Handy information displayed for each KC Model: • Name • # of KCs in the model • Created By Data. Shop generates two • Mapping Type • AIC & BIC Values KC models for free: • Single-KC • Unique-step These provide upper and lower bounds for AIC/BIC. Click to view the list of KCs for this model. 32

Dataset Info: Export a KC Model Select the models you wish to export and click the “Export” button. Model information as well as other useful information is provided in a tab-delimited Text file. Selecting the “export” option next to a KC Model will auto-select the model for you in the export toolbox. Export multiple models at once. 33

Dataset Info: Import a KC Model When you are ready to import, upload your file to Data. Shop for verification. Once verification is successful, click the “Import” button. Your new or updated model will be available shortly (depending on the size of the dataset). 34

Web Services • • • Why Web Services? ? Get Web Services Download Getting Credentials Authentication & Datashop. Client What is an ID? How to get a dataset ID How to see some transaction data Add a little Swing… Web Services URL 35

Why Web Services? ? • To access the data from a program – New visualization – Data mining – or other application 36

Get Web Services Download 37

Getting Credentials 38

Authentication & Datashop. Client • Put your token and secret access key in a file named ‘webservices. properties’ 39

What is an ID? • The Data. Shop API expects you to reference various objects by “ID”, a unique identifier for each dataset, sample, custom field, or transaction in the repository. • The ID of any of these can be determined by performing a request to list the various items, which lists the IDs in the response. • For example, a request for datasets will list the ID of each dataset in the “id” attribute of each dataset element. 40

How to get a dataset ID • Use Datashop. Client class provided in datashopwebservices. jar • Pass in a URL to form the request • Results include datasets that you have access to java –jar dist/datashop-webservices. jar “https: //pslcdatashop. web. cmu. edu/services/datasets” Handwriting/Examples Dec 2006 … 41

How to get a dataset ID java –jar dist/datashop-webservices. jar “https: //pslcdatashop. web. cmu. edu/services/datasets? access=edit” > datasets. xml 42

Open XML in browser and search 43

Back to command line 44

How to get a sample ID java –jar dist/datashop-webservices. jar “https: //pslcdatashop. web. cmu. edu/services/datasets/313/samples” All Data Default Sample that contains all transactions. % 11394 article. Tutor-B Default Sample that contains all transactions. liuliu@ANDREW. CMU. EDU 2707 45

How to see some transaction data Request a subset of columns for a given dataset and the ‘All Data’ sample which is the default java edu. cmu. pslc. datashop. webservices. Data. Shop. Client “https: //pslcdatashop. web. cmu. edu/services/datasets/313/transactions? limit=10&cols=problem_hierarchy, problem_name, step_name, outcome, i nput” Problem Hierarchy Problem Name Unit IWT_S 09 article. Tutor. B-A, Section Unit IWT_S 09 article. Tutor. B-A, Section … Step Name IWT Tests and IWT Tests and IWT Tests and Outcome Tutors Tutors Tutors Input article. Tuto article. Tuto 46

import edu. cmu. pslc. datashop. webservices. Datashop. Client; public class Web. Services. Demo. Client extends Datashop. Client { … private static final String DATASETS_PATH = "/datasets/"; private static final String TXS_PATH = "/transactions? headers=false” + "&cols=problem_hierarchy, problem_name, step_name, outcome, input"; private Web. Services. Demo. Client(String root, String api. Token, String secret) { super(root, api. Token, secret); }; public Tree. Map run. Report(String dataset. Id) { String path = DATASETS_PATH + dataset. Id + TXS_PATH; Http. URLConnection conn = service. Get. Connection(path); conn. set. Request. Property("accept", "text/xml"); Tree. Map map = new Tree. Map(); try { Input. Stream is = conn. get. Input. Stream(); Buffered. Reader reader = new Buffered. Reader(new Input. Stream. Reader(is)); String row = null; while ((row = reader. read. Line()) != null) { Transaction. Data. Subset t = Transaction. Data. Subset. create. Transaction(row); … 47

Add a little Swing… java –classpath “. . /dist/datashop-webservices. jar; . ” Web. Services. Demo. Client. UI dataset 313 48

To get more details… http: //pslcdatashop. org/about/webservices. html http: //pslcdatashop. org/downloads/ Web. Services. Demo. Client_src. zip 49

KDD Cup 2010 EDM Challenge

The datasets used for the challenge were: Dataset Students Steps File size Algebra I 2008 -2009 3, 310 9, 426, 966 3 GB Bridge to Algebra 2008 -2009 6, 043 20, 768, 884 5. 43 GB The competition ended on June 8. There were: – 655 registered teams – 130 teams who submitted predictions – 3, 400 submissions

Data. Shop - What’s in it for me? • Free tools to analyze your data • Free researchers to analyze your data • Real opportunities to validate ideas across multiple data sets

Thanks! - The Data. Shop Team • John Stamper – Data. Shop Technical Director • Alida Skogsholm – Data. Shop Manager, Developer • Brett Leber – Interaction Designer • Duncan Spencer – Data. Shop Developer • Shanwen Yu – Data. Shop Developer • Sandy Demi – QA (Quality Assurance – Testing)