Brett van de Sande Kurt Van Lehn

Brett van de Sande, Kurt Van. Lehn, & Tim Nokes University of Pittsburgh Cognitive Analysis of Student Learning Using Learn. Lab 1

Agenda I. Learn. Lab methodology II. Demonstration of Andes, an intelligent homework tutor III. Log File Analysis 2

Goal: To understand physics learning • Policy level – e. g. , Physics for high school freshman? • Instructional level – e. g. , How much assistance to give? – e. g. , How much practice per topic? – e. g. , How to handle errors? Our focus • Neurocognitive level – e. g. , Can neuroimaging distinguish deep from shallow studying of a text? 3

Traditional methods for studying learning • Design experiment – Modify text, classroom activities, tests… – e. g. , Project Scale-up • Lab experiment – Modify just one factor – Brief; money instead of grades, … 4

PSLC methods • Educational data mining – Logs from instrumented courses – Some analysis is automated Next • In vivo experiments – Control of variables – Instrumented courses 5

Instrumented courses (Called Learn. Lab courses) • Existing class + data collection – Homework done on a tutoring system or photocopied analyzed – Photocopies of quizzes, exams – FCI given before and after the course – Demographics, GPAs, Majors… – Handouts, slides, clicker data, … • Instructor, student & IRB cooperation – Anonymity 6

Existing Physics Learn. Lab Course(s) • US Naval Academy – Course take by all 2 nd year students – Learn. Lab is in 4 of about 20 sections – Profs. Wintersgill, Mc. Clanahan • Your course here 7

Basic data mining question • What features of students’ histories are statistically associated with learning gains? • e. g. , What are the differences between histories of Student A and Student F? Student A: 25% on Semester-long history pretest 85% on post-test Student F: 25% on Semester-long history pretest 20% on post-test 8

Knowledge decomposition hypothesis • Decompose knowledge to be learned into a set of knowledge components – e. g. , Newton’s third law – e. g. , Centripetal acceleration • Assume each knowledge component is learned independently – An approximation/idealization 9

Data mining with knowledge components (KCs) For each KC, find statistical associations between histories and gains. Student A A A KC 1 2 3 Pre-test 35% 15% 25% History … … … Post-test 85% 10% 20% B B B 1 2 3 50% 10% 25% … … … 20% 10% 80% 10

History decomposition hypothesis • Decompose the student’s history into events such that each event addresses only one (or a few) knowledge components. – – Reading a paragraph about Newton’s 3 rd law Drawing a reaction force vector Seeing the instructor draw a reaction force Drawing a centripetal acceleration vector • Assume that a KC’s learning gain depends only on that KC’s events 11

Events for 1 student on 1 KC (e. g. , Newton’s 3 rd law) Time Context Behavior 8/27/07 9: 05 FCI item 3 Incorrect 8/27/07 9: 12 FCI item 10 Incorrect 9/13/07 18: 06 Textbook, pg. 111 Highlighted 9/13/07 21: 11 Problem 5 -11, drawing FBD Omitted force on the hand due to block 9/14/07 9: 12 Lecture, slide 20 Taking notes 9/15/07 22: 05 Problem 5 -11, drawing FBD Draws force on the hand due to block etc. 12

Some events are not currently available Time Context Behavior 8/27/07 9: 05 FCI item 3 Incorrect 8/27/07 9: 12 FCI item 10 Incorrect 9/13/07 18: 06 Textbook, pg. 111 Highlighted 9/13/07 21: 11 Problem 5 -11, drawing FBD Omitted force on the hand due to block 9/14/07 9: 12 Lecture, slide 20 Taking notes 9/15/07 22: 05 Problem 5 -11, drawing FBD Draws force on the hand due to block etc. 13

More feasible data mining • Predict learning gains of a KC given the sequence of events relevant to that KC • On an event that assesses mastery of a KC, predict the student’s performance during that event given the sequence of preceding events relevant to that KC 14

Predicting correctness of events that assess mastery Context Event type P(Correct) FCI item 3 Assessment 0. 10 FCI item 10 Assessment 0. 12 Reading textbook, pg. 111, paragraph 3 Instruction Not applicable Problem 5 -11, drawing force Assessment on hand due to block 0. 30 Lecture, slide 20 Not applicable Instruction Problem 5 -11, drawing force Assessment on hand … with remedial then instruction feedback if needed 0. 55 15

Learning curves • Plot assessment events on x-axis – Ordered chronologically • Plot measure of mastery on y-axis – Usually aggregated across subjects e. g. , proportion of 100 subjects who performed correctly on this event 16

Frequency of correct An expected learning curve 1. 0 0. 5 0 1 2 3 4 5 6 Assessment events 7 8 17

Summary of PSLC educational data mining • Given knowledge to be learned – Decompose into knowledge components • Given students’ histories from an instrumented course – Divide into assessment/instruction events – such that one KC (or a few) per event • For each KC, find a function on a sequence of events that predicts the KC’s – learning gain during the course – learning curve 18

PSLC methods • Educational data mining – Logs from instrumented courses – Some analysis is automated • Andes produces logs with KCs • Data. Shop draws learning curves, etc. • Correlation ≠ Causation • In vivo experiments Next – Control of variables – Instrumented courses 19

Two major types of in vivo experiments • Short & fat – During one lesson or one unit • Long & skinny – During whole course – “invisibly” 20

Example of a short, fat, in vivo experiment (Hausmann 07) • During a 2 -hour period (usually used for lab work) • ~25 students in the room, each with a laptop and a headset mike • Repeat 3 times: – Study a video while explaining it into the mike – Solve a problem • 4 experimental conditions, varying the content of the video and the instructions for explaining it • Random assignment of students to conditions • Dependent measures include learning curves • Result: Instructions to self-explain worked best regardless of content of the video 21

Example of a long, skinny in vivo experiment (Katz 07) • During 8 weeks of a 13 -week course • Random assignment to 2 conditions: – Experimental group: After solving certain homework problems, the student discussed the solution with a natural language tutoring system – Control group: Extra homework problems • Result: Experiment > Control on some conceptual measures 22

Robust Learning • Immediate learning – During an immediate post-test – Similar content to training (near transfer) • Robust learning – Far transfer – Retention – Acceleration of future learning • Does manipulation of instruction on topic A affect rate of learning of a later topic, B? 23

Summary of PSLC methodology • Data mining – Instrumented (Learn. Lab) courses – Knowledge components – Instructional and assessment events – Learning curves • In vivo experiments – Short & fat vs. long & skinny – Robust learning 24

Agenda I. Learn. Lab methodology II. Demonstration of Andes, an intelligent homework tutor III. Log File Analysis Next 25

26

Define variables Draw free body diagram (3 vectors and body) Define coordinates (3 choices for this problem) Upon request, Andes gives hints for what to do next 27

Red/green gives immediate feedback for student actions Principle-based help for incorrect entry 28

29

# Log of Andes session begun Tuesday, July 17, 2007 12: 28 by [User] on [Computer]. . . 05: 03 DDE (read-problem-info "S 2 E" 0 0). . . 02: 35 Axes-671 64 335 143 296 02: 35 Axes-dlg Axes-671 || … 02: 38 C dir 40 02: 42 BTN-CLICK 1 OK 02: 42 DDE (assert-x-axis NIL 40 Axes-671 "x" "y" "z") 02: 42 DDE-COMMAND assoc step (DRAW-AXES 40) 02: 42 DDE-COMMAND assoc op DRAW-VECTOR-ALIGNED-AXES 02: 42 DDE-COMMAND set-score 39 02: 42 DDE-RESULT |T|. . . 10: 02 E 0 F 1_y+F 2_y=0 10: 02 EQ-SUBMIT 0 10: 02 DDE (lookup-eqn-string "F 1_y+F 2_y=0" 0) 10: 47 DDE-COMMAND assoc parse (= (+ Yc_Fn_BALL_WALL 1_1_40 Yc_Fn_BALL_WALL 2_1_40) 0) 10: 47 DDE-COMMAND assoc error MISSING-FORCES-IN-Y-AXIS-SUM 10: 47 DDE-COMMAND assoc step (EQN (= (+ Yc_Fw_BALL_EARTH_1_40 Yc_Fn_BALL_WALL 2_1_40 Yc_Fn_BALL_WALL 1_1_40) 0)) 10: 47 DDE-COMMAND assoc op WRITE-NFL-COMPO 10: 47 DDE-RESULT |NIL|. . . 10: 50 DDE-RESULT |!show-hint There is a force acting on the ball at T 0 that you have not yet drawn. ~e|. . . 16: 38 END-LOG problem name student action (draw axes) interpretation: compare to model green student action (equation) error analysis: intended action red session time 30

Demonstration by Tim Nokes # Log of Andes session begun Wednesday, April 18, 2007 21: 08: 07 by [user] on [computer]. . . 0: 02 DDE (read-problem-info "FARA 9" 0 0). . . 0: 13 Help-Hint 0: 13 DDE (Get-Proc-Help) 0: 13 DDE-COMMAND assoc (NSH NEW-START-AXIS 0) 0: 13 DDE-RESULT |!show-hint It is a good idea to begin most problems by drawing an axis. This helps to ground your work and will be useful later on in the process. ~e| … 0: 17 Begin-draw 50001 Axes-1 185 331. . . 0: 30 New-Variable resistance. . . 0: 39 DDE (define-variable "R" |NIL| |resistance| |R| |NIL| Var-2 "20 ohm") 0: 39 DDE-COMMAND assoc step (DEFINE-VAR (RESISTANCE R)) 0: 39 DDE-COMMAND assoc op DEFINE-RESISTANCE-VAR 0: 39 DDE-COMMAND assoc parse (= R_R (DNUM 20 ohm)) 0: 39 DDE-COMMAND set-score 3 0: 39 DDE-RESULT |T|. . 0: 50 DDE (lookup-vector "B" Unspecified B-field |s| NIL 0 |NIL| Vector-3) 0: 50 DDE-COMMAND assoc entry (VECTOR (FIELD S MAGNETIC UNSPECIFIED TIME NIL) ZERO) 0: 50 DDE-COMMAND assoc error DEFAULT-SHOULD-BE-NON-ZERO 0: 50 DDE-COMMAND assoc step (VECTOR (FIELD S MAGNETIC UNSPECIFIED TIME NIL) OUT-OF) 0: 50 DDE-COMMAND assoc op DRAW-FIELD-GIVEN-DIR 0: 50 DDE-COMMAND set-score 2 0: 50 DDE-RESULT |NIL|. . . 9: 51 DDE-RESULT |T| 9: 55 END-LOG 31

Agenda I. Learn. Lab methodology II. Demonstration of Andes, an intelligent homework tutor III. Log File Analysis Next 32

Model Solution Set Solution 1 Solution 0 Principle A Principle B Principle A Op 1 Op 3 Op 6 Op 7 Op 2 Op 3 Op 5 Op 8 Op 10 Principle C Op 10 Op 11 Op 12 Op 1 Op 3 Op 6 Op 7 Principle D Assumption: Opi = KC 33

# Log of Andes session begun Friday, July 27, 2007 14: 29: 38 by bobh on BOBH … 0: 02 DDE (read-problem-info "S 2 E" 0 0) problem name … 11: 45 Vector-dlg Vector-673 || … 11: 48 CLOSE type instantaneous 11: 48 SEL type 1 instantaneous 11: 51 BTN-CLICK 1 OK 11: 51 DDE (lookup-vector "a" instantaneous Acceleration |ball| NIL 0 |T 0| Vector-673) 11: 51 DDE-COMMAND assoc step (VECTOR (ACCEL BALL : TIME 1) ZERO) 11: 51 DDE-COMMAND assoc op ACCEL-AT-REST 11: 51 DDE-RESULT |T| … student actions green match model solution: assoc step = entry Assoc op = operator 34

14: 03 E 8 Fearth_y = m*g student actions 14: 11 EQ-SUBMIT 8 14: 11 DDE (lookup-eqn-string "Fearth_y = m*g" 8) 14: 11 DDE-COMMAND assoc parse (= Yc_Fw_BALL_EARTH_1_0 (* m_BALL g_EARTH)) 14: 11 DDE-COMMAND assoc error MISSING-NEGATION-ON-VECTOR -COMPONENT 14: 11 DDE-COMMAND assoc step (EQN (= Fw_BALL_EARTH_1 (* m_BALL g_EARTH))), (EQN (= Yc_Fw_BALL_EARTH_1_0 (Fw_BALL_EARTH_1))) 14: 11 DDE-COMMAND assoc op WT-LAW, COMPO-PARALLEL-AXIS 14: 11 DDE-COMMAND set-score 74 14: 11 EQ-F 8 14: 11 DDE-RESULT |NIL| error interpretation guess intended red 35

Review Video Match steps in video to log file 36

Researchable questions Timing Sequencing (order of steps) Hint Usage Problem solving skills Self-correction of errors Errors as window to mental state 37

Data. Shop https: //learnlab. web. cmu. edu/ > Launch Data. Shop > New user? Sign up now! > (Create account) 38