Studying and achieving robust learning with PSLC resources Ken Koedinger HCI & Psychology CMU Director of PSLC 1
This is the 4 th Annual PSLC Summer School • 8 th overall – ITS was focus in 2001 to 2004 • Goals: – Learning science & technology concepts – Hands-on project you present on Fri 2
Pittsburgh Science of Learning Center (PSLC) 170+ researchers from CA to Germany Ken Koedinger - Carnegie Mellon Co-Director Kurt Van. Lehn - University of Pittsburgh Co-Director Charles Perfetti - Chief Scientist & Future Co-Director Executive Committee: Vincent Aleven, Maxine Eskenazi (Diversity Director), Julie Fiez, David Klahr (Education Director), Marsha Lovett, Tim Nokes, Lauren Resnick + representatives of Jr. Faculty, Post-docs & Grad students Michael Bett – Managing Director
Advancing a Science of Academic Learning Challenge: Chasm between learning science & educational practice • Empirical – Lots of rigorous principle-testing lab studies – Lots of realistic classroom design research – Too few experiments combine both • Theoretical – Almost as many theories as there are results! – Need for “rigorous, sustained scientific research in education” (National Research Council, 2002) 4
Overview Next • Background – Intelligent Tutoring Systems – Cognitive Task Analysis • PSLC Methods, Resources, & Theory – – In vivo experimentation Learn. Lab courses Robust learning theoretical framework Enabling technologies • Summary 5
PSLC is about much more than Intelligent Tutors But tutors & course evaluations were a key inspiration Quick review … 6
Past Success: Intelligent Tutors Bring Learning Science to Schools! • Intelligent tutoring systems – Automated 1: 1 tutor – Artificial Intelligence – Cognitive Psychology • Andes: College Physics Tutor – Replaces homework Students: model problems with diagrams, graphs, equations Tutor: feedback, help, reflective dialog • Algebra Cognitive Tutor – Part of complete course 7
Cognitive Tutor Approach
Cognitive Tutor Technology • Cognitive Model: A system that can solve problems in the various ways students can Strategy 1: Strategy 2: Misconception: IF the goal is to solve a(bx+c) = d THEN rewrite this as abx + ac = d IF the goal is to solve a(bx+c) = d THEN rewrite this as bx + c = d/a IF the goal is to solve a(bx+c) = d THEN rewrite this as abx + c = d 9
Cognitive Tutor Technology • Cognitive Model: A system that can solve problems in the various ways students can If goal is solve a(bx+c) = d Then rewrite as abx + ac = d 3(2 x - 5) = 9 If goal is solve a(bx+c) = d Then rewrite as abx + c = d If goal is solve a(bx+c) = d Then rewrite as bx+c = d/a 6 x - 15 = 9 2 x - 5 = 3 6 x - 5 = 9 • Model Tracing: Follows student through their individual approach to a problem -> context-sensitive instruction 10
Cognitive Tutor Technology • Cognitive Model: A system that can solve problems in the various ways students can If goal is solve a(bx+c) = d Then rewrite as abx + ac = d 3(2 x - 5) = 9 If goal is solve a(bx+c) = d Then rewrite as abx + c = d Hint message: “Distribute a across the parentheses. ” Known? = 85% chance 6 x - 15 = 9 Bug message: “You need to multiply c by a also. ” Known? = 45% 2 x - 5 = 3 6 x - 5 = 9 • Model Tracing: Follows student through their individual approach to a problem -> context-sensitive instruction • Knowledge Tracing: Assesses student's knowledge growth -> individualized activity selection and pacing 11
Cognitive Tutor Course Development Process 1. 2. 3. 4. Client & problem identification Identify the target task & “interface” Perform Cognitive Task Analysis (CTA) Create Cognitive Model & Tutor a. Enhance interface based on CTA b. Create Cognitive Model based on CTA c. Build a curriculum based on CTA 5. Pilot & Parametric Studies 6. Classroom Evaluation & Dissemination 12
Cognitive Tutor Approach
Difficulty Factors Assessment: Discovering What is Hard for Students to Learn Which problem type is most difficult for Algebra students? Story Problem As a waiter, Ted gets $6 per hour. One night he made $66 in tips and earned a total of $81. 90. How many hours did Ted work? Word Problem Starting with some number, if I multiply it by 6 and then add 66, I get 81. 90. What number did I start with? Equation x * 6 + 66 = 81. 90 14
Algebra Student Results: Story Problems are Easier! Koedinger, & Nathan, (2004). The real story behind story problems: Effects of representations on quantitative reasoning. The Journal of the Learning Sciences. Koedinger, Alibali, & Nathan (2008). Trade-offs between grounded and abstract representations: Evidence from algebra problem solving. Cognitive Science. 15
Expert Blind Spot: Expertise can impair judgment of student difficulties 100 90 80 % making correct ranking (equations hardest) 70 60 50 40 30 20 10 0 Elementary Teachers Middle School Teachers High School Teachers 16
“The Student Is Not Like Me” • To avoid your expert blindspot, remember the mantra: “The Student Is Not Like Me” • Perform Cognitive Task Analysis to find out what students are like 17
Cognitive Tutor Course Development Process 1. 2. 3. 4. Client & problem identification Identify the target task & “interface” Perform Cognitive Task Analysis (CTA) Create Cognitive Model & Tutor a. Enhance interface based on CTA b. Create Cognitive Model based on CTA c. Build a curriculum based on CTA 5. Pilot & Parametric Studies 6. Classroom Evaluation & Dissemination 18
Tutors make a significant difference in improving student learning! • Andes: College Physics Tutor – Field studies: Significant improvements in student learning • Algebra Cognitive Tutor – 10+ full year field studies: improvements on problem solving, concepts, basic skills – Regularly used in 1000 s of schools by 100, 000 s of students!! 19
Prior achievement: Intelligent Tutoring Systems bring learning science to schools A key PSLC inspiration: Educational technology as research platform to generate new learning science 20
Logic of Pittsburgh Science of Learning Center (PSLC) • Support experimental studies that – Test fundamental principles, not whole courses – Are internally & externally valid • Create a theory of “robust learning” • Leverage technology & computational modeling 21
Overview • Background – Intelligent Tutoring Systems – Cognitive Task Analysis • PSLC Methods, Resources, & Theory – – Next In vivo experimentation Learn. Lab courses Robust learning theoretical framework Enabling technologies • Summary 22
PSLC statement of purpose • To yield theoretically sound and useful principles of robust learning, we have created Learn. Lab to facilitate in vivo learning experimentation. Outline of Overview 1. Robust learning 2. Theory Next 3. Method 4. Learn. Lab 23
What is Robust Learning? • Robust Learning is learning that – transfers to novel tasks – retained over the long term, and/or – accelerates future learning • Robust learning requires that students develop both – conceptual understanding & sense-making skills – procedural fluency with basic foundational skills 24
PSLC statement of purpose • To yield theoretically sound and useful principles of robust learning, we have created Learn. Lab to facilitate in vivo learning experimentation. Outline of Overview 1. Robust learning 2. Theory Next 3. Method 4. Learn. Lab 25
In Vivo Experiments Principle-testing laboratory quality in real classrooms 26
In Vivo Experimentation What is tested? Methodology Instructional solution • Where & who? – Lab vs. classroom • How? – Treatment only vs. Treatment + control • Generalizing conclusions: – Ecological validity: What instructional activities work in real classrooms? – Internal validity: What causal mechanisms explain & predict? Classroom – Instructional solution vs. causal principle Where? Lab Methodology features: • What is tested? Causal principle Lab experiments Design research & field trials 27
In Vivo Experimentation What is tested? Methodology Instructional solution • Where & who? – Lab vs. classroom • How? – Treatment only vs. Treatment + control • Generalizing conclusions: – Ecological validity: What instructional activities work in real classrooms? – Internal validity: What causal mechanisms explain & predict? Classroom – Instructional solution vs. causal principle Where? Lab Methodology features: • What is tested? Causal principle Lab experiments Design research & field trials In Vivo learning experiments 28
PSLC statement of purpose • To yield theoretically sound and useful principles of robust learning, we have created Learn. Lab to facilitate in vivo learning experimentation. Outline of Overview 1. Robust learning 2. Theory Next 3. Method 4. Learn. Lab 29
Learn. Lab A Facility for Principle-Testing Experiments in Classrooms 30
Learn. Lab courses at K 12 & College Sites • 6+ cyber-enabled courses: Chemistry, Physics, Algebra, Geometry, Chinese, English • Data collection – Students do home/lab work on tutors, vlab, OLI, … – Log data, questionnaires, tests Data. Shop Researchers Schools Learn Lab Chemistry virtual lab Physics intelligent tutor REAP vocabolary tutor 31
PSLC resources enable theoretical development … 32
PSLC statement of purpose • To yield theoretically sound and useful principles of robust learning, we have created Learn. Lab to facilitate in vivo learning experimentation. Outline of Overview 1. Robust learning 2. Theory Next 3. Method 4. Learn. Lab 33
Theoretical Framework Levels • Macro level – What instructional principles explain how changes in the instructional environment lead to changes in robust learning? • Micro level 34
Explanation at the macro-level Why is instruction IT better than IC? Instructional Method IT vs. IC Novice knowledge state Pre-test Learning Processes Normal Post-test Expert knowledge state Robust Learning measures: Transfer Long-term Accelerated retention future learning 35
Hausmann & Van. Lehn 2007 study: Macro level description • Research question: Does providing explanations or eliciting “selfexplanations” from students better enhance robust learning? • General instruction: Students alternate between – Watching videos of worked examples of physics problems – Solving new problems in the Andes intelligent tutor • Treatment variables: – Videos include justifications for steps or do not – Students are prompted to “self-explain” or paraphrase 36
Self-explanations => greater robust learning • Transfer to new electricity homework problems • Justifications: no effect! • Immediate test on electricity problems: • Instruction on electricity unit => accelerated future learning of magnetism! 37
Key features of H&V study • In vivo experiment – Ran live in 4 physics sections at US Naval Academy – Principle-focused: 2 x 2 single treatment variations – Tight control manipulated through technology • Use of Andes tutor => repeated embedded assessment without disrupting course • Data in Data. Shop (more later) 38
Theory Integration Strategy • PSLC has supported some 130+ studies • How do they fit together? 39
Macro-level of robust learning framework: Taxonomy of outcomes, processes, & treatments Robust Outcomes: Learning Sense. Making Learning Processes: Construction, elaboration, discrimination Co-Training Instructional Treatments: Collaboration, tell vs. ask, question insertion Visual-verbal coordination, example-rule coordination Foundational Skills Refinement of Features Strengthening Explicit Schedules, … instruction, learner control … 40
3 Original Research clusters Robust Learning Outcomes: Sense. Making Learning Processes: Construction, elaboration, discrimination Instructional Treatments: Collaboration, tell vs. ask, question insertion Interactive Communication Co-Training Visual-verbal coordination, example-rule coordination Coordinative Learning Foundational Skills Refinement of Features Strengthening Explicit Schedules, … instruction, learner control … Fluency and Refinement 41
Hausmann & Van. Lehn in Interactive Communication Cluster Robust Learning Outcomes: Sense. Making Learning Processes: Construction, elaboration, discrimination Instructional Treatments: Collaboration, tell vs. ask, question insertion Interactive Communication Co-Training Visual-verbal coordination, example-rule coordination Coordinative Learning Foundational Skills Refinement of Features Strengthening Explicit Schedules, … instruction, learner control … Fluency and Refinement 42
Interactive Communication Research Question • What properties of interactive communication promote robust learning? – 20+ projects/studies have addressed versions of this question 43
Use of wiki toward macrolevel theory development Experimenters & instructors: • Create study summary pages on wiki • Work out shared meanings of terms – Like, what does “self-explanation” mean? – Create & edit glossary pages in the wiki • Seek out commonalities with others – Tie their hypotheses to general principles – Create shared instructional principle pages 44
PSLC wiki supports theory integration Interactive Communication cluster page: Links to H&V study page: 45
PSLC wiki supports theory integration Self-explanation glossary entry 182 concepts in glossary 46
PSLC wiki supports theory integration Instructional Principle pages unify across studies Points back to Hausmann’s study page 47
Theoretical Framework Levels • Macro level – What instructional principles explain how changes in the instructional environment lead to changes in robust learning? • Micro level – Can learning be explained in terms of what knowledge components are acquired at individual learning events? 48
Key Micro-level Concepts • Knowledge component (unobservable) – A mental structure or process that a learner uses to accomplish steps in a task or a problem • Learning event (unobservable) – Points in time where knowledge components are learned or used • Instructional event (observable) – Element of learning environment designed to evoke a particular learning event – P(Learning_event | Instructional_event) < 1. 0 • Learner factors like metacognition & motivation affect this conditional probability 49
Kinds of Knowledge Components (KCs) Mental representations of: • Domain knowledge – Facts, concepts, principles, rules, procedures, strategies • Prerequisite knowledge – Feature encoding knowledge • Integrative knowledge – Schemas or procedures that connect other KCs • Metacognitive knowledge – About knowledge or controlling use of knowledge • NOT KCs: • Any external representation of knowledge – Textbook descriptions • Generic cognitive structures – Working memory • Continuous parameters on knowledge representations – Strength, level of engagement, implicit value of a goal, affect Beliefs & interests – What one likes, believes • Cross-cutting distinctions – Correct vs. incorrect – Verbal (explicit) vs. non-verbal (implicit) – Probabilistic vs. discrete 50
Macro-level analysis treats knowledge states & learning processes as black boxes Instructional Method IT vs. IC Novice knowledge state Pre-test Learning Processes Normal Post-test Expert knowledge state Robust Learning measures: Transfer Long-term Accelerated retention future learning 51
Macro-level analysis treats knowledge states & learning processes as black boxes Instructional Method IT vs. IC Novice knowledge state Pre-test Learning Processes Normal Post-test Expert knowledge state Robust Learning measures: Transfer Long-term Accelerated retention future learning 52
Unpacking learning: 3 kinds of events Instructional Event Novice knowledge state Pre-test Learning Event Assessment event Expert (desired) knowledge state Normal Post-test Robust Learning measures 53
PSLC’s Data. Shop • Vast open data repository, analysis tools, visualizations • Generates: – learning curves – statistical model fit (blue line) • Based on micro level analysis: – learning event opportunities – Averaged across knowledge components 54
Greatest impact of technology on 21 st century education? • Benefits to student learning from use of educational technology? • Perhaps • But my bet is: – Scientific advances coming from data mining of the vast explosion of learning data that will be coming from educational technologies 55
Back to H&V study: Micro-analysis Learning curve for main KC Self-explanation effect tapers but not to zero Example 1 Example 2 Example 3 56
Overview • Background – Intelligent Tutoring Systems – Cognitive Task Analysis • PSLC Methods, Resources, & Theory – – In vivo experimentation Learn. Lab courses Robust learning theoretical framework Enabling technologies • Summary Next 57
PSLC Enabling Technologies • Tools for developing instruction & experiments – CTAT (cognitive tutoring systems) • Sim. Student (generalizing an example-tracing tutor) – OLI (learning management) – Tu. Talk (natural language dialogue) – REAP (authentic texts) • Tools for data analysis – Data. Shop – Tag. Helper 58
Summary • Learning can be greatly improved! – Address robust learning – Increase certainty – Leverage technology • PSLC provides tools & processes to help learning researchers achieve these goals Robust Learning Principles Existing Solo Theories In Vivo Studies Learn. Lab Courses Enabling Technology 59
END 60
Скачать презентацию Studying and achieving robust learning with PSLC resources
e38157b5bcc4e9ad66b6cbf88354d094.ppt