Studying and achieving robust learning with PSLC resources Prepared by: Ken Koedinger HCII & Psychology, CMU Director of PSLC Presented by: Vincent Aleven HCII, CMU Member PSLC Executive Committee 1
7 th Annual Pittsburgh Science of Learning Center Summer School • 11 th overall – ITS was focus in 2001 to 2004 • Goals: – Learning science & technology concepts & tools – Hands-on project => poster on Fri 2
Vision for PSLC • Why? “rigorous, sustained scientific research in education” (NRC, 2002) Chasm between science & practice Indicators: Ed achievement gaps persist, Low success rate of randomized controlled trials • Underlying problem: Many ideas, too little sound scientific foundation • Need: Basic research studies in the field => PSLC Purpose: Identify the conditions that cause robust student learning – Field-based rigorous science – Leverage cognitive & computational theory, educational technologies 3
Builds off past success: Intelligent Tutors Bringing Learning Science to Schools! • Intelligent tutoring systems – Automated 1: 1 tutor – Artificial Intelligence – Cognitive Psychology • Andes: College Physics Tutor – Replaces homework Students: model problems with diagrams, graphs, equations Tutor: feedback, help, reflective dialog • Algebra Cognitive Tutor – Part of complete course 4
Tutors make a significant difference in improving student learning! • Andes: College Physics Tutor – Field studies: Significant improvements in student learning • Algebra Cognitive Tutor – 10+ full year field studies: improvements on problem solving, concepts, basic skills – Regularly used in 1000 s of schools by 100, 000 s of students! 5
President Obama on Intelligent Tutoring Systems! “we will devote more than three percent of our GDP to research and development. …. Just think what this will allow us to accomplish: solar cells as cheap as paint, and green buildings that produce all of the energy they consume; learning software as effective as a personal tutor; prosthetics so advanced that you could play the piano again; an expansion of the frontiers of human knowledge about ourselves and world the around us. We can do this. ” How close to this vision are we now? What else do we need to do? 6
Overview • PSLC Background Next – Intelligent Tutoring Systems – Cognitive Task Analysis • PSLC Methods & Tech Resources – In vivo experimentation – Learn. Lab courses, CTAT, Tag. Helper, Data. Shop • PSLC Theoretical Framework 7
Cognitive Tutor Approach
Cognitive Tutor Technology • Cognitive Model: A system that can solve problems in the various ways students can Strategy 1: IF the goal is to solve a(bx+c) = d THEN rewrite this as abx + ac = d Strategy 2: IF the goal is to solve a(bx+c) = d THEN rewrite this as bx + c = d/a Misconception: IF the goal is to solve a(bx+c) = d THEN rewrite this as abx + c = d 9
Cognitive Tutor Technology • Cognitive Model: A system that can solve problems in the various ways students can If goal is solve a(bx+c) = d Then rewrite as abx + ac = d 3(2 x - 5) = 9 If goal is solve a(bx+c) = d Then rewrite as abx + c = d If goal is solve a(bx+c) = d Then rewrite as bx+c = d/a 6 x - 15 = 9 2 x - 5 = 3 6 x - 5 = 9 • Model Tracing: Follows student through their individual approach to a problem -> context-sensitive instruction 10
Cognitive Tutor Technology • Cognitive Model: A system that can solve problems in the various ways students can If goal is solve a(bx+c) = d Then rewrite as abx + ac = d 3(2 x - 5) = 9 If goal is solve a(bx+c) = d Then rewrite as abx + c = d Hint message: “Distribute a across the parentheses. ” Known? = 85% chance 6 x - 15 = 9 Bug message: “You need to multiply c by a also. ” Known? = 45% 2 x - 5 = 3 6 x - 5 = 9 • Model Tracing: Follows student through their individual approach to a problem -> context-sensitive instruction • Knowledge Tracing: Assesses student's knowledge growth -> individualized activity selection and pacing 11
Cognitive Tutor Approach
Difficulty Factors Assessment: Discovering What is Hard for Students to Learn Which problem type is most difficult for Algebra students? Story Problem As a waiter, Ted gets $6 per hour. One night he made $66 in tips and earned a total of $81. 90. How many hours did Ted work? Word Problem Starting with some number, if I multiply it by 6 and then add 66, I get 81. 90. What number did I start with? Equation x * 6 + 66 = 81. 90 14
Algebra Student Results: Story Problems are Easier! Koedinger, & Nathan, (2004). The real story behind story problems: Effects of representations on quantitative reasoning. The Journal of the Learning Sciences. Koedinger, Alibali, & Nathan (2008). Trade-offs between grounded and abstract representations: Evidence from algebra problem solving. Cognitive Science. 15
“The Student Is Not Like Me” • To avoid your expert blind spot, remember the mantra: “The Student Is Not Like Me” • Perform Cognitive Task Analysis Use Data! to find out what students are like 17
Prior achievement: Intelligent Tutoring Systems bring learning science to schools A key PSLC inspiration: Educational technology as research platform to generate new learning science 19
Overview • PSLC Background – Intelligent Tutoring Systems – Cognitive Task Analysis • PSLC Methods & Tech Resources Next – In vivo experimentation – Learn. Lab courses, CTAT, Tag. Helper, Data. Shop • PSLC Theoretical Framework 20
PSLC Statement of Purpose Leverage cognitive and computational theory to identify the instructional conditions that cause robust student learning. 21
What is Robust Learning? • Achieved through: – Conceptual understanding & sense-making skills – Refinement of initial understanding – Development of procedural fluency with basic skills • Measured by: – Transfer to novel tasks – Retention over the long term, and/or – Acceleration of future learning 22
PSLC Statement of Purpose Leverage cognitive and computational theory to identify the instructional conditions that cause robust student learning. 23
In Vivo Experiments: Laboratory-quality principle testing in real classrooms 24
In Vivo Experimentation Methodology Important features of different research methodologies: • What is tested? – Instructional solution vs. causal principle • Where & who? – Lab vs. classroom • How? – Treatment only vs. Treatment + control • Generalizing conclusions: – Ecological validity: What instructional activities work in real classrooms? – Internal validity: What causal mechanisms explain & predict? 25
In Vivo Experimentation Methodology Lab Experiments Design Research Randomized Field Trials In vivo Experiments √ √ × × √ √ What? Instructional solution Causal principle √ Where & who? Lab √ Classroom × How? Treatment only √ √ × √ √ Internal validity √ × +/– √ Ecological validity × √ √ √ Treatment + control Generalizes how? 33
Learn. Lab A Facility for Principle-Testing Experiments in Classrooms 34
Learn. Lab courses at K 12 & College Sites • 6+ cyber-enabled courses: Chemistry, Physics, Algebra, Geometry, Chinese, English • Data collection – Students do home/lab work on tutors, vlab, OLI, … – Log data, questionnaires, tests Data. Shop Researchers Schools Learn Lab Chemistry virtual lab Physics intelligent tutor REAP vocabolary tutor 35
PSLC Technology Resources • Tools for developing instruction & experiments – CTAT (cognitive tutoring systems) • Sim. Student (generalizing an example-tracing tutor) – OLI (learning management) – Tu. Talk (natural language dialogue) – REAP (authentic texts) • Tools for data analysis – Data. Shop – Tag. Helper 36
PSLC Statement of Purpose Leverage cognitive and computational theory to identify the instructional conditions that cause robust student learning. 37
Overview • PSLC Background – Intelligent Tutoring Systems – Cognitive Task Analysis • PSLC Methods & Tech Resources – In vivo experimentation – Learn. Lab courses, CTAT, Tag. Helper, Data. Shop • PSLC Theoretical Framework Next 38
KLI Framework: Designing Instruction for Robust Learning The conditions that yield robust learning can be decomposed at three levels: Knowledge components Learning events Instructional events Get framework report at learnlab. org 39
Instructional events KLI Event Decomposition • Decompose temporal progress of learning • Observe/control instructional & assessment events • Infer learning events & changes in knowledge Explanation, practice, text, rule, example, teacher-student discussion KEY Ovals – observable Rectangle - inferred Solid line – cause Dashed line – inferences KC = Knowledge Component Assessment events Question, feedback, step in ITS Learning events KC accessible from long-term memory State test, belief survey Immediate performance Robust performance 40
KLI Framework: Designing Instruction for Robust Learning The conditions that yield robust learning can be decomposed at three levels: Knowledge components Learning events Instructional events 41
What’s the best form of instruction? Two choices? • More assistance vs. more challenge – Basics vs. understanding – Education wars in reading, math, science… • Researchers like binary oppositions too. We just produce a lot more of them! – – – – Massed vs. distributed (Pashler) Study vs. test (Roediger) Examples vs. problem solving (Sweller, Renkl) Direct instruction vs. discovery learning (Klahr) Re-explain vs. ask for explanation (Chi, Renkl) Immediate vs. delayed (Anderson vs. Bjork) Concrete vs. abstract (Pavio vs. Kaminski) … Koedinger & Aleven (2007). Exploring the assistance dilemma in experiments with Cognitive Tutors. Ed Psych Review. 42
More help Basics More challenge Understanding How many options What’s best? are there really? And what works Focused Gradually Distributed best when? practice widen practice Study examples 50/50 Test on problems Mix Study 50/50 Test Study Abstract Concrete Mix Immediate Delayed Abstract Immediate Delayed Block topics Fade in chapters Mix No feedback Block topics in chapters Interleave topics Explain Mix Ask for explanations Fade Explain No feedback Interleave topics Mix Ask for explanations 43
Derivation: Ø 15 instructional dimensions Ø 3 options per dimension Ø 2 stages of learning => 315*2 options 205, 891, 132, 094, 649 “Big Science” effort needed to tackle this complexity Cumulative theory development Field-based basic research with microgenetic data collection 44
An example of some PSLC studies in this space • Researchers like binary oppositions too. We just produce a lot more of them! – – – – Massed vs. distributed (Pashler) Study vs. test (Roediger) Examples vs. problem solving (Sweller, Renkl) Direct instruction vs. discovery learning (Klahr) Re-explain vs. ask for explanation (Chi, Renkl) Immediate vs. delayed (Anderson vs. Bjork) Concrete vs. abstract (Pavio vs. Kaminski) … 45
Learn by doing or by studying? • Testing effect (e. g. , Roediger & Karpicke, 06) – “Tests enhance later retention more than additional study of the material” • Worked example effect Kirschner, Sweller, & Clark (2006). Why minimal guidance during instruction does not work: …failure of… problem-based…teaching. – “a worked example constitutes the epitome of strongly guided instruction [aka optimal instruction]” – Paas and van Merrienboer (1994), Sweller et al. (1998), van Gerven et al. (2002), van Gog et al. (2006), Kalyuga et al. (2001 a, 2001 b), Sweller et al. (1998), Ayres (2006), Trafton & Reiser (1993), Renkl and Atkinson (2003), … the list goes on … • Theoretical goal: Address debate between desirable difficulties, like “testing effect”, and direct instruction, like “worked examples” • Limitation of past worked example studies have weaker control, untutored practice – PSLC studies compare to tutored practice 46
Worked Example Experiments within Geometry Cognitive Tutor (Alexander Renkl, Vincent Aleven, Ron Salden, et al. ) • 8 studies in US & Germany – Random assignment, vary single principle – Over 500 students – 3 in vivo studies run in Pittsburgh area schools • Cognitive Science ’ 08 Conference IES Best Paper Award 47
Ecological Control = Standard Cognitive Tutor Students solve problems step-by-step & explain 48
Treatment condition: Half of steps are given as examples Worked out steps with calculation shown by Tutor Student still has to self explain worked out step 49
Worked examples improve efficiency & understanding Lab results • 20% less time on instruction • Conceptual transfer in study 2 d =. 73 * In Vivo • Adaptively fading examples to problems yields better long-term retention & transfer 50
Worked example effect generalizes across domains, settings, researchers • Geometry tutor studies • Chemistry tutor studies in vivo at High School & College (Mc. Laren et al. ) – Same outcomes in 20% less time • Algebra Tutor study in vivo (Anthony et al. ) – Better long term retention in less time • Theory: Sim. Student model (Matsuda et al. ) – Problems provide learning process with negative examples to prune misconceptions • Research to practice – Influencing Carnegie Learning development – New applied projects with SERP, West. Ed 51
Processes of Learning within the KLI Framework The conditions that yield robust learning can be decomposed at three levels: Knowledge components Learning events Instructional events Learning events Fluency building, refinement, sense making 52
Learning Events in the Brain & reflected in Dialogue • Fluency building Memory, speed, automaticity • Refinement processes Classification, co-training, discrimination, analogy, non -verbal explanation-based learning • Sense-making processes Reasoning, experimentation, explanation, argument, dialogue Some PSLC Examples • ACT-R models of spacing, testing effects & instructional efficiency (Pavlik) • Sim. Student models of learning by example & by tutoring – Inductive logic prgrming, probabilistic grammars (Matsuda, Cohen , Li, Koedinger) • Transactivity+ analysis of peer & classroom learning dialogues (Rose, Asterhan, Resnick) 53
Knowledge components carry the results of learning • Knowledge component = an acquired unit of cognitive function or structure that can be inferred from performance on a set of related tasks. • Used in broad sense of a knowledge base – From facts to mental models, metacognitive skills 54
Example KCs with different features • Chinese vocabulary KCs: const->const, explicit, no rationale – If the Chinese pinyin is “lao 3 shi 1”, then the English word is “teacher” – If the Chinese radical is “日”, the English word is “sun” • English Article KCs: var->const, implicit, no rationale – If the referent of the target noun was previously mentioned, then use “the” • Geometry Area KCs: var->var, implicit & explicit, rationale – If the goal is to find the area of a triangle, and the base <B> and the height is <H>, then compute 1/2 * <B> * <H> – If the goal is to find the area of irregular shape made up of regular shapes <S 1> and <S 2>, then find area <S 1> and <S 2> and add Integrated KCs for mental models, central conceptual structures, strategies & complex planning 55
Kinds of Knowledge Components • Other kinds of KCs – Integrative, probabilistic, metacognitive, misconceptions or “buggy” knowledge 56
Knowledge components are not just about domain knowledge • Examples of possible domain-general KCs • Metacognitive strategy – Novice KC: If I’m studying an example, try to remember each step – Desired KC: If I’m studying an example, try to explain how each step follows from the previous • Motivational belief – Novice: I am no good at math – Desired: I can get better at math by studying and practicing • Social communicative strategy – Novice: When an authority figure speaks, remember what they say. – Desired: Repeat another's claim in your own words and ask whether you got it right Can these be assessed, learned, taught? Broad transfer? 57
Learning curves: Measuring behavior on tasks over time • • • Data from flash-card tutor Tasks present a Chinese word & request the English translation Learning curve shows average student performance (e. g. , error rate, time on correct responses) after each opportunity to practice 6 secs 3 secs 58
Empirical comparison of KC complexity Example KC types Chinese vocabulary KCs const->const, explicit, no rationale English Article KCs var->const, implicit, no rationale 6 secs 3 secs Time 6 -> 3 secs 10 -> 6 secs 10 secs 6 secs Geometry Area KCs var->var, implicit/explicit, rationale 14 -> 10 secs 14 secs 10 secs 59
Which instructional principles are effective for which kinds of knowledge components? Do complex instructional events aid simple knowledge acquisition? Do simple instructional events aid complex knowledge acquisition? 60
Prompted self-explanation studies across domains • Physics Course - field principles – Better transfer than providing explanations • Geometry Course - properties of angles Var->Var, explicit – Better transfer than just practice despite solving 50% fewer problems in same time • English Course - article use – Pure practice appears more efficient; self-explanation may help for long-term retention & for novices Var->Const, implicit • Cross-domain hypoth: Type of KC determines when self-explanation will be effective. 61
Summary • Obama: “learning software as effective as a personal tutor” • How close to this vision are we now? – Many fielded Intelligent Tutors – Students learn as much or more • What else do we need to do? – Expand to more areas => CTAT – More sophisticated interaction => CSCL – Use tutors to advance science & improve educational practice => In Vivo & EDM In other words … Take the PSLC Summer School! 62
63
To do • Other possibilities – Reduce in vivo slide by deleting redundancy between text and table – Define ITS as about the intelligence in the design – Add an opener? • Not techy enough – TEDx talk => KDD Cup => learning curves • Lots of issues -- gaming vs. not (Shih) • Tie to WE and SE – Get from HCI AB talk (CSCL example) Too much listing at the end 64
Learn. Lab Products Infrastructure created and highly used • Learn. Lab courses have supported over 150 in vivo experiments • Established Data. Shop: A vast open data repository & associated tools – 110, 000 student hours of data • 21 million transactions at ~15 second intervals – New data analysis & modeling algorithms – 67 papers, >35 are secondary data analysis not possible without Data. Shop 65
Typical Instructional Study • Compare effects of 2 instructional conditions in lab • Pre- & post-test similar to tasks in instruction Instruction Expert Novice Learning Pre-test Post-test 66
PSLC Instructional Experiments • Macro: Measures of robust learning Instruction Expert(desired) Novice Learning Pre-test Post-test: Post-test Long-term retention, transfer, accelerated future learning, or desire for future learning 67
PSLC Instructional Experiments • Macro: Measures of robust learning • Micro analysis: knowledge, learning, interactions Instruction Expert(desired) Novice Knowledge: Shallow percepts & concepts Pre-test Instructional Events Learning Assessment Events Knowledge: Deep percepts & concepts, fluent Post-test: Post-test Long-term retention, transfer, accelerated future learning, or desire for future learning 68
PSLC Instructional Experiments • Macro: Measures of robust learning • Micro analysis: knowledge, learning, interactions • Studies run in vivo as part of existing courses Instruction Expert(desired) Novice Knowledge: Shallow percepts & concepts Pre-test Course goals & cultural context Instructional Events Learning Assessment Events Knowledge: Deep percepts & concepts, fluent Post-test: Post-test Long-term retention, transfer, accelerated future learning, or desire for future learning 69
Develop a research-based, but practical framework • Theoretical framework key goals – Support reliable generalization from empirical studies to guide design of effective ed practices Two levels of theorizing: • Macro level – What instructional principles explain how changes in the instructional environment cause changes in robust learning? • Micro level – Can learning be explained in terms of what knowledge components are acquired at individual learning events? 70
Example study at macro level: Hausmann & Van. Lehn 2007 • Research question – Should instruction provide explanations and/or elicit “self-explanations” from students? • Study design – All students see 3 examples & 3 problems • Examples: Watch video of expert solving problem • Problems: Solve in the Andes intelligent tutor – Treatment variables: • Videos include justifications for steps or do not • Students are prompted to “self-explain” or paraphrase 71
Paraphrase Explan Selfexplain X No explan 72
Paraphrase Selfexplain Explan No explan X 73
Self-explanations => greater robust learning • Transfer to new electricity homework problems • Justifications: no effect! • Immediate test on electricity problems: • Instruction on electricity unit => accelerated future learning of magnetism! 74
Key features of H&V study • In vivo experiment – Ran live in 4 physics sections at US Naval Academy – Principle-focused: 2 x 2 single treatment variations – Tight control manipulated through technology • Use of Andes tutor => repeated embedded assessment without disrupting course • Data in Data. Shop (more later) 75
Develop a research-based, but practical framework • Theoretical framework key goals – Support reliable generalization from empirical studies to guide design of effective ed practices Two levels of theorizing: • Macro level – What instructional principles explain how changes in the instructional environment cause changes in robust learning? • Micro level – Can learning be explained in terms of what knowledge components are acquired at individual learning events? 76
Knowledge Components • Knowledge Component – A mental structure or process that a learner uses, alone or in combination with other knowledge components, to accomplish steps in a task or a problem-- PSLC Wiki • Evidence that the Knowledge Component level functions in learning … 77
Back to H&V study: Micro-analysis Learning curve for main KC Self-explanation effect tapers but not to zero Example 1 Example 2 Example 3 79
PSLC wiki: Principles & studies that support them Instructional Principle pages unify across studies Points to Hausmann’s study page (and other studies too) 80
PSLC wiki: Principles & studies that support them Hausmann’s study description: With links to concepts in glossary 81
PSLC wiki: Principles & studies that support them Self-explanation glossary entry ~200 concepts in glossary 82
Research Highlights • Synthesizing worked examples & self-explanation research – 10+ studies in multiple 4 math & science domains – New theory: It’s not just cognitive load! • Examples for deep feature construction, problems & feedback for shallow feature elimination This work inspired new question: Does self-explanation enhance language learning? Experiments in progress … • Computational modeling of student Learning – Simulated learning benefits of examples/demonstrations vs. problem solving (Masuda et al. , 2008) • Theory outcome: problem solving practice is an important source of negative examples • Engineering: “programming by tutoring” is more cost-effective than “programming by demonstration” – Shallow vs. deep prior knowledge changes learning rate (Matsuda et al. , in press) 83
Research Highlights (cont) · Computational modeling of instructional assistance · Assistance formula: Optimal learning (L) depends on L right level of assistance L = P*Sb+(1 -P)Fb P*Sc+(1 -P)Fc Assistance · Relevant to multiple experimental paradigms & dimensions of instructional assistance P · Direct instruction (worked examples) vs. constructivism (testing effect) Kirschner, Sweller, & Clark (2006). Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. Educational Psychologist · Concrete manipulatives vs. simple abstractions Kaminski, Sloutsky, & Heckler (2008). The advantage of learning abstract examples in learning math. Science. · Formula provides path to resolve hot debates 84
Research Highlights (cont) · Synthesis paper on computer tutoring of metacognition Koedinger, Aleven, Roll, & Baker. (in press). In vivo experiments on whether supporting metacognition in intelligent tutoring systems yields robust learning. In Handbook of Metacognition in Education. · Generalizes results across 7 studies, 3 domains, 4 populations · Posed new questions about role of motivation · Lasting effects of metacognitive support · Computer-based tutoring of self-regulatory learning · Technologically possible & can have a lasting effect · Students who used help-seeking tutor demonstrated better learning skills in later units after support was faded · Spent 50% more time reading help messages · Data mining for factors that affect student motivation · Machine learning to analyze vast student interaction data from full year math courses (Baker et al. , in press a & b) · Students more engaged on “rich” story problems than standard · Surprise: Also more engaged on abstract equation exercises! 85
Thrusts investigate overlapping factors Social context of classroom Novice Knowledge: Shallow, perceptual Metacognition Motivation Instruction Teacher Interaction Motivation Metacognition Learning THRUSTS Cognitive Factors Metacognition & Motivation Social Communication Comp Modeling & Data Mining Expert Knowledge: Deep, conceptual, fluent Metacognition Motivation 86
Thrust Research Questions • Cognitive Factors. How do instructional events affect learning activities and thus the outcomes of learning? • Metacognition & Motivation. How do activities initiated by the learner affect engagement with targeted content? • Social Communication. How do interactions between learners and teachers and computer tutors affect learning? • Computational Modeling & Data Mining. Which models are valid across which content domains, student populations, and learning settings? 87
4 th Measure of Robust Learning • Existing robust learning measures – Transfer – Long-term retention – Acceleration of future learning • New measure: – Desire for future learning • Is student engaged in subject? • Do they chose to pursue further math, science, or language? 88
END 89
Скачать презентацию Studying and achieving robust learning with PSLC resources
fd97e35da5856a62c8116e91d1f18973.ppt