Скачать презентацию Andes Tutoring Freedom Support and Accelerated Learning as
882ee7104ea3c1551fd12412b3b3e8c1.ppt
- Количество слайдов: 84
Andes Tutoring: Freedom, Support, and Accelerated Learning as Students Solve Complex Physics Problems Kurt Van. Lehn & Brett van de Sande School of Computing, Informatics and Decision Engineering Arizona State University 1
Andes development team u Developers – Brett van de Sande u Instructors/designers – Bob Shelby – Don Treacy u Experimenters – – – Scotty Craig Bob Hausmann Sandy Katz Tim Nokes Michael Ringenberg u Instructors 4 -year colleges Jim Culbertson – Arizona State University, AZ John Fontinella – US Naval Academy, MD David Guerra – St. Anselm College, NH Troy Hacker – US Air Force Academy, CO Andrew Heckler – Ohio State University, OH Gerd Kortemeyer – Michigan State University Ted Mc. Clanahan – US Naval Academy, MD Mary Wintersgill – US Naval Academy, MD 2 -year colleges Tom Wilbur – Ann Arundel Com. College, MD High Schools Sophia Gersham – Watchung Hills Reg. HS, NJ Paul Perkins – Belleview Christian School, WA David Richardson – Packer Academy, NY 2
Outline u Goals of the Andes project u Andes – the core u Andes – the surroundings u Andes – technology u Evaluation u Why does Andes succeed? Next 3
Why physics? u Required for the other sciences u Major source of attrition u Well-studied in cognitive science u Other courses are similar 4
A typical physics course Repeat: u Read chapter u Attend lectures u Solve problems u Do lab u Review for exam u Take chapter exam Tutoring needed Take finals: Standardized & conceptual 5
Why focus on problem solving? u Students spend most of their time on problem solving u Frustrating; May causes attrition u Bad habits can develop – “Symbol pushing” instead of deep understanding u Done right, it can elicit deep, conceptual understanding u Professional human tutors focus on it 6
What do professional tutors do? u Pick a good problem Andes could, but doesn’t – Neither trivially easy nor impossibly difficult – Problem targets a weakness of the student u Help the student solve it Andes does – Let student try each step – Give immediate feedback if step is wrong – Give hints sparingly u Reflective debriefing – What were main principles? – What did you learn? Under development 7
Repeat: Just these 2 as of today u Read chapter u Attend lectures u Solve problems u Do lab u Review for exam u Take chapter exam Comparison conditions How to evaluate Andes? 1. 2. 3. 4. Paper & human graders Paper & grading service Andes Professional human tutor Take finals: Standardized & Conceptual 8
Main goals: Andes should be a “workbook” where: u Instructors select & assign Andes problems – Need lots of problems, covering all textbooks u Students solve problems on Andes, not paper – getting immediate feedback and hints u Students learn more, compared to paper 9
More goals u Instructors agree with all of Andes’ advice u Instructors no longer need to grade homework u Students prefer Andes to paper u Andes is as effective as human tutors 10
Outline u Goals of the Andes project u Andes – the core u Andes – the surroundings u Andes – technology u Evaluation u Why does Andes succeed? Next 11
A typical physics problem A 2000 kg car at the top of a 20º inclined driveway 20 m long slips its parking brake and rolls down. If we ignore friction and drag, what is the magnitude of the velocity of the car when it hits the garage door? u University introductory physics courses u High school physics courses 12
Andes user interface Read a physics problem Draw vectors Type in equations Type in answer 13
Andes feedback and hints “What should I do next? ” “What’s wrong with that? ” Green means correct Red means incorrect Dialogue & hints 14
Frequently asked questions (slide 1 of 3) u Why is Andes sometimes called an intelligent tutoring system? – Tutors usually replace classes; Andes doesn’t. – Andes should be called a “step-based homework helper” u Is Andes the same as web-based grading services such as Web. Assign, Mastering Physics & LONCAPA? – They give feedback & hints on answers only – Andes gives feedback & hints on steps leading to answers 15
Most existing tutoring systems and web-based grading services accept only the answer W = 25 30° 40° x° y° u° z° w° Answer 45° What is the value of angle w? 16
With step-based systems (e. g. , Andes), students enter steps leading up to the answer Step 40+30+x=180 x=110 30° 40° x° y° u° z° x=z w° w+45+z=180 w+45+110=180 45° What is the value of angle w? w=180 -155 Step Step w=25 Answer Step 17
Step-based homework helpers usually give immediate feedback on each step w = 40 30° 40° x° y° u° z° OPPS! w° 45° What is the value of angle w? OK 18
Hints start general… w = 40 30° 40° x° y° u° z° w° Lines that look parallel often are not. 45° What is the value of angle w? OK 19
Hints become more specific. w = 40 30° 40° x° y° u° z° w° Try summing the angles of the triangle that include angle w. (see pg. 212) 45° What is the value of angle w? OK 20
Usually, the last hint tells the student exactly what to enter. w = 40 30° 40° x° y° u° z° w° You should apply the triangle sum rule by entering 45+z+w=180. 45° What is the value of angle w? OK 21
Frequently asked questions (slide 2 of 3) u Does Andes require students to do all the steps? – No. But skipping conceptual steps lowers the score. u Does it require doing steps in a particular order? – No, except that variables must be defined before being used in equations – The only way to define a vector component is to draw the vector u Are there correct solutions that Andes doesn’t accept. – No (we hope). 22
Frequently asked questions (slide 3 of 3) u Why does Andes focus on problem solving? Problem solving doesn’t improve conceptual understanding [sic]. Instruction really should focus on _____ instead of problem-solving. – Andes requires more conceptual steps than paper. – Reflective debriefing is being added – Perhaps instructors should let Andes to handle problem -solving so they can focus on _____. 23
Outline u Goals of the Andes project u Andes – the core u Andes – the surroundings u Andes – technology u Evaluation u Why does Andes succeed? Next 24
Two methods for accessing Andes problems u Stand-alone – Downloads as a standard Windows application u Open-Learning Initiative (Web-based LMS) 25
Standalone opening screen is a menu of physics topics (apprx. chapters) 26
After clicking on “Translational dynamics, ” get menu of problems 27
Two methods for accessing Andes problems u Stand-alone – Downloads as a standard Windows application u Open-Learning Initiative (Web-based LMS) – Free & open usage » OLI keeps no record of student’s usage Next – Authenticated member of a registered class » OLI keeps a grade book for the instructor 28
OLI top level screen Andes 29
OLI top level of Andes is a menu of topics (apprx. Chapters) 30
After clicking on “Translational Dynamics” module, get menu of problems 31
OLI grade book u Looks/acts like a spreadsheet – One row per student – One column per problem – Cell has student’s score on the problem u Can export to Excel or database u Clicking a cell displays student’s solution in instructor’s Andes. – Can check student’s work – Good for office hours 32
Outline u Goals of the Andes project u Andes – the core u Andes – the surroundings u Andes – technology u Evaluation u Why does Andes succeed? Next 33
The major components of an ITS User interface Problem to be solved Expert All correct steps in all orders Helper Response pattern for each student step Assessor P(mastery) for each knowledge compo.
The expert’s computation u Expert can be authors, students or an expert system u Solve the problem in all acceptable ways u Record steps taken u Record knowledge components used at each step Problem to be solved Expert All correct steps in all orders Helper Response patterns for each student step Assessor P(mastery) for each knowledge compo.
Andes uses an expert system u Knowledge base organized by “problem solving methods” – One per major physics principle – Often called as “principle schemas” – Approximately 100 principles covered in 1 -year course u u Can solve more that 500 physics problems For each, finds all acceptable solutions – Merges them into a solution graph u Pre-generates solution graphs & saves on disk – Allows regression testing (compare new to old) – Allows instructors to inspect 36
Problem solving method for Newton’s second law To apply Newton’s second law to
Problem & its solution graph Draw body for car A 2000 kg car at the top of a 20º inclined driveway 20 m long slips its parking brake and rolls down. If we ignore friction and drag, what is the magnitude of the velocity of the car when it hits the garage door? Draw coord. axes @ 20 deg Define given quantities d=20 m; m= 200 kg Apply conservation of energy vf^2 = 2*g*sin(20 deg) Solve equations for final velocity vf= 11. 59 m/s Enter answer: 11. 59 m/s Apply translational kinematics vi^2 = 2*a*d Apply Newton’s second law a= -g*cos(200 deg) • Draw weight force • Draw normal force • Draw acceleration 1. Draw free-body diagram 2. Apply F=m*a along x-axis 3. Project vectors onto x-axis 4. Apply weight law W=m*g 38
4 popular designs for the Expert u Hand-author Problem to be solved Expert All correct steps in all orders all possible solutions per problem – Auto. Tutor, CTAT u Rule-based AI problem solver + problem all possible solutions – Andes, Cognitive tutors u Hand-author one solution & use constraints to generalize to all possible solutions – Constraint-based tutors e. g. , SQL Tutor u Given paths log files from students, induce shortest/best – i. Lisp, Barnes tutor
The helper’s computation u When the student enters a step, match it to a correct step u Give feedback & hints as necessary u Record pattern response Problem to be solved Expert All correct steps in all orders Helper Response patterns for each student step Assessor P(mastery) for each knowledge compo
How to tell if student’s equation is correct? u u u Color by numbers algorithm Given student’s equation “Fw_x+m*g*sin(20 deg)” Substitute values from solution point – Fw_x 6704 N – g 9. 8 m/s^2 – m 2000 kg u Check arithmetic. – If checks, equation is correct u See paper for dealing with variables that don’t have values in the solution point 41
When the student needs a hint, how to choose next step? u Which solution branch is student probably following? u First PSM along path that is not finished? u First step inside the PSM that is not yet done? u But how to tell which equation-writing steps have been done already? 42
Matching equations via the indy check algorithm u Student enters “Fw_x+m*g*sin(20 deg)” – Row S below u Which equations in solution were combined? – Rows A, B, C, D – Because S’s gradient is a linear combination of their gradients f/ m f/ g f/ Fw_x gradient A Fw_x –Fw*cos(250º) 0 0 –cos 250º 1 (0, 0, 0. 342, 1) B Fw – mc*g –g –mc 1 0 (– 9. 8 , – 2000, 1, 0) C mc – 2000 1 0 0 0 (1, 0, 0, 0) D g – 9. 8 0 1 0 0 (0, 1, 0, 0) S Fw_x+mc*g*sin(20º) g*sin 20º mc*sin 20º 0 1 (3. 352, 684, 0, 1) Function f 43
3 main design issues for the Helper u Matching All correct steps in all orders Helper Response patterns for each student step student’s step to correct steps – Natural language: Use LSA, keywords, Atlas… – Math expressions: Substitute numbers for variables – Physical actions: Fuzzy, Bayesian u Handling pedagogically important student errors u Managing the student-tutor dialogue – Immediate feedback + hint sequences – Delayed feedback + student or tutor controlled debriefing – Adaptive, especially decision theoretic & fading
The assessor’s computation u Given – Response patterns for each step taken by the student – Old P(mastery) for each knowledge component u Calculate – New P(mastery) Problem to be solved Expert All correct steps in all orders Helper Response patterns for each student step Assessor P(mastery) for each knowledge compo. Currently, Andes doesn’t do such assessment
Where’s the scale-up bottleneck? Code size Expert User interface Helper Assessor Number of problems
Other scaling up issues = same as other reforms u Coordination with curriculum & standards u Teacher buy-in and training u Support u Etc…
Implementation: Summary u Expert – Expert system, not humans u Helper – Matching student equations is challenging u Assessor – Dynamic Bayesian networks Problem to be solved Expert All correct steps in all orders Helper Response patterns for each student step Assessor P(mastery) for each knowledge component
Outline u Goals of the Andes project u Andes – the core u Andes – the surroundings u Andes – technology u Evaluation u Why does Andes succeed? Next 49
Prior work with answer-only homework helpers u Compared to ordinary paper-based homework – Modest effect size: 0. 42 (Kulik et al. , 1983) u Compared to paper-based homework that is collected & graded – No benefits (Pascarella, 2002; Dufresne, Mestre & Rath, 2002) u Interpretation – Motivating students to do their homework provides benefits – But answer-only feedback & hints provides no additional benefits 50
Prior work with step-based homework helpers u Lisp Tutor (Corbett, 2001) and many others – Same homework problems & text – Experimenter’s exams only – Not a whole semester (only 5 lessons) u Cognitive Tutors (Koedinger et al; Carnegie Learning) – Whole year of high-school algebra, geometry – Both experimenter’s exams & standard exams – Curriculum confounded with tutoring system 51
Evaluation of Andes at the US Naval Academy u Fall semesters 2000, 2001, 2002 & 2003 u Only the homework modality was varied: Andes vs. paper-based – – Same textbook Similar lectures, labs, recitations Similar homework problems Same exams u Students were motivated to do paper-based homework – Either collected and graded – Or 1 homework problem on each quiz 52
Exams u Midterm exam – 1 hour, 4 problems – Scored on derivation & answer » Drawings (30%) » Variable definitions (20%) » Equations (40%) » Answers (10%) u Final exam – 3 hours, 50 problems – Multiple choice 53
Subjects; non-random assignment 2000 N Andes 140 N control 135 u Prior 2001 2002 2003 129 93 93 44 53 44 competence equal, all 4 years – Grade-point averages equal – Distribution of majors equal 54
Midterm exam results (All differences reliable, p <. 01) Effect size: 0. 61 55
Final exam: Methodological details u Andes coverage of the course increased to 70% by 2003, so used only that year’s final exam u Non-random sample – 89 Andes students (3 sections) – 823 non-Andes students (rest of course) – GPAs, Majors reliably different u Regressed out incoming GPA, Major 56
Final exam results Difference is reliable (p = 0. 028) Effect size = 0. 25 57
Effect sizes for subscores of midterm exam The more conceptual the subscore, 58 the greater the benefit
Benefits same regardless of GPA 59
Benefits varied by major on final exam but not on midterm exam 60
Interpretation of results u Engineering & science majors knew red path & preferred it for answer-only – Andes didn’t affect their final exam scores u Other majors did not have red path, so they used the blue path on answer-only – Andes increased their final exam scores u Problem Andes Diagram & variables Everyone used blue path on midterms – Andes increased everyone’s midterm exam scores – Biggest benefit for diagrams & variables – Smaller on equations; none on answer Andes Equations Prior math & physics Answer 61
Summary of results u Main result: Replacing graded paper homework with Andes provides benefits – Midterm exam effect size: 0. 61 – Final exam effect size: 0. 25 u Andes helps students learn conceptual skills – Effect sizes on conceptual subscores: 1. 21 & 0. 69 – Effect sizes on calculational subscores: 0. 11 & -0. 08 u Engineering & Science majors appear to have a non-conceptual method for solving problems – Competes with the conceptual method taught by Andes – They use it on the (answer-only) final exam – This dilutes the benefit of Andes on final exam 62
Andes’ effect sizes are typical of other step-based homework helpers 63
Take-home message on homework helpers u Baseline: – Paper-based with light or no human grading u Better: – Answer-based e. g. , Web. Assign u Best: – Step-based e. g. , Andes – 64
Outline u Goals of the Andes project u Andes – the core u Andes – the surroundings u Andes – technology u Evaluation u Why does Andes succeed? Next 65
Why these results? Hypotheses: u Baseline: – Paper-based with light or no human grading – Students often do not do their homework u Better: – Answer-based e. g. , Web. Assign – Students do homework, but with non-optimal methods u Best: – Step-based e. g. , Andes – Students do homework with optimal method 66
3 methods for doing homework 1. 2. 3. Get answers from friends Copy & edit another problem’s solution Generate each step oneself 67
1. Getting answers from friends u No learning, so should be discouraged u Even if the numbers in a problem are randomly generated, students circulate spreadsheets that calculate answers u Andes requires that students “show their work” – Andes can analyze time per step, too. – So Andes can make this kind of cheating very difficult 68
2. Copy and edit another problem’s solution u Students – – learn general solution schemas E. g. , for pulley problems, a = g(m 1 -m 2)/(m 1+m 2) Good for learning to solve algebra word problems High math students tend to use it (cf. USNA) But should learn physics principles, not problems u Andes mildly discourages – Must close current problem in order to open an old one – Example solutions are videos, not paper u Andes should implement “fading” 69
3. Generate each step oneself u If stuck, get hints (from Andes), or refer to textbook or videos of examples – Should focus on learning the principle, not just getting unstuck – Hint sequences start vague, become specific u Causes learning of principles, not problems 70
Hypothesized distributions of solution methods explains results 71
Take-home message, again u Baseline: – Paper-based with light or no human grading – Students often do not do their homework u Better: – Answer-based e. g. , Web. Assign – Students do homework, but with non-optimal methods u Best: – Step-based e. g. , Andes – Students do homework with optimal method 72
Outline u Goals of the Andes project u Andes – the core u Andes – the surroundings u Evaluation u Why does Andes succeed? u Future work Next 73
Future work u Andes 3 – Client/server (no downloading necessary) – Client runs in any browser & any OS – User interface ≈ Power. Point (for familiarity) – Compatible with Web. Assign & other services u Help others to author Andes problems u Support conceptual problem solving 74
Quantitative problem solving is necessary but not sufficient for deep understanding u Necessary – Avoids inert knowledge u Not sufficient – Question: » If a massive truck and a lightweight car have a headon collision, on which vehicle is the impact force greater? – Students who got A’s say: The car – Correct answer: Equal (Newton’s third law) u Many concept inventories (exams) exist 75
Does Andes improve scores on concept inventories? u Learning principles instead of problems should help u But not enough – USNA: FCI not significantly different 76
Why so little transfer from quantitative to conceptual problem solving? Quantitative prob. solving knowledge Conceptual prob. solving knowledge Traditional assumption lots of transfer Cognitive modeling little transfer 77
Solution: Add conceptual problem solving practice W=m*g Quantitative prob. solving knowledge Conceptual prob. solving knowledge If no mass, then no weight u Question: In the problem you just solved, what would be different if the car had no mass? 78
Katz & Connelly’s experimental addition to Andes u After students have finished an Andes problem, use a natural language dialogue system to discuss the problem’s solution u System asks lots of conceptual questions 79
Students first solve an Andes problem 80
Andes then asks about principles, alternative situations… u u u u Andes: Which major principle did you need to solve the problem? Student: Newton's 2 nd law Andes: Yes. The main difference between this problem and most of the previous Newtons 2 nd law problems is the fact that the direction of acceleration is _____. (fill in the blank) Student: centripetal Andes: Correct. All else being the same, how would the velocity at the lowest point of the swing change if the length of the rope were doubled to 4. 0 m? Student: it would also double Andes: Well, it would increase, but it would not double. Recall that the formula for centripetal acceleration involves the square of the velocity. … 81
Results of Katz/Connelly expt. § Successes § Treatment students significantly outperformed control students on a mostly qualitative post-test § Instructors would like reflective dialogue to become a permanent feature of Andes § Limitations § Currently covers only 8 weeks of Andes § No significant effect on FCI at the end of the semester § Next steps § Revise content to target FCI § Cover whole semester 82
Thanks for your attention! u At www. andestutor. org you can… – Download the stand-alone version of Andes – Try the OLI version of Andes – Download papers on Andes – View videos of Andes being used 83
User interface Questions? Problem to be solved Expert All correct steps in all orders Helper Response pattern for each student step Assessor P(mastery) for each knowledge compo. 84