Скачать презентацию Improving Pedagogical Efficiency and Effectiveness using an Automated Скачать презентацию Improving Pedagogical Efficiency and Effectiveness using an Automated

65505868f193deef4862532305437c1b.ppt

  • Количество слайдов: 19

Improving Pedagogical Efficiency and Effectiveness using an Automated Testing System Ramon Lawrence, Alyosha Pushak, Improving Pedagogical Efficiency and Effectiveness using an Automated Testing System Ramon Lawrence, Alyosha Pushak, Teresa Wrzesniewski University of British Columbia Okanagan

Introduction Technology can improve teaching efficiency and delivery, but the most repetitive and time-consuming Introduction Technology can improve teaching efficiency and delivery, but the most repetitive and time-consuming tasks of assignment and test preparation and marking are still mostly manual. Automated testing systems have the potential to dramatically reduce the amount of time spent on these repetitive tasks, freeing up instructor time for valuable student contact, and allowing for more uniformity on student and instructor achievement and evaluation. We have developed the Auto. Ed system that is an automated question generation and marking system and evaluated it in a first year Physics course. Ramon Lawrence, Alyosha Pushak, Teresa Wrzesniewski Improving Pedagogical Efficiency and Effectiveness using an Automated Testing System UBCO Research Week Presentation 2011 University of British Columbia Page Copyright© 2011 2

Background – Testing Systems There is a variety of technology for student testing: uclickers, Background – Testing Systems There is a variety of technology for student testing: uclickers, web-based learning, course management systems, etc. It is challenging to determine which technologies are appropriate for certain teaching styles and courses, and what are the correct products and systems that provide the appropriate payback for student learning and institutional deployment. Most of the question systems are limited to static questions. u. There is no learning value in repetitive practice. u. The questions do not encourage or improve learning; they simply test if previous learning has occurred. Ramon Lawrence, Alyosha Pushak, Teresa Wrzesniewski Improving Pedagogical Efficiency and Effectiveness using an Automated Testing System UBCO Research Week Presentation 2011 University of British Columbia Page Copyright© 2011 3

Background – Question Templates A question template is a question that tests a specific Background – Question Templates A question template is a question that tests a specific concept that can be re-used by varying its contents. Example: Calculate the distance travelled by a car in 2 hours that is travelling 60 km/h. Template: Calculate the distance travelled by a $x$ in $y$ that is travelling $z$. Randomization: For x, y, and z. Answer: y*z (with correct units) u. Incorrect answers can also be encoded with feedback. Ramon Lawrence, Alyosha Pushak, Teresa Wrzesniewski Improving Pedagogical Efficiency and Effectiveness using an Automated Testing System UBCO Research Week Presentation 2011 University of British Columbia Page Copyright© 2011 4

Previous Systems Previous question template systems: u. WWWAssign built by Dr. Larry Martin for Previous Systems Previous question template systems: u. WWWAssign built by Dr. Larry Martin for physics courses. Currently a commercial product called Web. Assign. u. Aplia - social sciences u. Mastering Physics u. Gradiance - computer science u. Carnege Learning – mathematics These products have textbook questions and provide web hosting. The cost is $15 -$100 per student per course. Course management systems such as Web. CT/Blackboard have question systems and some support for randomized questions. Ramon Lawrence, Alyosha Pushak, Teresa Wrzesniewski Improving Pedagogical Efficiency and Effectiveness using an Automated Testing System UBCO Research Week Presentation 2011 University of British Columbia Page Copyright© 2011 5

Previous Systems – Open Source There have been several university led efforts on building Previous Systems – Open Source There have been several university led efforts on building these systems and providing access. Quest Service at The University of Texas at Austin umany science/math questions uoutside users can get access by requesting UT Austin accounts LON-CAPA is an open source system built primarily at Michigan State University. u. It has hundreds of thousands of resources and questions that are shared by multiple institutions. Institutions are part of a sharing network for distributing questions and resources. Ramon Lawrence, Alyosha Pushak, Teresa Wrzesniewski Improving Pedagogical Efficiency and Effectiveness using an Automated Testing System UBCO Research Week Presentation 2011 University of British Columbia Page Copyright© 2011 6

Selecting a System Selecting an appropriate system depends on one key decision: Who pays? Selecting a System Selecting an appropriate system depends on one key decision: Who pays? 1) Student pays uanother cost in addition to textbooks paid by students ustudents will be reluctant to pay especially if not used extensively ucommercial systems are great from an instructor point of view 2) Institution pays u. Highly expensive for the institution so likely limited to open source systems. Both of which have their own restrictions. u. An in-house system is the only other choice. Ramon Lawrence, Alyosha Pushak, Teresa Wrzesniewski Improving Pedagogical Efficiency and Effectiveness using an Automated Testing System UBCO Research Week Presentation 2011 University of British Columbia Page Copyright© 2011 7

Auto. Ed System Overview The Auto. Ed system was designed and built by Alyosha Auto. Ed System Overview The Auto. Ed system was designed and built by Alyosha Pushak (undergrad honours) and Dr. Lawrence in computer science. u. It is written completely in Java and uses open source systems for the database and web server. u. The database stores all user question and answers. u. Users interact with the system using a web browser. Students can: u. Practice questions and do assignments as given by the instructor. Students receive immediate feedback. Instructors can: u. Create practice questions and assignments. u. Examine student assignment progress and marks at any time. Ramon Lawrence, Alyosha Pushak, Teresa Wrzesniewski Improving Pedagogical Efficiency and Effectiveness using an Automated Testing System UBCO Research Week Presentation 2011 University of British Columbia Page Copyright© 2011 8

Auto. Ed – Student Test Ramon Lawrence, Alyosha Pushak, Teresa Wrzesniewski Improving Pedagogical Efficiency Auto. Ed – Student Test Ramon Lawrence, Alyosha Pushak, Teresa Wrzesniewski Improving Pedagogical Efficiency and Effectiveness using an Automated Testing System UBCO Research Week Presentation 2011 University of British Columbia Page Copyright© 2011 9

Auto. Ed – Question Authoring Ramon Lawrence, Alyosha Pushak, Teresa Wrzesniewski Improving Pedagogical Efficiency Auto. Ed – Question Authoring Ramon Lawrence, Alyosha Pushak, Teresa Wrzesniewski Improving Pedagogical Efficiency and Effectiveness using an Automated Testing System UBCO Research Week Presentation 2011 University of British Columbia Page Copyright© 2011 10

Auto. Ed – Student Completion Rate Ramon Lawrence, Alyosha Pushak, Teresa Wrzesniewski Improving Pedagogical Auto. Ed – Student Completion Rate Ramon Lawrence, Alyosha Pushak, Teresa Wrzesniewski Improving Pedagogical Efficiency and Effectiveness using an Automated Testing System UBCO Research Week Presentation 2011 University of British Columbia Page Copyright© 2011 11

Physics Evaluation The Auto. Ed system was evaluated in Phys 112 in Fall 2010: Physics Evaluation The Auto. Ed system was evaluated in Phys 112 in Fall 2010: u 2 paper assignments, 2 electronic assignments, 1 choice u. Electronic assignments were converted from paper assignments by adding randomization. Surveys conducted at the first and last classes. u. First survey: 105 out of 111 surveys completed (95. 5% return) u. Second survey: 76 out of 89 surveys completed (85. 4% return) Ramon Lawrence, Alyosha Pushak, Teresa Wrzesniewski Improving Pedagogical Efficiency and Effectiveness using an Automated Testing System UBCO Research Week Presentation 2011 University of British Columbia Page Copyright© 2011 12

Physics Evaluation Results u. Format choice: 90% of the students used Auto. Ed for Physics Evaluation Results u. Format choice: 90% of the students used Auto. Ed for all three assignments and 92% used it for the last assignment. u. Paper or electronic preference: 70% chose electronic while an additional 15% chose both as equal. u. Experience: Over 72% of the students chose satisfactory or very satisfactory when asked about their experience using the system while only 3% rated it as very unsatisfactory. u. Help learning: Almost half of the students (45%) reported that Auto. Ed helped their learning. u. Completion: Students reported that Auto. Ed helped them complete their assignments (81%) and hand their assignments in on time (84%). u. Other uses: Over 92% of the students thought Auto. Ed should be used again in Physics 112, and 87% felt that it could be used in other classes (chemistry, biology, mathematics, English). Ramon Lawrence, Alyosha Pushak, Teresa Wrzesniewski Improving Pedagogical Efficiency and Effectiveness using an Automated Testing System UBCO Research Week Presentation 2011 University of British Columbia Page Copyright© 2011 13

Insights and Improvements 1) Watch technical issues with the system and questions. u. Issues Insights and Improvements 1) Watch technical issues with the system and questions. u. Issues with incorrect marking and significant figures. u. Question preparation takes time and all answers must be checked before releasing the assignment. 2) Users will use the Internet to find answers. u 5% of users wanted the system to block Internet searches. u. Copy-and-paste electronic question versus typing it in! u. Effectiveness as an evaluation tool as more answers are online? 3) Students really do wait until the last minute. u. By removing the overhead of assignment generation and collection/marking, more frequent, smaller assignments is better. Ramon Lawrence, Alyosha Pushak, Teresa Wrzesniewski Improving Pedagogical Efficiency and Effectiveness using an Automated Testing System UBCO Research Week Presentation 2011 University of British Columbia Page Copyright© 2011 14

PHYS 122: Paper vs. Electronic Ramon Lawrence, Alyosha Pushak, Teresa Wrzesniewski Improving Pedagogical Efficiency PHYS 122: Paper vs. Electronic Ramon Lawrence, Alyosha Pushak, Teresa Wrzesniewski Improving Pedagogical Efficiency and Effectiveness using an Automated Testing System UBCO Research Week Presentation 2011 University of British Columbia Page Copyright© 2011 15

PHYS 122: Question Writing Q 3: Using Kirchhoff’s rules for the circuit below find: PHYS 122: Question Writing Q 3: Using Kirchhoff’s rules for the circuit below find: a)The current in each resistor. [8 marks] b) The potential difference between points c and f [2 marks] Randomize R 3 from 4. 00 kΩ to 10. 0 kΩ, increments 1. 0 kΩ. Ramon Lawrence, Alyosha Pushak, Teresa Wrzesniewski Improving Pedagogical Efficiency and Effectiveness using an Automated Testing System UBCO Research Week Presentation 2011 University of British Columbia Page Copyright© 2011 16

PHYS 122: Question Writing (2) Q 2. A conductor suspended by two flexible wires PHYS 122: Question Writing (2) Q 2. A conductor suspended by two flexible wires (as shown below) has a mass per unit length of 0. 040 kg/m. What current must exist in the conductor for the tension in the supporting wires to be zero when the magnetic field, B = 3. 60 T points into the page? Give your answer with 3 sig. figs. [5 marks] Randomize B from 2. 60 T to 4. 60 T, intervals 0. 10 T. Ramon Lawrence, Alyosha Pushak, Teresa Wrzesniewski Improving Pedagogical Efficiency and Effectiveness using an Automated Testing System UBCO Research Week Presentation 2011 University of British Columbia Page Copyright© 2011 17

Future Work and Conclusions It is possible to build a customized question answer system Future Work and Conclusions It is possible to build a customized question answer system with minimal cost. Although the upfront costs and time is higher, the payback in terms of student costs and instructor flexibility is significant. We are currently evaluating the system in several other courses and will release it as open source to the community. The system will be maintained and extended for many years. Limited time offer: u. We will create the questions for you over the summer for any courses that you may be interested in using the system for! Ramon Lawrence, Alyosha Pushak, Teresa Wrzesniewski Improving Pedagogical Efficiency and Effectiveness using an Automated Testing System UBCO Research Week Presentation 2011 University of British Columbia Page Copyright© 2011 18

Auto. Ed – Demonstation Let's have a quick tour of the Auto. Ed system. Auto. Ed – Demonstation Let's have a quick tour of the Auto. Ed system. Ramon Lawrence, Alyosha Pushak, Teresa Wrzesniewski Improving Pedagogical Efficiency and Effectiveness using an Automated Testing System UBCO Research Week Presentation 2011 University of British Columbia Page Copyright© 2011 19