- Количество слайдов: 90
Part II: The 3 T’s of Online Assessment: Tools, Techniques, and (Saving) Time Curtis J. Bonk, Professor, Indiana University President, Course. Share http: //php. indiana. edu/~cjbonk; [email protected] edu Vanessa Dennen, Assistant Professor Florida State University [email protected] edu
Do you have a strategic plan for evaluation and assessment?
Bonk et al. ’s (in press) Online Learning Assessment and Evaluation Model Bonk, C. J. , Wisher, R. A. & Champagne, M. V. (in press). Toward a comprehensive model of e-learning evaluation: The Components.
What to Evaluate? 1. 2. 3. 4. 5. 6. 7. 8. Learner—attitudes, learning, jobs. Instructor—popularity, course enrollments. Training—internal and external. Task--relevance, interactivity, collaborative. Tool--usable, learner-centered, friendly, supportive. Course—interactivity, completion rates. Program—growth, long-range plans. Organization or University—cost-benefit, policies, vision.
1. n n n n Measures of Student Success (Focus groups, interviews, observations, surveys, exams, records) Increased Comprehension & Achievement High Student Attitudes High Retention, Completion Rates in Program Jobs Obtained, Internships Enrollment Trends for Next Semester Grades, Achievement, Certifications Computer Log Activity; e. g. , Number of Posts, Participation, Messages/day, Time in System
1. Student High-End Success n n n Message complexity, depth, interactivity, questioning Collaboration skills Problem finding/solving and critical thinking Challenging and debating others Case-based reasoning, critical thinking measures Portfolios, performances, PBL activities
Assessments Possible n n n n n Quizzes and Tests Peer Feedback, Mentoring, Responsiveness Tasks Attempted or Completed, Usage, etc. Discussion/Forum Participation Writing, Blogs, Weekly Reflections Cases and Problems Group Work Web Resource Explorations & Evaluations Performances, Portfolios, etc.
Issues to Consider… 1. 2. 3. 4. 5. 6. 7. Bonus pts for participation? Pts for peer evaluation of work? Assess improvement? Are tests timed? Allow retakes if lose connection? How many retakes? Give unlimited time to complete? Cheating? Is it really that student? How measure competency and demonstrate learning online?
2. Instructor Success n n High student evals, more signing up, student recommendations High student completion rates Utilize Web to share teaching Course recognized with awards
3. Training: Outside Support n n n Training (Faculty. Training. net) Courses & Certificates (JIU, e-education) Reports, Newsletters, & Pubs Aggregators of Info (Course. Share, Merlot) Global Forums (Faculty. Online. com; GEN) Resources, Guides/Tips, Link Collections, Online Journals, Library Resources
3. Training Inside Support… n n n Instructional Consulting Mentoring (strategic planning $) Small Pots of Funding Laptops Summer and Year Round Workshops Colloquiums, Tech Showcases, Guest Speakers, Awards, Recognitions n Newsletters, guides, active learning grants, annual reports, faculty development, brown bags
5 -ULO 3 US RIDIC Technology Use Model of 4. Tasks (RIDIC): n n n Relevance Individualization Depth of Discussion Interactivity Collaboration-Control-Choice. Constructivistic-Community
RIDIC 5 -ULO 3 US Model of Technology Use 5. Tech Tools (ULOUS): n n n Utility/Usable Learner-Centeredness Opportunities with Outsiders Online Ultra Friendly Supportive
6. Course Success n n n Few technological glitches/bugs Adequate online support Increasing enrollment trends Course quality (interactivity rating) Monies paid Accepted by other programs
7. Online Program or Course Budget (i. e. , how pay, how large is course, tech fees charged, # of courses, tuition rate, etc. ) n Indirect Costs: learner disk space, phone, accreditation, integration with existing technology, library resources, on site orientation & tech training, faculty training, office space n Direct Costs: courseware, instructor, help desk, books, seat time, bandwidth and data communications, server back-up, course developers, postage
7. Program: Online Content Considerations n n n Self-Paced or Live mentors? Interactive or content dumping? Individual or Collaborative? Lecture or problem-based learning? Factual or performance assessment?
8. Institutional Success n E-Enrollments from n n n new students, alumni, existing students Additional grants, new State monies Press, publication, partners, attention Cost-Benefit model, ROI Faculty attitudes, training, support Acceptable policies, ADL compliant
Let’s Focus at the Individual Level What Online Testing Tools Do You Use?
What Can Online Tests Do? n n n Assess student progress Allow for self-assessment Provide standards for success Timed testing and retesting Opportunity for instructor commenting
Test Selection Criteria (Hezel, 1999) n n n n Easy to Configure Items and Test Handle Symbols Scheduling of Feedback (immediate? ) Provides Clear Input of Dates for Exam Easy to Pick Items for Randomizing Randomize Answers Within a Question Weighting of Answer Options
More Test Selection Criteria n n n Recording of Multiple Submissions Timed Tests Comprehensive Statistics Summarize in Portfolio and/or Gradebook Confirmation of Test Submission
More Test Selection Criteria (Perry & Colon, 2001) n n n Supports multiple items types—multiple choice, true-false, essay, keyword Can easily modify or delete items Incorporate graphic or audio elements? Control over number of times students can submit an activity or test Provides feedback for each response
More Test Selection Criteria (Perry & Colon, 2001) n n Flexible scoring—score first, last, or average submission Flexible reporting—by individual or by item and cross tabulations. Outputs data for further analysis Provides item analysis statistics (e. g. , Test Item Frequency Distributions).
How Use Technology to Assess?
What Assess Online? 1. 2. 3. 4. Traditional Online Exams Self-Test Exams Learner-Content Interactions Guided Explorations: Virtual Tours and Timelines 5. Cases and Vignettes 6. Blogs 7. Chats with Visual Representations 8. Soft Skill Simulations 9. Virtual Reality and Role Play Simulations 10. Cyber Fashion Shows & Music Performance
1. Traditional Online Testing
Using Web. CT Quizzes in a High. Demand Environment (Brothen & Wambach, Technology Source, May/June 2003) “Several reviews and metaanalyses…have found superior student learning in PSI compared to traditional lecture/discussion methods. ” Here, students read a textbook and when they are ready, they take chapter quizzes; after they master one chapter, they move on to the next. ”
2. Online Self-Testing
3. Learner-Content Interactions (Option 6 Designers)
4. Guided Explorations: Virtual Tours and Timelines
4. Guided Explorations: Interactive Adventure Content (Andrew Revkin, New York Times, May 25, 2003)
5. Case-Based Learning: My Patient. com and Sim. Teacher
6. Blogs (diaries, writing)
7. Visual with Chat: Learningbydoing. net Participants: a facilitator of online therapy, students at all levels, a doctoral candidate in DE, administrators, teachers, lecturers, researchers, a physicists, a professor of Psychology, a professor of Mathematics, a consultant in training, an HR trainer, and a psychotherapist. We were located in Herzelia, a beach town north of Tel Aviv, Stanford California, Baltimore, Montreal, and Ismir, Turkey.
Games and Simulations “There’s something new on the horizon, though: computerbased soft skills simulations, which let learners practice skills such as negotiation and team building. ” Clark Aldrich, The State of Simulations, Sept. 2001, Online Learning
8. Simulations: Virtual Univ Adminstrator & Virtual Leader
9. Virtual Worlds/Virtual Reality l. Avatars--representations of people l. Objects--representations of objects l. Maps--the landscape which can be explored l. Bots--artificial intelligence
10. Online Performances (e. g. , Cyber Fashion Shows)
Which might you use? How would you use?
Online Survey Tools for Assessment
Sample Survey Tools n n n Zoomerang (http: //www. zoomerang. com) Survey. Monkey (http: //surveymonkey. com) Question. Mark (http: //www. questionmark. com/home. html) n Survey Solutions from Perseus (http: //www. perseusdevelopment. com/fromsurv. htm) n Infopoll (http: //www. infopoll. com)
Sample Survey Tools n Active Feedback n (http: //www. activefeedback. com/af) n Survey. Key (http: //www. surveykey. com) EZSurvey from Raosoft (http: //www. raosoft. com/) n Survey. Share n n n (http: //Survey. Share. com; from Courseshare. com)
Survey Student Opinions (e. g. , Info. Poll, Survey. Solutions, Zoomerang, Survey. Share. com)
Online Survey in Blackboard
Web-Based Survey Advantages n n n Faster collection of data Standardized collection format Computer graphics may reduce fatigue Computer controlled branching and skip sections Easy to answer clicking Wider distribution of respondents
Why Conduct Online Surveys n n Formative assessment of class Increase student voice and ownership in class Involve students from other locations Quickly gather answers to questions
Web-Based Survey Problems: Why Lower Response Rates? n n n Low response rate Lack of time Unclear instructions Too lengthy Too many steps Can’t find URL
Survey Tool Features n n n n Support different types of items (Likert, multiple choice, forced ranking, paired comparisons, etc. ) Maintain email lists and email invitations Conduct polls Adaptive branching and cross tabulations Modifiable templates & library of past surveys Publish reports Different types of accounts—hosted, corporate, professional, etc.
Web-Based Survey Solutions: Some Tips… n n n n Send second request Make URL link prominent Offer incentives near top of request Shorten survey, make attractive, easy to read Credible sponsorship—e. g. , university Disclose purpose, use, and privacy E-mail cover letters Prenotify of intent to survey
Increasing Cheating Online ($7 -$30/page, http: //www. syllabus. com/ January, 2002, Phillip Long, Plagiarism: IT-Enabled Tools for Deceit? ) n http: //www. academictermpapers. com/ http: //www. termpapers-on-file. com/ http: //www. nocheaters. com/ n http: //www. cheathouse. com n n http: //www. realpapers. com/ http: //www. pinkmonkey. com/ (“you’ll never buy Cliffnotes again”)
Sample Paper Assistance (i. e. , Cheating) Sites
Sample Paper Assistance (i. e. , Cheating) Sites
n “In recent years there has been a sharp increase in students cutting and pasting material into papers from Web sites without attribution, or purchasing term papers from online term-paper mills. ”
Cell Phones to Cheat on Accounting Exam (1/30/03)
Plagiarizing Plagiarism (June 8, 2003; Dartmouth Review) n n “Michael Heberling—an author at the Chronicle and president of Baker College for Graduate Studies—read an article written in Syllabus magazine that looked remarkably like his own piece. The subject of the plagiarist’s piece: plagiarism. ” the offending author, a graduate student, apologized for any passages accidentally cut-and-pasted into her own article. Her excuse: ‘deadline pressure. ’
When Values Collide, NY Times, p. 4, July 15, 2003 n n “Some of the tutors make questionable decisions. Writing papers for the players, if that is true, is very questionable. ” “I guess it is difficult, if not impossible, to stop the faculty members that are devout fans and give any grade they want. ”
Ok, Million Dollar Question: How can you reduce cheating online?
1. Reducing Cheating Online http: //www. syllabus. com/ January, 2002, Phillip Long, Plagiarism: IT-Enabled Tools for Deceit? ) http: //www. syllabus. com/article. asp? id=5916 n n n Ask yourself, why are they cheating? Do they value the assignment? Are tasks relevant and challenging? What happens to the task after submitted—reused, woven in, posted? Due at end of term? Real audience? Look at pedagogy b 4 calling plagiarism police!
Ways to Reduce Cheating Online (How to Proctor from a distance, Dan Carnevale, Chronicle of Higher Ed, Dec. , 12, 1999) 2. Use proctored exams 3. Require cameras during discussions and tests 4. Get to know your students better through discussions and chats 5. Give random quizzes using chat tool 6. Write a long essay at start of semester to serve as an index of writing style 7. When in doubt, place document into a search engine (Eve 2 or Turnitin. com)
Resources for Reducing Cheating Online ($7 -$30/page, http: //www. syllabus. com/ January, 2002, Phillip Long, Plagiarism: IT-Enabled Tools for Deceit? ) n n n http: //www. turnitin. com/ (software, $100, free 30 day demo/trial) http: //www. copycatch. freeserve. co. uk/ (free in UK) http: //www. canexus. com/ (Eve 2 software; essay verification engine, $19. 95) http: //www. plagiarism. org/ (resource) http: //www. academicintegrity. org/ (assoc. ) http: //sja. ucdavis. edu/avoid. htm (guide)
Turnitin Testimonials "Many of my students believe that if they do not submit their essays, I will not discover their plagiarism. I will often type a paragraph or two of their work in myself if I suspect plagiarism. Every time, there was a "hit. " Many students were successful plagiarists in high school. A service like this is needed to teach them that such practices are no longer acceptable and certainly not ethical!”
Check Papers with Eve 2
Do Dennen & Bonk Plagiarize?
More Ways to Reduce Cheating Online (How to Proctor from a distance, Dan Carnevale, Chronicle of Higher Ed, Dec. , 12, 1999) 8. Vary items in exam 9. Have timed exams 10. Make course too hard to cheat 11. Random selection of items for item pool 12. Use mastery learning for some tasks 13. Assign collaborative tasks
Still More Ways to Reduce Cheating Online 14. Use test passwords 15. Rely on computer IP# screening 16. Set expectations (e. g. , scholarly integrity, syllabus procedures) 17. Emphasize consequences (e. g. , stories of past offenders) 18. Have students make a vow of no cheating (e. g. , University of Virginia)
Academic Honor Codes!
More Ideas: (Promoting Academic Integrity) (June 16, 2003, TLTL, IU Anti-Plagiarism Strats) 19. Help students recognize citation probs Ø Examples, cases, Web sites, 1 credit course 20. Help Faculty and AIs recognize problems Ø Examples, cases, practice grading , discuss 21. Provide practice in citing properly Ø Interactive workshops, discussions 22. Design assignments to minimize dishonesty and poor citation Ø Provide frameworks, well designed prompt 23. Hold students accountable 24. Plan for special situations 25. Use available support
Plagiarism Test (Ted Frick, Indiana University)
Still More Ways to Reduce Cheating Online (Mc. Murty (2001) E-Cheating: Combating a 21 st Century Challenge, Tech Horizons in Ed, 29(4), 36 -41. 26. Require prewriting steps such as an outline and rough draft 27. Require electronic paper submission 28. Design writing assignments with high specificity, not open-ended (harder to find a match in a pool) 29. Use Google to search for phrases that do not appear to be from a particular student 30. Peruse paper help (i. e. , cheat)
Tips on Authentification 31. Check e-mail access against list 32. Use password access 33. Provide keycode, PIN, or ID # 34. Futuristic Other: Palm Print, fingerprint, voice recognition, iris scanning, facial scanning, handwriting recognition, picture ID
The Ultimate in Plagiarism? ? ?
Fake Degrees (Oakley, 2003)
Fake Degrees (Burks Oakley, 2003) n http: //www. fakedegrees. com/ http: //www. myodegrees. com/ n http: //www. boxfreeconcepts. com/magicmill/ n n n http: //www. blackmarket-press. com/ http: //www. bogusphd. com/ http: //www. coastalretreat. com/degrees/funny. html www. counterfeitlibrary. com/cl/qual. asp
Sample Fake Degree Sites
Imagine the Assessments! American Inter. Continental University (AIU)
Final advice…whatever you do…