Development and Deployment of a Web-Based Course Evaluation

Development and Deployment of a Web-Based Course Evaluation System Jesse Heines and David Martin Dept. of Computer Science Univ. of Massachusetts Lowell Miami, Florida, May 26, 2005

The All-Important Subtitle Trying to satisfy. . . the Students the Administration the Faculty and the Union presented in a slightly different order from that listed in the paper 2

The All-Important Subtitle 3

The All-Important Subtitle 4

Paper-Based System Reality Distributed and filled out in classrooms 5 Thus, virtually all students present that day fill them out However, absentees never fill them out

Paper-Based System Reality Distributed and filled out in classrooms Collected but not really analyzed At best, Chairs “look them over” to get a “general feel” for students’ reactions Professors simply don’t bother with them 6 lack of interest and/or perceived importance simple inconvenience of having to go get them and wade through the raw forms

Paper-Based System Reality Distributed and filled out in classrooms Collected but not really analyzed Lose valuable free-form student input because those comments are often. . . 7 downright illegible so poorly written that it’s simply too difficult to try to make sense of them

Paper-Based System Reality 8 Distributed and filled out in classrooms Collected but not really analyzed Lose valuable free-form student input However, these comments have the greatest potential to provide real insight into the classroom experience

Paper-Based System Reality 9 Distributed and filled out in classrooms Collected but not really analyzed Lose valuable free-form student input However, these comments have the greatest potential to provide real insight Bottom Line #1: The paper-based system pays little more than lip service to the cry for accountability in college teaching

Paper-Based System Reality 10 Bottom Line #2: We’re all already being evaluated online whether we like it or not. . .

11

Web-Based System Goals Collect data in electronic format 12 Easier and faster to tabulate More accurate analysis Possibility of generating summary reports

Web-Based System Goals Collect data in electronic format 13 Easier and faster to tabulate More accurate analysis Possibility of generating summary reports Retrieve legible free-form responses Allow all students to complete evaluations anytime, anywhere, at their leisure, and even if they miss the class in which the evaluations are distributed

What We Thought If we build it, they will come. . . 14 . . . but we were very wrong!

Student Issues 1. Maintain anonymity 2. Ease of use 3. Speed of use 15

Student Issues 3. Maintain anonymity 2. Ease of use 1. Speed of use We guessed wrong on the relative priorities of these issues. 16

Student Issues Our main concern: 17 Prevent students from “stuffing the ballot box” One Student = One Survey Submission

Student Issues Our main concern: Major concern that appeared after the system was deployed: 18 Prevent students from “stuffing the ballot box” One Student = One Survey Submission Simply getting students to participate There appeared to be a great deal of apathy, particularly in non-technical courses

Student Login Evolution Fall 2003

Student Login Evolution

Administration Issues System quality and integrity “Buy in” from the deans But the real issue was. . . Dealing with the faculty union 23

Faculty Issue #1 Control of which courses are evaluated Contract wording: “The evaluation will be conducted in a single section of one course per semester. . At the faculty member’s option, student evaluations may be conducted in additional sections or courses. ” 24

Union Issue #1 In 2004, all surveys were “turned on” by default, that is, they were all accessible to students on the Web In 2005, the default is inaccessible 25 This was a breach of the contract clause stating that “evaluation will be conducted in a single section of one course” Use of the system thus became voluntary As of May 20, 2005 (end of final exams), 95 professors (25% of the faculty) in 40 departments had made 244 course surveys accessible to students

Faculty Menu

Faculty Issue #2 Control of what questions are asked Contract wording: “Individual faculty members in conjunction with the Chairs/Heads and/or the personnel committees of academic departments will develop evaluation instruments which satisfy standards of reliability and validity. ” 27

Union Issue #2 In 2004, deans could set questions to be asked on all surveys for their college 28 This was a breach of the contract clause stating that faculty would develop questions “in conjunction with the Chairs/Heads and/or department personnel committees” In 2005, all college-level questions are now at the department level so that only Chairs can specify required questions Deans then had essentially no access to

Faculty Menu

Faculty Question Editor

Faculty Add Question Form

Survey as Seen by Students

Faculty Issue #3 Control of who sees the results Contract wording: 34 “Student evaluations shall remain at the department level. At the faculty member’s option, the faculty member may submit student evaluations or a summary of their results for consideration by various promotion and tenure review committees. The faculty member shall become the sole custodian of these student evaluations at the end of every three academic years and shall have the exclusive authority and responsibility to

Results as Seen by Faculty

Union Issue #3 Data was collected without faculty consent All survey response data for the Fall 2004 semester were deleted on February 15, 2005, unless the faculty member explicitly asked that it be kept 36 This was a breach of the contract clause stating that “student evaluations shall remain at the department level” What’s going to happen with this semester’s data has not yet been determined

Faculty Menu

Lessons Learned/Confirmed No matter what you do, there will be those who object 38 You must remain open-minded and flexible Practice good software engineering so that the software can be easily modified It’s really worth it to work with the many power factions to garner support Every system needs a “champion” Be prepared to spend a huge amount of time on system support

Support, Support 39

Thank You Jesse M. Heines, Ed. D. David M. Martin, Ph. D. Dept. of Computer Science Univ. of Massachusetts Lowell 40 {heines, dm}@cs. uml. edu http: //www. cs. uml. edu/{~heines, ~dm}