Quality Management in E-learning Towards an integrated approach

Quality Management in E-learning: Towards an integrated approach Dr Jill Fresen and Lesley Boyd Dept of Telematic Learning and Education Innovation, University of Pretoria Independent Quality Assurance Consultant

Problem “Quality assurance and online delivery are hot topics in universities, yet until recently discussions of each have had little to do with each other. ” Reid (2003) Academic Exchange Quarterly, 7(1), 17 -23.

Agenda Background of QA in e-learning Methodology of project at UP Lessons learnt Towards integration Where to next?

Agenda Background of QA in e-learning Methodology of project at UP Lessons learnt Towards integration Where to next?

Terminology Contact Education Distance Education E-learning face-to-face Other terms: Websupported learning paper-based Web-based learning Internet-based learning Online learning Technology-enhanced learning

E- Learning @ UP Web-support Multi-media (on CD-Rom) Video production Video conferencing Interactive Television Graphic services Photographic services

Role players Stakeholders management government funders community Clients lecturers students Practitioners project manager instructional designer education consultant programmer graphic artist library consultant

Who are ourclients? Lecturers • • instructional design educational support training technical support Students • end users • training • technical support

Case Studies University of Southern Queensland Distance Education Centre (DEC) first ISO 9001 accreditation University of Bangor in Wales - tools for judging pedagogical quality Frameworks, benchmarks and best practice indicators: critical success factors for e-learning

Agenda Background of QA in e-learning Methodology of project at UP Lessons learnt Building blocks for the future Where to next?

Methodology of the UP project Apply standard QA theory to e-learning applications, by implementing a Quality Management System Investigate what factors promote quality e-learning Implement systematic evaluation and customer feedback systems

Methodology of the UP project Apply standard QA theory to e-learning applications, by implementing a Quality Management System Investigate what factors promote quality e-learning Implement systematic evaluation and customer feedback systems

QMS Approach - 1 Pro-active evaluation and improvement rather than external accountability Ownership and commitment from all practitioners in the E-education Unit

QMS Approach – 2 formal online Documented, auditable, visible Minimise paperwork process based Understand the entire ID process from beginning to end and how awareness of this helps people to work together effectively. consensus driven Based on consensus ownership not prescription ISO 9000 cognisant Recognise useful principles behind ISO 9000

QMS – Timescales 2001 TLEI management request a formal quality approach to be implemented Nov 2001 – May 2002 Training Workshops for all management and staff May 2002 – Sept 2003 Production of procedure documentation & development of online QMS Sept 2003 – loading of procedure documents into online QMS Oct 2003 Launch of ‘live’ QMS 2003 – 2005 ‘Death valley’ of implementation 2006 Self evaluation of system – recognised as benchmark for services provided to clients

QMS – Objectives To provide a defined framework for all role players to work together consistently along the entire project timeline. To enable everyone, including new staff, to understand ‘the way things are done around here’. To identify together areas for improvement. To provide an integrated and simple method to access and use supporting documentation eg checklists, forms, templates. To ensure that the right tools are available to allow for comprehensive checks and to minimise errors. To try and catch any errors as soon as possible before it’s too late or too expensive to fix them. To evaluate completed projects and help to assess their impact on teaching and learning at UP. To learn lessons which may help to improve future projects. To share more with each other about ways of doing things. To demonstrate to any external stakeholders (e. g. auditors or UP management) that TLEI has a formal quality management system in place to control e-education projects.

QMS - Methodology QA training workshops Formalised intuitive documentation such as the Instructional Design Toolkit and the Project Timeline QMS Triangle

Ons slakkie

Quality Management System Measurements and Targets Feedback Loop ACT Strategy Policies Objectives Processes Procedures Standards, checklists, forms PLAN DO Detailed work instructions Records Boyd, 2001 – adapted from Waller, Allen & Burns, 1993 CONTROL

ISO 9000 model Value adding activities Information flow SABS (2000)

Project Timeline Based on traditional ADDIE model of instructional design Illustrates the major PROCESS

Project Timeline Analysis

Procedures Each step documented into a PROCEDURE using: • • task teams (jigsaw and pizza) a procedure template a fully worked example a set of self evaluation questions

Procedures Rapid prototyping and crossreferencing between task teams All procedures then uploaded into the live QMS with a document control master list Live system launched and left to sink or swim!

Methodology of the UP project Apply standard QA theory to e-learning applications, by implementing a Quality Management System Investigate what factors promote quality e-learning Implement systematic evaluation and customer feedback systems

Assumptions Technology infrastructure • reliability, accessibility, availability • accuracy, stability of student data Commitment, motivation of clients Computer literacy of clients Information literacy of clients Positive attitude of lecturers Consultation and support to lecturers Sound instructional design practice Sound teaching and learning practice Commitment to cont. improvement

Exogenous factors Quality of the institutional LMS Stability of national telecoms infrastructure Class size Workload of lecturers and students Recognition and incentives for lecturers

Critical Success Factors Six studies from literature Six categories: • • • Institutional Factors Technology Factors Lecturer Factors Student Factors Instructional Design Factors Pedagogical Factors

Methodology of the UP project Apply standard QA theory to e-learning applications, by implementing a Quality Management System Investigate what factors promote quality e-learning Implement systematic evaluation and customer feedback systems

Four levels Reaction Learning Behaviour Results Customer satisfaction Return on investment Kirkpatrick, D. 1998

Client satisfaction Student feedback survey – online at end of each semester Lecturer interviews – pilot group of 22 active users

Student feedback survey July 2003 baseline data Overall Experience survey: 4 650 respondents Response rate 27. 4 % • • • Technical adequacy Educational support Affective domain Online communication Perceived learning

Student frustration - 2003 Frustration index: 73% 17% 10% 83% of students experience moderate to high levels of frustration

Frustration index factors insufficient computers available insufficient printers available extent of technical difficulties insufficient support from student CD inadequate student training in Web. CT an impersonal learning experience slow response from classmates feelings of annoyance and/or stress

Student satisfaction - 2003 Satisfaction index: 44% 43% 13% 87% of students experience moderate to high levels of satisfaction

Satisfaction index factors feeling comfortable communicating via online tools feeling more freedom to express oneself than in a traditional classrooom learning from the contributions of other students promoting one’s ability to work as a team or group member increased ability to plan one’s own work enriching learning environment

Longitudinal measures Measures of student satisfaction, 2003 – 2005 2003 2004 2005 I have access to my own computer (either at home, at work or in the residence) 80. 5% 78. 6% 73. 1% I find it difficult to find a computer on campus when necessary 85. 1% 50. 3% 36. 5% I find it difficult to find a printer on campus when necessary 73. 3% 45. 3% 42. 2% My technical difficulties were solved within 24 hours 43. 7% 79. 3% 76. 1% I felt comfortable communicating via online tools 72. 9% 80. 0% 82. 8% Web-supported communication helped me to express myself more than I would have in a traditional classroom 79. 1% 57. 9% 60. 9% I found web-supported learning to be an enriching learning experience 81. 9% 77. 7% 78. 7% I experienced feelings of annoyance or stress during 80. 0% this web-supported learning experience 37. 9% 32. 5% I found the opportunities for ‘anywhere, anytime’ learning on the web to be convenient 87. 0% 88. 3% 73. 8%

Lecturer feedback survey Pilot interviews: 22 lecturers Benefits • useful for organisation and admin • communication and interaction • excellent support and service Problems • library / copyright issues • human element is difficult Lessons learnt • training for staff and students • e-learning adds value

Agenda Background of QA in e-learning Methodology of project at UP Lessons learnt Towards integration Where to next?

Lessons: Risks formal Resistance to generation of documentation versus perceived benefits online Time takenpressures for development of Work load Expectation that consultant should be online system process based best practice procedures providing Once created, it must be maintained and kept Resistance to ideasexpectations; process; associated Inappropriate to driven ofinternal functionality and consensusup to date forinappropriateeg that the with system will and commerce guarantee quality to HE? improvements external visibility ISO 9000 cognisant Inability to reach consensus amongst role players or to drive this forward Automatic dismissal and perceptions of Expectation that consultant is bureaucracy as soon as ISO 9000 term should be providing ‘answers’ mentioned! Tendency to overlook the value of the system – that it is only a document management tool

Lessons: Benefits Gartner hype cycle: Source: http: //news. bbc. co. uk/1/hi/technology/3577746. stm#graphic

Lessons: Benefits 1. Consensus that the QMS is on the slope of enlightenment 2. Self evaluation and improvement of each step in the process; consensus on ‘the way we do things around here’ 3. Rationalisation of pre-existing documents, checklists and forms 4. The QMS is a repository for the most up to date versions of all resources for planning, developing & evaluating projects 5. System informs new staff on UP procedures and enables them to feel self sufficient more quickly

Lessons learnt 1. Fundamental ID model (process) 2. Importance of Analysis and Evaluation 3. Timely training in the basics of QA theory and practice 4. Doubts about the need for a formalised QMS 5. Time required for reflection on practice 6. Roles and responsibilities of all role players 7. Manage lecturers’ unrealistic expectations 8. Understand that the system will go through a change management cycle

Agenda Background of QA in e-learning Methodology of project at UP Lessons learnt Towards integration Where to next?

Conceptual framework Applies process theory, critical success factors, and four levels of evaluation into an integrated framework

Conceptual framework inputs process Institutional factors Instructional Design Process Technology factors Lecturer factors ID factors Student factors Pedagogical factors outputs measures Products websupported courses & multimedia products feedback loop Lecturer satisfaction Student satisfaction

Agenda Background of QA in e-learning Methodology of project at UP Lessons learnt Towards integration Where to next?

Where to next? Use the QMS and the conceptual framework to produce measures about effectiveness, efficiency and contribution of e-learning to teaching and learning Use measures to prioritise improvements Implement summative evaluation as a joint exercise between all role players Integrate evaluation of e-learning with institutional programme review

Further reading Fresen (2005) Quality assurance practice in web-supported learning in higher education. Ph. D thesis. Available at http: //upetd. up. ac. za/UPe. TD. htm Fresen & Boyd (2005) Caught in the web of quality. International Journal of Educational Development, 25(3), 317 -331. Fresen (in press) A taxonomy of factors to promote quality websupported learning. International Journal on E -learning. To appear in 2006.

Where to next? Quality in e-learning means learning transformed and enhanced with the integration of technology where ‘e’ means: empowered enabled engaged enlightened emotional enthusiastic Dr Ulf-Daniel Ehlers: Presentation at E-learning in Post-Secondary Education Conference, Alberta, Canada June 2005

Sharing experiences Jill Fresen: jill. fresen@up. ac. za Lesley Boyd: lgboyd@eject. co. za Online QMS: http: //www. up. ac. za/telematic/quality. htm