Benchmarking in e-learning an overview Professor Paul Bacsich

Benchmarking in e-learning: an overview Professor Paul Bacsich Matic Media Ltd and Middlesex University, UK ACODE 39: Quality in Teaching and Learning, 17 -18 November 2005, Melbourne 1

The Menu l UK e-learning – why listen? l Benchmarking overview l Pick & Mix system l MBS case study l Conclusions ACODE 39: Quality in Teaching and Learning, 17 -18 November 2005, Melbourne 2

Myself l Consultant to several UK agencies & universities l Adjunct professor at Middlesex University – Global Campus and School of Computing Science l Open University for 25 years l One of the former Directors of UK e. Universities, which aimed to be a global provider of e-learning l Current work includes developing a global benchmarking methodology for e-learning – Already piloted at Manchester Business School – Presented to EU conference in Brussels, and in Colombia, ALT-C, HEA and Sydney Uni ACODE 39: Quality in Teaching and Learning, 17 -18 November 2005, Melbourne 3

Not in my talk! l Costs of e-learning (Activity Based Costing) l Competitor analysis of e-learning providers l What went wrong with UK e. Universities? – Two volumes of reports (35 chapters, over 2500 pages) and research overview available – More soon ACODE 39: Quality in Teaching and Learning, 17 -18 November 2005, Melbourne 4

Why listen to UK? l UK has many years experience of quality management in universities, via various organisations – Latest is Quality Assurance Agency for Higher Ed l This has guidelines for quality in e-learning l UK has substantial experience in distance learning and e-learning including global delivery l Benchmarking is part of Higher Ed policy for e-learning (HE Academy, under way now) l UK-Australian collaboration/co-funding on a number of issues including e-Framework ACODE 39: Quality in Teaching and Learning, 17 -18 November 2005, Melbourne 5

The main UK agencies l QAA – for quality l JISC – for support of ICT in universities – UKERNA to run the JANET high-speed network across all UK l Higher Education Academy (HEA) – for pedagogy l Some smaller agencies: – Leadership Foundation for HE (LFHE) – Observatory for Borderless HE (OBHE) ACODE 39: Quality in Teaching and Learning, 17 -18 November 2005, Melbourne 6

Quality Assurance Agency UK l Covers all four UK home nations l “Code of practice for the assurance of academic quality and standards in higher education” l See www. qaa. ac. uk/academicinfrastructure/ code. Of. Practice/ l Not much on pedagogy – this is left to the discretion of the academic l Only 1 private uni in UK, over 100 public ones ACODE 39: Quality in Teaching and Learning, 17 -18 November 2005, Melbourne 7

QAA in e-learning l “Collaborative provision and flexible and distributed learning (including e-learning)” l September 2004 l BUT – Some feel it says too little, others do not want to be restricted – It was too late – 4 years? – No international comparisons (whereas research has) ACODE 39: Quality in Teaching and Learning, 17 -18 November 2005, Melbourne 8

Pedagogy l Higher Education Academy l “works with universities and colleges, discipline groups, individual staff and organisations to help them deliver the best possible learning experience for all students” l Runs Subject Centres for each subject l Advising on e-learning since early 2005 – Slowish progress ACODE 39: Quality in Teaching and Learning, 17 -18 November 2005, Melbourne 9

JISC and JANET l Joint Information Systems Committee – ICT for universities and colleges (not schools) – England, Scotland, Wales, N Ireland l JANET is the UK National Academic and Research Network (JANET) l JISC funds JANET via UKERNA company ACODE 39: Quality in Teaching and Learning, 17 -18 November 2005, Melbourne 10

In UK, universities compete - and now in e-learning l Universities want to judge how well they are doing in e-learning l Funding agencies and public want to know l But universities don’t want to tell if they are doing badly! l And universities (like people) are not good at judging themselves ACODE 39: Quality in Teaching and Learning, 17 -18 November 2005, Melbourne 11

Benchmarking l Like Activity Based Costing (ABC), it has been around for many years l Unlike ABC, but like BPR, quality, excellence, etc; no one is now sure what it means… ACODE 39: Quality in Teaching and Learning, 17 -18 November 2005, Melbourne 12

Back to Basics (Xerox) a process of self-evaluation and self-improvement through the systematic and collaborative comparison of practice [process] and performance [metrics, KPIs] with competitors [or comparators] in order to identify own strengths and weaknesses, and learn how to adapt and improve as conditions change. ACODE 39: Quality in Teaching and Learning, 17 -18 November 2005, Melbourne 13

Benchmarking (in Universities) l There are several reports that will tell you how to do benchmarking in general – Higher Education Academy (UK) – Learning and Skills Development Agency (UK) – Department of Education Training and Youth Affairs (Australia)/Sydney Uni ACODE 39: Quality in Teaching and Learning, 17 -18 November 2005, Melbourne 14

Benchmarking in e-Learning There are few published reports re approaches l My surveys and proposals – http: //www. alt. ac. uk/altc 2005/timetable/files/527/Benchm ark_overview. doc l E-Learning Maturity Model (NZ) – Marshall l NUTN/Hezel emerging work (Jan 2006? ) l Work by OECD and OBHE l National Learning Network (UK) – colleges l So far unpublished work (Sydney/OU and ACODE) ACODE 39: Quality in Teaching and Learning, 17 -18 November 2005, Melbourne 15

Best Practice in e-Learning l There a few reports (US): – APQC/SHEEO Study 1998 (US) – IHEP “Quality on the Line” 2000 (US) l And several projects (EU): – BENVIC – SEEQUEL – Swiss Virtual Campus @ Lugano: MINE (adapting the IHEP work for EU) – E-xcellence (EADTU and others) ACODE 39: Quality in Teaching and Learning, 17 -18 November 2005, Melbourne 16

Benchmarking e-learning A global “synthesis” incorporating what work has been done elsewhere ACODE 39: Quality in Teaching and Learning, 17 -18 November 2005, Melbourne 17

Focus of my work l Focussed purely on e-learning l But not to any particular style (e. g. DL) l Oriented to institutions past the “a few projects” stage l Suitable for desk research as well as “invasive” studies l Suitable for single- and multi-institution studies l Started work in Jan 2005, already piloted at Manchester Business School against 12 competitors world-wide ACODE 39: Quality in Teaching and Learning, 17 -18 November 2005, Melbourne 18

Processes or Outputs? l Outputs: measure first (can be done by desk research) l Processes: later (best done in clubs or invasive studies) l Inputs: not of so much interest to students; but of course of great interest to funders ACODE 39: Quality in Teaching and Learning, 17 -18 November 2005, Melbourne 19

Metrics or Bureaucratic l Use a 6 -point scale – 5 from Likert plus 1 more for “excellence” l Backed up by metrics where possible l Also contextualised by narrative l Some issues of judging “best practice”; judging “better practice” is easier – e. g. VLE convergence l Some criteria are rather “criteria bundles” ACODE 39: Quality in Teaching and Learning, 17 -18 November 2005, Melbourne 20

Other Decisions l Explicit (otherwise you are not trying) l Independent or collaborative l Internal or external l Horizontal – focus on processes across whole institution – but can look at individual projects, missions and departments to get “range of scores” ACODE 39: Quality in Teaching and Learning, 17 -18 November 2005, Melbourne 21

How Many Benchmarks? l It is like ABC: how many activities? l Answer: Not 5, not 500 l Better answer: Well under 100 – Composite some criteria together – Remove any not specific to e-learning – Be careful about any which are not provably critical success factors – Institutions may wish to add specific ones to monitor their objectives and KPIs. ACODE 39: Quality in Teaching and Learning, 17 -18 November 2005, Melbourne 22

How Many do Others Have? l LSDA (UK) has 14 – but colleges l IHEP (US) has 24 – but old l APQC/SHEEO (US) had 14 – but older l EMM (NZ) has 43 – but some are being merged and some are outside core elearning area l OECD has many but several are “taxonomic” not critical success factors ACODE 39: Quality in Teaching and Learning, 17 -18 November 2005, Melbourne 23

Pick and Mix System l Based on survey of “best of breed” ideas l 6 -point scale (Likert + excellence) l Backed up by narrative and metrics l 18 core criteria (e-learning specific) l Can easily add more in same vein for local needs l Output and student-oriented aspects covered l Focussed on critical success factors l Methodology-agnostic l Requires no long training course to understand – But must know and be undogmatic about e-learning ACODE 39: Quality in Teaching and Learning, 17 -18 November 2005, Melbourne 24

“Adoption phase” (Rogers) 1. Innovators only 2. Early adopters taking it up 3. Early adopters adopted; early majority taking it up 4. Early majority adopted; late majority taking it up 5. All taken up except laggards, who are now taking it up (or retiring or leaving) 6. First wave embedded, second wave under way (e. g. m-learning after e-learning) ACODE 39: Quality in Teaching and Learning, 17 -18 November 2005, Melbourne 25

“Training” 1. No systematic training for e-learning 2. Some systematic training, e. g. in some projects and departments 3. U-wide training programme but little monitoring of attendance or encouragement to go 4. U-wide training programme, monitored and incentivised 5. All staff trained in VLE use, training appropriate to job type – and retrained when needed 6. Staff increasingly keep themselves up to date in a “just in time, just for me” fashion except in situations of discontinuous change ACODE 39: Quality in Teaching and Learning, 17 -18 November 2005, Melbourne 26

“Accessibility” 1. e-learning material and services is not accessible 2. Much e-learning material and most services conform to minimum standards of accessibility 3. Almost all e-learning material and services conform to minimum standards of accessibility 4. All e-learning material and services conform to at least minimum standards of accessibility, much to higher standards 5. e-learning material and services are accessible, and key components validated by external agencies 6. Strong evidence of conformance with letter & spirit of accessibility in all countries where students study Too aspirational, too international, too regulated? ACODE 39: Quality in Teaching and Learning, 17 -18 November 2005, Melbourne 27

Case Study Jan-Apr 2005 Manchester Business School within Manchester U (done by Matic Media Ltd) ACODE 39: Quality in Teaching and Learning, 17 -18 November 2005, Melbourne 28

Methodology l Externally-focussed (internal going on now) l Looked at 12 “comparator” business schools (2 UK, 10 non-UK) – no time to discuss l Focus on speedy desk research (Web+DB) l Focus on criteria susceptible to that – plus “narratives of good practice” l Aim: to learn lessons for MBS ACODE 39: Quality in Teaching and Learning, 17 -18 November 2005, Melbourne 29

A few MBS conclusions l Numeric (not so interesting) – “taxonomic” l Tabular (see next slide) l Lots of case study narrative (but structured) l Top-level conclusions include: – Saturation wireless networks universal – e-Portfolios used in “sandstone-level” – Alumni get same IT systems as students ACODE 39: Quality in Teaching and Learning, 17 -18 November 2005, Melbourne 30

Criterion Level 1 Factor Adoption phase (Rogers) Leuven? VLE stage Tools use 2 3 Illinois Leuven Virginia 4 UBC Virginia Illinois Michigan 6 Babson? Babson UBC Babson IUPUI Leuven e-Learning Strategy Virginia Organisation Virginia Technical support to academics 5 Penn State Illinois Twente Babson UBC IUPUI Penn State Babson ACODE 39: Quality in Teaching and Learning, 17 -18 November 2005, Melbourne 31

Work in progress l Presentation at UK HEA Town Meeting l Discussions with EU projects on “quality” and “excellence” l Implications of report on UKe. U Committee for Academic Quality (in e-Learning) l Keynote at EFQEL conference l Workshop and Presentation at Online Educa Berlin (Nov/Dec 2005) l Detailed comparison of methodologies (NB costing) l See how this can be taken into account for UK HEA strategy for benchmarking e-learning – 12+60 HEIs to be involved in 2006 ACODE 39: Quality in Teaching and Learning, 17 -18 November 2005, Melbourne 32

Thank you for listening Any questions? Professor Paul Bacsich bacsich@matic-media. co. uk www. cs. mdx. ac. uk/staff/profiles/ p_bacsich. html ACODE 39: Quality in Teaching and Learning, 17 -18 November 2005, Melbourne 33