Learning and Teaching Enhancement Unit Aligning the Ducks

Learning and Teaching Enhancement Unit Aligning the Ducks: Sustaining the quality of CAA in a Period of Growing Demand Glenis Lambert LTEU, Canterbury Christ Church University Glenis Lambert LTEU June 2009

Learning and Teaching Enhancement Unit Glenis Lambert LTEU June 2009

Learning and Teaching Enhancement Unit Outline • The history of online summative tests at CCCU using short-answer questions delivered online • A reflection, 2005 – 2009 • What do we want from CAA? • How do we match up to this today? • The way forward Glenis Lambert LTEU June 2009

Learning and Teaching Enhancement Unit Background: Canterbury Christ Church University Faculties of Health and Social Care, Education, Undergraduate Programmes +++ Large numbers of students on placement Growing student numbers Diverse student body Growing numbers of collaborative programmes Multi-location campus Central Learning and Teaching Enhancement Unit with Learning Technologists for each Faculty Glenis Lambert LTEU June 2009

Learning and Teaching Enhancement Unit 2005 What we did 2006 • • Bought Question. Mark/Perception Mapped with existing University policies and managed conflicts Using BS ISO/IEC 23988: 2007 Wrote a policy document Set up staff procedures and guidance with full documentation for staff 2007 What happened in the pilot phase • 2008 Over the academic year 8 end of course exams in 18 sessions were successfully delivered, totalling 660 individual examinations. 5 of these were organised and delivered by the departments, three by the examinations office. 2009 Glenis Lambert LTEU June 2009

Learning and Teaching Enhancement Unit 2006 What we did 2005 2006 2007 2008 Slight increase in number of tests Consolidated procedures What went wrong Lack of staff confidence in online environment led to high support demand Some institutional procedures did not fit the CAA mode of delivery, resulting in complications for the student Human error! What we Learned Stakeholder confidence in support and procedures help develop practice Buy-in at all levels is essential Clear fall-back strategies are essential BUT: 2009 Exit surveys revealed student satisfaction with online examinations All of the fall-back procedures worked well The systems/software performed well Glenis Lambert LTEU June 2009

Learning and Teaching Enhancement Unit 2007 2005 What we did Delivered over 5, 000 summative tests and a few other surveys. Support load on LTEU began to be unsustainable. 2006 What went wrong 2007 2008 Staff still not engaging with processes and procedures Infrastructure made process expensive Some institutional procedures did not fit the CAA mode of delivery, resulting in complications for the student Human error! “Last minute” culture led to increased support demand Infrastructure began to creak! 2009 Glenis Lambert LTEU June 2009

Learning and Teaching Enhancement Unit 2008 What we did to try to manage the support load 2005 2006 2007 2008 2009 Put an embargo on all new tests and set out to reinforce our policies by working with stakeholders to devolve some of the activities carried out by LTEU Bought the Blackboard Bridge to reduce the administrative load Encouraged more academics to do “open” assessments which reduced the infrastructure and invigilator load, but not the support load, although this changed its emphasis. (Bertolo and Lambert, 2007) What we learned Our ambition to control the testing environment was probably never going to happen It was going to take a long time to get all stakeholders on board There needed to be a central agency to co-ordinate all aspects of the process Support time for CAA was always going to be high Infrastructure/development costs were an ongoing commitment We had yet to muster a persuasive argument to get total institutional buy -in Glenis Lambert LTEU June 2009

Learning and Teaching Enhancement Unit 2008 2005 2006 2007 What went well We have rarely had to resort to “fall-back” positions, e. g. paper Our quality procedures have been robust and applied throughout Our planning procedures have raised stakeholder confidence Documentation produced for staff has been welcomed resulted in clear definitions of responsibilities Staff development has led to improvements in all areas 2008 2009 Glenis Lambert LTEU June 2009

Learning and Teaching Enhancement Unit 2009 What we learned 2005 2006 We needed to think carefully about what we were doing and whether we should continue to do it. 2007 2008 2009 Glenis Lambert LTEU June 2009

Learning and Teaching Enhancement Unit What do we require from CAA? Report on e. Assessment Quality (REAQ) Overview There is wide consensus that e-assessment promises many potential benefits to the JISC community, ranging from lowered costs, higher productivity, and faster feedback, through to assessments which are more accurate, more detailed, and more robust under critical scrutiny and audit. There is equally wide concern that these benefits seem relatively slow to materialise, and may be negated by poorer quality assessment which fails to reliably provide acceptably valid measures of actual student achievement and capability, particularly at the higher levels of Bloom’s taxonomy of the cognitive domain. The issues confronting staff concerned with the quality management of eassessment suggest a pressing need for evidence, for information, and for guidance. http: //www. jisc. ac. uk/whatwedo/projects/reaq. aspx Glenis Lambert LTEU June 2009

Learning and Teaching Enhancement Unit What do we require from CAA – the Institution’s view? • • • Assessed tasks should attract sufficient student time and effort Tasks should engage the student in productive learning activity Students' effort should be spread as evenly as possible Orientation to the task should help students to perceive the demands of an assessment task Communication through guidance provided should offer clarity on expectations Sufficient and timely feedback should be provided Feedback should focus on learning not on the marks or the student Feedback should be closely linked to assessment criteria Feedback should be given at a level accessible to the student's sophistication Students should be required to receive and respond to feedback http: //www. canterbury. ac. uk/support/learning-teaching-enhancement-unit/assessmenthandbook/pages/section 1/11 -3 -strategies. asp Glenis Lambert LTEU June 2009

Learning and Teaching Enhancement Unit Can CAA ever be “fit for our purposes? ” Develops students’ abilities to evaluate own progress, direct own learning Emphasises authentic and complex assessment tasks Uses high-stakes summative assessment rigorously but sparingly Offers “low-stakes” confidence building opportunities and practice Is rich in formal comment, (e. g. tutor comment, self-reflective logs) Is rich in informal feedback (e. g. peer review, collaborative project work) Above all, it should enhance learning. Glenis Lambert LTEU June 2009

Learning and Teaching Enhancement Unit Quality Assurance Procedures ‘Openness to challenge is a critical cultural necessity for good risk management and compliance – it is in fact more important than any framework or set of processes’. Paul Moore, Evidence to Treasury Select Committee (February 2009) “Over the last twenty years, the higher education sector has been subject to structural and cultural changes associated with a focus on accountability and value for money which have profoundly affected academic life…. . In this new reality the term ‘quality’ became synonymous with data collection, performance scrutiny and a massive increase in bureaucracy. ” George Gordon and Catherine Owen, Cultures of Quality Enhancement: a short overview of the literature for higher education policy makers and practitioners Scottish Higher Education Enhancement Committee (SHEEC) and QAA Scotland (2009) Glenis Lambert LTEU June 2009

Risks involved in the continuing growth of CAA: Learning and Teaching Enhancement Unit • High costs of quality-assurance not met by institution What are thereliable delivery not met by institution • High costs of risks? • Concentration on “box ticking” accountability does not lead to enhanced learning Risks of NOT using or encouraging the growth of CAA: • Opportunity for enhancing learning and feedback lost • Students’ expectations of HE not met (Dermo 2009) Glenis Lambert LTEU June 2009

What we want from CAA Embedding Strategy Policy Robust Learning and Teaching Enhancement Unit Embedding In addition: That it is used to provide speedy feedback Strategy to students That it is used as a means of enhancing Policy learning Robust Reliable Fit for purpose Fit for Purpose Glenis Lambert LTEU June 2009

Where we are at present Embedding Strategy Policy Robust Reliable Learning and Teaching Enhancement Unit Where people have used CAA it has become embedded in their practice, but summative use isn’t leading to formative use. There is a growth in the strategic use of CAA, but largely to reduce marking load. Policy documents are now in existence, but they need reviewing in the light of support demand. We are confident that our policies and procedures do result in adequately assured assessments where they are applied, but the infrastructure and support services are sometimes more fragile The software and systems have proved to be reliable. Infrastructure and the application of policy and procedures less so. “Quick Burn” implementation (Warburton 2009) leading to some Fit for purpose questionable practice Feedback not built in to tests, little formative testing Glenis Lambert LTEU June 2009

Learning and Teaching Enhancement Unit Conclusions The support and maintenance costs of quality-assuring CAA are high and ongoing It is easy to quality-assure CAA if it is only seen as a method of providing accountability for assessment practices and procedures If the major requirement of CAA is to enhance learning, it can be costeffective, when “opportunity costs” are factored in (Ridgway et al. 2004). To achieve the benefits of CAA for learning enhancement staff development has to be the central strategy for assuring quality. A “student satisfaction” ethos needs to be central to mitigating failures Glenis Lambert LTEU June 2009

Learning and Teaching Enhancement Unit Forward plans Raising awareness through Enhancement Theme on Assessment and Feedback Embedding DEBUT staff development project Work with staff on planning of new courses via FQOs Development of e-learning strategy Strategy Benchmarking e-learning, including use of CAA Policy Robust Reliable Fit for purpose Inclusion of policy and procedure documents in staff development pack Continue to work with all stakeholders to emphasise the importance of team structures Staff development for all stakeholders including Examinations Office, User Technology etc. In the absence of more resources, designate a number of rooms as testing labs to make infrastructure support more efficient Make sure all stakeholders know about fall-back procedures Maintain central consultancy role in LTEU for construction of questions and tests DEBUT staff development project Glenis Lambert LTEU June 2009

Learning and Teaching Enhancement Unit Glenis Lambert LTEU June 2009

Assessment Handbook, Canterbury Christ Church University Learning and Teaching Enhancement Unit http: //www. canterbury. ac. uk/support/learning-teaching-enhancement-unit/assessmenthandbook/pages/section 1/11 -3 -strategies. asp Bertolo, E & Lambert, G: 2007 Implementing CAA in Chemistry: A Case Study Procedings of 11 th CAA Conference, Loughborough University 2007 Dermo, J. (2008) “Implementing Online Assessment: Finding the Right Path for an HE Institution” in Ladwa, A (ed) E-Learning in HE available online at http: //www. rscyh. ac. uk/Documents/HEbooklet 2_000. pdf (last accessed June 2009) Gordon, G & Owen, C, 2009, Cultures of Quality Enhancement: a short overview of the literature for higher education policy makers and practitioners Scottish Higher Education Enhancement Committee (SHEEC) and QAA Scotland Parsons, R, (2004) Ensuring Quality and Efficiency with Online Assessments JISC http: //www. enhancementthemes. ac. uk/documents/events/20040416/Parsonspaperrevised. doc. Report on e. Assessment Quality (REAQ), JISC 2009 http: //www. jisc. ac. uk/whatwedo/projects/reaq. aspx Accessed June 2009 -07 -02 Warburton 2009, Quick win or slow burn: modelling UK HE CAA uptake Assessment & Evaluation in Higher Education, 1469 -297 X, Volume 34, Issue 3, Pages 257 – 272 References Glenis Lambert LTEU June 2009

Learning and Teaching Enhancement Unit What we want from questions and tests Embedding Strategy Policy That online assessments are viewed as routine, in the same way as presentations, paper tests etc. That short, formative online tests are a routine part of assessment strategies That policies and procedures are routine elements of assessment procedure That all questions and tests are valid and peer-reviewed Robust Reliable That staff can rely on support and staff development in the preparation of assessments That all elements are “fit for our purposes” Fit for purpose Glenis Lambert LTEU June 2009

Stage 1: This is a duck: True or False? Learning and Teaching Enhancement Unit The question: a digression. Stage 2: This bird is: a) A coot d) A swan c) A duck e) A diver Stage 3: Identify this common duck and match it with its species a) Mallard i) Marmaronetta b) Pochard ii) Netta c) Teal iii) Aythya Glenis Lambert LTEU June 2009

Learning and Teaching Enhancement Unit Barriers to improving question quality • Many academic staff operate in a text-based environment and find the black and white nature of computer-marked questions challenging • Learning technologists do not have detailed subject knowledge • Staff development to enable the development of question -setting skills takes time. Glenis Lambert LTEU June 2009