Distance Learning Unit Faculty of Engineering and Design

Distance Learning Unit Faculty of Engineering and Design Towards Enhanced Feedback Practices in a Virtual Environment – a case study Mick Hancock Peter Mellett Rachel Wood Alex Copping May 2011

Context. . . • A modular distance learning M-level programme in Construction Management • 300 students in 45 countries worldwide • Blended learning format – teaching and learning materials include printed ‘workfiles’, textbooks and online activities. • Students meet online and at residential schools to establish ‘collaborative communities of practice’ • Student support includes a full-time dedicated Bath-based Programme Administrator and external, contracted, module-specific online tutors who are usually academic subject specialists. . and Generalisability • How do you attempt to answer questions of the sort ‘How can I improve the quality of my/our practice? ’ • How ‘distant’ are your colleagues and other critical staff? • How can you bring them ‘closer’ so that change is owned by all within a sense of corporate endeavour?

The Distance Learning MSc programme International Construction Management (ICM)

ICM – Core Modules Assessment Profiles • Module 1 – Assignment 60% / Open Book Exam 40% • Module 2 – Assignment 60% / Open Book Exam 40% • Module 3 – Online Negotiation Task 50% / CW 50% • Module 4 – Assignment 50% / Closed Book Exam 50% • Module 5 – Assignment 50% / Open Book Exam 50% • Module 6 – Assignment 50% / Closed Book Exam 50% NB. Assignments and examinations are written and marked by the relevant tutor. Assignments are updated/amended as circumstances dictate/allow.

ICM Core Units – tutors’ questionnaire response scores Table 1: Start of the TDF project Adding value to summative assessment through enhancing feedback practices Question: To what extent do the assessment practices on your module – Score / 10 M 1 M 2 M 3 M 4 M 5 M 6 1 encourage self-assessment? 1 2 8 8 4 2 2 encourage teacher and peer dialogue around learning? 5 5 3 5 7 3 3 help to clarify what good performance is (goals, criteria, expected standards)? 3 4 8 1 2 5 4 provide opportunities to close the gap between current and desired performance? 3 5 2 7 5 5 5 deliver high quality information to the students about their learning? 7 5 7 8 4 4 6 encourage positive motivational beliefs and self-esteem? 5 3 5 8 1 1 7 generate information about student learning that can be used to reshape the learning experience? 1 1 7 2 8 8 NB. modules are studied in the sequence 1 to 6. The assessment for Module 3 (Construction Law) is based on a Moodle-mediated negotiation exercise between student groups, whereas all others are based more directly on the study materials.

Appendix 1: Summary of Tutors’ responses to Moodle quiz Self-evaluation of Assessment Practice Fred (M 1) Question 1: To what extent do the assessment practices on your unit encourage selfassessment? Jim (M 2) Pete (M 3) Arthur (M 4) Pru (M 5) Cynthia (M 6) 1/10 I don't see that the students are encouraged to do any of these things. 2/10 I understand respect the need for self-assessment tasks in a distant taught package such as ICM, but I have a F/t job and a life to keep going as well. Hence I do not always achieve all of these pedagogic goals. I have experimented with the idea of students giving feedback to one another before submitting the final (summative) assessment, but find it time consuming and relatively selective as many students do not participate. I would also suggest that the Multiple choice questions that conclude each section of the work file in Module 2 are a means of contributing to self assessment. 8/10 I think the module is pretty good as getting the students to self -assess. There is the reflective part at the end of the negotiation exercise and there are milestones through the rest of the task as well. The practical scenario is less reflective. I think you can over do it - the dissertation forum where they talk about each other's work is having some success but I don't think it takes the place of student/tutor interaction. 8/10 I seem to recall a fairly recent student survey which gave the students an opportunity to comment on the positive and negative aspects of their previously received feedback - but I might be wrong / confused with another institution The discussion forum indirectly enables students to reflect and make judgements about the quality of their work. This forum also provides an opportunity for students to learn from each other. Not sure that it is sold quite that way though. The M 4 Handbook is very explicit concerning milestones and the need to build a portfolio or work and experience 4/10 Module 5 has been set as per other modules. No specific practice to encourage self assessment has been put into Module 5. 2/10 The assessment is very traditional and they have criteria to set against but little in the approach to encourage this. Maybe core reason is I do not see this as my role as there is plenty on the site for them to self assess, I do not see that the assessment approach could or should help this. I am tutor for their first module and many struggle simply to write an 'academic' essay (for various reasons). As they have not yet had any experience of what is required of them, their ability to be selfcritical is limited. Their wishes for feedback are almost certainly. . . whatever is needed to get a better grade. They have no 'portfolio' of work to choose from Current assignments are not designed to be developed in a way that would enable achievement milestones to be included. In any case, time on each module seems is short and I suspect that there is not enough money to pay the tutors to spend so much extra time on such an approach Time is also probably too short for students to give feedback on each other's work. . . they are too busy trying to complete the module.

Question 2: To what extent do the assessment practices on your unit encourage teacher and peer dialogue around learning? 5/10 This is a difficult one because I am not aware that the students are actually asked to do any of these things, but the opportunity is there for them to do at least some of them e. g. share and discuss tutor feedback with their peers and suggest strategies to improve performance next time; identify feedback comments that they found useful and to explain how they helped give each other descriptive feedback on their work in relation to published criteria before submission; Each of these can be done through the various forums etc. . within Moodle, but I am not aware of students being encouraged to do so. Question 3: To what extent do the assessment practices on your unit help to clarify what good performance is (goals, criteria, expected standards)? 3/10 Students are provided with carefully constructed criteria sheets and performance level definitions. and I do talk about criteria and required standards at the Introductory workshop (although unfortunately many students have already completed the module before they attend so it's too late). However, i am not aware of any of the other examples given here being used. 5/10 I regard supporting the discussion forum as a key part of e-tutoring; and I commit at least two hours per week when the module is running. In my weekly response I aim to summarise the discussion that has taken place, confirm positive activity and contributions where appropriate, and encourage clarity and an academic approach. I also think that the way the programme material is presented and written helps students think about the scale of the problem. For example, one of the exercises that often raises questions of qualification requires students to find official construction data relating to the nation they live or work in. 3/10 There is not a great deal of dialogue around the negotiation exercise which is that happy thing - something the students grasp and run with by themselves and then present their findings to us. The practical scenario used to have a stage where they needed approval of their scenarios but I am not sure if this is still used or how useful it actually was. 1 -10/10 I cannot be specific about this question for the following reasons: I imagine this is dictated by the extent of use / capability of Moodle. This being a Distance Learning programme, it is as good as the weakest link which might be the communications medium I think (but have no evidence to support) that student contact details are made available to one another I have seen messages referring to pub nights in BC but have never been and so cannot comment on their success rates 7/10 A range of self reflection questions are built into Module 5 and students do submit responses and other students comment on these responses. One challenge is that only about 50% of students actually bother to post responses to the questions and there is no sanction to make all students contribute. 3/10 there is some feedback on the assessment feedback but this is limited 4/10 This is something that I struggle with, since e-moderating appears to be moving the goal posts. I recognise a need to support and identify progress towards students producing their best results, but I shy away from explicitly telling them how to achieve this with model answers and the like. Having access to the marking criteria and a clear assignment brief is obviously essential; but providing exemplar assignments opens a can of worms! However, I can see room for discussing drafts and confirming the direction that students are making. I also can see a place for exercises relating to effective referencing but have not had the time to set this up electronically. 8/10 We use the grading sheets where they can see what a good mark is and what we reward. My only concern here is that there is some benefit it not disclosing these - a very clever student can tailor what they do to meet the criteria rather than perform how they would normally if they did not know the criteria. In some ways I prefer to give the feedback at the end when they can see how they did against undisclosed criteria - criteria which the right approach would have led them towards anyway. 1/10 The Grading System provided to me is fine. I have posted a guide for students writing assignments. This seems to have worked, judging by the 14 assignments recently assessed I am not aware of any exemplar assignments or for the opportunity to assess one another's work Again the discussion forum does offer the facility for discussion amongst students of results and feedback. I have previously been emailed with specific issues 2/10 This is very weak. 5/10 There is quite well developed performance criteria but limited to one way.

Good Feedback A positive focus Start off with a positive encouraging comment. Balance negative with positive comments. Turn all criticism into positive suggestions. Suggesting where next Make general suggestions on how to go about the next assignment. Suggest specific ways to improve the assignment. Suggest follow-up work and references. Ask questions which encourage reflection about the work. Making connections with the student Be prompt. Use informal, conversational language. Offer help with specific problems. Offer the opportunity to discuss the assignment and comments. Offering justification Explain all your comments. Include a brief summary of your view of the assignment. Relate specifically to the learning outcomes and assessment criteria. Explain the mark or grade, and why it is not better (or worse!)

Good Assessment for Learning… …Engages students with assessment criteria …Supports personalised learning …Focuses on student development …Ensures feedback leads to improvement …Stimulates dialogue …?

Your plans… • • What must you keep? What could you change? What must you change? Justifications Now, soon, and a bit further on What evidence will you collect to tell the team your ‘new’ thing has been a success?

The ICM Assessment Development Problem • Tutors do not have sufficient ‘spare’ time to engage with this project as envisaged – and fulfil their obligations to ICM students as currently constituted – and meet their other mainstream professional commitments. • External distance learning tutors operate as academics working in a semiautonomous mode; there are dangers that change actioned internally may be perceived as rule by diktat and meet resistance. The ICM Feedback Problem • All feedback opportunities are limited where an assessment task may be intrinsically infertile or its intention is obscurely expressed.

The Tutor’s Assessment Mantra Bearing in mind the requirement to ensure that all assessment briefs should allow students to give the best possible account of their learning. . . . can I demonstrate that my assessment activities discriminate accurately and fairly?

Effecting change • How do we bring about change in assessment practice and feedback practice – whereby internal staff do the development work and tutors hold the changes as their own? • How do we establish an inclusive ‘us’ in the place of two lots of ‘them and us’ seen from opposing sides?

Buy two, get two (three) free! Bearing in mind the requirement to ensure that all assignment briefs allow students to give the best possible account of their learning, can I show that my assessment activities discriminate accurately and fairly? Assignment statement + Marking criteria Marking process + Feedback process. . and feedforward

ICM – Core Module Assignment Profile Table 2 Types of output required M 1 A commentary of your experience of engaging with the workfile M 3 M 4 x M 5 M 6 x A written account M 2 x An academic discussion x An academic essay x A critical exploration x Extended commentary on a scenario x A discussion paper to be presented to the board of directors x A paper x Critique x Essay x A critical appraisal x A report for the Chief Executive x • Do tutors have a common understanding of the key terms? • Are the requirements made explicit to the students or do they depend on a tacit knowledge of academic terms, of which students may not share our understanding?

Bloom & Co.

ICM Core Module Assignments – Action Verb Analysis Table 3 Action verbs M 1 Explore* M 2 M 3 M 4 M 5 x Consider* x Reflect on* x Discuss* x x Recommend* Remember x List x Describe x Identify Understand x x Comment upon Analyse x x x Give an outline x x x Justify x Describe the benefits and disbenefits x Evaluate x Critically appraise Critically review Critically assess x x Explain the differences (i. e. contrast) Evaluate x x Elaborate on Explain** Analyse M 6 x x x *These terms are not so easy to classify according to Bloom’s Taxonomy and may indicate a range of behaviours depending on the context. **Explain can require understanding or synthesis or evaluation.

Starting the glossary from existing terms • • • • academic discussion academic essay account analyse comment upon commentary consider critical appraisal critical exploration critically appraise critically assess critically review critique describe discuss • • • • discussion paper elaborate on essay evaluate explain explore identify Justify list outline paper recommend reflect on report

Gently coaxing What we would like you to do 1. Review and comment on the Glossary of terms so that we can develop a shared understanding of key terms (to be published to students in due course) by 1 st April 2. Review your own assignment brief, taking account of the need for clarity of meaning and explanation of requirements so there is no room for ambiguity, and bearing in mind the range of skills required across the programme. Submit a new assignment brief and marking criteria by 1 st April. Below you will find links to each of the assignment briefs and assessment criteria. These include annotations by Peter Mellett and Rachel Wood with potential points for clarification.

Key words and appropriate responses Key Words Appropriate response Define Describe Identify Indicate List Outline Recognise Review Select State Remember, either from the previous text or from your own experience, recall data or information i. e. state: etc. Understand data and information as you: etc. Apply your knowledge and understanding, demonstrating. . Analyse Characterise Classify Compare Confirm Contrast Correlate Differentiate Discriminate Distinguish Examine Explain Explore Identify Illustrate Outline Prioritise Relate Select Analyse material/concepts/rules as you: etc. Evaluate etc. Create etc. • Terminology • Individual facts • Rules • Group data into identifiable or logical subgroups • Identify violation of rules or logic • Construct diagrams of the logical relationships between entities • Categories • Structure • Procedure • Explain the logic of a series of events • Separating materials or concepts into component parts so the organisational structure can be understood

Summary • The more ‘distant’ colleagues and other staff are, the more difficulty you will have acting as an agent of change. • Be careful about the assumptions make when using the word ‘we’. • Approach change from the perspective of an action-reflection cycle so that you do not make false assumptions about progress i. e. ask (and respond to) the sequential questions: 1. What is my concern? 2. Why am I concerned? 3. What do I think I can realistically do about it? 4. What kind of evidence can I collect to help me make some judgments about what is happening? 5. How can I check that my judgment about what has happened is reasonably fair and accurate? • Affirm all contributions, however small, from ‘distant’ colleagues and clearly incorporate them into the evolving dialogue about change – and in the amended teaching and learning materials/processes that result and which those ‘distant’ colleagues ultimately will have to use.