Webinar Series This session is part of e-Assessment

Webinar Series This session is part of e-Assessment Scotland 2013 21 Aug 2013: Assessment-as-learning: introducing the Conversation Sim Robert Nelson and Phill Dawson (Monash University, Australia) Your hosts Professor Geoff Crisp, Dean Learning and Teaching, RMIT University geoffrey. crisp[@]rmit. edu. au Dr Mathew Hillier, Teaching and Educational Development Institute, University of Queensland mathew. hillier[@]uq. edu. au

The conversation sim a new approach to assessment-as-learning Phillip Dawson & Robert Nelson Monash university OPVCLT http: //conversationsim. org

What is a conversation sim? • A teaching tool that simultaneously assesses • A participatory paradigm for presenting information • An inversion of the MCQ that uses the MCQ algorithm • Educational engine: problem suggestion decision reflection

According to conversationsim. org A conversation sim is a way of teaching and assessing complicated content using standard online resources. Through any LMS, you can make the computer simulate conversation, engaging students in discussions that are infallibly on-topic and deeply thoughtful. At the same time as conducting the discourse for you, the conversation sim will perform assessment, yielding a grade that reflects the student’s grasp of critical ideas and sympathy for the syllabus. It isn’t new software but a way of using the resources that already lie to hand. They operate efficiently and are functionally robust.

The conversational sim structure problem suggestion decision reflection • Built around areas of doubt or difficulty –curious conundrums more than rules • Also works if the subject matter has objective closure — the conversation is conceived around misconceptions

Structure for webinar • What a conversation sim is – already more or less done • Time to do a sim – let’s go live next • Our journey toward the concept – practical and theoretical • The project with the OLT – research and implications

Point of departure: a tutorial • Somebody asks a question • Somebody else makes a suggestion toward an answer • Somebody expresses agreement or discomfort or extrapolates • Somebody argues either way with reason • An air of making a decision attends • • The question is put again Another suggestion is made toward an answer Argument follows The thought of making a decision prompts further responses

Socratic dialogue • A conversation of sorts – not totally organic but interrogative and analytical • Guided by teacher’s agenda – energized by students’ interventions • Students’ participation more an invitation than a controlling share in debate – how would it not be patronizing?

The question paradox • Good question has us stumped? – how profitable is that, unless you proffer answer? – which defeats the purpose! • Does a good question rely on an impasse? – in which case what happens to curiosity? • Surely a negative and uncreative approach to questions – good question leads to another good question – dynamic and organic, proposing a conversation

A question wants something • In quest of something – a fulfilment is sought for something that builds • Questions are unsatisfying when autonomous – because apparently without consequence – or incapable of demonstrating consequence • Come to life in a sequence – act out volition, a will to argue, to reach expression through exchange • Always in search of a participant

Dubious subject matter • The key is doubt, uncertainty, a decision among likely options, a conundrum • Organize a conversational question sequence through the LMS (VLE) • Identify problems that lead somewhere • Doubt paradoxically gives you a handle • Ideal ingredient for the conversation sim

Four Moodle moments at Monash • Research Graduate Supervision (HED 5070) • Monash Orientation for Higher Objectives (MOHO) • Teaching Associate Conversational Thinking (TACT) • Responsible Research (RST 0005)

Research integrity training • must be online – otherwise very expensive to deliver • with patchy local interpretations, quality control • dependent on enthusiasm of conscripted teachers • hard to monitor attendance and assess • preferably universal – otherwise admin burden over exemptions • endless argie-bargie and chasing people

Learning outcomes 1. an appreciation of the values in the Australian Code for the Responsible Conduct of Research, especially 2. a critical view of research ethics and some of the processes for looking after it 3. a practical and mindful understanding of intellectual property and good data management 4. insight into good general practice in relation to research integrity and 5. an appreciation of publication conventions, collaborative research and research cultures that reconcile personal ambition, ethics and scrupulosity in scholarship.

Classical online OHS method • Present The Code, pdf + online video lecture • Test to see if content is assimilated by MCQs – demoralizing and dry as dust • nobody helps; uninspiring, sometimes irrelevant – shallow: what’s to ponder? what’s to intrigue? • as soon as you learn, you forget (learning for tests) – resented: you brought me here for this? • I labour frustratedly so that your arse is covered

Teaching The code more than rules • ‘maintain a climate in which responsible and ethical behavior in research is expected. ’ (1. 1) • ‘respect the truth and the rights of those affected by their research’ (1. 6) • ‘manage conflicts of interest so that ambition and personal advantage do not compromise ethical or scholarly considerations’ • how do we teach ‘beneficence’?

for every candidate to be an ethicist

An example of the method Why should I have to?

A situation Ethics concerns everybody but not all processes concern every investigator. For example, if I’m never likely to use human subjects in my research, why should I bother with all of those elaborate and intricate formalities—so rich in laborious paperwork—that govern research involving humans?

A response That’s what I was thinking. Surely going over all this stuff is onerous and pointless if I’m an electrical engineer, a mathematician or a renaissance historian. To say that ethics concerns everyone is true; but so does the law. Just because the law is universal throughout its jurisdiction, it doesn’t mean that I have to become a lawyer.

You have to respond • Yes • No • Maybe

Professor Moodle says I’m not so fond of this response. To pursue the analogy, it’s true that we don’t all need to be lawyers to live a lawful life—and heavens forbid that we’re ever tempted by crime—but we still need to know something about the law and have a general sense of its principles. It also helps to know certain laws (like with roads and tax) not just to keep out of trouble but to belong to a society where we can listen and think about the concerns of our fellow citizens.

The same situation gets another response Even though you’re a chemical engineer or a renaissance historian, you’re also a member of a large council, which is the intellectual college of the university. The university is nothing without your place in it as an academic who speaks and approves or disapproves. Also, you have a career which isn’t necessarily confined to a single discipline but to the governance of intellectual process: it’s that collegial economy of opinion which ultimately takes responsibility for everything that happens in the university.

You have to respond • Yes • No • Maybe

Moodle comes back at you I love this response. Also, on a practical level, research is especially mobile when research graduates and honours students come into our ambience; because they often have projects that do involve humans. . . and if you’re thinking that ethics isn’t in your field, you’ll be less likely to notice that your candidate’s research has ethical implications.

A third response is put to you I can see that responsibility for ethics might want to be shared among all the scholars of the university; but surely judgements about the discipline are best left to the experts in the field. I would feel a great lack of humility telling social scientists or medicos how to handle human ethics. Actually, when I think about it that way, it’s none of my business.

You have to respond • Yes • No • Maybe

Moodle answers: No. This response is very sweet but not congruent with ethical responsibility. We could equally say the same thing over social justice and feel selfeffacing because we aren’t in public policy or economics; but then what happens during fascism? I prefer to think that ethics are about a community feeling a certain way. And in that sense, the voice from outside the discipline is especially valuable, because it isn’t filled up with the assumptions that have won acceptability in the field.

A fourth and final response Even though you know that you’re a chemical engineer or a renaissance historian—and we have no desire to change that!—you never know when you find yourself switching roles and you go from a technical or theoretical or textual study to a context which has human subjects.

You have to respond • Yes • No • Maybe

Moodle gives feedback again That’s correct. An obvious example is educational research, which applies to all of the disciplines, like information technology or baroque musicology. When we study the pedagogy of such fields (which are themselves free of live human subjects) we are quickly tempted to engage with live data sets. Suddenly the assumed remoteness from human ethics breaks down.

Our project • Successful bid with OLT ‘Assessment as learning through conversation simulation’ –Write a manual of how to write conversation sims conversationsim. org – Bring in champions to write sims • and learn from their experience – Publish analysis and L&T reasons

Assessment-as-learning • qua essay, very old – instructive and creative presentation – subsequently evaluated and marked • But automated system emulating tute? • Almost assessment-as-learning? – first define as a learning schedule – rather, evidence of learning clocks up credit • genuine rigorous discriminatory assessment?

To question assessment • Our interrogation the roots of testing – fear that learning is assessment-led – rigour of bell-curve + ambition of student = manipulation or educational gaming • In fact assessment is more modern than old – more for the sake of the examiner than the examined • ‘A contribution to the history of assessment: how a conversation simulator redeems Socratic method’, Assessment & Evaluation in Higher Education, June 2013

To understand conversation • Maybe schematizing conversation for the sake of digital flow helps us understand conversation itself • To make educational process more conversational: paradox of digital? • Much research to be done! Try it yourself at conversationsim. org