ASSESSING SPEAKING PURPOSES AND TECHNIQUES Elena Onoprienko

ASSESSING SPEAKING – PURPOSES AND TECHNIQUES Elena Onoprienko Yulia Polshina Tatiana Shkuratova

Outline v v v Key questions Nature of speaking Speaking as a skill Test purposes and different test types Speaking test tasks (advantages and disadvantages) v Washback effect

Key questions What is Speaking? Why assess Speaking? Construct Task Types How (Test)? Purpose Scoring criteria How (Score)?

Nature of Speaking v Spoken language v Speaking as interaction v Speaking as a social activity v Speaking as a situation-based activity

What is Speaking? A part of the shared social activity of talking (Luoma, 2004: 29) In comparison with writing, speaking is v v Transient Dynamic Interpersonal Content dependant Less v Planned v Complex v Formal v Lexically dense

Speaking vs Writing The main differences are in two sets of conditions: processing and reciprocity. • Processing is connected with time; speaking is going on under pressure of time. • Solution to this problem – reciprocity. Speakers take turns and create the text together. Bygate, 1987

Spoken language v Pronunciation v Spoken grammar v Lexis

Pronunciation v Speech is judged on the basis of pronunciation. v What is standard? – Native speaker vs non-native speaker. v Communicative effectiveness, which is based on comprehensibility and probably guided by native speaker standards but defined in terms of realistic learner achievement, is a better standard for learner pronunciation. (Luoma S. , 2004) v What to include into assessment of pronunciation? v Pronunciation – individual sounds, pitch, volume, speed, pausing, stress and intonation

Spoken Grammar v Grammar is easy to judge because it is easy to detect in speech and writing. v Speakers do not usually speak in sentences. v Speech consists of idea units connected with and, or, but, or that v Planned vs unplanned speech – complex structures vs short idea units v The internal structure of idea units - topicalisation and tails create an impression of naturalness

Features of Spoken Lexis v ‘Simple’ and ‘ordinary’ words are common in normal spoken discourse and mark a highly advanced level of speaking skills. (Luoma S. , 2004) v Generic words (important for the naturalness of talk) v Vague words v Fixed conventional phrases v Small words (the more – the better perceived fluency)

Slips and errors Normal speech contains a fair number of slips and errors such as mispronounced words, mixed sounds, and wrong words due to inattention. (Luoma S. , 2004).

Speaking as a skill v Task fulfillment/content v Fluency v Accuracy v Vocabulary and grammar range v Interaction

Speaking as meaningful interaction v Speaking is both personal and a part of the shared social activity of talking. v The openness of meanings is not only a convenience in speech; it is also an effective strategy for speakers. (Luoma S. , 2004) v Chatting vs information-related talk v The role of speaking situations v Roles, role relationships and politeness

Why assess speaking No single answer v Different groups of language learners have different needs e. g. some groups of adult language learners » International travellers: language for travel, leisure » Migrants: survival skills, access to employment » Students: exams, academic communication, social interaction » Professionals: workplace communication, presentations, v Different users have different purposes when they seek information from tests v But most users of language do need to speak (Green, A. , 2012)

Test purposes and different test types Test Purposes v Proficiency tests v Achievement tests v Placement tests v Diagnostic tests

What do we need to decide before giving a speaking test? v What aspects of language we want to assess v How to elicit ratable language samples from testtakers suitable for the aspects of language We need to decide: v Rating criteria [marking categories, levels, descriptors] [Holistic scales vs. Analytical scales] v Elicitation techniques / Test format (types of questions, task types)

Performance testing in second language proficiency assessment is traditionally used to describe the approach in which a candidate produces a sample of spoken or written language that is observed and evaluated by an agreed judging process (Mc. Namara 1996)

What is Performance Testing? 1. Sample of written or spoken language 2. Simulates behaviour in the real world © Not like paper-and-pencil ‘objective’ tests 3. Observed and evaluated by agreed judging process (Green, A. , 2012)

Speaking tasks v A communicative task is a piece of classroom work which involves learners in comprehending, manipulating, producing or interacting in the target language while their attention is principally focused on meaning rather than form… (Nunan 1993: 59) v Speaking tasks can be seen as activities that involve speakers in using language for the purpose of achieving a particular goal or objective in a particular speaking situation (Bachman and Palmer 1996: 44)

Types of information-related talks Factually-oriented talk v v description narration Instruction comparison Evaluative talk v v explanation justification prediction decision (Bygate, 1987)

Communicative functions Microfunctions according to CEF v Giving and asking for factual information (describing reporting, asking); v Expressing and asking about attitudes (agreement/disagreement); v Suasion (suggesting, requesting, warning) v Socialising (attaching attention, addressing, greeting, introducing) v Structuring discourse (opening, summarising, changing the topic) v Communication repair (signalling non-understanding, appealing for assistance, paraphrasing) (Council of Europe, 2001: 123, Luoma, 2004: 33)

Features of a speaking task v Input, or material used in the task v Roles of the participants v Settings, or classroom arrangements for paired or group work v Actions, or what is to happen in the task v Monitoring, or who is to select input, choose role or setting, alter actions v Outcomes as the goal of the task v Feedback given as evaluation to participants • Candlin (1987) from G. Fulcher (2003)

Speaking test tasks format v Individual v Open-ended tasks v Paired v Structured tasks v Group

Advantages and disadvantages of an interview + tester’s control over + opportunity for an examinee to show the range of their speaking skills - it’s costly in term’s of tester’s time; - interviewer’s power over an examinee

Advantages and disadvantages of paired formats + Capable of eliciting more symmetrical contribution to the interaction from test-takers + Capable of eliciting a much richer and more varied language functions + Positive reaction from test-takers (less anxious), a sign of positive washback effect + Practical: time-efficient, cost-effective, less burden and less training for the examiners - The amount of responsibility given to examinees, who are not trained in interview techniques

Advantages and disadvantages of group formats + Well-received by learners; +Support learning; - Difficult to administer and manage (size of the groups and mixture of learners’ abilities); - Difficult to monitor the progress of the testing;

Speaking test tasks v Oral Presentation (Verbal Essay, Prepared monologue) v Information Transfer (Description of Picture Sequence, Questions on a single Picture, Alternative Visual Stimuli) v Interaction Tasks (Information Gap: Student – Student, Student – Examiner, Open Role Play, Guided Role Play) v Interview (Free, Structured) v Discussion (Student-Student, Student-Examiner) (O’Sullivan, 2008: 10 -11)

Framework for designing test tasks v Operations (activities/skills) Informational routines (e. g. telling a story) and improvisational skills (negotiation of meaning and management ofinteraction) v Conditions under which the tasks are performed (e. g. time constraints, the number of people involved and familiarity with each other) v Quality of output, the expected level of performance in terms of various relevant criteria, e. g. accuracy, fluency or intelligibility. (C. Weir, 1993: 30)

Developing criteria for assessment speaking v The importance of double marking for reducing unreliability is undeniable. v These criteria need to reflect the features of spoken language interaction the test task is designed to generate. v The criteria used would depend on the nature of the skills being tested and the level of detail desired be the end users. The crucial question would be what the tester wants to find out about a student’s performance on appropriate spoken interaction tasks. (C. Weir, 1993: 30)

Rating criteria: v Phonological control v Grammatical accuracy v Vocabulary range v Fluency (Taken from ‘Common European Framework of Reference for Languages’, Council of Europe 2001) • Test format: interview format with the following structure 1. Openings (1 minute) 2. Conversation on familiar topics (3 minutes) The interviewer asks the candidate to talk about him/herself. 3. Picture Description (2 minutes) The interviewer asks the candidate to describe a photo. 4. Conversation on topics from the given picture (5 minutes) The interviewer asks the candidate questions linked to the picture (from general to extended questions). 5. Closings (1 minute) (Fumiyo Nakatsuhara)

Scoring v Holistic scale e. g. Trinity College Bands A, B, C, D v Analitic scale e. g. IELTS Fluency and coherence Lexical resources Grammatical range and accuracy pronunciation

Holistic rating scales v Positive features – Practicality: fast – Rating holistically may be more naturalistic v Disadvantages – No useful diagnostic information: single score – Not always easy to interpret: raters not required to use same criteria to arrive at score (Green, A. , 2012)

Analytic rating scales v Positive features – Can provide diagnostic information if scores reported separately – Potentially clear, explicit and detailed – Usually more reliable (multiple scores) – Useful in training raters to focus on our construct – Potentially useful in guiding learners v Disadvantages – Time-consuming – May overburden raters (Green, A. , 2012)

The role of an interviewer interrater / intrarater reliability The solution – training raters v Understanding criteria for assessment v Agreement with other raters v Consistency of performance

Washback Effect: effect of testing on teaching and learning Positive / negative washback v Positive – stimulating classroom teaching of what is tested; v Negative – test reliability is affected.