ASSESSINGOnoprienko Yulia Polshina Tatiana SPEAKING PURPOSES Prepared

ASSESSINGOnoprienko, Yulia Polshina, Tatiana SPEAKING – PURPOSES Prepared by Elena Shkuratova AND TECHNIQUES Based on material by Fumiyo Nakatsuhara

Outline • • Key questions Nature of speaking Speaking as a skill Test purposes and types of test Speaking test tasks Scoring Washback effect

KEY QUESTIONS

Key questions What is Speaking? Why assess Speaking? Construct Task Types How (Test)? Purpose Scoring criteria How (Score)?

THE NATURE OF SPEAKING

Nature of Speaking • • Spoken language Speaking as interaction Speaking as a social activity Speaking as a situation-based activity

What is Speaking? A part of the shared social activity of talking (Luoma, 2004: 29) In comparison with writing, speaking is More • Transient • Dynamic • Interpersonal • Content dependent Less • Planned • Complex • Formal • Lexically dense

Speaking vs Writing The main differences are in two sets of conditions: processing and reciprocity. • Processing is connected with time speaking is going on under greater pressure of time. • Solution to this problem in spoken language – reciprocity. Speakers take turns and create a text together. (Bygate, 1987)

Spoken language • Pronunciation • Spoken grammar • Lexis

Pronunciation • Speech is judged on the basis of pronunciation. • What is standard? – Native speaker vs non-native speaker. • Communicative effectiveness, which is based on comprehensibility and probably guided by native speaker standards but defined in terms of realistic learner achievement, is a better standard for learner pronunciation. (Luoma, 2004) • What to include in assessment of pronunciation? • Pronunciation – individual sounds, pitch, volume, speed, pausing, stress and intonation

Spoken Grammar • Grammar is easy to judge because it is easy to detect in speech and writing. • Speakers do not usually speak in sentences. • Speech consists of idea units connected with and, or, but, or that • Planned vs unplanned speech – complex structures vs short idea units • The internal structure of idea units - topicalisation and tails create an impression of naturalness

Features of Spoken Lexis v ‘Simple’ and ‘ordinary’ words are common in normal spoken discourse and mark a highly advanced level of speaking skills. (Luoma, 2004) v Generic words (important for the naturalness of talk) v Vague words v Fixed conventional phrases v ‘Small words’ (the more – the better perceived fluency)

Slips and errors Normal speech contains a fair number of slips and errors such as mispronounced words, mixed sounds, and wrong words due to inattention. (Luoma, 2004).

SPEAKING AS A SKILL

Speaking as a skill What is skillful speech? • Task fulfillment/content • Fluency • Accuracy • Vocabulary and grammar range • Interaction

Speaking as meaningful interaction • Speaking is both personal and a part of the shared social activity of talking. • The openness of meanings is not only a convenience in speech; it is also an effective strategy for speakers. (Luoma, 2004) • Chatting vs information-related talk • The role of speaking situations • Roles, role relationships and politeness

Why assess speaking No single answer • Different groups of language learners have different needs • e. g. some groups of adult language learners • International travellers: language for travel, leisure • Migrants: survival skills, access to employment • Students: exams, academic communication, social interaction • Professionals: workplace communication, presentations, • Different users have different purposes when they seek information from tests • But most users of language do need to speak

TEST PURPOSES AND TYPES OF TEST

Test purposes and types of test Test Purposes • Proficiency tests • Achievement tests • Placement tests • Diagnostic tests

What do we need to decide before giving a speaking test? • • What aspects of language we want to assess How to elicit ratable language samples from test-takers suitable for the aspects of language We need to decide: • • Rating criteria [marking categories, levels, descriptors] [Holistic scales vs. Analytical scales] Elicitation techniques / Test format (types of questions, task types)

Performance testing in second language proficiency assessment is traditionally used to describe the approach in which a candidate produces a sample of spoken or written language that is observed and evaluated by an agreed judging process (Mc. Namara, 1996)

What is Performance Testing? Sample of written or spoken language Simulates behaviour in the real world - Not like paper-and-pencil ‘objective’ tests Observed and evaluated by agreed judging process

Speaking tasks • A communicative task is a piece of classroom work which involves learners in comprehending, manipulating, producing or interacting in the target language while their attention is principally focused on meaning rather than form… (Nunan 1993: 59) • Speaking tasks can be seen as activities that involve speakers in using language for the purpose of achieving a particular goal or objective in a particular speaking situation (Bachman and Palmer 2010)

Types of information-related talk Factually-oriented talk Evaluative talk • description • explanation • narration • justification • Instruction • prediction • comparison • decision (Bygate, 1987)

Communicative functions ‘Microfunctions’ according to CEFR • Giving and asking for factual information (describing reporting, asking); • Expressing and asking about attitudes (agreement/disagreement); • Suasion (suggesting, requesting, warning) • Socialising (attaching attention, addressing, greeting, introducing) • Structuring discourse (opening, summarising, changing the topic) • Communication repair (signalling non-understanding, appealing for assistance, paraphrasing) (Council of Europe, 2001: 123, Luoma, 2004: 33)

Features of a speaking task • Input, or material used in the task • Roles of the participants • Settings, or classroom arrangements for paired or group work • Actions, or what is to happen in the task • Monitoring, or who is to select input, choose role or setting, alter actions • Outcomes as the goal of the task • Feedback given as evaluation to participants Candlin (1987) cited by Fulcher (2003)

Speaking test task formats • Individual • Open-ended tasks • Paired • Structured tasks • Group

Advantages and disadvantages of an interview + tester’s control over interaction + opportunity for an examinee to show the range of their speaking skills - it’s costly in terms of tester’s time; - interviewer’s power over an examinee

Advantages and disadvantages of paired formats + Capable of eliciting more symmetrical contribution to the interaction from test-takers + Capable of eliciting a much richer and more varied language functions + Positive reaction from test-takers (less anxious), a sign of positive washback effect + Practical: time-efficient, cost-effective, less burden and less training for the examiners - The amount of responsibility given to examinees, who are not trained in interview techniques

Advantages and disadvantages of group formats + Well-received by learners; + Support learning; - Difficult to administer and manage (size of the groups and mixture of learners’ abilities); - Difficult to monitor the progress of the testing.

SPEAKING TEST TASKS

Speaking test tasks • Oral Presentation (Verbal Essay, Prepared monologue) • Information Transfer (Description of Picture Sequence, Questions on a single Picture, Alternative Visual Stimuli) • Interaction Tasks (Information Gap: Student – Student, Student – Examiner, Open Role Play, Guided Role Play) • Interview (Free, Structured) • Discussion (Student-Student, Student-Examiner) (O’Sullivan, 2008: 10 -11)

Framework for designing test tasks • Operations (activities/skills) Informational routines (e. g. telling a story) and improvisational skills (negotiation of meaning and management of interaction) • Conditions under which the tasks are performed (e. g. time constraints, the number of people involved and familiarity with each other) • Quality of output, the expected level of performance in terms of various relevant criteria, e. g. accuracy, fluency or intelligibility. (Weir, 1993: 30)

SCORING

Developing criteria for assessing speaking • The importance of double marking for reducing unreliability is undeniable. • These criteria need to reflect the features of spoken language interaction the test task is designed to generate. • The criteria used would depend on the nature of the skills being tested and the level of detail desired be the end users. The crucial question would be what the tester wants to find out about a student’s performance on appropriate spoken interaction tasks. (Weir, 1993, p. 30)

Rating criteria • Phonological control; Grammatical accuracy; Vocabulary range; Fluency (Council of Europe 2001) • Test format: interview format with the following structure 1. Openings (1 minute) 2. Conversation on familiar topics (3 minutes) The interviewer asks the candidate to talk about him/herself. 3. Picture Description (2 minutes) The interviewer asks the candidate to describe a photo. 4. Conversation on topics from the given picture (5 minutes) The interviewer asks the candidate questions linked to the picture (from general to extended questions). 5. Closings (1 minute) (Nakatsuhara, 2012)

Scoring Holistic scale Analytic scale e. g. Trinity College e. g. IELTS Bands A, B, C, D Fluency and coherence Lexical resources Grammatical range and accuracy Pronunciation

Holistic rating scales v Positive features – Practicality: fast – Rating holistically may be more naturalistic v Disadvantages – No useful diagnostic information: single score – Not always easy to interpret: raters not required to use same criteria to arrive at score

Analytic rating scales v Positive features – Can provide diagnostic information if scores reported separately – Potentially clear, explicit and detailed – Usually more reliable (multiple scores) – Useful in training raters to focus on our construct – Potentially useful in guiding learners v Disadvantages – Time-consuming – May overburden raters (Green, 2012)

The role of an interviewer interrater/ intrarater reliability The solution – training raters • Understanding criteria for assessment • Agreement with other raters • Consistency of performance

WASHBACK EFFECT

Washback Effect: effect of testing on teaching and learning Positive / negative washback Positive – test stimulates classroom teaching of important skills; Negative – narrow focus on teaching just for the test.