Promoting Sustainable Excellence in Testing and Assessment of

Promoting Sustainable Excellence in Testing and Assessment of English Financed by the TEMPUS programme of the European Union MODULE 1 UNIT 7 1

Assessing skills: Writing: Purpose and Techniques § Lecture materials were compiled by § Professors Elena Solovova and Maria Verbitskaya (Moscow State University); § Olga Simonova and Inna Chmykh (Surgut State University) § Some slides have been taken from Dr. Tony Green’s presentation (CRELLA)

Plan of the lecture § 1. Why teach writing? § 2. Stages of teaching writing. Writing: Process VS Product. § 3. Why test writing? Types of tests & their purpose for different test users. § 4. Writing test specification. § 5. Rating scales in assessing writing skills 3

Why teach writing? 1. Children (K-11): § writing as a means of learning; § writing as a Process; § writing as a Product; 2. Adults: § socio-cultural experience (travelling, migration, etc. ); § academic experience (furthering one’s education); § professional experience (career development). 4

Stages of learning & teaching writing 5 Graphics (level of letters/their combinations) Spelling (word level) Note-taking (words/chunks/sentences/plan, etc. ) Writing as a communicative skill

Micro skills (Brown H. Douglas Language assessment. Principles and Classroom practices p/221) 1. produce graphemes and orthographic patters 2. produce writing at an efficient speed to suit the purpose 3. produce an acceptable core of words and use appropriate word-order patterns 4. use appropriate grammar system (tense, agreement, pluralisation, etc. ) 5. express a particular meaning in different grammar forms. 6. use cohesive devices in written discourse. 6

Task types 7 Copying Dictations Gap filling (word/ word combination/ sentence level) Scrambled words/ sentences Sequencing/ ordering Find and correct the mistakes Short answer and sentence completion MCQ MM Paraphrasing Grammatical transformations Vocabulary exercises Cloze Picture-cued tasks

Macro skills 1. Use the rhetorical form and conventions of written discourse. 2. Appropriately accomplish the communicative functions of written texts according to form and purpose. 3. Convey links and connections between events. 4. Communicate the main idea and supporting ideas. 5. Communicate given information and the new information, generalization and exemplification. 6. Distinguish between literal and implied meanings when writing. 7. Correctly convey culturally specific references in the context of the written text. 8. develop and use a battery of writing strategies such as: assessing the audience’s interpretation; using pre-writing devices; writing the first draft; using paraphrases and synonyms; seeking peer and instructor’s feedback; using feedback for revising and editing 8

Types of tasks 9 Copying Dictations Gap filling (word/ word combination/ sentence level) Scrambled words/ sentences Sequencing/ ordering Find and correct the mistakes Short answer and sentence completion MCQ MM Paraphrasing Grammatical transformations Vocabulary exercises Cloze Picture-cued tasks

Writing : Process vs Process § drill /controlled /guided ex-s; § brainstorming, planning, all draft materials. Product § different types of written texts within EGP, EAP, ESP language courses (letters/ e-mails/ 10 questionnaires, transactional letters; memos/ minutes/ business plan; essays, reports, articles, presentations, etc. )

Genres of written language and text types (Brown H. Douglas Language assessment. Principles and Classroom practices p/219) § 1. Academic writing: general subject reports, essays, compositions, term/ § § 11 course papers, academically focused journals, short-answer test responses; technical reports (lab reports); theses, dissertation, etc. 2. Job-related writing: messages/ phone messages; letters/e-mails/ transactional letters; memos; minutes; reports( job evaluation, project reports); schedules, labels, signs; announcements, advertisements; manuals, etc. 3. Personal writing: letters, emails, greeting, cards, invitations; messages, notes, reminders, shopping lists; financial documents (checks, tax forms, loan applications); forms, questionnaires, medical reports, immigration documents; diaries, personal journals; fiction, etc.

Assessing writing : Key questions What is Writing? Why assess Writing? Construct Task Types How (Test)? Purpose Scoring criteria How (Score)?

Why assess Writing? No single answer – Different groups of language learners have different needs e. g. some groups of adult language learners – – International travellers: language for travel, leisure Migrants: survival skills, access to employment Students: notes, essays, dissertations Professionals: emails, reports, letters, memos – Different users have different purposes when they seek information from tests – But most users of language do need to write

14 Why should assess writing? SOCILAL NEEDS International travellers: language for travel, leisure Migrants: survival skills, access to employment Professionals: emails, reports, letters, memos 2. ACADEMIC NEEDS Students: mobility, exchange programs, notes, essays, course works, dissertations National exams 3. EDUCATIONAL NEEDS Teachers: monitoring, control, assessment Learners : progress check, skills development

15 Language Competence Organisational competence Grammatical competence vocabulary syntax graphology Textual competence cohesion rhetorical organisation Pragmatic competence Functional competence ideational functions manipulative functions heuristic functions imaginative functions Sociolinguistic competence dialects/ varieties registers natural or idiomatic expressions cultural refs & figures of speech

Performance testing in writing § samples of written language that simulate reallife behaviour (are authentic in situation and interaction); § complex events involving unpredictable variables; § observed & evaluated by agreed judging process, where facets of measurement interact to produce score § (Mc. Namara, 1996) 16

Writing test specifications (after Christine Coombe) § § § § 17 Topic Text type Length Areas to be assessed Timing Weighting Pass level ?

Parts of writing test tasks § § 18 Rubric Prompt/ input Expected response Post-task evaluation

19 Rubric the instruction and the set of criteria for evaluation of the writing task, that mostly come from specification It is supposed to: Ø Specify a rhetorical pattern, length, time; Ø Indicate the resources available for students (dictionaries, spell checker, etc. ) and the delivery method (paper and pencil, etc. ) Ø Indicate whether a draft or an outline is required; Ø Include the overall weighting of the writing task in the exam (Christine Coombe (2010)

Writing prompt the stimulus the students must respond to’ Hyland (2003 p. 221); 3 types of prompt patterns Kroll and Reid (1994 p. 233) Ø Base prompt (see the handouts) Ø Framed prompt (see the handouts) Ø Text-based prompt (see the handouts) 20

Expected response and relationship between input and outcome (Bachman & Palmer, 1996) § § § Length Purpose Text type Functions Linguistic features § Input-outcome interaction Ø Reactivity: degree of reciprocity; Ø Scope: amount of input needed to respond; Ø Directness : direct/ indirect correlation / real world knowledge involved apart from the input. 21

Task analysis Academic Writing Task 2 Upbringing WRITING TASK 2 You should spend about 40 minutes on this task, Write about the following topic: Children who are brought up in families that do not have large amounts of money are better prepared to deal with the problems of adult life than children brought up by wealthy parents. To what extent do you agree or disagree with the opinion? Give reasons for your answer and include any relevant examples from your own knowledge and experience. Write at least 250 words. 22

Rating scales Ø Ø Ø 23 § Holistic (see handouts) Cambridge ESOL +fast (practicality) – not easy to interpret/ remember all to arrive at score (see handouts) § Analytical (see handouts) Parameters (task fulfillment, content, organization, cohesion, register, accuracy, etc. ) and Criteria (3 -2 -1 -0 and their description) +clear, explicit, detailed diagnostic info, reliable, good for guiding learners/-time consuming

Holistic rating scales § Positive features – Practicality: fast – Rating holistically may be more naturalistic § Disadvantages – No useful diagnostic information: single score – Not always easy to interpret: raters not required to use same criteria to arrive at score

Analytic rating scales § Positive features – Can provide diagnostic information if scores reported separately – Potentially clear, explicit and detailed – Usually more reliable (multiple scores) – Useful in training raters to focus on our construct – Potentially useful in guiding learners § Disadvantages – Time-consuming – May overburden raters

How to reduce subjectivity § § § § 26 specification of the content of the assessment adopting standard procedures basing judgments on specific defined criteria requiring multiple judgments undertaking appropriate training moderation of judgments checking validity, reliability by analyzing assessment data.

THANKS FOR YOUR ATTENTION! 27