UNIVERSITY OF JYVÄSKYLÄ CEFLING The linguistic basis of

UNIVERSITY OF JYVÄSKYLÄ CEFLING The linguistic basis of the Common European Framework levels: Combining second language acquisition and language testing research

UNIVERSITY OF JYVÄSKYLÄ FUNDING 2007 - 2009 13 February, 2009 kmantyla@campus. jyu. fi

UNIVERSITY OF JYVÄSKYLÄ VENUE http: //www. jyu. fi/hum/laitokset/kielet/cefling/en Summer school 8. -11. 6. 2009, Jyväskylä n Department of Languages Finnish and English n Centre for Applied Language Studies n Department of Teacher Education n. SLATE-network Belgium, France (Univ. Of Paris III), Germany, Italy, The Netherlands, Sweden, UK

UNIVERSITY OF JYVÄSKYLÄ CEFLINGS RESEARCHERS n Riikka Alanen n Hannele Dufva n Ari Huhta n Paula Kalaja n Maisa Martin n Katja Mäntylä n Mirja Tarnanen PROJECT SECRETARIES n Laura Hartikainen n Riikka Purola 13 February, 2009 DOCTORAL STUDENTS n Taru Kynsijärvi n Helena Miettinen n Sanna Mustonen n Nina Reiman n Marja Seilonen n Henna Tossavainen MA STUDENTS n Anni Kettunen n Sarianna Kivilahti n Leena Nyyssönen n Vilja Paavola n Nina Pajunen n Minna Roiha n and others kmantyla@campus. jyu. fi

UNIVERSITY OF JYVÄSKYLÄ RESEARCH TOPIC How does foreign or second language proficiency develop? How could that development be described as stages of achievement? Focus: writing 13 February, 2009 kmantyla@campus. jyu. fi

UNIVERSITY OF JYVÄSKYLÄ Research question 1/4 What combinations of linguistic features characterise learners’ performance at the proficiency levels defined in the Common Framework and its Finnish adaptations? 13 February, 2009 kmantyla@campus. jyu. fi

UNIVERSITY OF JYVÄSKYLÄ Research question 2/4 To what extent do adult and young learners who engage in the same communicative tasks, at a given level, perform in the same way linguistically? To what extent are the adult-oriented CEFR levels and their Finnish adaptations for young learners equivalent? 13 February, 2009 kmantyla@campus. jyu. fi

UNIVERSITY OF JYVÄSKYLÄ Research question 3/4 To what extent are the pedagogical tasks found in the teaching materials in the Finnish comprehensive school comparable with the tasks defined in the CEFR and the new curriculum? 13 February, 2009 kmantyla@campus. jyu. fi

UNIVERSITY OF JYVÄSKYLÄ Research question 4/4 What are the linguistic and communicative features that teachers (or National Certificates raters) pay attention to when assessing learners with the help of the Finnish adaptations of the CEFR scales? How do these features relate to the linguistic and communicative analysis of the same performances? 13 February, 2009 kmantyla@campus. jyu. fi

UNIVERSITY OF JYVÄSKYLÄ RATIONALE CEF very influential in Finland: - school curricula - adult education curricula - National Certificates of Proficiency - citizenship requirements 13 February, 2009 kmantyla@campus. jyu. fi

UNIVERSITY OF JYVÄSKYLÄ SUBJECTS AND LANGUAGES n Written performances of adults taking the National Certificate of Proficiency examination (3 texts per subject) n Similar texts on similar tasks from young learners (grades 7 – 9, ages 13 -16) n L 2 Finnish – L 1 varies n L 2 English – L 1 Finnish or Swedish (possibly native speaker control groups) 13 February, 2009 kmantyla@campus. jyu. fi

UNIVERSITY OF JYVÄSKYLÄ RATING OF THE DATA n three – four trained raters per writing sample n inclusion for main data: complete interrater agreement or two in agreement, one + or – one level n Problems with certain levels – (not enough A 1 for adults and C 1 and C 2 for young writers) 13 February, 2009 kmantyla@campus. jyu. fi

UNIVERSITY OF JYVÄSKYLÄ CODING OF THE DATA n CHILDES (. chat format, CLAN tools) n Basic coding for all data, structural features as needed (several for Finnish, fewer for English at the moment) n Automatic analyses for English only 13 February, 2009 kmantyla@campus. jyu. fi

UNIVERSITY OF JYVÄSKYLÄ 13 February, 2009 kmantyla@campus. jyu. fi

UNIVERSITY OF JYVÄSKYLÄ RQ ¼: What combinations of linguistic features characterise learners’ performance at each level Finnish: sentence types extension of local case use development of infinitive phrases development of negation and morphological marking English: questions negative forms relative clauses agreement collocations derivative skills (wordformation) Finnish vs. English: formulaic sequences cohesive features 13 February, 2009 kmantyla@campus. jyu. fi

UNIVERSITY OF JYVÄSKYLÄ RQ 2/4 To what extent do adult and young learners perform in the same way linguistically? /To what extent are the adultoriented CEFR levels and their Finnish adaptations for young learners equivalent? n Raters use both scales in separate rounds of rating. Results are statistically compared and raters interviewed. CEFR scale A 1 -C 2 Finnish adaptation A 1. 1 - C 1. 1 (more linguistically oriented) 13 February, 2009 kmantyla@campus. jyu. fi

UNIVERSITY OF JYVÄSKYLÄ RQ ¾ To what extent are the pedagogical tasks found in the teaching materials in the Finnish comprehensive school comparable with the tasks defined in the CEFR and the new curriculum? n Textbooks and other teaching materials are analysed both for tasks and for the progression of structural skills. 13 February, 2009 kmantyla@campus. jyu. fi

UNIVERSITY OF JYVÄSKYLÄ RQ 4/4 What are the linguistic and communicative features that teachers/trained raters pay attention to when assessing learners/ How do these features relate to the linguistic and communicative analysis of the same performances? n Teachers and trained raters are systematically interviewed re. their arguments for certain level, for decision making between levels etc. 13 February, 2009 kmantyla@campus. jyu. fi

UNIVERSITY OF JYVÄSKYLÄ Plans for the future n A longitudinal study to complement the cross-sectional data n http: //www. jyu. fi/hum/laitokset/kielet/kesakoulu 2009 Language acquisition and assessment 8 -11 June, 2009 Plenary speakers Charles Alderson, Lancaster University, UK James Milton, Swansea University, UK. Gabriele Pallotti, University of Modena and Reggio Emilia, Italy Rob Schoonen, University of Amsterdam, The Netherlands Ineke Vedder, University of Amsterdam, The Netherlands 13 February, 2009 kmantyla@campus. jyu. fi

UNIVERSITY OF JYVÄSKYLÄ An example of what we’ve done Assessing lexical awareness: EFL learners and English word-formation (together with Ari Huhta, ari. huhta@campus. jyu. fi) n How to test word formation skills? How do the three methods n n used in the study function? What is the relationship between the word-formation skills and overall written proficiency of Finnish school pupils ? (What kind of knowledge do they have on English wordformation? ) (How do their word-formation skills develop? ) (Is there any difference between Finnish and Swedish speaking participants? ) 13 February, 2009 kmantyla@campus. jyu. fi

UNIVERSITY OF JYVÄSKYLÄ Word-formation and SLA n Word-formation and SLA in general n Role in teaching English in a Finnish school? n Derivation chosen because • Productivity of the method • The participants familiar with it at least implicitly 13 February, 2009 kmantyla@campus. jyu. fi

UNIVERSITY OF JYVÄSKYLÄ Participants n 7 th graders • 13 -14 –year-olds, have studied English as a FL for at least 4 years • 162 completed the word formation tests, 87 of whom also completed several writing tasks • (next step: 8 th and 9 th graders) n Different parts of Finland 13 February, 2009 kmantyla@campus. jyu. fi

UNIVERSITY OF JYVÄSKYLÄ Word-formation test 1 n Three written word-formation tests (revised after piloting) 1. Sentences / sentence pairs in English with a Finnish translation of the target word (productive gap-filling): – – I am ____ (varma) that he will get the job in London. He will _____ (varmasti) get the job in London. – sure - surely 13 February, 2009 kmantyla@campus. jyu. fi

UNIVERSITY OF JYVÄSKYLÄ Word-formation test 2 (final version) 2. Sentences with non-words with explanations in Finnish (gap-filling): • Example in Finnish (with Finnish real words) – She could bourble animals very well because she was a good ____ bourble____. (henkilö, joka tekee lihavoidun sanan kuvaamaa toimintaa/työtä) – (a person who does the action described by the bolded word) 13 February, 2009 kmantyla@campus. jyu. fi

UNIVERSITY OF JYVÄSKYLÄ Word-formation test 3 – 3. A list of prefixes from which the participants were to choose suitable ones to fill in the gaps in sentences anti- de- disinim- – – ilirinterintramega- minimismononeo- nonpolypostpre- He did not follow the instructions. He had ___ understood them. 13 February, 2009 kmantyla@campus. jyu. fi proretransun-

UNIVERSITY OF JYVÄSKYLÄ Writing tasks Ø Email to a friend Ø Email to one’s teacher Ø Email to a store Ø Opinion piece Ø Narrative piece 13 February, 2009 kmantyla@campus. jyu. fi

UNIVERSITY OF JYVÄSKYLÄ Marking word-formation tests n Double marking n Scoring: – Productive gap-fill test : 0 -1 -2 -3 -4 Non-words based test and List-choice based test: 0 -1 -2 n The respondents were very creative: minigabl n Spelling errors more or less ignored in scoring: unbelievubl, unbelievevabl, unbelievobl; understant, anderstand n (cf. shore sure, deffreno different) 13 February, 2009 kmantyla@campus. jyu. fi

UNIVERSITY OF JYVÄSKYLÄ Analysis of the word formation tests n Tia. Plus programme (CITO, the Netherlands) – for classical item analyses – for norm-referenced tests 13 February, 2009 kmantyla@campus. jyu. fi

UNIVERSITY OF JYVÄSKYLÄ Characteristics of the 3 tests Items Mean Standard Std. score Deviat. Error of (percent) ion Mean Cronbach’s Alpha for Average 40 -item /total test correlation 19 70% 21. 0 1. 6 . 87 . 93 . 55 Non-words 8 based test 31. 5% (27%) 23. 6 (24. 4) 2. 0 (1. 9) . 70 (. 74) . 92 (. 93) . 58 (. 61) List-choice 12 based test 36% 20. 6 1. 6 . 74 . 90 . 51 All 3 tests together 51% . 90 . 91 (. 47) Productive gap-fill test 39 13 February, 2009 kmantyla@campus. jyu. fi

UNIVERSITY OF JYVÄSKYLÄ Correlation between word-formation tests and the rating of writing skill WRITING SKILL (on CEFR scale) Mean rating across 4 raters and all tasks completed by student Productive gap-fill test . 691 - items tapping the base form of word (9) . 578 (. 589, if item A 17 removed) - items tapping the . 713 inflected form of word (10) Non-words based test . 575 (n = 76) List-based test . 675 n = 87 p =. 000 13 February, 2009 kmantyla@campus. jyu. fi

UNIVERSITY OF JYVÄSKYLÄ Mean word-formation test scores (%) across CEFR levels (based on writing) A 1 (n= 27) A 2 (n= 42) B 1 (n= 15) B 2 (n= 2) Productive gap-fill test 46 75 87 100 - items tapping the base form of word (9) 57 78 84 100 - items tapping the inflected form of word (10) 37 73 90 100 Non-words based test 13 28 38 81 List-based test 15 32 47 71 13 February, 2009 kmantyla@campus. jyu. fi

UNIVERSITY OF JYVÄSKYLÄ Correlations between word-formation tests N = 140 / 162 A (total) A 1. items A 2. items Productive tapping the gap-fill test base form inflected form A. Productive gap-fill test 1. 00 A 1. Items tapping the base form A 2. Items tapping the inflected form C. Listbased test (. 902) (. 966) . 596 . 615 1. 00 . 765 . 497 . 554 1. 00 . 555 . 596 1. 00 . 628 B. Non-words based test C. List-based test 13 February, 2009 B. Nonwords based test 1. 00 kmantyla@campus. jyu. fi

UNIVERSITY OF JYVÄSKYLÄ Comparison of the three wordformation test methods Pros Cons Productive gap-fill test very familiar test type memorising words? marking difficult (if very fine-tuned) the relationship between items Non-words based test focus only on wordformation relative difficulty unfamiliar test type List-based test quick and easy to take and mark focus on word-formation (relatively) memorising words? unfamiliar test type? difficult to write easy items? 13 February, 2009 kmantyla@campus. jyu. fi

UNIVERSITY OF JYVÄSKYLÄ Overall conclusions about the wordformation tests n Productive gap-fill: possibly useful for testing word-formation but more work needed to develop / select suitable items n Non-words test: promising (e. g. as reliable as the others; construct relevant) but there is a ’threshold’ to overcome for the test-taker, i. e. to understand what it is about n List-based gap-fill: promising (fairly construct relevant), but the difficulty of the words need to suit students’ level better n Which level of affixes/words? Relationship with the word frequency? 13 February, 2009 kmantyla@campus. jyu. fi