Developing a task-based assessment of EAP pragmatics Soo

Developing a task-based assessment of EAP pragmatics Soo Jung. Youn (soojung@hawaii. edu ) University of Hawai‘i at Mānoa 3 rd International Conference on Task-Based Language Teaching Lancaster University, UK

Investigating EAP Pragmatics Even with high L 2 proficiency, L 2 are in great “the study of language from the point of view of the users, need of especially of the choices they make, the constraints they encounter L 2 pragmatics in sessions, Advising using language in social interaction, and the effects their use of instruction language has on other participants e-mails to professor, the. A longitudinal in the act of little attention communication” case study of p. 301, italics added) on systematic (Crystal, 1997, e-mail writing tutorials literacy dvlpmt (Bardovi-Harlig investigation of & Hartford, (Chen, 2006) learning needs 1990, 1996, 2005) “referring to nonnative speakers’ comprehension and production of + speech acts, and how that L 2 -related speech act knowledge is pedagogic & acquired” (Kasper & Dahl, 1991, p. 216) assessment 2 tasks

Assessment of Pragmatics • Development of six prototype pragmatic measures and rating scales (Hudson, Detmer, & Brown, 1992, 1995) • Development of general pragmatic tests (Roever, 2005, 2006; Liu, 2007; Tada, 2005) Remaining Concerns: 1) how the test batteries are being used and their pedagogical value 2) not fully aligned with pragmatic needs of a specific context 3)How can these concerns be addressed? 3

Task-based Language Assessment • Fundamental Question: Why and how are task-based assessment being used in particular L 2 educational contexts? (Norris, 2002, 2008) Deal with complex, integrative, task-specific needs • Examples of empirical TBLA studies DSL in Belgium Reform an entire for high-stakes foreign language University-level decisions with curriculum second & foreign summative uses - both pedagogical TBLA of TBLA (Norris et al. , 1998; & assessment Brown et al. , 2002) (Gysen & Van Avermaet, 2005) (Byrnes, 2002) TBLA • Provide learning opportunities • Evaluate effectiveness • Rethink the program … 4

Study Purpose & Context • To engage in the process of identifying EAP pragmatic needs, specifying intended use of assessment tasks, developing assessment tasks & rating scales • EAP program context: - English Language Institute (ELI) at the University of Hawai‘i at Mānoa - provides academic English instruction for international & immigrant & (U. S. citizens) – in need of assistance with EAP - academic skill-based curriculum 5

Research Questions (a) What are major EAP pragmatic needs as perceived by students, instructors, and the program? (b) What are the intended uses of assessments of EAP pragmatics within a university EAP program? (c) What kinds of assessment tasks and rating criteria can be developed to meet both the EAP pragmatic learning needs and intended assessment uses in the target program? 6

Study Procedure 1) Needs analysis 2) A specification of intended uses of EAP pragmatic assessment 3) Development of pragmatic learning outcomes proposal 4) Development of pragmatic target tasks 5) Development and validation of EAP pragmatic assessment tasks and task-dependent rating scales 7

Needs Analysis - Procedure Semi-structured interviews 12 ELI Students a) Focus group b) Interviews Questionnaire synthesis 20 EAP pragmatics-related situations 102 ELI Students 7 ELI Instructors 3 ELI Administrators (Grad/UG: 43. 1%/ 56. 9%) 8

Questionnaire Main Findings • 1(not at all necessary)-2 -3 -4 (very necessary) scale: “learning need” • Cronbach alpha = 0. 9 • Lowest mean = 2. 9 (out of 4) • Five most needed tasks (by students) 1. Refuse politely to professor’s request 2. Request a recommendation letter 3. Write a cover letter to apply for a job 4. Write an appropriate e-mail to professor 5. Ask clarifying questions to professors appropriately 9

Any gaps among informants? • Do all informants mention same needs? • Among the least needed tasks by students. . Know how to appropriately comment, criticize, and compliment on classmates’ in class work But, mentioned and emphasized by ELI teachers!! 10

Graduate vs. Undergraduate Students Recommendation letter request Write an e-mail Write a cover letter 11

Across Different Levels Recommendation letter request Write a cover letter 12

Intended Uses of EAP Pragmatic Assessment What is being Who use? • ELI students • ELI instructors • Prospective instructors & students assessed? • Overall pragmatic ability with a range of EAP tasks that place common pragmatic demands on students Why assessment? Who/What is being impacted? • Not intended to reform, but strengthen the program • Satisfaction of needs • Evaluate effectiveness • Professional development • Raising awareness • • Diagnose Self-assessment Pedagogical tasks Measure students’ learning progress and achievement • Teaching materials 13

Assessment Tasks 2. Write an e-mail to a potential 3. Write an e-mail employer to refuse professor’s to send application request packet 1. Write a 2. recommendation 4. Write constructive 3. request e-mail comments on cover letter 4. to professor based on identified written by classmate 5. Task 1 target tasks: 7. Role-play with a professor in situations of making requests and refusal 6. Role-play with a classmate in situations of making suggestions and disagreement 5. Give oral peer-feedback on classmate’s request e-mail to professor 14

Task-Dependent Rating Criteria • Analytical rating criteria with detailed descriptions • Each task has different rating criteria • Feedback from domain experts • 3 (good) – 2 (able) – 1 (inadequate) e. g. ) 1. Write a recommendation letter request e-mail to professor Task rating criteria 15

Procedure Examinee Participation • 40 students participated (low – high proficiency) • Individually completed all 7 tasks (about 1 hour to 1 hour 15 min) • Either use a computer or hand-write • Role-play audio-recorded Rating • 3 raters (experienced ESL/EFL teachers with MA in ESL degrees) • Had three consecutive separate training sessions • Asked to keep monitoring, to document any difficulties, reasoning of rating 16

FACETS summary Write a constructive comment on cover ltr written by a classmate most difficult 0. 90 easiest Role-play with a professor (request, refusal) 17

Concluding Marks - so what? • Ongoing process • Future investigations (e. g. , how these assessment are actually put to use, helpful to teachers & learners? , impact on the program) • Need more empirical studies (e. g. , do actually assessment tasks promote L 2 pragmatic learning? If so, how? ) • Rethink roles of assessment 18

Mahalo! (soojung@hawaii. edu) Sincere appreciation to: Dr. John M. Norris & ELI, HELP, Research Participants Funded by University of Hawai‘i at Mānoa Art & Sciences Student Research Award Graduate Student Organization Grant 19

Reference Bardovi-Harlig, K. , & Hartford, B. S. (1990). Congruence in native and nonnative conversations: Status balance in the academic advising session, Language Learning, 40, 467 -501. Bardovi-Harlig, K. , & Hartford, B. S. (1996). Input in an institutional setting, Studies in Second Language Acquisition, 18, 171 -188. Bardovi-Harlig, K. , & Hartford, B. S. (Eds. ) (2005). Interlanguage pragmatics: Exploring institutional talk. Mahwah, New Jersey: Lawrence Erlbaum Associates. Brown, J. D. , Hudson, T. , Norris, J. , & Bonk, W. J. (2002). An investigation of second language taskbased performance assessments. Honolulu, HI: Second Language Teaching and Curriculum Center. Byrnes, H. (2002). The role of task and task-based assessment in a content-oriented collegiate foreign language curriculum. Language Testing, 19, 419 -437. Chen, C. -F. E. (2006) The development of e-mail literacy: From writing to peers to writing to authority figures. Language Learning & Technology, 10(2), 35 -55. Gysen, S. , & Van Avermaet, P. (2005). Issues in functional language performance assessment: The case of the certificate Dutch as a foreign language, Language Assessment Quarterly, 2, 51 -68. Hudson, T. , Detmer, E. , & Brown, J. D. (1992). A framework for testing cross-cultural pragmatics (Technical Report #2). Honolulu: University of Hawai‘i, Second Language Teaching and Curriculum Center. Hudson, T. , Detmer, E. , & Brown, J. D. (1995). Developing prototypic measures of cross-cultural pragmatics (Technical Report #7). Honolulu: University of Hawai‘i, Second Language Teaching and Curriculum Center. Liu, J. (2007). Developing a pragmatic test for Chinese EFL learners. Language Testing, 24, 391 -415. Norris, J. M. (2000). Purposeful language assessment. English Teaching Forum, 38(1), 18 -23. Norris, J. M. (2002). Interpretations, intended uses and designs in task-based language assessment, Language Testing, 19, 337 -346. Norris, J. M. , Brown, J. D. , Hudson, T. , & Yoshioka, J. (1998). Designing second language performance assessments. Technical Report #18. Honolulu, HI: Second Language Teaching and Curriculum Center. Roever, C. (2006). Validation of a web-based test of ESL pragmalinguistics. Language Testing, 23, 229 -256. 20