NOTE This Power Point presentation has been modified

NOTE: This Power. Point presentation has been modified by removal of all high-resolution graphics, to reduce the storage and downloading requirements (sorry, no pretty photos!). Evaluating Task-Based Language Programs Colloquium – TBLT 2009

The colloquium Why bother with TBLT program evaluation? Three presentations + clarification questions (2: 00 -3: 30): Re-framing the evaluation of task-based language education Evaluating a TBLT Spanish immersion program Evaluation of TBLT in Flanders Open audience-panel discussion (3: 30 -3: 50)

John M. Norris University of Hawai‘i at Mānoa TBLT 2009 Please cite as: Norris, J. M. (2009, September). Reframing the evaluation of task-based language education. Paper presented at the refereed colloquium “Evaluating task -based language programs”, at the 3 rd International Conference on Task-Based Language Teaching, Lancaster, UK (September 14, 2009). Re-framing the evaluation of task-based language education

TBLL v. TBLT: Disconnects between inquiry and practice

What is task-based language learning (TBLL)? 1. Societal need for change in language education… Value Outcomes Methods 2. Emerging notions of L 2 acquisition… Processes Impediments Indicators

What is task-based language learning (TBLL)? Practices 1. Societal need for change in language education… Value Observations Outcomes Methods 2. Emerging notions of L 2 acquisition… Proposals Discussions Hypotheses Processes Impediments Indicators Findings Opportunity for a researched language pedagogy

What is task-based language learning (TBLL)? Pedagogic principles, such as… Promote learning by doing, experiential learning Use task as the unit of analysis for instruction & assessment Provide rich L 2 input Elaborate (rather than simplify) L 2 input Respect learner-internal syllabuses Enable inductive/chunk learning Promote collaborative-cooperative interaction Provide focus on form, negative feedback (e. g. , Doughty & Long, 2003; Ellis, 2003)

What is task-based language teaching (TBLT)? Rationales and Principles Philosophy of Education Cognitive Psychology Sociocultural theory SLA Applied to Task-Based Language Teaching Learners Needs Curriculum Instruction Materials Curriculum theory Planning and policy L 2 Education Programs Assessment Inform? ? ? Teacher development

What is the role of task-based inquiry? GENERALIZABLE Generate theory Understand what works, where, when, & why SITUATED Test hypotheses Task-Based Language Learning Task-Based Language Teaching Improve teaching practice DISCRETE Discover robust, if small, truths Inform curriculum, course design HOLISTIC

Challenges for task-based inquiry 1. The scope of task-based research does not match the scope of our claims about (for or against) TBLT. 2. The focus of theoretical task-based research does not relate to the situated realities of task-based teaching. Needs Goals, outcomes Curriculum Scope, sequence Materials Instruction Teachers TBLT Education Programs Resources Practices History, training Learners Individual differences Assessment Intended uses, users

Framing TBLT inquiry through program evaluation

Inquiry through evaluation Research emphasizes theoretical, conclusion-oriented inquiry Evaluation operationalizes decisionoriented inquiry Cronbach & Suppes (1969)

Inquiry through evaluation Evaluation is the gathering of information about any of the variety of elements that constitute educational programs, for a variety of purposes that include primarily understanding, demonstrating, improving, and judging program value; evaluation brings evidence to bear on the problems of programs, but the nature of that evidence is not restricted to one particular methodology. Norris (2006) MLJ Perspectives Inquiry frame and focus Inquiry impetus Inquiry question prioritization

Inquiry through evaluation Cronbach et al. (1980) “The evaluator will be wise not to declare allegiance to either a quantitative-scientificsummative methodology or a qualitativenaturalistic-descriptive methodology. ” (p. 7) Paradigms Epistemology 1 Methodology 1 Epistemology 2 Methodology 2

Inquiry through evaluation Pragmatism Cronbach et al. (1980) “The evaluator will be wise not to declare allegiance to either a quantitative-scientificsummative methodology or a qualitativenaturalistic-descriptive methodology. ” (p. 7) Who? What? Why? Method 1 Method 4 When? Method 2 Method 3 Method 5

Inquiry through evaluation 1. Participation – stakeholders, representatives, primary intended users 2. Prioritization – challenges, questions in immediate need of answers 3. Instrumentation – what data will answer the questions? 4. Collection – how can we get data in available time/resources? 5. Interpretation – what do findings mean in context? 6. Utilization – what decisions & actions are taken? Participation by language educators is essential A educators are ultimately uses for Languagefocus on specific intendedresponsible throughout evaluation if contextual relevance forevaluation findings is essential from the what happens in language education. is is to make any difference. outset, if evaluationsought.

Context: Program features Learner needs, institutional resources, program goals and outcomes, curriculum, materials, instruction, assessment, teachers, teacher development, learners, etc. Context: Intended uses Evaluating TBLT Programs Understand Improve in situ Educate Demonstrate worth Hold accountable Questions + Methods Empower Values clarification (Test theory) Implementation Process-product Outcomes Context: Intended users Teachers Administrators Curriculum writers Learners Parents/public Funders (Researchers) Patton (1997) Utilization -focused evaluation

Learning from evaluation examples: the brief history of TBLT

Learning from TBLT evaluation Evaluating the Communicational Teaching Project – Prabhu’s “Bangalore Project” (See Prabhu, 1987) Context English L 2 Education in Bangalore, India; Seeking Improvement via Innovation Theory L 2 learning by processing meaning; Unconscious grammar construction by learners Program Project/taskbased work; 4 experimental schools; Implemented 1979 -1984

Learning from TBLT evaluation Evaluating the Communicational Teaching Project – Phase 1 (See Beretta & Davies, 1985) Initial inquiry, final year of the project: “To assess, through appropriate tests, whethere is any demonstrable difference in terms of attainment in English between classes of children who have been taught on the CT project and their peers who have received normal instruction in the respective schools. ” Beretta & Davies (1985)

Learning from TBLT evaluation Evaluating the Communicational Teaching Project – Phase 1 (See Beretta & Davies, 1985) Purpose Test theory Demonstrate method effectiveness Findings Claims? ? ? Structures test: Control > CTP Task-based learners achieved as much or more than traditional on all but the least functional outcomes What do we really know? ? ? Methods Contextual grammar: Control = CTP Quasi-experimentation Dictation: Control = CTP Class/method comparison List/Read comp: CTP > Control Outcome achievement assessments Task-based test: CTP > Control Task-based instruction is successful Warranted claims? ? ?

Learning from TBLT evaluation Evaluating the Communicational Teaching Project – Phase 2 (See Beretta, 1986, 1990, 1992) Purpose Findings Understand program implementation Lack of comparability (intact classes, no baseline data) Illuminate relation with apparent outcomes Methods Retrospective interview protocols Teacher level of concern questionnaires Document analysis More qualified teachers in CTP classes Implementation of CTP highly variable (over time, between classes, with structures) More confident teachers = better results

Learning from TBLT evaluation Evaluating the Communicational Teaching Project – Lessons Learned (See Beretta, 1992) Theory testing, methods comparisons, “what works” claims are rarely feasible in real educational programs Apparent differences in learning achievement, behaviors, etc. can only be explained by observation of multiple factors as they are experienced in real program contexts Even poorly executed evaluations (e. g. , post-hoc) can shed light on how programs function and help explain why learning does or does not occur Teachers (beliefs, training, commitment, time) play a key role in implementing programs: what they actually do must be understood

Learning from TBLT evaluation Evaluating a university French curriculum – Student perspectives (See Towell & Tomlinson, 1999) Context French FL Education, UK university, Salford; Restructuring advanced FL teaching Theory Input, text, task; Learning through formfunction mapping in tasks Program Task-based syllabus; Multiple levels at university; Implemented 1988 -96

Learning from TBLT evaluation Evaluating a university French curriculum – Student perspectives (See Towell & Tomlinson, 1999) Multiple iterations of development, implementation, evaluation, revision: “Curriculum design, evaluation, application and enhancement is a slow process, and subject to a number of extraneous influences which make it impossible to measure with totally scientific precision…use of diaries and questionnaires on the first occasion enabled a number of lessons to be learned and these helped considerably in creating a second application where the testimony of the student population through a detailed questionnaire shows the success of the operation. ” Towell & Tomlinson (1999)

Learning from TBLT evaluation Program: Initial TBLT long group projects Program: Revised TBLT staged, short projects Methods: S T A G E • Learner diaries 1 Findings: • Learner surveys • Assessments/exams S T A G E • Learner surveys 2 Findings: • Focus groups • Assessments/exams • Projects too long (6 wks) • Increased satisfaction • Training in group work • Higher learning of skills • Staged task objectives • Improved oral translation • Gains in text/task learning • Written translation? • Developing accuracy?

Learning from TBLT evaluation Evaluating a university French curriculum – Student perspectives (See Towell & Tomlinson, 1999) Student/learner perspective on teaching with tasks sheds important light on the realities of implementation (how + how well) Building evaluation activities into curricular delivery from the outset (e. g. , student diaries), enables longitudinal insights about change, development, response to instruction Triangulated learner feedback (diaries, self-assessments, questionnaires, exams) can lead to effective improvements in curriculum and task design, and in turn to higher evaluations Learners can change how they learn—acculturating to TBLT—especially when tasks, instructions, assessments are intentionally designed and staged to do so

Learning from TBLT evaluation Evaluating TBLT for EAP – Developmental evaluation in Thailand (See Mc. Donough & Chaikitmongkol, 2007) Context English FL education, Thai university; Improving EAP instruction Theory Integrated-skills, communication; Life-long learning; Learner needs + interest orientation Program Task-based syllabus; English department; Implemented 12 months

Learning from TBLT evaluation Evaluating TBLT for EAP – Developmental evaluation in Thailand (See Mc. Donough & Chaikitmongkol, 2007) Inquiry for developing and improving TBLT experiences: “…relatively few empirical studies have documented how teachers and learners react to entirely task-based courses, as opposed to the use of individual tasks…The purposes of this case study were (a) to identify teacher and learner reactions to the course and (b) to describe how their concerns, if any, were addressed. ” Mc. Donough & Chaikitmongkol (2007)

Learning from TBLT evaluation Evaluating TBLT for EAP – Developmental evaluation in Thailand (See Mc. Donough & Chaikitmongkol, 2007) Methods: Findings: Uses: • Learner task evaluations (repeated) • Increased learner independence, language skills, learning strategies • Intro unit on language learning • Learning notebooks • Decreased grammar obsession • Class observations • Non-specific real-world relevance • Teacher’s guide to instruction + workshop • Need time to adjust (T&L) • Enhanced task guidelines, built-in feedback opportunities • Teacher/student interviews • More support, guidance from teachers • Reduced number of tasks • Observer field notes • Too much to cover, disparate materials • Consolidated materials • Student course evaluations

Learning from TBLT evaluation Evaluating TBLT for EAP – Developmental evaluation in Thailand (See Mc. Donough & Chaikitmongkol, 2007) Cycles of evaluation planned into TBLT innovation, and carried throughout, can lead to increased likelihood of effectiveness Systematic evaluation (a) from multiple stakeholder perspectives and (b) focused on multiple program elements (materials, preparedness, outcomes) enables balanced change Teachers and learners both require support in implementing TBLT, especially during early phases of introducing task-based instruction TBLT based on learner needs can work well in EAP contexts, especially when evaluation is used to support ongoing effectiveness of delivery from the outset

Learning from TBLT evaluation Evaluating TBLT teacher training – Cyclical evaluation in Belgium (See Van den Branden, 2006) Context Dutch SL Education in Flanders; Nationwide; K-16; Ensuring Educational Access, Equity Theory Large-scale Task-Based LT Innovation; Improving Functional DSL Abilities Program School-based Teacher. Training Programs; Enabling Change, 1994 -2003

Learning from TBLT evaluation Van den Branden (2006): “…the teacher tries to act as a true interactional partner, negotiating meaning and content with the students, eliciting and encouraging their output, focusing on form when appropriate and offering them a rich, relevant and communicative input” (p. 217). Evaluation PROBLEMS Teacher cognition • What do they theorize about TBLT? • How do they learn about TBLT? Teacher action • Are they willing to change with syllabus? • How do they adopt/adapt TBLT in practice? Teaching context • What are the social constraints on T-Dev? • How can T-Dev be optimized? Evaluation USES Understand teachers Illuminate context Improve T-dev program Encourage teacher agency Ensure teaching success Enable TBLT learning Demonstrate outcomes

Learning from TBLT evaluation Program: Theoretical inservice training S Methods: T • Teacher survey A G • Training observation E Findings: 1 • Transmission model • Short term (3 hrs. ) • “Try that with my students”…Postcoursal depression! Program: TB training + syllabus support S T A G E 2 Methods: • Teacher logs, interviews, classroom observations Findings: +awareness of TBLT +student enthusiasm ? teacher adoption -teacher control -task complexity groupwork Program: Training + coaching + agency S T A G E 3 Methods: • Coaching obs, classroom obs, coach/ teacher interviews Findings: +conscious decisions +TBLT adaptation +self-evaluation ? teacher control transfer groupwork

Learning from TBLT evaluation Program: Sustained implementation of TBLT T-Dev with coaching, support S T A G E Methods: 4 +incorporation of TBLT correlated with higher Dutch L 2 proficiency outcomes • Pre-post student learning outcomes, teacher surveys, classroom observations Findings: +3 -year gains in DSL higher in TBLT intensive adopting schools ? mixed incorporation of TBLT across schools, teachers

Learning from TBLT evaluation Evaluating TBLT teacher training – Cyclical evaluation in Belgium (See Van den Branden, 2006) Long-term evaluation of TBLT sheds light on how ideas are implemented, how participants change, and what support is needed Multi-directional evaluation (political, social, school, individual) increases our capacity to explain why task-based ideas work or do not Persistent follow-through on evaluation findings (use) underlies effective innovation Teachers can learn to engage with TBLT, but change takes time, requires individualized support, and must be valued

Learning from TBLT evaluation What have we learned? Reframing evaluation in TBLT From summative to intentional From assessment-driven to multi-methodological From external to participatory From method-testing to program-illuminating From one-shot to longitudinal, cyclical From theoretical conclusions to educational decisions

Research, evaluation, and the future of task-based education

TBLL research Sociocultural, cognitive, and other theories provide useful starting points for thinking about language teaching and learning, and offer principles for building educational programs Task-based language learning research helps in that it raises our awareness about particular factors that we should pay attention to in the instructed L 2 learning process Task-based language learning research cannot tell us much about how or why language education programs work; findings from TBLL research should not be interpreted as direct implications for TBLT education

TBLT evaluation Answers questions & informs decisions of local interest Sheds light on how TBLT ideas work in practice Intentional Provides truths situated in rich contexts of programs Evaluative Relates outcomes to TBLT delivery and other factors Inquiry Focuses on scope that is meaningful to teachers, learners Tests and informs innovation on the ground, in situ Empowers participants to learn, and learn to change Forces an honest accounting of TBLT

TBLT evaluation Resources: It takes time and money to do evaluation well and to sustain it within L 2 educational programs. Challenges For TBLT Evaluation Training: Effective evaluation calls upon skills that may not be easily available among personnel at hand. Dissemination: There are few venues for publishing evaluation reports, thereby limiting learning. Actual uses: There are many possible uses/needs for evaluation that we are not sufficiently attuned to, yet.

Cheers! (Mahalo!) References Beretta, A. (1986). Program-fair language teaching evaluation. TESOL Quarterly, 20, 431 -445. Beretta, A. (1990). Implementation of the Bangalore Project. Applied Linguistics, 11(4), 321 -340. Beretta, A. (1992). What can be learned from the Bangalore evaluation? In J. C. Alderson and A. Beretta (eds. ), Evaluating second language education (pp. 250 -273). Cambridge: Cambridge University Press. Beretta, A. , & Davies, A. (1985). Evaluation of the Bangalore Project. ELT Journal, 39(2), 121 -127. Cronbach, L. J. , & Associates. (1980). Toward reform of program evaluation. San Francisco: Jossey-Bass. Cronbach, L. , & Suppes, (1969). Research for tomorrow's schools: Disciplined inquiry for education. New York: Macmillan, 1969. Doughty, C. , & Long, M. H. (2003). Optimal psycholinguistic environments for distance foreign language learning. Language Learning & Technology, 7, 50 -80. Ellis, R. (2003). Task-based language learning and teaching. New York: Oxford University Press. Mc. Donough, K. , & Chaikitmongkol, W. (2007). Teachers’ and learners’ reactions to a task-based EFL course in Thailand. TESOL Quarterly, 41(1), 107 -132. Norris, J. M. (2006). The why (and how) of student learning outcomes assessment in college FL education. Modern Language Journal, 90(4), 590 -597. Norris, J. M. (2009). Task-based teaching and testing. In M. Long and C. Doughty (Eds. ), Handbook of language teaching (pp. 578 -594). Cambridge: Blackwell. Patton, M. Q. (1997). Utilization-focused evaluation (3 rd ed. ). Thousand Oaks, CA: Sage. Prabhu, N. S. (1987). Second language pedagogy. Oxford University Press, Oxford. Towell, R. , & Tomlinson, P. (1999). Language curriculum development research at university level. Language Teaching Research, 3(1), 1 -32. Van den Branden, K. (2006). Training teachers: Task-based as well? In K. Van den Branden (ed. ), Task-based language teaching in practice (pp. 217 -273). Cambridge: Cambridge University Press. http: //www 2. hawaii. edu/~jnorris