Learning to value evaluation In language teaching John

Learning to value evaluation In language teaching John M. Norris University of Hawai´i at Mānoa

Dealing with change In language education

Language learning in crisis “We cannot afford to seek out foreign language skills after a terrorist attack occurs. The failures of communication and understanding have already done their damage. We must provide an ongoing commitment to language education and encourage knowledge of foreign languages and cultures. ” Daniel Akaka, U. S. Senator from Hawaii

Language learning in crisis The present state of language education and proficiency in Australia is seriously inadequate for our current and emerging needs, and far behind comparable levels in our peers and competitors. The size of this gap and the work and time required to close it has led some to describe the situation as a crisis.

Language learning in crisis At present, though, the English-speaking abilities of a large percentage of the population are inadequate, and this imposes restrictions on exchanges with foreigners and creates occasions when the ideas and opinions of Japanese people are not appropriately evaluated.

Language learning in crisis “We are lousy at foreign languages and shouldn’t be”

Language learning in crisis “Viva la English” Tribune Media: 10 -26 -2007 Andy Rooney “I think that the fewer languages we have, the better off civilization will be. ”

Language learning in crisis More than half of the world's 7, 000 languages are expected to die out by the end of the century BIOCULTURAL DIVERSITY? Most of what we know about species and ecosystems is not written down anywhere, it's only in people's heads David Harrison (2007) – Living Tongues Institute

Language learning in crisis “English language learners (ELLs) are the fastest growing student population in America. Today, one out of every nine students is learning English as a second language…By 2025, English language learners will make up one out of every four students in our classrooms. ” Margaret Spellings (2005) – U. S. Secretary of Education Number of children studying Chinese in U. S. schools: 24, 000 Number of children studying English in Chinese schools: 200, 000

Language learning in crisis Number of ‘newcomer’ children who need to study Japanese in schools in 1991 = 4, 463 Number of ‘newcomer’ children who need to study Japanese in schools in 2002 = 19, 764 (4 X) Foreign students in Japanese universities in 1985 = 19, 741 Foreign students in Japanese universities in 2006 = 100, 804 (5 X) Brazilians residing in Japan in 1986 = 2, 135 Brazilians residing in Japan in 2005 = 302, 080 (141 X)

Language learning in crisis Who are we teaching and what do they need to learn? Are we seeking survival skills, intercultural competence or quality of mind? Language Teaching & Learning Why bother with language learning at all; what does it offer? What’s the relationship between L 1, L 2, social, & What’s the most academic effective way to teach development? language(s)? How do we develop teachers to meet current demands?

Language learning in crisis In the U. S. … High-quality teachers MLA White Paper NCATE – TESOL/ACTFL Teacher Development Program Standards Foreign Languages and Higher Education: New Structures for a Changed World

Language learning in crisis MEXT “The Ministry of Education, Culture, Sports, Science and Technology (MEXT) shall settle on an action plan to improve English education within the fiscal year 2002 with the aim of fostering "Japanese with English abilities". As of the fiscal year 2003, the employment of excellent assistant language teachers of foreign nationality as regular teachers is to be promoted. ” In Japan …

Why worry about evaluation? “What we assess is what we value” --Lauren Resnick “How we choose to assess will determine what gets valued” --Norris

Why worry about evaluation? Top 3 sources of pressure for evaluation in U. S. college FL programs: 1. University administration 2. The dean “As part of its reaccreditation, the university has required all undergraduate programs to create and implement outcomes-oriented assessment plans. ” (survey respondent) 3. Accreditation process “Time-consuming. Takes away from the business of teaching. Many aspects of learning can’t be measured. ” (survey respondent)

Why worry about evaluation? All universities, junior colleges and colleges of technology conduct self-checks and self-evaluations concerning education and research All universities, junior colleges and colleges of technology are regularly evaluated and accredited by organizations certified by the Minister of Education, Culture, Sports, Science and Technology

Why worry about evaluation? “This interest was fuelled primarily by a reduction in public spending to cope with recession, internal and external pressures for performance improvement, and constant demand to review, renovate, and reform the way the public sector does business in order to achieve social accountability. ” Nagao, Kuji-Shikatani, & Love (2005) Founded in 2000

Why worry about evaluation? ニュース 公立中、英語で授業４％だけ　英語の授業の大半を英 語で行っている公立中学校は約４％にとどまり、「授 “Ministry survey finds 業の大半を英語で行う」とする文部科学省の目標に遠 schools' English く及ばないことが、同省の調査で分かった。 　また、 teaching falling short” ビジネス界で英語力の指標とされる「ＴＯＥＩＣ」で ７３０点以上の英語教員は、中学で１割、高校で２割 にも満たず、英語教育のお寒い実態が浮き彫りになった。 調査は今年２月、全国の公立中学校約１万２００校と、 公立高校約３８００校を対象に実施した。授業での英 語の活用実態を調べたのは初めて。

Why worry about evaluation? NO COLLEGE STUDENT CHILD LEFT BEHIND College Leaving Exam “Does the Spellings Commission think about language education at all? ” Michael Holquist (ADFL 2007 Summer Seminar West)

Why worry about evaluation? Rate. My Professors Quality Rating Categories ☺How easy? ☺How fair? ☺How good? HOW HOT? ? ?

Why worry about evaluation? Drake University language programs… • Low enrollments, student dissatisfaction, poor external reviews • Faculty refuse instructional development support • Faculty refuse to create strategic plan for improvement No acknowledgement of need to change No engagement with evaluation findings NO MORE LANGUAGE PROGRAMS, NO MORE FACULTY!

Why worry about evaluation? Despite such problems, U. S. college FL survey respondents desired increased use of evaluation for: 1. Understanding & improving program outcomes 2. Understanding & improving program functions 3. Improving FL education on the whole 4. Understanding & Improving the worth of the program 5. Raising awareness about FL programs

Why worry about evaluation? And some expressed a professional ethic to engage in evaluation: “We have a social and moral responsibility towards our students and towards society at large to state as clearly as we can what it is that we do for them and why what we do is valuable. ” (Anonymous survey respondent)

Evaluation as an agent of change Within this milieu, what is the role to be played by evaluation? Key Challenge: ØDefine ØDelimit ØDetract ØDismiss Value of Language Education Key Opportunity: ØEnable ØEnhance ØEngender ØEmpower

Traditions, trends, and the status quo In language program evaluation

Traditions of language program evaluation Epistemology “The experimenting society” Methods Measure L 2 outcomes, compare programs Program focus Products of program completion Evaluation purpose Summative: judge program worth Knowledge: identify best methods Evaluation use Perpetuate, fund, or shut down Evaluation practice Short-term, external, Jet-In Jet-Out Expert JIJOE

Traditions of language program evaluation …but why did the program fail (or succeed)? …but how can you conduct a rigorous experiment on actual classrooms, teachers, and learners? …but you cannot compare our program with their program—they are so different! …but you don’t really understand our program— you were just here for a week! …but that test really doesn’t measure what we teach! …but what can you tell me about what language teaching method really works the best for my learners in my school? …but what if we don’t have the option of closing down the program—how do we make improvements and what do we need to improve?

Traditions of language program evaluation Proliferation of language program evaluators (and texts)… • Alderson & Beretta (1992) Evaluating second language education • Rea-Dickins & Germaine (1992) Evaluation • Brown (1994) The elements of language curriculum • Weir & Roberts (1994) Evaluation in ELT • Lynch (1996) Language program evaluation Changing emphases in evaluation practice… • Focus on formative (improvement) purposes • Attention to process of language teaching/learning • Use of multiple methods (qualitative + quantitative +…) • Pragmatic problem-solving approach • Integration into curriculum and context

Traditions of language program evaluation Rea-Dickins (1994): “If evaluation in English Language Teaching is to be effective, we will see a stronger integration of evaluation within practice, as part of an individual’s professionalism, and an increase in collaborative activity where teachers (and other relevant participants) are actively engaged in the monitoring process. ” Importance of language teacher participation

Countervailing trends in evaluation Swender (2002), on FL teacher professional development: “After all, if teachers do not know how to measure what students can do with language, how will they be able to determine whether their students are measuring up to the expectations of the 21 st century”. Technocratic measurement problem

Countervailing trends in evaluation Web. CAPE Foreign Language Placement Exam Standardized assessment problem: One size fits most purposes & settings

Countervailing trends in evaluation IELTS TSE – SPEAK – SLEP – TOEIC Standardized assessment problem: One size fits most purposes & settings

Countervailing trends in evaluation The ‘measurement mindset’ in teacher practice: “Although this student still has problems with grammar, the ideas are there. He is working through the choices the community has about their need for a better water supply system. Hmm, this is difficult. I just wish his grammar errors weren’t so bad then I could give him an ‘A’. ” Bernie Mohan example, TBLT 2007 Conference

Countervailing trends in evaluation What is the appropriate proficiency level to adopt as a student learning outcome for the 2 -year language requirement? INTERMEDIATE - LOW ? ? ?

Countervailing trends in evaluation Accountability movement: using standardized tests to hold teachers and students to performance expectations “We’re going to stand strong on accountability” --Margaret Spellings, U. S. Secretary of Education “If you want to hold schools accountable and make sure they are learning, you have to test ” --Robert Black, spokesperson for the Gov. of Texas Test Misuse Problem

Countervailing trends in evaluation "There has been an explosion of mandates for more and more standardized tests with very little evidence to support their use" --Walter Haney of Boston College's Center for the Study of Testing, Evaluation and Educational Policy.

Countervailing trends in evaluation “Bilingual education ends up being monolingual education in the language of the high-stakes test, until the test is over. " --Deborah Palmer, University of Texas

Countervailing trends in evaluation “…parents are dealing with children vomiting on the morning of the tests …" --Gloria Pipkin, Florida Coalition for Assessment Reform

Countervailing trends in evaluation Student Learning Outcomes Assessment Outcomes embody the essential purpose of an educational program: developments in knowledge, skills, dispositions of learners Requires rethinking of educational programs as something more than the delivery of experiences or the exposure of learners to information Calls for articulation of curriculum and instruction in support of targeted outcomes, demands integrated thinking Provides a clear statement of educational program value; answers the question “How do you know? ” with evidence of educational effectiveness

Countervailing trends in evaluation Large public institution Accreditation pressures to assess learning We have to assess our “liberal studies” core, ASAP! Huge expenditure, $$$, time, effort How about an electronic portfolio? Thousands of student portfolios created BUT…

Countervailing trends in evaluation Electronic Portfolio …faculty didn’t understand it …students thought: 1. *!%#$!@* 2. waste of time …administrators wanted to do something with it but weren’t sure exactly what NEVER GOT USED

Countervailing trends in evaluation Barrington (2003), on outcomes assessment in the liberal arts: “To design and administer (intellectually honest) assessment plans that will measure such capabilities with a dozen or more standardized ‘learning objectives’ is next to impossible” leading to “pestilent repercussions” for the truly valued learning objectives that constitute the liberal arts, in that it “discourages teaching such skills because they are difficult to measure”. Perception problem

Traditions, trends, and the status quo Back where we Language program evaluation started Status quo 1. Focus on doing… 1. Not useful—not used! 2. Based on standardized testing 2. Potential negative washback, reductionism, waste 3. Reactive v. proactive praxis 3. Not relevant to curriculum & instruction, program values 4. Driven by external impetuses 4. Not ours—done to us, not for or with us 5. Technocratic measurement emphasis 5. Not perceived as worth the effort by faculty 6. Little scholarly investment 6. Minimal professional development

Re-envisioning evaluation in language education

Resolving terminological confusion Measurement is the consistent elicitation of quantifiable indicators of well-defined constructs via tests or related observation procedures; it emphasizes efficiency, objectivity, and technical aspects of construct validity. Norris (2006) MLJ Perspectives

Resolving terminological confusion Assessment is the systematic gathering of information about student learning in support of teaching and learning…It may be direct or indirect, objective or subjective, formal or informal, standardized or idiosyncratic…It provides locally useful information on learners and learning to those individuals responsible for doing something about it. Norris (2006) MLJ Perspectives

Resolving terminological confusion Evaluation is the gathering of information about any of the variety of elements that constitute educational programs, for a variety of purposes that include primarily understanding, demonstrating, improving, and judging program value; evaluation brings evidence to bear on the problems of programs, but the nature of that evidence is not restricted to one particular methodology. Norris (2006) MLJ Perspectives

The nature of useful evaluations Action oriented: Actions are taken based on evaluation findings Manageable & feasible: Adapted to available time and resources Pragmatic: Context relevant use & focus Participatory: Active involvement of key stakeholders Democratic: Negotiated decision making Responsive: Evaluation responds to Educational & Transformative: primary intended users’ Users learn by participating purposes Clear & understandable: Transparent processes and outcomes

Evaluative assessment Light (2001) on outcomes assessment: “…a process of evaluating and improving current programs, encouraging innovations, and then evaluating each innovation’s effectiveness. The key step is systematic gathering of information for sustained improvement. And always with an eye toward helping faculty or students work more effectively. ” Richard Light (2001, p. 224)

Acknowledging multiple legitimate purposes Resisting change Holding accountable Raising awareness Empowering teachers Justifying $ requests PURPOSES Developing programs Improving teaching Revising materials Demonstrating value Revising curriculum Diagnosing need Accrediting schools Articulating courses

Aligning empirical methods to purposes Surveys Language Tests Interviews Focus groups Document analyses Performance assessment Student journals METHODS Portfolios Language Profiles Meetings Delphi technique Observations Teacher logs Self assessment

Approaching useful evaluation Who is asking for that information? Received view: Begin by asking… Evaluative vision: Begin by asking… What are the objectives or outcomes targeted by the Who is in a position to utilize information for the language program? betterment of the language program? How can they be measured? What questions do they have about their learners, Who is held teachers, courses? What challenges do they face? Who is doing the measuring and interpreting? responsible?

Proceduralizing useful evaluation 1. Participation – stakeholders, representatives, primary intended users 2. Prioritization – challenges, questions in immediate need of answers 3. Instrumentation – what data will answer the questions? 4. Collection – how can we get data in available time/resources? 5. Interpretation – what do findings mean in context? 6. Utilization – what decisions & actions are taken? Participation by relevant intended uses for is A focus on specific language educators Language educators are ultimately responsible essential throughout all is essential from the if evaluation findings phases of evaluation for what happens in language education. outset, contextual relevance is sought. if evaluation is to make any difference.

Why bother? Deepen understanding of language programs by participants Enhance communication among program stakeholders Raise awareness & buy-in among teachers, students, others about evaluation Increase the likelihood that data will be used by increasing investment in it Decrease frequency and number of (useless) evaluations Decrease the pejorative effects of evaluation by increasing Actually do something on the relevance and meaning basis of evaluation!

What does it look like? Value added by evaluation in language education

Example 1: Improving teacher induction practices Rapid turnover in GTAs Variable teaching experience Flexible induction practices “Lack of preparedness” Context • English Language Institute • U. S. University • Diverse International Students • Required ESL coursework • M. A. Graduate TA Instructors High student & uni expectations for ELI Feeling of ‘sink or swim’ teaching Teacher-led internal evaluation ELI Administration = Intended users Teacher pre-service needs? Admin’s induction goals? Improvements in induction practices?

Example 1: Improving teacher induction practices Findings + Uses Methods Interviews: • Admin üInduction partially successful: identified weaknesses üInduction outcomes not explicit: admin created based on their goals + teacher wants Transformations Teachers empowered, support improved ELI Administrator: “I can now see how evaluation is connected to the Focus groups: üTeachers value more induction: increase program”Admin values evolution of the required • Experienced evaluation teachers Surveys: • Admin • Former + new teachers üTeachers value variety of practices: enhance availability (online resources, observation, meeting with other teachers) üNew teachers unaware: early hiring, pre-work internships, annotated syllabi Dedicated teacher evaluator position created

Example 2: Perpetuating language studies Context Institution doubts FL contribution Threats of program closure Monitor teaching and learning Learners invested in FL learning? Instructors prepared to teach for proficiency? Pressure to assess outcomes • Stanford language center • 44 foreign languages • 1 -year undergraduate language requirement • Internationalization of the curriculum • Teach for proficiency, ability to use the FL, performance outcomes SLC Director-led evaluation Institution, administration, faculty = Intended users Students achieving outcomes? Teachers focusing on proficiency? Program perpetuation, improvement?

Example 2: Perpetuating language studies Findings + Uses Methods Students: üUncertain outcomes of FLs: set proficiency standard on national scale (ACTFL) Transformations SOPI tests Program survival, (simulated growth, support Program Director: “…focusing on student üStudents enroll in FLs: proficiency enrollments on steady increase; interviews) achievements has reverberative effects on request funding, staffing $$ professional development in its many facets as associate Teachers: Teachers üAssessment used to ensure professionalism with well Oral as on the perception of the efficacy of proficiency: publish outcomes evaluation proficiency language programs throughout an institution. ” by language; basis to seek testing certification support where needed +Student proficiency requirements üInstructors teach for proficiency: reward structure, $ +Web-based dissemination üStudents evaluations very high: perpetuate requirement and proficiency focus +Student course evaluations Institutional respect garnered for language teaching

Example 3: Innovating in teacher development Context High public demand for Dutch Poor Dutch L 2 outcomes Belgium • Dutch SL education in Flanders • Large-scale K-12 education National curricular change • Ensuring functional L 2 abilities Teacher understanding? • Task-based teaching innovation Teacher willingness? Teacher development? Innovator-led evaluation • Providing access to ed/society • Teacher pre-service training • Teacher in-service support Teacher trainers, schools, teachers How do teachers learn/change? What are the constraints on change? How can teacher-dev be optimized? Improvements in Dutch SL learning?

Example 3: Innovating in teacher development Findings + Uses Methods Teachers: Interview, Observation, Survey, Log Transformations üTeacher TBLT dev failing: training not task-based, too comment, early “In the short; change training stages: Teacher cognition Headteacher beginning, teachers were suspicious”…“Try that illuminated üTeachers & students adapt to new training, but variably: with my students!” Trainings: Observation, Interview individualize training, provide school-based support Teacher agency Teacher comment, later stages: “You have a respected üTeachers fear loss of control, training, a meeting, structure: adopt things out, and lack of you can try gradualist approach, lots ofand Task-based learning then there is another meeting you can Students: Prehappening effectively test, post-test guidelines & coaching Dutch L 2 proficiency, Observation 10+ years, cycles of use evaluate. I like that. ” üStudents’ scores increase most in TBLT classes: disseminate to Innovation doesn’t teachers & schools & public happen over night!

What can we learn from these examples? Received View Useful Evaluations • Generic, one-size-fits all Contextualized: specific language programs • Accountability-driven Intentional: multiple purposes and uses • Measurement-based Diversified: methods articulated to uses • One-shot, absolute Iterative: change takes time • Problem-identifying Problem-solving: improve via understanding • Imposed, required Engaging: stakeholders (especially educators) take interest in and act upon evaluation

Facing change: Learning to value evaluation

University of Hawaii, National Foreign Language Resource Center Summer Institute 2007 What is the value of evaluation in language education? Provides a framework for discussion Sheds light on how programs function Increases awareness, communication Makes student learning more efficient Facilitates solving of problems Encourages heightened commitment Democratizes, unifies, engages… “enables the field to articulate and demonstrate— internally and externally—the unique contributions of language studies in a pluralist and globalized world. ”

What can we do? Clarify roles & make space for evaluation in language programs Encourage, enable, and engage in professional development Generate and share examples, participate in the discourse Hold evaluation accountable to language programs, teachers, learners USE it or lose it… FIND OUT MORE!

University of Hawai‘i at Mānoa, Second Language Studies

http: //www. nflrc. hawaii. edu/evaluation

University of Hawaii

Mahalo!