Integrating learning theory in an era of accountability

Integrating learning theory in an era of accountability testing Drew Gitomer ETS Conference on Educational Testing in America: State Assessment, Achievement Gaps, Federal Policy and Innovations September 8, 2008 Copyright © 2007 Educational Testing Service

The Problem We Are Confronting • Accountability assessments grounded in an outdated scientific model for conceptualizing proficiency, teaching it, and assessing it • Accountability is based on a limited set of limited proxies • Interim assessments, formative assessments, and teacher professional development that are emulating the less desirable characteristics of those accountability tests (aka the irony of alignment) 2

The Vision Can have an assessment system that contributes to educational practice by: • Documenting what students have achieved (“assessment of learning”), • Helping identify how to adjust instruction (“assessment for learning”), and • Engaging students and teachers in worthwhile educational experiences in and of themselves (“assessment as learning”) 3

Coherence of Three Components • Public accountability functions • Formative assessment • Professional development What is coherence? 4

Current Focus of the Project • • Reading, writing and mathematics Middle school Primary work in Maine (Portland) Beginning work in other places • Today’s talk focuses on reading and the work cited is led by Kathy Sheehan and Tenaha O’Reilly 5

The Foundation: A Strong Conceptual Base • Built on: – Cognitive-scientific research • Principles • Domain-specific competency models • Developmental models as available – Aligned with State standards 6

The Reading Competency Model: The Skills Branch Prerequisite Reading Skills Understand & Use Texts Basic Comprehension Skills Reading to Learn Applied Comprehension Skills Required Skills Learning to Reading to Do Pre-Reading Strategies Model Building Strategies for Going Beyond the Text Informational Text Conv. & Characteristics Persuasive Text Literary Text 7

The Reading Competency Model: The Strategies Branch Deliberate, conscious, effortful actions that readers can implement to repair breaks in comprehension & enhance understanding. Prerequisite Reading Skills Required Skills Basic Comprehension Skills Applied Comprehension Skills Understand & Use Texts Pre-Reading Strategies Ask/answer pre-reading questions Model Building Strategies Use Graphic Organizers Strategies for Going Beyond the Text Adopt a Critical Stance Informational Text Conv. & Characteristics Persuasive Text Literary Text 8

The Reading Competency Model: The Text Branch Prerequisite Reading Skills Required Skills Basic Comprehension Skills Applied Comprehension Skills Pre-Reading Strategies Model Building Strategies for Going Beyond the Text Informational Text Conv. & Characteristics Persuasive Text Literary Text Knowledge of conventions that contribute to reading ease or difficulty 9

The Text Branch (Based on Sheehan, Kostin & Futagi, in press) Informational Genre Research/Persuasive Literary The Text Vocabulary Difficulty Drivers Ensure that: (1) targeted skills are assessed within a gradeappropriate span of text complexity; and (2) assessment tasks focus on those aspects of text variation that are likely to have a significant impact on reading performance. Syn. Complexity Cohesion Structure, Negation, etc. 10

The Skills Tested by Most Existing Reading Comprehension Tests Recognize Common Words Prerequisite Reading Skills Decode Unfamiliar Words Read Fluently Understand & Use Texts Locate/Retrieve Inf. Basic Comprehension Skills Und. Diff. Voc. /Lit. Term. Infer Relations Amg Concepts Und. & Use Discourse Struc. Evaluate/Critique Applied Comprehension Skills Integrate Inf. Across Texts Integrate Inf w Bk. Know. 11

Key Features - Reading Design • Measure the full range of skills, including prerequisite skills • Purposeful, authentic, scenario-based tasks • Support reading across the curriculum – Informational Texts: Excerpts from the introductory chapters of science and social studies textbooks, newspaper & magazine articles, etc. – Persuasive Texts: Editorials, Letters to the editor – Literary Texts: Narratives, memoirs, poems • Select task formats that encourage use of targeted reading strategies • Include a mixture of expressive (CR) and receptive (mc) tasks 12

Implementation: The Assessment Design • Period accountability assessments (PAAs) • Foundational tasks – Constructed response and selected response – Scaffolded structure • Supplemental tasks • Computer Administration • Automated Scoring 13

CBAL vs. (stereo)typical assessments Traditional CBAL Single Measurement Multiple Occasion Measurement Occasions Many short items A few long tasks (mostly) unrelated Centered around a common theme Representative of a Based on a domain Competency Model Homogeneous Heterogeneous Response Types 14

A Sample PAA • Spoken Module – Provide mastery evidence relative to the Prerequisite Skills – Short Tasks: Word Recognition, Decoding, Oral Fluency – Students speak into a headset, responses are scored as Words Correct per Minute (wcpm) • Comprehension Module – Provide mastery evidence relative to the Model-Building & Applied Comprehension Skills – All texts & tasks revolve around a meaningful scenario • e. g. , You have to write a report about the scientific method for your science class – Tasks employ innovative response formats: SR and CR – Multi-step thinking, scored via partial credit scoring rubrics 15

The Scenario You have to write a report about the Scientific Method for your science class. Since you enjoy reading about American History, you decide to focus your report on Ben Franklin’s use of the Scientific Method. Read the following passages to learn more about the Scientific Method and about Ben Franklin’s scientific experiments. • Four Texts – A passage about Ben Franklin adapted from a Social Studies textbook – An article about the Scientific Method extracted from an Encyclopedia – A newspaper article about 3 winners of the Intel Science Competition – A diagram from a student lab report 16

Preliminary Results – Construct Validation TOWRE: a standardized measure of Prerequisite Skills Gates-Mac. Ginitie: a std. measure of Model-Building Skills TOWRE The Spoken Module The Comprehension Module Gates z’ 0. 78 0. 63 2. 69 ** 0. 46 0. 79 5. 18 *** 17

The Assessment is Also Designed to Yield Diagnostic Information that Teachers can Use to Adjust Instruction 18

Formative System • Is built upon the same conceptual base as the Accountability System • Gets information from the Accountability System • Is used at the district’s and teacher’s option • Is available on demand • Performance in the formative system will not be used for accountability purposes 19

Structure of formative assessment • Library of resources including professional development component – Elaborated foundational tasks – for each task component • Clarification of learning intentions – connection to competency model • Criteria for considering student responses • Common conceptual challenges • Potential pedagogical strategies • Potential additional resources 20

Challenges • • Psychometric Automatic scoring Educational infrastructure Political accountability 21