Assessment Principles and Practices Reliability Test

Assessment Principles and Practices

Reliability • Test reliability • Student related reliability—test taking strategies • Test administration reliability • Rater reliability ØInter-rater reliability: scorers not applying the same standards---lack if attention to scoring criteria, inexperience, bias ØIntra-rater reliability---unclear scoring criteria, fatigue, bias

Validity • How well an assessment elicits the performance that matches that of the course or unit of study being tested.

Content Validity • Cleary define achievement being measured • Covers as many objectives as are taught in the course

Content Validity • Adequate # of samples of S performance • Clearly framed set of standards for evaluating performance(0 -1, 1 -5. etc. ) • Provide FB to the S

Objectives identified & well framed 1 Ss should be able to demonstrate some reading comprehension. 2 To practice vocabulary in context. 3 Ss will have fun through a relaxed activity and thus enjoy their learning. 4 To give Ss a drill on the /i/-/I/ contrast. 5 Ss will produce yes/no questions with a final rising intonation.

Ss should be able to demonstrate some reading comprehension. • Modal should=ambiguous • expected performance not stated

To practice vocabulary in context. • Everyone can fulfill the act of ‘practicing’ • No standards are stated or implied

Ss will have fun through a relaxed activity and thus enjoy their learning. • Can not be assessed

To give Ss a drill on the /i/-/I/ contrast. • Just a T note on the type of activity to be used

Ss will produce yes/no questions with a final rising intonation. • Framed in a form that lends itself to assessment • Includes performance verb • Includes specific linguistic target

Construct Validity • Assessment reflects theoretical constructs Theoretical constructs—psychological constructs (motivation, self-esteem) linguistic constructs (proficiency, communicative competence) FOR EXP. ) Test for oral proficiency should test: 1 Pronunciation 2 Fluency 3 Grammatical accuracy 4 Vocabulary use 5 Sociolinguistic appropriateness Justification: theoretical construct: 5 major components of oral proficiency FOR EXP. ) Lexical objective of chapter: communicative use of vocabulary

Consequential Validity • Social consequences (intended and unintended) on test interpretation and use • Accuracy in measuring intended criteria • Impact on the preparation of test takers • Effect on learners (motivation, independent learning, study habits, attitude towards school work, subsequent performance in course)

Face Validity Extent to which Ss view the assessment as ‘fair, relevant, useful for improving learning’ 1 Well constructed format, familiar tasks 2 Doable within allotted amount of time 3 Items clear, uncomplicated 4 Directions crystal clear 5 Task relevant to course work (content val. ) 6 Difficulty level—reasonable challenge

Authenticity 1 2 3 4 5 REAL WORLD TASKS Language of test—as natural as possible Items contextualized (not isolated) Topics meaningful-relevant & interesting Thematic organization of topics—episode, story line Tasks represent or closely represent real world tasks

“Going to” 1. What____this weekend? a. are you going to do b. you are going to do c. you’re gonna do 2. I’m not sure. _____anything special? a. you are going to do b. are you going to do c. is going to do 3. My friend Burcu and I ____ a party. Would you like to come? a. am going to b. are going to go to c. going to 4. I’d love to! _______ a. What’s it going to be? b. Whose going to be? c. Where’s it going to be? 5. It is _______to be at Esra’s house. a. going b. go c. gonna

1. There are three countries I would like to visit. One is Italy. a. The other is New Zealand other is Nepal. b. The others are New Zealand Nepal. c. Others are New Zealand Nepal. 2. When I was twelve, I used______ everyday. a. swimming b. to swimming c. to swim 3. When Osman creates a website, he always creates it____. a. artistically b. artistic c. artist 4. When Sarah broke her leg, she asked her spouse to _____her to work. a. driving b. to drive c. drive

WASHBACK Effect tests have on instruction: How Ss prepare for test(‘cram’ for tests; ‘teach to the test’) Beneficial WB: diagnose strengths & weakness Informal assessment—interactive FB Formal assessment—interactive or not (A, B, C/#) WB reinforces principles of Language Acquisition (intrinsic motivation, autonomy, language ego, strategic investment, interlanguage, self confidence, etc…)

WASHBACK • • Give praise and constructive criticism Written comments—not just a letter/# grade Strategy for improvement Create assessment to provide FB for Ss (they gain a sense of accomplishment & challenge)

Scenario 1. Standardized multiple choice –S receives report form: listening, grammar, proof-reading, reading comprehension 2. Timed impromptu test of written English-S receives a holistic score ranging from 0 -6. 3. One-on-one oral interview to assess overall oral ability-S receives a holistic score ranging from 0 -5. Raterreliability Test Content Face reliability Validity Authenticity

Scenario 4. S reads 3 paragraphs and responds to 10 multiple-choice general comprehension questions. S receives score report showing which answers correct/incorrect. 5. S gives 5 min. prepared oral presentation in class. T evaluates by filling in a rating sheet indicating S success in delivery, rapport, pronunciation, grammar, & content 6. S listens to 15 min. lecture video & takes notes. T provided written comments on individual S notes. Raterreliability Test reliability Content reliability Face Validity Authenticity

Scenario 7. S write a takehome(overnight) 1 page essay on assigned topic. T reads paper and comments on organization & content only & returns essay to S for subsequent draft. 8. S creates multiple draft of 3 pg. essay, peer & T reviewed & turns in a final version. T comments on grammatical & rhetorical errors only, and returns to S. 9. S assembles a portfolio of materials over a semester-long course. T conferences w/S on portfolio at end of semester. Rater. Test reliability Content reliability Face Validity Authenticity

Scenario 10. S writes a dialogue journal over the course of a semester. T comments on entries every 2 11. Multiple-choice listening quiz provided by textbook w/taped prompts, covering the content if 3 week module. T provides S score w/no indication of right/wrong 12. S given 10 vocabulary items and directed to write 10 sentences w/items. T only marks correct/incorrect. Raterreliability Test reliability Content reliability Face Validity Authenticity