Evaluating Reading Proficiency Gain and Program Effectiveness A

Evaluating Reading Proficiency Gain and Program Effectiveness: A Three-tiered Web-Based Reading Assessment Approach Helen Shen and Chen-hui Tsai The University of Iowa 1

Introduction This presentation reports the evaluation system for “The comprehensive web-based Chinese Reading Program” supported by a DOE grant. • The program • The evaluation 2

The Program: Purpose • To remedy the acute dearth of appropriate Chinese reading materials (beyond the class textbook) • To furnish a superior Chinese reading environment in which reading can take place anytime, anywhere • To improve users’ vocabulary knowledge, reading skill, and cultural knowledge through extensive reading 3

The Program: The Targeted Users • College students who are taking Chinese courses at beginning, intermediate, and advanced levels • High-school students who are taking Chinese AP courses • Individuals who work in government agencies or private organizations and who wish to improve their reading skills in a systematic way 4

The Program: Content • Three levels of reading materials (900 Lessons) • 90 achievement tests • 15 proficiency tests 5

The Program: Format • Flash-embedded HTML files • Image files • NJStar files • Audio HTML files 6

The Program: Academic strength • Proficiency-guided definable reading levels • Cultural theme-based reading unit organization • Fluency-oriented progressive reading skill building • Emphasizing Intentional vocabulary gain • Supporting Constructive responses • Three-tiered web-based assessment system 7

The Program: The Design 8

The Evaluation A Three-tiered web-based assessment system is developed to assess both individual reading achievement and the overall program effectiveness • The flash-based built-in reading outcome assessment for individual users • The File. Maker Pro reading database • The Quia web survey tool 9

The Flash-based Reading Outcome Assessment for Individual Reading Proficiency Gain • Vocabulary pre- and post tests • Multiple choice and open-ended Reading comprehension questions • Reading speed and comprehension rate (REI) report • Achievement tests • Proficiency tests 10

The File. Maker Pro Database This reading database allows us to record: • User’ contact information • User’s reading activities (reading time, reading comprehension score, reading lessons completed, vocabulary tests score. ) • Built-in on-line survey 11

The Evaluation: Quia Web Survey • The Quia survey data allow us to collect users’ additional quantitative and qualitative feedback on certain arising issues during the use of the program. http: //www. quia. com/sv/246903. html 12

The Evaluation: Data Analysis Data from File. Maker Pro • Distribution of Users in the world • Distribution of Reading levels and topics among users 13

The Evaluation: Data Analysis Distribution of Users in the world (78 countries; 1771 valid entries) Africa America Asia Europe Oceania South Africa 8 USA 1171 China 204 United Kingdom 84 Australia 78 Argentina 4 Canada 77 Thailand 48 Germany 46 New Zealand 11 Algeria 2 Brazil 25 Singapore 33 France 42 Fiji 1 14

The Evaluation: Data Analysis Distribution of Reading levels and topics among US users (2741) Reading Level L 1 Beginning L 2 Intermediate L 3 Advanced Frequency of Read lessons 1712 840 189 Popular units U 1 / 720 打招呼和交往礼节 U 11/ 493 中国风俗 U 10/ 中国各民族 U 5/ 266 做饭和吃饭 U 1/ 116 中国家庭 U 20/ 14 中国法制和司法 U 13/ 旅游 U 3/ 42 中国音乐和音乐家 U 1/ 12 中国现代城乡生活 U 6/8/9/16 -30 U 2 -9/1216/18/24/26/28/30 Units not read U 8/18/22 114 125 15

The Evaluation: Data analysis Data from The Quia web survey: One month reading activity and survey (February 2009) • Distribution of users and age groups in the world • Qualitative assessment of program effectiveness • Quantitative assessment of program effectiveness 16

The Evaluation: Data analysis Distribution of users’ nationality and age groups (valid entries: 87), Feb. 09 Africa America Asia Europe Oceania Nationality 1 37 35 11 3 Age Range Under 20 - Under 20 above 50 –above 50 21 -30 Age group ranking 21 -30 Under 20 31 -40 Under 2040 21 -30 31 -40 Under 20 17

The Evaluation: Qualitative Purpose Of helpfulness The part like most Problem Suggestion Teacher Improve reading Qualitative data Use skills Articles in various topics The Include pinyin sound file in the reading does not text work for me Student use Improve listening skills Articles are interesting; very close to real life Self study Expand vocabulary Audio files Pleasure reading Improve reading speed Njstar files Improve cultural knowledge The interface is appealing and easy to use more “beginning” level topics Make sound files downloadable 18

The Evaluation: Quantitative Asia: from a 7 -point scale ; 35 users images are clear. 19

The Evaluation: Quantitative America: From a 7 -point scale; 37 users images are clear. 20

The Evaluation: Quantitative T-test on the comparison of the ratings between Asia and US Population Item number M SD t Sig. Asia User 12 5. 43 . 22 -1. 13 . 272 USA User 12 5. 54 . 26 21

Conclusion: The three-tiered web-based assessment system is an excellent approach conducting the formative assessment for the program, which helps us to know users needs, expectations, and problems they are encountering so that we can adjust the program based on users’ direct and indirect feedback. This system contributes to maintain a long-lasting high quality program. 22

The reading program is free to all users!!! http: //chinesereadingworld. org http: //collections. uiowa. edu/chinese Contact information: Helen-Shen@uiowa. edu Chen-hui-Tsai@uiowa. edu 23

The End

HTML file Back

NJStar Back

Audio files Back

File. Maker Back

Go to Post-test Pretest

Posttest Back

Go to Open Q multiple

Back Open-ended Questions

Back Final Report

Achievement Test Back

Proficiency Test Back

File. Maker Back

Built-in Survey Back

Quia Survey