Measuring Learning with WBIEG Level-2 Evaluation Toolkit By

Measuring Learning with WBIEG Level-2 Evaluation Toolkit By Violaine Le Rouzic, Evaluation Officer, WBIEG March 27, 2005

What is a Level-2 evaluation? Objective Ø To help refine a course by: ü Measuring how much participants learned ü Assessing what participants learned Evaluation Design Ø Test all participants’ knowledge of course contents at the start & the end of the course Ø Same difficulty on pre- & posttests, but different items, to avoid pretest recall effect Main Indicator Ø Group’s Learning Gain = Post-test – Pre-test 2

What is WBIEG Level-2 evaluation Toolkit? Ø A set of guidelines, templates, databases, and macro enabling course teams to assess their participants’ learning. Ø Adapt psychometrics to WB context to measure learning with fair confidence (Tradeoff between science and feasibility) 3

Main Toolkit features Ø To measure participant group learning at the course Ø Practical, short, step-by-step Ø Tasks divided between content experts & assistants Ø Accounts for short preparation time, few test takers Ø No evaluation knowledge required Ø Need basic Word, Excel & Internet browsing skills Ø Test form templates in over ten languages Not: theoretical, state of the art psychometrics Not: for certification of individual participants 4

For whom is the Toolkit? Ø World Bank course teams (for courses with external and/or internal participants) Ø World Bank managers or any WB staff who want to compare learning outcomes data across various criteria (division, years, etc. ) Ø Course teams in other organizations can use the evaluation tools on external WB web site (but the test items and results database is for World Bank users only. ) 5

When to use the Toolkit? Ø Level-2 evaluation is feasible: ü Learning knowledge/skills is the main objective ü Learning objectives are clear before the course ü Every participant follows the same curriculum Ø Worth investing in Level-2 evaluation: ü Course will be offered again ü Long enough (at least 1 week recommended) ü Many participants (30 or more recommended) Ø Enough resources and commitment: ü Two staff weeks’ time for the evaluation ü Commitment to use the evaluation results 6

Toolkit’s 13 Steps http: //web. worldbank. org/WBSITE/EXTERNAL/WBI/0, , content. MDK: 20270021~page. PK: 209023~pi. PK: 335094~the. Site. PK: 213799, 00. html 7

1. Plan the evaluation For course director Ø Course team resources needed: ü 1 week of the content expert team (can be split) ü 1 week of assistants Less time required on subsequent L 2 evaluations Ø Through the course cycle ü From early design stage to course re-design ü Most time for test development before delivery ü Start at early course design stage http: //siteresources. worldbank. org/WBIINT/Resources/Plan-for-Level-2. pdf 8

2. Map the test For course director Ø Build a test specification matrix to determine: ü Which content areas should be tested ü To which cognitive domain each area relates ü How many test items are needed per content area and cognitive domain (Recommended minimum total: 20 item pairs per test) Ø Objective: ü Make the test representative of the course content http: //siteresources. worldbank. org/WBIINT/Resources/How-to-build-matrix. pdf 9

3. Review past items (optional) For content experts of World Bank only Ø Consult a database with over 5, 000 items used in over 100 WB courses with Level-2 evaluations ü Search for keywords in offering titles or items Ø Potential benefits ü Save time if some items fit your needs ü Identify issues to avoid from past items ü Get ideas on writing new items MCaution ü Item quality is context-specific, don’t re-use blindly! http: //intranet. worldbank. org/WBSITE/INTRANET/UNITS/WBIINT/0, , content. MDK: 20191931~page. PK: 135700~pi. PK: 135698~the. Site. PK: 136975, 00. html 10

4. Write items For content experts Ø Match the content area and cognitive domain of the test specification matrix Ø Test items use multiple-choice format Ø All items have five response options (Last option is always “I don’t know. ”) Ø Average difficulty level Ø Clearly stated http: //siteresources. worldbank. org/WBIINT/Resources/How-to-write-items. pdf 11

5. Pair items For content experts Ø For each item, write an equivalent item ü Same difficulty level ü Same content area ü Same cognitive domain ü Same length ü Same format Ø Examples in guidelines Ø Objective: Make pre- and post-test equivalent http: //siteresources. worldbank. org/WBIINT/Resources/How-to-pair-items. pdf 12

6. Pilot tests For content experts (with assistants) Ø Have volunteers take the tests (or part of the test) before the course. Volunteers can be: ü Other content experts (to check key) ü Alumni ü Participant look-alike ü Non-content experts MBUT NOT the actual participants! Ø Collect comments and demographics with tests responses. Test without, then with key. http: //siteresources. worldbank. org/WBIINT/Resources/How-to-pilot-items. pdf 13

7. Review test items For content experts (with assistants) Ø Use the pilot test responses and the Toolkit checklist to review each item for: ü content ü wording ü using statistical item analysis (if any) Ø Finalize the items http: //siteresources. worldbank. org/WBIINT/Resources/How-to-review-items. pdf 14

8. Produce test forms For assistants Ø Use automated template to randomly assign items to either pre- or post-test Ø Use test form templates (customize the templates, as needed) Ø Use formatting and production guidelines MPoor test form production can ruin all results! http: //siteresources. worldbank. org/WBIINT/Resources/How-to-prepare-test-forms. pdf 15

9 &10. Collect test forms For any organizer on site Ø Collect pre-test at course start and post-test at course end Ø Have all participants answer Ø Have all participants write their codes on both forms to match results by respondent Ø Explain evaluation objectives & confidentiality http: //siteresources. worldbank. org/WBIINT/Resources/How-to-collect-pre-test. pdf http: //siteresources. worldbank. org/WBIINT/Resources/How-to-collect-post-test. pdf 16

11. Compute results For assistants Ø Follow tabulation guidelines Ø Enter responses in tabulation template Ø Click macro for automatic item analysis http: //siteresources. worldbank. org/WBIINT/Resources/How-to-tabulate-responses. pdf 17

11 A. Results example: Learning gain: a course learning gain green if statistically significant; Compare with orange, if not. and post-test score other courses Pre- and post-test scores (matched respondents) Post-test scores (all respondents) 18

11 B. Results example: distribution of a course’s pre- and post-test scores Post-test to the right of pre-test: participants learned 19

11 C. Results example: post-test responses by item Key Items’ text Responses 20

Check: item confusing or misconception not overcome 11 D. Results example: a course item analysis Check: item too hard or course did not teach this well Check pre-post equivalence Confused high scorers Most items statistically OK Compare reliability (WB only) 21

12. Send results to WBIEG For assistants (WBI courses only) Ø WBIEG will: üCheck quality of processing üInclude test items and results in database üReport evaluation efforts and results on request http: //web. worldbank. org/WBSITE/EXTERNAL/WBI/0, , content. MDK: 20270039~page. PK: 209023~pi. PK: 335094~the. Site. PK: 213799, 00. html 22

13. Interpret results For course director & content experts Ø Review and interpret results using: üInterpretation guidelines üResults database (WB only) üToolkit glossary Ø Decide how to improve next offering and next test http: //siteresources. worldbank. org/WBIINT/Resources/How-to-interpret-results. pdf 23

Thanks to: Ø Developers: Joy Behrens, Guangbin Liu, WBIEG staff Ø Advisors: Marlaine Lockheed, William Eckert, Sukai Prom-Jackson, Zhengfang Shi, Gary Echternacht … Main contact: Violaine Le Rouzic 24