A Ten-Year Review of CBT By Casey Marks

A Ten-Year Review of CBT By Casey Marks, Ph. D. , NCSBN Geraldine Piskorski, MBA, ASCP Board of Registry Anthony R. Zara, Ph. D. , PV Presented at the 2004 CLEAR Annual Conference: 21 st Century Regulation: “Show Me” What Works September 30, 2004 ASCP Board of Registry_______ ASCP Certification – The Standard of Excellence September 30, 2004 CLEAR Conference

Session Agenda 1. 2. 3. 4. 5. 6. 7. Speaker backgrounds Before transition to CBT Transition Operations Refinement CBT lessons learned Conclusions ASCP Board of Registry_______ ASCP Certification – The Standard of Excellence September 30, 2004 CLEAR Conference

Speaker Background Casey Marks, NCSBN Geraldine Piskorski, ASCP Board of Registry Anthony Zara, Pearson VUE ASCP Board of Registry_______ ASCP Certification – The Standard of Excellence September 30, 2004 CLEAR Conference

What We’re Trying to Accomplish 1. Share experiences 2. Compare and contrast large vs. small programs’ issues 3. Share solutions 4. Outline successful practices ASCP Board of Registry_______ ASCP Certification – The Standard of Excellence September 30, 2004 CLEAR Conference

Some Industry Trends 1. 2. 3. 4. 5. 6. Exploding numbers of programs More complicated item types Security concerns Globalization Maturing technologies, e. g. , Internet Increased choices for program sponsors ASCP Board of Registry_______ ASCP Certification – The Standard of Excellence September 30, 2004 CLEAR Conference

Issues Before Transition to CBT Small program Large program • ASCP/BOR background • • • Size and depth of item bank Variable vs. fixed length test Opportunity to review Size of Examinee population Availability of computer testing facilities • Cost of delivery NCSBN background NCLEX facts Pre-CAT research program Barriers? ASCP Board of Registry_______ ASCP Certification – The Standard of Excellence September 30, 2004 CLEAR Conference

Foundational Pre-CBT Research • • • Dimensionality IRT calibration Scale stability Comparability of results Early on, much psychometric research was necessary; now a bit less, but still need to validate findings with your program. • Mode of administration – What flavor of CBT? (CAT not right for everyone) ASCP Board of Registry_______ ASCP Certification – The Standard of Excellence September 30, 2004 CLEAR Conference

Operational Research • • Examinee interface Stopping rules Test length Item pre-testing plan Test timing Candidate impact Beta testing Here is where your program’s specifics get analyzed and determined ASCP Board of Registry_______ ASCP Certification – The Standard of Excellence September 30, 2004 CLEAR Conference

Transition Issues Small program Large program • Transition from testing two days each year to year-round testing • Staffing • Stakeholder communications • CBT benefits • Create roadmap to success • Organizational processes • Stakeholder communications ASCP Board of Registry_______ ASCP Certification – The Standard of Excellence September 30, 2004 CLEAR Conference

Benefits of CBT • More frequent administrations – (daily testing) • Measurement accuracy and efficiency • Advanced item types can be used • Scheduling on demand • Increased security • Instant candidate results • Professional testing environment • Reduced time between retakes ASCP Board of Registry_______ ASCP Certification – The Standard of Excellence September 30, 2004 CLEAR Conference

Roadmap for CBT • • • Needs analysis Stakeholder analysis Building consensus Working decision processes Communications Leverage the experience of others when analyzing your program. ASCP Board of Registry_______ ASCP Certification – The Standard of Excellence September 30, 2004 CLEAR Conference

Organizational Processes • Organizational Decision Process – Understanding needs – Options for solutions • Contracting Process – RFP or directed proposals – Provider options – Focus on outcomes ASCP Board of Registry_______ ASCP Certification – The Standard of Excellence September 30, 2004 CLEAR Conference

Operations Issues Small program Large program • Psychometrics • Administration • Stakeholder support • Batch to continuous model – Test development – Psychometrics – Administration – Stakeholder support – Staffing ASCP Board of Registry_______ ASCP Certification – The Standard of Excellence September 30, 2004 CLEAR Conference

Program Operations • Batch to Continuous: Different Mindset – Test Development • Constant pool rotation -- ability to introduce content more quickly • More items needed • Continuous item development/reviewing • Item writer recruitment ASCP Board of Registry_______ ASCP Certification – The Standard of Excellence September 30, 2004 CLEAR Conference

Program Operations • Batch to Continuous: – Psychometrics • Constant item maintenance • Different item exposure concerns • Candidate volume spikes/dips due to content changes and passing standard changes • Ability to remove underperforming problematic/underperforming items during administration ASCP Board of Registry_______ ASCP Certification – The Standard of Excellence September 30, 2004 CLEAR Conference

Program Operations • Batch to Continuous: – Test Administration • • • Individual issues/incidents Ongoing test center monitoring Ratio of investigations to candidates increases Time from exam registration to result Time-dependent registration policies – Application process – Fee changes – Rescheduling issues ASCP Board of Registry_______ ASCP Certification – The Standard of Excellence September 30, 2004 CLEAR Conference

Program Operations • Interface with stakeholders – Test accommodations – Communicating program changes – Procedure/documentation changes • Stakeholder Support – Direct support – Information/communication • • Accessible Timely Constant Rumors/urban legends ASCP Board of Registry_______ ASCP Certification – The Standard of Excellence September 30, 2004 CLEAR Conference

Program Operations • Batch to Continuous: – Program Staffing • Dedicated • Specialized • Outsourcing – Customer Service • The on-demand mentality • Expectations ASCP Board of Registry_______ ASCP Certification – The Standard of Excellence September 30, 2004 CLEAR Conference

CBT Program Refinement Small program Large program • Immediate pass/fail results at test center • Providers - more options • Quality of graphics • • Ongoing research Item types Providers Service ASCP Board of Registry_______ ASCP Certification – The Standard of Excellence September 30, 2004 CLEAR Conference

Program Refinement Research • Ongoing Research – Methodology and content – Staff and/or test service – External • Implementation dates seemingly “arbitrary” ASCP Board of Registry_______ ASCP Certification – The Standard of Excellence September 30, 2004 CLEAR Conference

Program Refinement – Item Types • Advanced item types – Media – Stimuli – Response formats ASCP Board of Registry_______ ASCP Certification – The Standard of Excellence September 30, 2004 CLEAR Conference

Program Refinement Providers • Service Provider Change – Between 1994 and 2002 same CBT provider – New test development and administration provider • (2 for 1) in 2002 – – – New center network New test driver/candidate interface Candidate support Data reconciliation between test services New application/administration/reporting/results processing systems – Seamless… ASCP Board of Registry_______ ASCP Certification – The Standard of Excellence September 30, 2004 CLEAR Conference

Program Refinement - Service • Innovative item types (beyond MCQ) – Utilizing same exam administration parameters • More frequent administrations – 2 to 4 to 8 per year • Additional testing time – Opportunity to demonstrate competence under optimal exam termination rules • International administration – Outside U. S. and territories starting in 2005 ASCP Board of Registry_______ ASCP Certification – The Standard of Excellence September 30, 2004 CLEAR Conference

CBT Lessons Learned Small program Large program • Importance of good communication with all stakeholders • Size of examinee population • Improved benefits to examinees • Benefits obtained worth it? • Program complexities • Stakeholder satisfaction? • Service? • Etc. ASCP Board of Registry_______ ASCP Certification – The Standard of Excellence September 30, 2004 CLEAR Conference

Challenges • Large program, yet perfection is needed/expected daily… – More moving pieces, more expertise needed – Change is harder to integrate – The more complex the program, the more difficult to educate stakeholders – Timing of program changes never convenient for all stakeholders ASCP Board of Registry_______ ASCP Certification – The Standard of Excellence September 30, 2004 CLEAR Conference

General Conclusions • Any large vs. small program things? • Any “gotchas”? • Need to consider all program stakeholders when considering procedure changes ASCP Board of Registry_______ ASCP Certification – The Standard of Excellence September 30, 2004 CLEAR Conference

Conclusions • CAT is a good thing for the NCLEX… – Boards of Nursing enjoy: • Convenience and speed of system • The ability to focus on exam content not psychometric and/or exam administration – Candidates like: • Quick registration, scheduling and results receipt • Scheduling options • Personalized service – NCSBN appreciates: • Security benefits • The ability to test content in new and novel ways ASCP Board of Registry_______ ASCP Certification – The Standard of Excellence September 30, 2004 CLEAR Conference

Discussion ASCP Board of Registry_______ ASCP Certification – The Standard of Excellence September 30, 2004 CLEAR Conference

Contact Information Casey Marks, Ph. D. Director of Testing Services NCSBN email: cmarks@ncsbn. org Gerry Piskorski, MBA Director, Applicant, Certificant and IT Activities ASCP email: gpiskorski@ascp. org Anthony R. Zara, Ph. D. Vice President Testing Services Pearson VUE email: tony. zara@pearson. com ASCP Board of Registry_______ ASCP Certification – The Standard of Excellence September 30, 2004 CLEAR Conference