State System Initiatives for Academic Quality Reviewing the Options

State/System Initiatives for Academic Quality: Reviewing the Options Peter T. Ewell National Center for Higher Education Management Systems (NCHEMS) University of North Carolina Strategic Directions Initiative December 12, 2012

Topics for Discussion n What Others Are Doing (Approaches, Purposes) n Strengths and Drawbacks of Particular Approaches n Some Lessons from Experience

State Purposes for Quality Assessment n Regulating Student Flow (SD “Rising Junior”) n Assessing Institutional Performance (WV CLA) n Performance Funding (TN Schedule) n “Institution-Centered” Mandate (VA) n Technical Assistance (WV Accreditation Review)

How Many States are Doing What? n Common Standardized Test (5) n Common Student Surveys (6) n Academic Performance Indicators > Retention (44), Completion (44), Transfer (34) > Graduate Placement (17) n Policy Requiring Institutional Assessment (21)

What Are Institutions Doing? n Standardized Tests (39%) n National Surveys (76%) n Local Surveys (52%) n Alumni Surveys (43%) n Course Embedded Assessment (81%) n Portfolios (84%) n Employer Surveys (62%)

Standardized Tests n Advantages: • Allows Comparison or Benchmarking • Professionally Developed n Drawbacks • Limited Guidance for Improvement • Uncertain Alignment with Taught Curriculum • Hard to Motivate Students to Take them Seriously

Student Surveys (National and Local) n Advantages: • Allows Comparison or Benchmarking (If National) • Professionally Developed (If National) • Can Point to Things to Improve n Drawbacks • Self-Reported Learning and Behaviors May be Unreliable • Hard to Get Good Response Rates

Academic Performance Indicators n Advantages: • Clear and Succinct Communication • Allows Comparison or Benchmarking n Drawbacks • Limited Guidance for Improvement • Susceptible to Misinterpretation or Manipulation • Can Induce Action to Fix the Number, Not the Problem

Alumni Surveys n Advantages: • Targeted on Outcomes that Matter (Job and Graduate School Placement) • Credible to External Audiences (“Satisfied Customers”) • Can Point to Things to Improve n Drawbacks • Self-Reported Learning May be Unreliable • Hard to Get Good Response Rates

Institution-Centered Approaches n Advantages: • Institutional Buy-In and Support • Allows Tailoring Methods to Mission • Can Point to Things to Improve n Drawbacks • Can’t Compare Results Across Campuses • What to Report to Stakeholders and the Public?

Some Lessons from Experience n Balance Purposes: Accountability and Improvement n Multiple Approaches are Best (“Triangulation”) n Include a Common Core of Outcomes (e. g. DQP) n Think Carefully About Public Reporting (Unit of Analysis, Comparison) n Make the Consequences for Institutions Clear n Build in a Pilot and a Formal Periodic Review of the Approach After X Years