Using Data and Accountability to Identify Gaps Set

Using Data and Accountability to Identify Gaps, Set Goals, and Improve Quality Working Poor Families Project June 2007 State Policy Academy on Adult Education Center for Law and Social Policy Amy-Ellen Duke aduke@clasp. org

State policies impacting program improvement: Data as the starting point n Develop comprehensive and integrated data system for measurement of needs and results n Identify and explicitly state goals n Collect data that show whether goals are being achieved n Set standards, measure progress, and evaluate performance (develop indicators of program quality, choose content standards, produce report cards, or other descriptive reports) n Reward or penalize performance (performance-based funding) n Identify and implement policy and institutional changes and align resources in order to better meet goals

States’ Role in Improvement and Management INFLUENCING FACTORS STATE ADULT EDUCATION ACTIVITIES INVOLVING THE USE OF DATA INTENDED OUTCOMES FEDERAL LEGISLATION, REGULATIONS, POLICIES · MARKET ADULT EDUCATION, OBTAIN FUNDING TARGET POPULATIONS OF LEARNERS RECEIVE EDUCATIONAL SERVICES · DEVELOP POLICY, GUIDANCE · MANAGE FUNDING PROCESS · CONDUCT MONITORING, COMPLIANCE REVIEW STATE LEGISLATION, REGULATIONS, POLICIES · IMPLEMENT ACCOUNTABILITY REQUIREMENTS · PROVIDE TRAINING, TECHINICAL ASSISTANCE TO ADULT EDUCATION PROGRAMS QUALITY ADULT EDUCATION SERVICES ARE DELIVERED · DEVELOP SPECIAL INITIATIVES STATE MANDATES, PRIORITIES · CONDUCT INTERAGENCY ACTIVITIES TO SUPPORT, EXTEND ADULT EDUCATION SERVICES · SUBMIT DATA TO FEDERAL AND STATE ENTITIES From Judith Alamprese, Planning State Change and Improvement Initiatives ADULT LEARNERS ACHIEVE GOALS AND DEMONSTRATE OUTCOMES

Using data to state need and determine where interventions are necessary n WA study: “tipping point” for substantial earnings payoff from college is at least 30 vocational credits plus a credential. Other research supports this. $2, 700 and $1, 700 more per year (respectively) for workforce students entering with HSD or GED n Even larger increases for lower skilled students and those with limited English; ESL students earn $7, 000 more per year and ABE students $8, 500 more annually n

Using data to state need and determine where interventions are necessary n By contrast, those in short-term customized training don’t gain enough skills to reach tipping point n Earn $3, 800 less per year than those who reach the tipping point; $6, 800 less per year if they started from ABE (SBCTC, 2005, outcomes for 35, 000 students) n “Tipping Point” workers need = mid-level skills employers need n Provided evidence for state to invest in particular interventions, including enhanced FTE to take IBEST statewide

Balancing student and program goals n Many adult ed/ESL students’ first priority is to get job skills and higher paying employment through completion of GED; large percentage also have college enrollment as a secondary goal n WIA accountability measures (influence program goals) n n States must collect and set performance standards for four core follow-up measures (focus on outcomes, not program processes and indirectly addressed program quality) To remain eligible for fed funding, local providers must be able to document their ability to achieve state performance standards

Accountability challenge n Balancing various goals, ensure diversity of learner needs is met n Since adult ed/ESL system is so decentralized , there is little consistency or guarantee of rigor in content or quality of instructional methods, which makes accountability challenging n Leads to questions of what to teach and assess and if content standards are necessary n Can lead to program improvement and influence resource priorities

Federal solution to data & accountability needs: National Reporting System n States must collect and set performance measures: n Basic measures n n n Educational gain High school completion Entered postsecondary education Entered employment Retained employment n Criticisms: Doesn’t measure program quality, fails to adequately capture spectrum of learner gains n Forced states to take measurement seriously; many adopted additional measures

Adopting additional performance measures n Balance between reasonable, attainable standards and challenging providers to improve performance n States have incorporated qualitative indicators into operation of accountability systems (most aggressive states have strongest data capacity) n n KS considers student outcomes and program operations AZ measures gains in family literacy, computer literacy, corrections, work-based learning; increases in number of FT teachers, teacher pay, caliber of teachers; decreases in class size; implementation of research-based “best practices”

Content standards and their connection to performance n Standards-based reform not limited to NCLB n Content standards articulate what students need to know and be able to do to achieve their goal n Belief that content standards with aligned assessments and accountability mechanisms will provide expectations of learning and will lead to continually higher levels of achievement n Examples of Content Standards: Equipped for the Future, Work Keys, OVAE n Aligned to what: academic standards or contextualizing adult ed with workforce?

Content standards in Adult Ed: Equipped for the Future n A standards-based reform initiative aimed at improving the quality of adult education (NRS requires standardized methods for assessing learner outcomes) and building the capacity of the system to better assist adults in accomplishing their life goals n Framework for what constitutes a set of standards for teaching, assessing, and improving adult literacy programs based on 3 adult roles: worker, family member, citizen n 16 standards in 4 skill areas: communication, decision-making, interpersonal, and lifelong learning n Over 18 states have adopted EFF standards as a means of policy change: setting goals and aligning curriculum, instruction, assessment, and accountability (programs in over 40 states are using the framework, as well)

Performance-based funding and program improvement n Performance-based funding can help serve state’s accountability needs by ensuring resources are effectively used by local providers n Linking funding to program performance has encouraged providers to reexamine the relationship between curriculum content, classroom instruction, and testing practices and student outcomes—greater alignment n Performance-based funding has radically altered approach to adult education n Most state adult ed directors report gains for adult learners after implementation of perf-based funding

Performance-based funding and program improvement n Some states use performance results to allocate some resources to local providers (15 states in 2003) n Funding strategy can focus on learner outcomes, process indicators, and/or performance incentives n Rewards of core or bonus funding for high performance n Deny or withdraw some or all of the funding for poor performers n Ensure state sets standards that make allowances for variation in clients served n Kansas-half of federal funding tied to outcomes of students at lowest levels of literacy; an incentive for programs to include the hard-to-serve who often produce learning gains at slower rates

Performance-based funding: Kentucky n Program performance funding n NRS indicators used: Completing NRS educational levels, entering PSE/training, earning GED n County programs must meet their enrollment goals and at least 70% of NRS performance indicators to qualify for performance funding (increase from 50 to 70% reflects new focus on quality and student outcomes) n Student performance funding n New opportunity for programs to increase their funding through achieving specific student outcomes n Examples: each GED attainment, students making multiple learning gains, each GED graduate transitioning to PSE, KY Employability Certificate and KY Manuf Skills Standards certificate attainment

High-stakes performance accountability systems—some considerations n States with diverse pops of adult learners or with large pops of adults with low literacy levels sometimes have trouble documenting learner progress n Hold programs to same standards, while making allowances for variations in the client pop served n Requires use of common set of measures (what’s appropriate? ) n Must build technical capacity of providers to collect and report data

High-stakes performance accountability systems—some considerations n Effective data management system important n Danger of shifting adult education priorities from serving local needs to primarily providing services in which learner progress can be easily documented n Can lead to decrease in numbers & types of providers applying for funding n Can harm smaller providers with less capacity (CBOs may have limited resources for data collection or marginal quality, but sometimes do better due to wrap-around supports)

Using data to improve quality and build support: State report cards n Purpose: a user-friendly tool for measuring yearly progress and highlighting program accountability efforts; should drive program improvement n Include outcomes and other data that reflect program quality, provide basis for evaluation, present contextual data or descriptive info that promotes understanding n Include outcome measures beyond NRS n n Addl employment and educational measures, such as improving type of job or achieving skills (more accurate if have access to UI records), certificate completion Instruction measures, program and teacher measures, student measures

What works? Lessons from data n Through policy and content of state contracts, state can encourage programs across the system to adopt practices that will ensure program quality (Comings 2007, Bos et al 2002) n Improving program quality n Set standards for duration and intensity of instruction (100150 hours to advance one level) n Set standards with clear expectations for students; establish student’s goal early n Continuous improvement loop through use of data, benchmarks, assessments, professional development, etc. n Align instruction to student goals/needs and ensure students move along path—ensure progress toward reaching goals

What works? Lessons from data (cont) n Leveraging resources and forming partnerships to market and expand program offerings and services to adults n Targeting instruction and content to meet specific needs of individuals and those in the community (local labor market) n Combining basic skills education with occupational and job training, with wrap-around supports

Policy options to support student success n State funds can encourage promising models and contracts can require them: Case management model of advising for at-risk students n Use of cohorts (such as learning communities) n Avoid open entry/exit n Flexible delivery modes (evening, weekend, distance) n Opportunities for integrating work and learning n Mix of instructional methods n Advising that includes career counseling & action plan n Provide traditional supportive services (child care, transp) n Increasing intensity of instruction n

Defining program quality: Focus on inputs and process (Massachusetts) n Emphasizing quality over quantity n Served as many as 40, 000 in 1991, but many receiving as few as 12 hours/year. Cut number of students served to 12, 000 n Decision that new funding would be put into program and staff development, and only once quality improved would number of students served increase n Defined quality and set guidelines for programs n Monitors process and outcomes n Program funding based on demonstrating satisfactory performance on student participation, goals, and educational gains.

Defining program quality: Focus on inputs and process (Massachusetts) n Guidelines for local programs cover: n Student services (incl instructional hours and class size, counseling) n Program and staff development (reqs for paid time for staff and program development, creation of community partnerships) n Administration (requirements for matching funds, minimum coordination time, minimum recommended salary rates) n Minimum contact hours n Suggestions for additional policies to strengthen services, i. e. , child care and transportation assistance

Setting goals to increase outcomes. Kentucky n Emphasis on increasing quantity n Part of Kentucky’s Go Higher Campaign, set goal of 40% of GED grads transitioning to PSE (went from 12% in late 1990 s to 21% in 2004, require programs to report on it) n Awards performance funding to counties that meet or exceed goals; programs that fail can be terminated at any time, all providers reapply for funding every 2 years

Setting goals to increase outcomes. Kentucky n Adult Education Report Card measures (higher goals than required): Are more Kentuckians participating in adult ed programs? n Are more adult ed students meeting their educational goals? n Are more adult ed students advancing onto PSE within 2 years of completion of GED? n Are more adult ed students prepared for employment? n Are KY’s communities and economy benefiting? n

Kentucky

Rethinking goals in light of data: KY in 2007 n New framework with greater focus on helping adults gain academic skills needed for PSE and higher skilled jobs n Lower enrollment goals n n n Enrollment goals previously drove system with 60, 000 served in 2001, 115, 000 served in 2005 Average attendance hours for adults enrolled more than 12 hours was half of the national average Greater emphasis on quality student outcomes n Failed to meet GED targets (28% drop in GED grads from 2000 -05)

Rethinking goals in light of data: KY in 2007 n Revised formula funding and new opportunities for n n programs to earn performance funding More flexibility in providing adult education services All students, including those in family literacy and corrections education programs count toward targets Want programs to think more about program design, recruitment and retention strategies, and staffing needs Performance funding addresses program and student performance