Performance Measurement Under Title 1 -B of the

Performance Measurement Under Title 1 -B of the Workforce Investment Act Regional Training Richard West Social Policy Research Associates

Topics to Be Covered • WIA Performance Measures – Definitions and calculations • Challenges – Working with wage records – Supplemental data – When outcomes occur and are measured – Understanding performance reports – Implementing new data collection processes – What about those JTPA exiters included in WIA performance measures?

17 Performance Measures • Separate measures for funding streams and youth age groups – Adults – Dislocated workers – Youth under 19 at registration – Youth 19 to 21 (under 22) at registration. • Individual served with multiple funding streams included in all relevant measures. • (Nearly) Common measures for adults, dislocated workers, and older youth.

• For adults, dislocated workers, older youth: – Entered employment rate – Employment retention rate 3 x 4=12 – Earnings change (Adults, Older Youth) or earnings replacement rate (DWs) – Credential attainment • For younger youth: – Goal attainment – Diploma attainment – Retention • Customer satisfaction – Participants – Employers 3 2 17

Performance Measures Keyed to the Exit Quarter • Exit quarter is the quarter containing the exit date. • Individuals exit if: – Formally withdraw from or complete services OR – There is no service for 90 days (WIA or tracked partner-funded services) • Exit date is the date of withdrawal or completion or the date of the last service.

WIA Exit Last Service WIA Participation No services WIA Exit Date End of 90 -day period

Time Periods Used for the Measures § Measures that use "real time" data. § Exiters who leave during the program year. § Applies to measures that do not use UI wage records: § Younger youth goal attainment § Younger youth diploma § Customer satisfaction § Measures that use UI wage records. § Exiters during the last 3 quarters of the previous PY and the first quarter of the current PY. § October 1 to September 30 § Applies to all other measures

Exit Periods for Performance 1 Previous PY 2 3 4 1 Current PY 2 3 4 Real Time Measures Wage Record Measures EE 2 Q Lag

Challenge: Working with Wage Records

Sources of Employment Data • Wage records the primary source for entered employment and retention; • Wage records do not include: – Individuals in uncovered employment, such as self-employment – Federal government employees and others outside the state UI system • May use out-of-state wage records – Bilateral agreements with nearby states – Wage Record Interchange System (WRIS)

When Are Wage Records Good Enough? • Measures sensitive to incomplete wage records • Example: – Wage records 90% complete – “True” employment is 70% – Then measured employment is: 0. 9 x 70 = 63% A seven percentage point bias • Really want records close to 99% complete • Typically a one or two quarter lag before wage records are good enough

WRIS • Enables states to obtain out-of-state wage records efficiently and cost-effectively. • WRIS is currently fully operational with 20 pilot states • By July 1, 2001 all states will be able to join. • Expected to be self supporting – Cost of participation has not been determined. – An initial pricing study is available on • Workforce. ATM. org, as is other more information on WRIS

How Does WRIS Work? • State submits a list of SSNs for matching • WRIS matches those SSNs against an index to identify the states with records for those SSNs. • Wage request is sent by WRIS to only those states with records for an SSN. • Receiving states send wage records back to WRIS. • Requesting states can download the information from WRIS.

Challenge: Developing a Realistic Process for Collecting Supplemental Data

Sources of Employment Data 2 • Supplemental data sources may be used if not person not found in wage records. – Case management, follow-up services, participant surveys. – Records sharing or automated record matching with other administrative databases. • Note: records that provide information on quarterly earnings will be considered as wage records, – Must be collected within 30 days after individual determined to not be in wage records

Supplemental Data “By the Book” • State gets wage record data • Local area is told who is not in wage records • Local area collects supplemental data within 30 days • Supplemental data is reported to state

Data Collection Timing for Employment Quarters After Exit 1 2 3 4 5 6 7 8 9 Get Wage Records Exit EE 2 Q Lag SD RTN 2 Q Lag SD 30 days to get supplemental Follow-up Services Can get supplemental early during follow-up services

“Real World” Supplemental Data • Local areas collect supplemental data during follow-up services – For everyone – For those likely to not be in wage records: • Self-employed, other noncovered jobs • Out of state employment • State gets wage record data and tells local areas who is not in wage records • Use supplemental for those not in wage records • Supplemental data reported to state

Calculating Performance Measures

Global Exclusions • Exclusions from all performance measures: – Adults & dislocated workers who received only self-service and informational core services – Institutionalized at exit (Item 622 = 1) – Health/medical exits (Item 622 = 2) – Deceased (Item 622 = 3) – National Emergency Grants only – Rapid Response only (except additional assistance) – Some statewide programs only

What Statewide Programs Can Be Excluded? • Very specialized statewide programs with their own specific goals, for example: – Incumbent worker training authorized under Section 134(a)(3)(A)(iv)(I), or – A project for chemically dependent TANF recipients. • These statewide programs are also excluded from WIASRD reporting (if they are the only program)

In What Performance Groups do Statewide Participants Count? YES Youth Displaced Homemaker Item 308 = 1 YES Adult or Youth Incumbent Worker Item 309 = 1 only YES Excluded Other Statewide Item 310 = 1 YES Adult or Dislocated Worker Rapid Response only Item 311 = 1 YES Excluded Rapid Response Additional Assistance Item 312 = 1 YES Dislocated Worker Youth Statewide? Item 307 = 1

Ambiguous Cases--Suggested Approach • If already in a performance group, ignore these statewide programs Other Statewide Item 310 = 1 NO YES Has Dislocation Date? Item 125 or also in NEG grant Item 313 a YES Displaced Homemaker Item 308 = 1 NO Age >= 22? Age < 18? NO YES Youth Services? YES NO Dislocated Worker NO Adult YES Youth

Entered Employment Rate Number Employed in Quarter after Exit divided by Number of Exiters – Wage records the primary data source – Supplemental data may be used for persons not found in wage records – Exclude adults, older youth employed at registration – Exclude older youth in advanced training/ postsecondary education (but not employed) in quarter after exit.

Entered Employment Rate Adults and Dislocated Workers Dislocated Worker Measure? NO YES Employed at Registration? Item 115 = 1 NO Employed in Quarter After Exit ? Item 601 = 1 YES Excluded YES Numerator ES Y NO Denominator

Entered Employment Rate Older Youth Employed at Registration? Item 115 = 1 YES Excluded NO Employed in Quarter After Exit Item 601 = 1 YES Numerator Y ES NO In Advanced Training or Postsecondary Education in Quarter After Exit Item 623 = 1 YES NO Denominator Excluded

Employment Retention Of those employed in the quarter after exit: Number Employed in 3 rd Quarter after Exit divided by Number Employed in Quarter after Exit – Based on those employed in quarter after exit – Wage records the primary data source – Can use supplemental data if not in wage records. – Excludes older youth in advanced training/ postsecondary education (but not employed) in 3 rd quarter after exit.

Employment Retention Rate Adults and Dislocated Workers Employed in Quarter After Exit Item 601 = 1 NO Excluded YES Numerator YES ES Y Employed in 3 rd Quarter After Exit Item 608 = 1 NO Denominator

Employment Retention Rate Older Youth Employed Quarter After Exit? Item 601 = 1 NO Excluded YES Numerator YES Employed 3 rd Quarter After Exit Item 608 =1 Y ES NO In Advanced Training or Postsecondary Education in 3 rd Quarter After Exit Item 624 = 1 YES NO Denominator Excluded

Earnings Change Adults and Older Youth Of those employed in the quarter after exit: Earnings in 2 nd and 3 rd Quarters after Exit minus Earnings in 2 nd and 3 rd Quarters before Registration divided by Number employed in quarter after exit – Wage records only • Exclude if supplemental data used to determine employment: – in either the 1 st or 3 rd quarter after exit. • Exclude if preprogram quarters too far back to get data – Exclude older youth in advanced training/ postsec. education (but not employed) in 3 rd quarter after exit.

Earnings Replacement Dislocated Workers Of those employed in the quarter after exit: Earnings in 2 nd and 3 rd Quarters after Exit divided by Earnings in 2 nd and 3 rd Quarters before Dislocation – Wage records only • Exclude if supplemental data used to determine employment: – in either the 1 st or 3 rd quarter after exit. • Exclude if preprogram quarters too far back to get data • Use 2 nd and 3 rd quarters before registration if no dislocation date or dislocation date after registration date – Add earnings over all included exiters before dividing

Earnings Change--Adults Earnings Replacement--DWs Employed in Quarter After Exit Item 601 = 1 NO Excluded Supplemental Data Used Q 1 Item 602 = 1 or 2 NO Supplemental Data Used Q 3 Item 609 = 1 or 2 YES NO YES Numerator and Denominator

Earnings Change Older Youth NO Excluded YES NO Employed 3 rd Quarter After Exit Yes Item 608 =1 NO In Advanced Training or NO Postsecondary Education in 3 rd Quarter After Exit Item 624 = 1 YES Supplemental Data Used Q 1 Item 602 =1 or 2 Supplemental Data Used Q 3 Item 609 =1 or 2 NO Included YES Employed Quarter After Exit? Item 601 = 1

Credential Rate • Flexible definition of credential – Standard credentials – Certificates that recognize the successful completion of training services …. • Credential attainment by end of 3 rd qtr after exit • Positive outcome is attain credential and: – Enter employment (adults, dislocated workers) – Enter employment or in advanced training/postsec. educ. in quarter after exit (older youth) • Includes: – Adults/DWs who received training. – All older youth

Credential--Adults and DWs Of those who receive training: Number who enter employment and attain credential divided by Number of exiters Credential--Older Youth Number who enter employment and or are in advanced training or postsecondary education in the quarter after exit divided by Number of exiters

Employment and Credential Rate Adults and Dislocated Workers Received Training? Item 333 is a valid date NO Excluded YES Employed in Quarter After Exit Item 601 =1 NO Denominator YES Attained a credential Item 621 = 1, 2 , 3, 4, 5, or 6 YES NO ES Y Numerator

Credential Rate--Older Youth Employed in Quarter After Exit Item 601 =1 YES NO In Advanced Training or Postsecondary Education in Quarter After Exit Item 623 = 1 NO YES Denominator Attained a credential Item 621 = 1, 2 , 3, 4, 5, or 6 NO YES Numerator

Younger Youth Goal Attainment • Outcome attained during participation • 3 goal areas: – Basic skills • A basic skills goal must be set if the youth is deficient in basic skills – Work readiness skills – Occupational skills • Flexibility in defining goals • Youth has a year to attain the goal • Maximum of 3 goals set per year (for performance) – First year begins on registration date and ends a year later. – Goals used for performance must be identified up front

Younger Youth Goal Attainment-2 Number of goals attained divided by Number of goals attained or reached anniversary/exit date without attaining • Add numerator and denominator over all youth before dividing. • Include all in-school youth and out-ofschool youth assessed as needing basic skills, work readiness skills, or occupational skills

HS Diploma/GED Attainment Of youth who were not HS grads at registration: Number of exiters who attained diploma divided by Number of exiters not still in school at exit • Diploma can be attained during participation or by end of quarter after exit • Exclude youth still in secondary school at exit

Younger Youth Diploma Rate HS grad at registration? Item 123 > 11; Item 130 = 2 or 4 YES Excluded NO Attained Diploma? Item 673=1 or 2 YES Numerator Y ES NO In Secondary School at Exit? Item 673 = 3 YES NO Denominator Excluded

Younger Youth Retention • In one or more qualifying outcomes in 3 rd quarter after exit – Postsecondary education (Item 676 = 1) – Advanced training (Item 676 = 2) – Military service (Item 676 = 3) – Qualified apprenticeship (Item 675 = 4) – Unsubsidized employment (Item 676 = 5) • Exclude youth in secondary school at exit • Not related to outcome in quarter after exit

Younger Youth Retention Rate In Secondary School at Exit? Item 673 = 3 YES Excluded YES Numerator NO ES NO Y Retained in 3 rd Quarter After Exit Item 676=1, 2, 3, 4, or 5 Denominator

Challenge Understanding When Outcomes Occur and When they Are Measured

When Outcomes Occur PY 2000 Q 1 Entered Employ. Q 2 Exit Q 3 PY 2001 Q 4 Q 1 Q 3 Q 4 EE RTN Retention 12 Mo Credential Youth Diploma Q 2 When outcome occurs =When Measured

Challenge Understanding Performance Reports What you get when you ask for “the most recent data available”

Previous PY 1 2 3 4 1 Current PY 2 3 4 Real Time Measures Exiters Cust Sat Yth Diploma Yth Goals Wage Record Measures EE RTN Earnings Credential Look at Data Available May 15

Challenge: Implementing New Data Collection Processes

Outcome Data Collection Activities • Wage records – Collect quarterly (both in and out of state) • Supplemental employment data – Collect during follow-up services – Collect quarterly after not found in wage records if necessary • Credential – Collect during participation – Update after the 3 rd quarter after exit • Diploma – Collect during participation – Update after the quarter after exit

Challenge: What about those JTPA Exiters Included in WIA Performance Measures?

JTPA exiters from October 1, 1999 to June 30, 2000 are included in : • • – All adult, dislocated worker, and older youth measures – Younger youth retention measure. Are you getting wage records for them? Are you collecting supplemental employment data? (Do you need it to meet your goal? ) Are you collecting credential data? Are you collecting younger youth retention outcomes? It’s not too late now, but …. . .