Social Dynamics, LLC Bringing Complex issues into Focus DEI Grantee Conference DEI Evaluation March 12, 2013 In partnership with Altarum Institute Berkeley Policy Associates Mathematica Policy Research
Social Dynamics, LLC Bringing Complex issues into Focus Overview of the Session § Welcome & Introductions • ETA/ODEP Partnership • DEI Evaluation Team • Grantees’ Direct Link to the DEI Evaluation: State Evaluation Liaisons § Overview of the DEI Evaluation Design § Six Main Research Questions of the DEI Evaluation § Partnership & Collaboration Objectives of the DEI Evaluation Team
Overview of the Session (continued) Social Dynamics, LLC Bringing Complex issues into Focus § DEI Evaluation Synthesis Reports § What we Learned § DEI Data Quality • What do we mean by data quality? • Why is it important? § Systems Change Component of the Evaluation • • • What is it? How will it be used? SCCS Structure
Social Dynamics, LLC Bringing Complex issues into Focus Welcome & Introductions ODEP/ETA Partnership § ODEP—Division of Policy Planning & Research • Dr. Richard Horne, Supervisory Research Analyst • Dr. Janet Voight, Contracting Officer’s Representative § ETA—Disabilities Program • Randee Chafkin, M. A. , Senior Program Specialist • ETA Regional FPOs
DEI Evaluation Team Social Dynamics, LLC Bringing Complex issues into Focus ODEP COR ODEP J. Voight R. Horne Project Director Subject Matter Experts D. Stapleton G. Livermore D. Klayman Site Visit Coordination SD TA & Grantee Communications SD Data System Altarum Qualitative Analysis BPA Statistical Analysis SD R. A. & LWIB Selection Mathematica
Grantees’ Direct Link to the DEI Evaluation: State Evaluation Liaisons Social Dynamics, LLC Bringing Complex issues into Focus Each grantee has a State Evaluation Liaison who will: § Respond promptly to DEI Evaluation-related questions from State, LWIB, and American Job Center personnel; § Ensure smooth implementation of the DEI Data System at the American Job Center level; and § Coordinate, design, and schedule technical assistance related to the DEI Evaluation (e. g. , Data System site visits, webinars, conference calls, etc. ).
Social Dynamics, LLC Bringing Complex issues into Focus DEI Evaluation Liaisons Kim Barlow, M. P. H. IL, KS, NJ Dr. Robert Bartolotta AR, IN, LA, MN Dr. Robert Bleimann HI, WA Danielle Herbert, M. A. FL, MA, RI Dr. Becca Jackson AK, OH, TN, SD Dr. Douglas Klayman CA, WI Dr. Susan Labin IA Marilia Mochel, M. A. DE, ME, NY, VA KBarlow@socialdynamicsllc. com RBartolotta@socialdynamicsllc. com RBleimann@socialdynamicsllc. com DHerbert@socialdynamicsllc. com Rjackson@socialdynamicsllc. com dklayman@socialdynamicsllc. com Slabin@socialdynamicsllc. com MMochel@socialdynamicsllc. com 1 -855 -990 -1105 (toll free)
Social Dynamics, LLC Bringing Complex issues into Focus The DEI Evaluation Design The DEI Evaluation is a comprehensive implementation/process, outcome, and impact study. § Implementation: Collection of information on systems change and the strategies implemented by each of the DEI grantees (site visits/telephone interviews). § Outcomes: Focus on customer-level outcomes based on administrative data (WIASRD, W-P, DEI Data System). § Impact: Focus on comparing DEI pilot sites with comparison sites (WIASRD, W-P, DEI Data System).
Social Dynamics, LLC Bringing Complex issues into Focus The DEI Evaluation Design: Random Assignment § Random assignment ensures that the DEI pilot and comparison sites are equivalent in terms of all extraneous variables. In other words, the characteristics of LWIAs in both groups are the same. The comparison group serves to illustrate: What would have happened in an equivalent group of LWIAs in the absence of the DEI? 9 9
Social Dynamics, LLC The DEI Evaluation Design (continued) Bringing Complex issues into Focus § Site Visits: Years 1 and 3 to discuss start-up and implementation issues. (Component: Implementation) § Mid-Term Telephone Interviews: Year 2 interviews with DEI State Leads to learn how grantees use the selected strategies to improve their public Workforce Development Systems and better serve PWDs. (Component: Implementation) § WIASRD, Wagner-Peyser, and the DEI Data System: Customer outcome data will be sent quarterly to the Evaluation Team. (Component: Outcome & Impact)
Social Dynamics, LLC Six Main Research Questions Bringing Complex issues into Focus 1. What start-up issues were identified? How were these issues resolved? Sources: Site visits and telephone interviews (Component: Implementation) 2. What are the impacts of the DEI grants on customer utilization and disability employment outcomes such as employment, earnings, and employment retention? Source: Administrative data—WIASRD/W-P/DEI Data System (Components: Customer Outcomes & Impact) 3. What services and strategies appear to be the most successful in helping customers achieve employment outcomes? Sources: Site visits, telephone interviews, and administrative data (Component: Implementation)
Social Dynamics, LLC Six Main Research Questions (continued) Bringing Complex issues into Focus 4. What are the challenges that need to be addressed to improve the employment outcomes of PWDs? Sources: Site visits and telephone interviews (Component: Implementation) 5. What systems changes need to be made to overcome these challenges? Sources: Site visits and telephone interviews (Components: Implementation & System Change Outcomes) 6. What best practices have the pilot sites identified that address these systems changes? Sources: Site visits and telephone interviews (Component: Implementation)
Partnership & Collaboration Objectives of the DEI Evaluation Team Social Dynamics, LLC Bringing Complex issues into Focus The DEI Evaluation is a partnership between DOL, the DEI grantees, and the Evaluation Team. There are five partnership objectives: 1. To support the collection of pilot and comparison DEI Evaluation data in LWIAs and American Job Centers through telephone, on-site, and webinar technical assistance. 2. To provide ongoing support for implementing the DEI Evaluation and using results for program monitoring and improvement.
Social Dynamics, LLC Partnership & Collaboration Objectives (continued) Bringing Complex issues into Focus 3. To involve grantees and stakeholders in the evaluation process and use their input to inform the analysis of findings related to program context and systems change. 4. To provide annual presentations on DEI Evaluation results (when available) to inform local planning activities. 5. To maintain open lines of communication throughout the program period.
DEI Evaluation Synthesis Report(s) Social Dynamics, LLC Bringing Complex issues into Focus What is the Annual Synthesis Report? § Informs ODEP and ETA of progress made to-date on DEI goals; § Provides useful information on DEI start -up (on both the program and the evaluation side); and § Documents emerging systems changes.
What we Learned Social Dynamics, LLC Bringing Complex issues into Focus (Examples from Years 2– 3 R 1 & R 2 States) Rounds 1 & 2 Grantees: Moving Forward § Numerous R 1 and R 2 grantees have made significant strides in creating awareness of the needs of CWDs, developing their network of partners, and extending training opportunities statewide. • A DRC in ME provides training to executive-level LWIB staff, which is described as “revolutionary” by those who attend. “It gave us, for the first time, a comprehensive understanding of what we need to do to create better access to services for customers with disabilities. ” • To promote the sustainability of the DEI, IL DRCs train American Job Center staff on the availability of benefits and how to handle TTW customers.
Social Dynamics, LLC Bringing Complex issues into Focus What we Learned (continued) Rounds 1 & 2 Grantees: Moving Forward – continued § In NJ, one LWIA distributes DEI resources across four different people who play the DRC role. This initiative increased the availability and visibility of the DRC position and is expected to improve the knowledge of American Job Center staff through informal and formal training opportunities. § DRCs in NY are now all CWIC-trained! § Several states, including DE, ME, SD, and TN, have created customer-focused IRTs that bring multiple agency partners to the table (in-person) on a customer-by-customer basis. This arrangement has led to the braiding of funds that increase the availability of programs and services.
Social Dynamics, LLC Bringing Complex issues into Focus What we Learned (continued) Rounds 1 & 2 Grantees: Moving Forward – continued § TN uses a statewide resource mapping initiative to identify available resources (and the need for new resources) in order to build the capacity for increasing access and availability to programs and services for CWDs. § An LWIA in NJ developed a comprehensive, customer-driven monitoring process to ensure that American Job Center CWDs help to shape the kinds of trainings, programs, and services the American Job Center offers.
Social Dynamics, LLC Bringing Complex issues into Focus What we Learned (continued) Variation in Implementation § The DEI requires that grantees implement specific strategies and infrastructure arrangements, but provides few requirements as to how exactly this implementation is to happen. This approach gives the grantees the flexibility to build on existing strategies and to explore new and innovative strategies in serving PWDs and, in year 2, gave us and DOL valuable information about promising practices in this regard. § Because of this flexibility, we expected to—and did—find a great deal of variation across the grantees. § DRCs may either work at the American Job Center/consumer (local) level or at the LWIA (system) level. In most cases, the State defines the role for the DRC, but how this role is operationalized varies (often by LWIA). Stark differences in the DRC role occur in the same state and in contiguous LWIAs.
Social Dynamics, LLC Bringing Complex issues into Focus What we Learned (continued) Variation in Implementation – continued I. Local-level (American Job Center/consumer) DRC responsibilities vary. There are three types: a. Focuses exclusively on TTW. b. Works only with CWDs. c. Plays support (helps other staff) and employment counselor role. Best Practice of Local-Level DRCs § Provide informal training and support to increase the overall level of knowledge of CWDs in American Job Center staff rather than being the “the funnel” for CWDs who enter the American Job Center.
Social Dynamics, LLC Bringing Complex issues into Focus What we Learned (continued) Variation in Implementation II. continued System-level DRCs spend most of their time overseeing the work of American Job Center employment counselors, helping them to identify key training and benefits opportunities for CWDs. They are more likely than locallevel DRCs to offer formal training to American Job Center staff and network outside of the American Job Center with the LWIB and other local systems (e. g. , education, juvenile justice, employer community, etc. ) Best Practice of System-Level DRCs § DRC implementation to increase the knowledge of staff who have the skills to work with CWDs (AK). Relatedly, some states utilize supervisory (LWIA-level) DRCs (IL, ME, VA) whose job it is to oversee local DRCs and/or American Job Center employment counselors to build capacity.
Social Dynamics, LLC Bringing Complex issues into Focus What we Learned (continued) Examples of Variation in Implementation § There is also substantial variation in how DEI strategies are implemented at the LWIA level. • Integrated Resource Teams (IRTs) are customer-centric, and meet in-person with the DRC, agency/organizational team members, customer, and (if youth) parents/guardians. • IRTs are focused on problem/resolution. DRCs separately contact “the individuals that are important to what we’re trying to do in terms of funding. ” This approach focuses on the problem at hand (e. g. , customer needs certificate-based training, customer needs GED to get a job). This approach does not provide an opportunity to brainstorm ideas with customer and partners and involves less customer input. • Customized employment is defined very generally as a way to fit a job around an individual. We have not seen the more intensive approach that includes Discovery and Negotiation in the states that selected this strategy.
Social Dynamics, LLC Bringing Complex issues into Focus What we Learned (continued) Examples of Variation in Implementation - continued • Asset Development is being defined broadly as both informal discussions and formal trainings on savings accounts, banking, and financial literacy. LWIAs do not always use a specific curriculum, but more likely incorporate modules into more general trainings on job readiness. • The Guideposts for Success are used informally by youth states as a general structure for youth services. At this point, there is no evidence that they have been incorporated into youth programming. • Disability disclosure remains a challenge. In some LWIAs, DRCs are coaching employment counselors to try to obtain information on customers’ disabilities in order to help them get certain services, while in other states DRCs and employment counselors rely on self-disclosure only. This type of variability may affect the number of American Job Center-enrolled CWDs.
Social Dynamics, LLC Bringing Complex issues into Focus What we Learned (continued) Concerns of R 1 & R 2 States § Some American Job Centers are concerned that their EN status is being usurped by the State EN, as funds collected by American Job Centers ENs go to the State. § In a number of states, existing providers that are ENs are unhappy with the new EN status of their local American Job Centers. • This arrangement puts the system into a competitive mix. § CWIC loss mentioned as a deficit by several LWIA staff. § In youth states, permission to speak with a customer under the age of 18 requires parental permission, which complicates the process of engaging youth in Workforce Development services.
Social Dynamics, LLC Bringing Complex issues into Focus What we Learned (continued) Concerns of R 1 & R 2 States – continued § Some youth states have had difficulty placing formerly incarcerated youth and have spent considerable time working with youth to have their criminal records expunged. § Several American Job Center staff expressed concerns about the lack of awareness of mental disabilities. “Employers don’t seem to think that mental health issues are important. They don’t think of these customers as having a disability, which creates difficulties when symptoms affect behavior. ”
Social Dynamics, LLC Bringing Complex issues into Focus Data Quality What is Data Quality? § The degree to which a dataset can provide accurate and reliable information. § Content quality identifies erroneous information (e. g. , missing, out-of-range entries). § Predictive quality measures the extent to which a dataset can accurately estimate population parameters.
Social Dynamics, LLC Bringing Complex issues into Focus Data Quality (continued) Why is Data Quality Important? § Better position to accurately measure the impact of the DEI. § Ensures we can produce analyses on important subgroups (e. g. , TTW customers). § Ensures that your work is accurately represented in annual and final reports.
Data Quality (continued) Social Dynamics, LLC Bringing Complex issues into Focus What Improves Data Quality? § Consistency in how each Data Element is defined. • Is the perceived barriers question asked the same way by each grantee? § Completion of all required Data Elements. • Are all Data Elements being completed by staff at each American Job Center? § Consistency in American Job Center operations. • Do all LWIA American Job Centers ask the “disability question”? • Is data collected from customers at approximately the same time intervals? • Is the eligibility criteria for certain services approximately the same across American Job Centers?
Social Dynamics, LLC Bringing Complex issues into Focus Data Quality (continued) How can the DEI Evaluation Team help with Data Quality? § Careful review of all DEI grantee American Job Centers to ensure that they collect the required DEI Data Elements. § Provide Data Quality Reports that pinpoint where errors occur. § Quarterly Data Quality Reports sent to DEI State Leads. § Webinars for State, LWIB, and American Job Center staff on DEI data collection requirements. § Issue Briefs on data quality and data collection requirements. § Toll-Free Telephone Technical Assistance available to DEI State Leads, State IT staff, and American Job Center staff.
Social Dynamics, LLC Bringing Complex issues into Focus Data Quality (continued) Data Quality and Statistical Control The DEI Evaluation Team will monitor errors rates using an error tolerance band of 3% with +/-1 percentage tolerance; an acceptable error rate for DEI data is 4%. The objective is to determine what proportion of records have mean error rates above 4% (upper-specification limit), and to correct, isolate, and resolve these errors. Percentage of customers expected to be outside the upper tolerance zone of 4%. 4 3 2 0 3+-1 percentage points 2% Tolerance Band 4% To determine the proportion of case records that have a mean error rate above 4% (upperspecification limit): Z = (8 - 5) / S. D. 2. 78 = 1. 08 or 8% of case records have errors.
Data Quality (continued) Social Dynamics, LLC Bringing Complex issues into Focus Data Quality and Statistical Control - continued Pareto analysis can be used to detect the most frequently occurring sources (or vital causes) of error and the data reporters that have the highest error rates (vital cases). Pareto analysis does this by arranging the dataset according to priority of importance. Where should we be spending our time? Percent of Error Ordered by Type of Error (Pareto Distribution) Cumulative Distribution/ Pareto Curve Cases with Missing, Duplicate, and/or Invalid SSNs 30% of N 0 Percent of Cases by Type of Error 40 20 0 LWIB 4 LWIB 1 60 LWIB 3 Percent of Total Error Missing Dis status 20 Invalid SSN 40 Duplicate SSN 60 Missing SSN 80 Percent of Total Cases 90 LWIB 2 100 LWIB 5. . . . k
Social Dynamics, LLC Bringing Complex issues into Focus Data Quality (continued) What can we do to Maximize Data Quality? § We will contact you quarterly to report on any data collection issues. § We will provide you with LWIA-level information on, for example: Number/% records with complete WIASRD, W-P, and DEI Data Element data for all customers, and for CWDs only. Specific variables or Elements that are not being completed regularly. § If issues arise, we can participate in conference calls with particular LWIAs or American Job Centers.
Social Dynamics, LLC Bringing Complex issues into Focus Data Quality (continued) What can you do to Maximize Data Quality? In addition to collaboration between the Evaluation Team, State Leads, and IT staff, several states have taken additional steps to help ensure data quality. For example: § NY had a webinar to introduce staff to the mechanism for collecting DEI Data Elements. During this call, the importance of complete data was emphasized. § VA permanently adopted all of the DEI Data Elements into their everyday common intake form. § WA is assigning specific data entry duties to Americorps volunteers, who will be specially trained to enter DEI data.
Systems Change & the DEI Evaluation Social Dynamics, LLC Bringing Complex issues into Focus § ODEP and ETA have extensive experience working on systems change issues. § Social Dynamics adapted systems change approaches from ODEP/ETA to create a DEI systems change coding scheme and rubric. Handout § What do we mean by “systems? ” • Systems are aligned networks of organizations, individuals, processes, and relationships that are grounded in values and principles that provide PWDs with access to services and supports. • Systems involve collaboration across agencies, employers, Workforce Development staff, others groups, and individuals for the purpose of improving access to services and employment and expanding the array of services and supports.
Social Dynamics, LLC Systems Change & the DEI Evaluation (continued) Bringing Complex issues into Focus § Functional systems have/create the necessary context, structure, and expertise to achieve their goals. They: • Adapt/change according to social, political, economic, and organizational circumstances; • Harness the different functional areas and expertise of its members; and • Work toward a common goal of providing integrated services to PWDs that lead to more effective employment policies and services for PWDs.
Social Dynamics, LLC Systems Change & the DEI Evaluation (continued) Bringing Complex issues into Focus General Indicators Context Social and political context that leads to resources that Components Provide for the design and implementation of effective strategies and Adapted from Coffman, 2007 Relationships Create better linkages among stakeholders across the Workforce Development System, which leads to Capacity Subsystems and processes that facilitate continuous improvement and monitoring Scope So that the system can produce broad impacts on PWDs.
Systems Change & the DEI Evaluation (continued) Social Dynamics, LLC Bringing Complex issues into Focus General Questions Context 1. Has the DEI changed the social and political context of LWIAs? 2. Has the DEI produced changes in the way resources are used to enhance policies or practices that will improve system components, capacity, and scale? Components 1. Did the DEI grantees implement selected strategies as intended? 2. Did these strategies produce their intended outcomes for PWDs? Adapted from Coffman, 2007 Relationships 1. Did the DEI create linkages across the system that did not exist prior to implementation that improved access to needed services/ resources? 2. Did the DEI enhance existing relationships that improved access to needed services/ resources? 3. Did these linkages produce their intended outcomes on the system? Capacity 1. Did the DEI produce enhancements in the capacity of participating LWIAs to meet the needs of PWDs? 2. Did the enhancements meet sustainability and quality objectives? Scope 1. Did the DEI lead to better access to and availability of services and resources for PWDs? 2. Did the DEI result in the intended impacts for PWDs at a system-wide population level?
The SCCS Development Process Social Dynamics, LLC Bringing Complex issues into Focus § Clients (ODEP and ETA) have extensive experience working on systems change procedures. • Customized Employment (ODEP) • Self-Employment for People with Disabilities (ODEP) • Disability Workforce Development System TA (ODEP) • Disability Program Navigator Initiative (ETA) § Rather than reinventing the wheel, Social Dynamics adapted ODEP’s existing systems change framework so that it aligned with the DEI.
The ODEP Systems Change Framework Social Dynamics, LLC Bringing Complex issues into Focus ODEP’s 9 System Change Domains 1. Capacity 2. Coordination/Integration 3. Customer Choice 4. Employer Outreach 5. Employer Partnerships 6. Employer Support 7. Development or Adaptations of Existing Practices 8. Dissemination of Effective Practices 9. Sustainability
Social Dynamics, LLC Bringing Complex issues into Focus Additional SCCS Domains § System Boundaries — defined by the members and the relationships that perpetuate, challenge, and/or enhance system operations (Midgley, 2007). Defines who the members of the system are and their respective roles. § System Density — the proportion of an entity’s (organization or individual) ties (or partnerships) to other members of the system out of the total number of ties across the system (Barabasi, 2002). An indicator of the amount of collaboration across the system.
Additional SCCS Domains (continued) Social Dynamics, LLC Bringing Complex issues into Focus § System Centrality — the number of direct connections to each member of the system. Entities with high centrality are typically the leaders and facilitators of the system (Borgatti et al. , 2009). Helps to identify the leaders of the system and their respective roles. Why are they the leaders? What do they do to enhance system effectiveness? Leaders may include organizations/agencies or individuals.
SCCS Structure Social Dynamics, LLC Bringing Complex issues into Focus § SSCS is used to code qualitative data from interviews and focus groups with members of LWIA systems. Inter-rater reliability is obtain in two ways: 1. Team consensus is achieved through site visit team discussions. • Each team member completes the SCCS using the data from each site visit. A score is given by each team member using a rubric.
Social Dynamics, LLC Bringing Complex issues into Focus SCCS Structure (continued)
Social Dynamics, LLC Bringing Complex issues into Focus SCCS Structure (continued) 2. Weighted Kappa (linear weights for ordinal data). Degree of agreement between two site visitors who classify LWIAs’ “system” ratings into one of four ordinal groups at baseline and follow-up. Weight (w 1, 2) quantifies the degree of concordance between the ordinal groups 1 & 2, 1 & 3, and 1 & 4. W 12 = 1. 0 W 12 =. 75 W 12 =. 50 W 12 =. 00
Social Dynamics, LLC 481 North Frederick Ave. , Suite 410 Gaithersburg, MD 20877 Phone: (301) 990 -1105 www. socialdynamicsllc. com Bringing Complex issues into Focus To Contact the DEI Evaluation Team: Douglas Klayman, Ph. D. Project Director dklayman@socialdynamicsllc. com 301 -990 -1105 x 105 Toll free: 1 -855 -990 -1105
Скачать презентацию Social Dynamics LLC Bringing Complex issues into Focus
e68c2afb0f9388a658df0003590bb6b4.ppt