How Career Development Services Are Assessed The Canadian

How Career Development Services Are Assessed! The Canadian research group on measuring the impact of career development services (Robert Baudoin, Lynne Bezanson, Bill Borgen, Liette Goyer, Bryan Hiebert, Vivian Lalande, Kris Magnusson, Guylaine Michaud, Céline Renald, and Michel Turcotte) ***Names listed in alphabetical order*** Canadian group - impact of services 05

Why this interest? • International symposia in 1999 and 2001 • The OCED study on career development, 2002 -2004 • The International Centre for Career Development and Public Policy, 2003 • Canada-wide symposium on career development, continuous education and human resource development, 2003 Canadian group - impact of services 05 2

What do we mean by career development? • The aim of high-calibre career development services is to help individuals to: • become informed about the various training and work options; • choose the options that correspond best to their interests, values, aptitudes and beliefs as they make occupational choices and identify their aspirations in an uncertain and changing environment. Canadian group - impact of services 05 3

What career development services contribute • High-calibre services that can: • contribute to an individual’s well-being and overall development. • stimulate motivation for studies and work • help maintain good mental health for work and studies • create a sound foundation for building effective skill development and training systems. Canadian group - impact of services 05 4

Benefits of career development for society • Spin-off benefits may include: • Fewer learners operating in “trial and error” mode • Higher success rates in programs, courses, and obtaining certificates and degrees • Less time on social assistance • Lower healthcare costs • Fewer social drop-outs • Happier employees, less sick leave, higher retention rates, greater involvement in training and learning Canadian group - impact of services 05 5

Why is assessment important? • To obtain funding • To have a clearer idea of actual results as opposed to expected results • To request the investment needed to achieve the expected results. Canadian group - impact of services 05 6

Lack of solid data • At this time, we unfortunately don’t have much information on how career development services are assessed. • Before suggesting ways of solving the problem, it is crucial to know more about the current state of assessment practice and its impact. Canadian group - impact of services 05 7

Responses to the need for solid data • Fall 2004: online survey: • How do service providers assess the impact of their services? • Our goal: To better understand the state of current practice in Canada with respect to documenting the results of career development activities and services. Canadian group - impact of services 05 8

The purpose of the survey: 4 main questions 1. How important is it to assess the impact of career development services? 2. How can the impact of these services be determined? 3. What are the results from such services? 4. How can these results be measured? Canadian group - impact of services 05 9

Data collection • • Questionnaire for agencies (online) Questionnaire for practitioners (online) Questionnaires in both official languages Telephone interviews with employers and policymakers Canadian group - impact of services 05 10

What we mean by “results” • A reaction or specific product that results from an activity • A result can be one or other of the following : • Changes in the client’s skills: demonstrated by changes in the client’s knowledge, abilities or characteristics • Changes in the client’s situation: for example, job situation, educational situation, etc. • General changes for clients and the community: • Changes in terms of financial situation, social integration, impact on families, community economic development, etc. Canadian group - impact of services 05 11

What we mean by “activities” Any activity carried out with the intention of helping clients to make changes to their individual situations – in other words, any action that produces a result for a client or a group of clients. Canadian group - impact of services 05 12

Survey sample • 173 agencies; 214 practitioners; 9 policymakers and 7 employers • The types of service offered include all customary services – individual interviews, group sessions, assessment, LMI and employment development • Target age-group: mostly young people and adult jobseekers, followed by young people at the post-secondary level or finishing high school • Type of agency: mostly NGOs (50. 3%), followed by noneducational provincial organizations (23. 3%) • Agency size: 41% with over 10 employees; 20% with 6 -10 employees; and 39% with fewer than 6 employees Canadian group - impact of services 05 13

What importance is put on measuring the results or effectiveness of the services provided? 97. 6% of the agencies responded that it was “quite important” (24. 4%) or “very important” (73. 2%) 84% of the agencies report on the impact or effectiveness of their services. Canadian group - impact of services 05 14

There is a significant relationship between the type of service provider and the importance placed on measuring impact and determining its actual value • Agencies: Primary and secondary schools are less inclined to respond “very important” and post-secondary institutions are less inclined to respond “quite important. ” • Practitioners: Practitioners in educational institutions are less inclined to respond “very important, ” and the NGOs are more inclined to respond “very important. ” Canadian group - impact of services 05 15

Agencies and practitioners: What are the 3 most important results you report on? 1. Changes in the client’s job or educational situation; ** 2. Development of skills; financial independence, self-confidence; ** 3. 4. 5. 6. 7. and, to a lesser extent: Number of clients served Client satisfaction Programs completed Service delivery Cost/benefit ratio Canadian group - impact of services 05 16

Agencies and practitioners: Types of evidence for results • Assessment/follow-up reports: Reports from client or a partner; investigations, interviews • Frequencies: number of clients served/month; number of clients finding jobs; number of action plans; number of clients completing programs To a lesser extent: • • Observation of changes in the client Cost/benefit analysis No difference between agency and practitioner responses 33% of the practitioners did not respond Canadian group - impact of services 05 17

Comments received • Assessment is difficult • Certain results can’t be measured • Too much importance put on employment statistics – to the detriment of other types of results • There is need for research on career development services and on specific populations Canadian group - impact of services 05 18

What results do you obtain that are not measured or reported? • Developing the client’s ability to act • Skill development (e. g. , self-confidence, independence, changed attitudes towards their future and their perception of the labour market) • Financial independence • Creation of support networks • More job opportunities and other kinds of opportunities for clients To a lesser extent: benefits for the community; client satisfaction; increase in demand for political lobbying (agency); program implementation and development (agency); repeat clients (practitioner); professional development (practitioner) Canadian group - impact of services 05 19

What proves that you have achieved these results? • Anecdotal evidence • Verbal reports from clients • Verbal reports from employers • Observations • Observed changes in the client’s attitudes • Recurring theme in the data: belief that several of the results observed or reported are either not quantifiable or are very difficult to measure (and, as a result, are not taken seriously) Canadian group - impact of services 05 20

What difficulties do you experience in proving or measuring results? • Complexity • Difficulty in determining the results of the services provided • Belief that certain results cannot be measured • Assessment considered to be of little importance • Lack of allotted resources (money and time) • Little recognition of the importance of collecting solid data • Lack of training in evaluation methods • Difficulty in obtaining feedback from clients • Loss of contact with clients; clients don’t want to give feedback Canadian group - impact of services 05 21

Difficulties (cont’d) • Lack of uniformity or agreement between the agencies and the funders on the type of results to be collected • Little linkage between service delivery and assessment (agencies) • Absence of assessment protocols or formal processes for completing effectiveness evaluations • Lack of experience in, and limited access to, models of best practices (practitioners). • Rapidity of changes (practitioners) Canadian group - impact of services 05 22

What things would you like to report on, if given the opportunity? • The social impact of services on individuals, families, the community, and employers • Development of the ability to act, and client growth and satisfaction • Changes in behaviour and attitudes • Quality of services • Components of the process; time needed to achieve the goals • Longitudinal impact • Link between local and national results Canadian group - impact of services 05 23

The other side of the coin What do policymakers and employers think? Canadian group - impact of services 05

What do policymakers want? • Client results • e. g. , client satisfaction, intelligent decisions, and client’s ability to identify career opportunities. • Indicators of client results • e. g. , number of jobs obtained, number of successful transitions • Services that make a difference in client’s lives; the change would not have occurred without the service provided • Value added of the services in terms of return on investment or economic improvement • Longitudinal studies to make comparisons between those who receive services and those who don’t Canadian group - impact of services 05 25

Other information that policymakers are interested in • How services are delivered • Number of clients served; number of places offered in programs, client assessment • Level of service provided and degree of correspondence between level of service and the client’s expressed need • Monitoring and assessing contracts Canadian group - impact of services 05 26

What do employers want? • Employees who are well-trained, competent, committed, motivated and satisfied • Increased productivity, less employee turnover, skill-based job descriptions, improved internal employee mobility • Feedback from employees and service providers on what worked and what didn’t Canadian group - impact of services 05 27

What policymakers and employers agree on • Need to collect solid data linking results and services • Need to develop better methods for assessing results. Canadian group - impact of services 05 28

Next steps • Develop detailed impact assessment models • Develop clear, valid and reliable tools • Create an analytical framework for comparing data (meta-analysis) • Create a mechanism for disseminating and promoting service impact assessment information. Canadian group - impact of services 05 29

Thank you! The Canadian research group on measuring the impact of career development services Canadian group - impact of services 05