Tracer Study Approach in Evaluation of BTVET Presented

Tracer Study Approach in Evaluation of BTVET Presented To Uganda Evaluation Association & ECD Project Kampala Evaluation Talk Thursday, 2 nd October 2014 1

Presenter • • Muhwana Wilberforce, MWSES CONSULT Ntinda, Steka House, 1 st Floor Plot 3122 Ntinda – Kiwatule road wamuhwana@gmail. com; mwssconsult@rocketmail. com; 0772429679; 0701242652 Social Sector Development Consultancy Education, Health, Community Livelihoods Policy/project formulation, Baseline surveys, Mid and End Line Evaluation, Impact Assessments, Tracking studies Government ministries, Institutions, Donor agencies, NGOs 2

Study team MUHWANA WILBERFORCE (LOCAL CONSULTANT) – Kampala, Uganda CLAUDIA MUMMENTHEY (INTERNATIONAL CONSULTANT) – Cape Town, South Africa Sourced by BTC separately Teamed up to harmonise the methodology for the evaluation Delivered a single inception report Supported by 7 RAs and 3 data entrants 3

Background to the study • • The Business, Technical, Vocational Education and Training (BTVET) Subsector is under the Ministry of Education and Sports (Mo. ES), BTVET Department and was established under TVET Act, 2008. BTVET offers training under an overlapping three - tier system: – (i) P 7 leavers to enroll in technical schools & community polytechnics, – (ii) ‘O’ leavers enroll in technical institutes & vocational training institutes, – (iii) ‘A’ leavers enroll in technical colleges; and – (iv) recently non-formal short courses across the various levels. 4

Background to the study – cont’d The sub-sector fell short of the overall goals and objectives regarding the production of a ‘critical mass’ of graduates that could readily be absorbed into the existing labour market. Internally the sector and subsector was grappling with issues of low quality of education and irrelevant curricula (UNESCO, 2005). Externally the BTVET system was not corresponding with the needs of the private sector (ESAPR, 2010). The private sector was not involved in curriculum development thus there was no realignment of knowledge and skills acquisition to the real needs of the labour market. There was widespread unfavourable perception and recognition of the BTVET programs among the public 5

Purpose of the study Mo. ES embarked on developing a 10 -year BTVET Strategic Plan in 2011 Six studies had been conducted ex- ante The Tracer Study conducted belatedly ◦ To generate data and information about the effectiveness of different BTVET programmes in Uganda as an input into the strategic planning process of identifying potential improvement areas for the overall system. 6

Objectives of the study • Overall objective: To provide an in-depth analysis of the employment destinations of graduates BTVET programmes. – To undertake a comparative analysis of factors affecting external effectiveness and relevance of training provided by the public and private sectors, – To identify employment destinations/career paths of completers of BTVET programmes and factors determining post – training careers 7

Objectives of the study – cont’d ◦ Accurately assess the potential significant differences in the external effectiveness of different BTVET programmes 8

Tracer study Tracing of a “reachable” (i. e. within the given restrictions of data availability and time constraints of the assignment) and representative cohort of BTVET graduates from the existing BTVET system using a scientific approach. ◦ Effectiveness – ability to equip learners with the skills and knowledge required in the labour market thus leading to employment readily. 9

Design of the Study • • Consensus on study expectations Methodology – Industry model – Scope – Sample Structure – Data collection tools – Data collection – Data recording and analysis – Data Quality 10

Consensus on Study Expectations • • Desk –based review of reports by strategy development group Debriefing by Strategy development group Debriefing/interviews with key stakeholders – Mo. ES, WB, BTC, GTZ/PIVOT On-site visits – BTVET institutions – public, private, rural & urban – Helped to understand expectations, quality of BTVET programmes, available data; planning for the study 11

Methodology Industry model: Input process focus on: ◦ ◦ output, with Training competence and infrastructure (input) training processes (processes) Training outcome (output/outcome) Overall satisfaction and image/reputation of BTVET 12

Methodology • • Scope: GRADUATES (i. e. learners who had passed final examination; learners who had completed the training but not yet received certificate) Current EMPLOYERS of graduates. Contextual elements: – Established the readiness of BTVET graduates being absorbed into in the labour market (i. e. formally employed or selfemployed) and/or accessing further education and training opportunities within the system. – Identified and examined the factors and circumstances that enhance and/or inhibit the placement of graduates into employment and/or further education and training opportunities. – Included the evaluation of BTVET delivery processes, interaction between BTVET institutions and employers and expectations of graduates and employers, among others. 13

Sample structure BTVET Graduate Cohort ◦ Multi dimensional & stratified: (i) By institutions – public + private = 650 ◦ arbitrary 10% representative sample – 65; ◦ Public – 33 + private – 32 = 65 (ii) By regional distribution - UGAPRIVI Classification 7 regions (i. e. Central (incl. Kampala), East, North, South West, West Nile and Western 14

Sample structure Region % region al dstn No. Public Inst. Sample private public inst. Central 124 24% 31 8 8 East 73 14% 18 4 4 North 73 14% 18 4 4 South 56 11% 15 3 4 South West 72 14% 18 4 4 West Nile 77 15% 20 6 9 6 5 West 43 8% 11 3 4 3 Total 518 100% 131 32 33 No. Private Inst. 15

Sample structure- cont’d • BTVET Graduate Cohort – (iii) By year of completion – distribution across different years of completion (2007 -2010) – (iv) By learning pathways –formal, informal, non-formal – (v) By exit examinations- UNEB, DIT – (vi) By different occupational training areas – (vii) By gender distribution – male- 80%, female – 20% • Target sample – telephone contacts for at least 60 graduates in each institution were compiled; 3629 contacts compiled; client’s target was 1500; 1415 graduates were interviewed by phone 16

Sample structure- cont’d • Employers – UBOS data on sector growth – agriculture, industry, services (potential for employment) – UBOS data on distribution of workforce by industry – Kampala & suburbs were covered – good concentration of businesses/companies – Formal & informal – Size of the enterprises • Target sample – contact details for 60 employers; 100 contacts compiled (more than 60 within Kampala); 63 interviewed, 12 not responded, 25 not contacted 17

Data collection tools Structured questionnaires for graduates and employers ◦ Inclusive of industry model focal areas ◦ 5 - point Likert-type scales ◦ Qualitative responses 18

Data collection • • • 7 RAs deployed in 7 regions to collect bio data (especially phone contacts) on all target graduates in the selected institutions; used a fixed template 3 RAs deployed to collect profile data on employer organisations (especially phone & e-mail contacts) 7 RAs administered the questionnaires to the graduates using telephonic interviews (English and in local languages) – 7 weeks, 50 interviews per week/ 10 per day • 2 RAs administered questionnaires to employers physically, collected completed questionnaires, sent via email – based on respondents’ preference – 3 weeks, 20 interviews per week/4 per day 19

Data recording and analysis • • • M&E web-based survey platform- Metior Solutions. 3 data recorders trained for ½ day Each data recorder was given account on the platform Entered each questionnaire and uploaded on the platform Quantitative and qualitative data was analysed using platform’s own instant analysis tools Descriptive statistics on quantitative data and summary reports on qualitative data provided instantly 20

Data quality • • • Training of RAs for 3 days including simulation & feedback Initial supervised/observed telephonic interviews – 5 interviews after training Confirmation of interviews – 5 in every 50 Paper questionnaires thoroughly checked Instant platform-based feedback on data entry – incomplete questionnaires, inconsistencies, errors in entries, etc 21

Key findings • Overall satisfaction and BTVET image – 59% of graduates were satisfied – 48% of employers were satisfied • Training quality and delivery (efficiency) – 55% of graduates provided a good rating – 35% of employers rated private institutions as excellent – 38% of employers rated public institutions as good • Overall BTVET outcome (effectiveness) – 64% of graduates felt that skills acquired made them ready for the labour market – 63% of employers felt that graduates were ‘prepared’ for applying the skills and knowledge but not up to the required levels of employment 22

Recommendations Establish effective BTVET Management Information System Upgrade, modernise and sufficiently resource existing training infrastructure Intensify and strengthen cooperation/partnerships between training institutions and industry Enhance female participation Focused BTVET programmes on local skills demands 23

Challenges of the evaluation • Lack of baseline data on both graduates and employers – A base population frame of contacts details had to be established before sampling • EMIS BTVET data on graduates not disaggregated by training programmes – Called for prior field visit to public and private institutions • Lack of up-to-date contact data on public and private institutions across 7 regions – individual research approaches on institutions had to be conducted • Obtaining employer interviews in the formal sector was cumbersome as it involved formal approval – Many were left out 24

Challenges of the evaluation Some employers required repetitive follow – ups and/or solicitation for nominal incentives ◦ Discarded from the list after failure through persuasion Less eagerness for participation among graduates in the central region ◦ opted for extra interviews in South and South West to compensate 25

Rationale for evaluation approach Tracer study periodically reaffirms quality, relevance and effectiveness of BTVET sub sector/related services Tracer study adequately informs direction for prioritisation Telephonic interviews minimise time consumed in ‘snowballing’ Metior platform facilitates quick delivery of analysed data particularly in restrictive ‘time bound’ situations 26