Development of Monitoring Evaluation Framework for ISCP-Tz

Development of Monitoring & Evaluation Framework for ISCP-Tz Sosthenes Sambua Diligent Consulting Ltd TCI Conference Capetown, 28 October 2008

8 pilot Cluster Initiatives - 2006 v v v v Arusha Seeds - Arusha Cultural Heritage Tourism - Bagamoyo Eastern Region Mushrooms - Kibaha Korogwe Small scale Sisal farming - Korogwe Morogoro Fruit & Vegetable Processors – Morogoro Metal Works – Morogoro Nutraceuticals (Functional Foods) – Dar es Salaam Zanzibar Sea Weeds - Zanzibar

11 second batch CIs v v v ICT - Dar es Salaam Educational Services – Dar es Salaam Small Scale Mining – Kilindi Oil Seeds Processing - Dodoma Rice Processing - Morogoro Wood Carving – Dar es Salaam Textile & Garments – Dar es Salaam Cassava Processing – Kibaha Cultural Heritage Tourism - Tanga Building Construction – Dar es Salaam Bio-fuels - Dar es Salaam & Morogoro

20 New CIs October 2008 v OIL SEEDS v v v v v SINGIDA OIL SEEDS - MANYARA RICE PROCESSING - MANYARA BEEKEEPING - MANYONI WINE PRODUCTION - DODOMA FURNITURE - MOROGORO FURNITURE - DAR-ES-SALAAM ICT DAR-ES-SALAAM CASHEWNUTS - MTWARA BASKETRYIRINGA BASKETRY - SINGIDA

. . . 20 New CIs October 2008 v ONIONS PRODUCTION - IRINGA v MUSHROOM v v v v RUVUMA MUSHROOM - PEMBA FISH PROCESSING & MARKETING – SAME FISH FARMING – MWANGA POULTY FARMING – DAR ES SALAAM CONTAINER GARDENING – DAR ES SALAAM METAL WORKS – DAR ES SALAAM FRUIT PRODUCTION - IRINGA

Do we need Monitoring and Evaluation Framework?

Monitoring is answering the question “how do you know where you are going, and tell when you arrive? ”

The process % Working with ISCP-Tz Secretariat % Defining overall objectives % Defining outcomes – not more than 5 % Defining intervention logic % Defining assumptions % Defining indicators % Defining questions % Defining monitoring methods

What is monitoring v Monitoring is about defining a project v Monitoring is about generating relevant data v Monitoring is a process of motivating people v Monitoring is “hand maid” to evaluation v Monitoring is a systematic process of collecting and collating information v Monitoring is inseparable from evaluation

What is evaluation % Evaluation is the art of asking interesting and provocative % % % questions. Evaluation is about evidence. Evaluation is about causation. Evaluation is about different perspectives. Evaluation is about reflection. Evaluation is about learning.

Myths about evaluation % Evaluation is about proving the value of your work % Evaluation is only for funders % Evaluation is all numbers % Evaluation is retrospective % Evaluation reduces creativity % Evaluation is negative

Good evaluation % Meaningful % High quality % User-friendly % Useful

Seven benefits of evaluation % Assurance % Knowledge % Insight % Control % Integration % Motivation % Publicity

Two roles for evaluation Learning Auditing

Two roles for evaluation AUDITING LEARNING Retrospective Real time External evaluators Everyone’s job End of term surveys Integrated with MIS Quantitative Mixed method What Why Compliance Improvement

Facets of evaluation % Thinking: Framing, focusing, conceptualising, interpreting, synthesising % Systems: Planning, gathering data, entering data on a computer, analysing data, disseminating information % People: Motivating, training, involving, reassuring, listening, informing, influencing % Action: Making recommendations, implementing recommendations

Defining M&E framework for ISCP-Tz . . .

A Systematic approach % Framework % Programme logic (goals, outcomes, impact) % Assumptions % Indicators % Questions % Issues % Methods % Forms % Questionnaires

A Systematic approach % Forms % Questionnaires for Process evaluation % Questionnaires for National Programme evaluation % Questionnaires for Cluster Initiative evaluation % Questionnaires for Cluster Firm evaluation % Guidelines for Case Studies % To demonstrate progress, impact and best practices % To facilitate information dissemination

ISCP-Tz Programme: Goal To stimulate and facilitate the development of innovation systems and innovative clusters in Tanzania, leading to wealth creation, jobs and change of mindset Funds & Access to expertise Cluster formation & development Improved productivity, quality & consistency Access to better markets within & abroad Improvement of members’ wealth & livelihood

ISCP-Tz Programme support go CIs B. Building A. Mustering Support C. Creating Momentum D. Extending the Base Training cluster facilitators, Trust building, New technologies/techniques, Business skills & techniques, Benchmarking, expert advice Monitoring & Evaluation, Networking and sharing ISCP-Tz support to clusters E. Sustaining Momentum

ISCP-Tz Programme Logic Funds & Access to expertise Cluster formation & development Improved productivity, quality & consistency Access to better markets within & abroad Improvement of members’ wealth & livelihood

ISCP-Tz inputs into CI development % Marketing ISCP-Tz % Training cluster facilitators % Trust building % New technologies & techniques % Business skills & techniques % Benchmarking % Expert advice % Monitoring & Evaluation % Networking and information sharing

ISCP-Tz programme outcomes % 1. ISCP-Tz has identified appropriate value chains % % % and cluster facilitators 2. Firms, academia and public agencies recognise the need to cooperate as a strategy to achieve competitiveness 3. Firms recognise the need to utilise new technologies or techniques and are able to do so effectively 4. Firms/ cluster leadership are marketing effectively demonstrated by increased profitable sales to up markets domestically and abroad 5. Research on innovation systems is influencing policies 6. ISCP-Tz has laid the foundations for the continuation of sustainable cluster initiatives

Table 1: Chain of reasoning and assumptions in the logic Steps in logic chain Funds & Access to expertise Cluster formation and development Improved productivity, quality and consistency of products/services Access to better markets within and abroad Improvement of members’ wealth and incomes and livelihoods Firms, academia and public sector are willing and do participate in the cluster activities Cooperation & training leads to improvements in cluster firms’ attitudes and behaviour… ISCP-Tz is able to identify value chains with most potential to sell internationally Costs can be contained or reduced such that profitability of participating firms increases Competent facilitators and BDSPs are available to undertake capacity building … which leads to specialisation, diversification, innovation and improved quality & consistency… ISCP-Tz is able to identify and engage suitable New ways of working and cooperating continue cluster facilitators & consultants after ISCP-Tz support ends … without significantly raising costs Firms in clusters co-operate while competing Assumptions in logic ISCP-Tz markets the programme effectively to a wide range of potential stakeholders Competent are available to advise firms on need for new technologies & techniques New technologies and techniques are appropriate Cluster firms (and cluster leaders) are able to and effective market effectively Competent are available to advise firms on business & marketing skills, traceability, etc After introduction, firms continue to use new technologies and techniques Improved quality and productivity increases demand from up market customers Bought-in supplies (including, for example, Cluster firms are consistently able to meet packaging, traceability compliance, transport) are quality, volume and delivery requirements of available and affordable up market customers Firms can find additional capital required both for Prices are competitive whilst reflecting the capital equipment and working capital quality of the product ISCP-Tz is able to partner effectively with other stakeholders (including loan funds, advisers, etc)

Table 1: Key questions for process evaluation Objectives Were the objectives of clusters clear? Were the objectives understood by the stakeholders? Were the objectives the right ones? Should they have been narrower or broader? Are the activities sufficiently focused on building competitiveness? What are the lessons for future interventions? Intake process Was the programme marketed appropriately? Were the intake criteria clear to cluster firms? Was the intake process at the right level of detail? What are the lessons for future programmes? Assessment process Were the cluster proposals of a high quality? Did the selected cluster initiatives meet the intake criteria? What are the lessons for future programmes? Decision making How effectively does the NSC operate as a decision making body? Does the NSC have the appropriate information? Does the NSC have the appropriate skills? Does the NSC have the appropriate systems? Are NSC members committed to the programme? Capacity building support What was the level of support provided by ISCP-Tz? Was there synergy between the support provided by ISCP-Tz and other projects? Were there barriers to firms/clusters taking up support? What was the quality of support given? Did firms feel that the capacity building support met their needs? What are the lessons for future interventions? Technology review What was the level of support provided by ISCP-Tz? Was there synergy between the support provided by ISCP-Tz and other sources? Were there barriers to firms introducing and utilising new technologies or techniques? What was the quality of support given? Did firms feel that the new technology or technique met their needs? What are the lessons for future programmes? Resourcing Were the resources appropriate in quality, quantity and timing? Value for money What is the total running cost compared to other support programmes? What are the unit costs per key outcome compared to other support programmes? Context What were the dimensions of the initiative, organisations and environment that affected implementation and impact (including gender, region, location, education)? How did the mechanisms and effects vary with different types of clusters, target audiences, organisations and contexts? What were the barriers and enabling factors that affected achievement of outcomes and impacts? Was the initiative well coordinated with other programmes serving the same, similar or complementary objectives? Mechanisms How did the initiative motivate participants to create the intended effects? How can these mechanisms be replicated? Learning Were the initiative elements appropriate? Were they well implemented? Was the initiative value for money compared to the opportunity cost? Could this initiative learn from, or share good practice with, different programmes which have similar programme elements (benchmarking)? What are the lessons for future programmes?

Proposed targets Objectives and outputs Means of verification Assumptions Objectively verifiable indicators Programme objective Stimulate and facilitate the development of innovation systems and innovative clusters in Tanzania, leading to wealth creation, jobs and change of mindset Increased research on innovation systems 2 p. a ISCP-Tz reports Increase in use of research findings in policy reviews Cluster evaluation case studies Rise in incomes of entrepreneurs of cluster firms Cluster reports Increase in jobs in cluster firms Increase in linkages between cluster firms and jobs in other firms in supply chain as result of programme Value of purchases by cluster firms from other firms in the cluster Evidence of increased cooperation among cluster firms Evidence of improved quality of products/processes/services Increased productivity Outcomes 1. ISCP-Tz has identified appropriate value chains and cluster facilitators 30 Cluster facilitators identified, trained and participating in programme 40 Firms participating in each cluster (male led; female led; rural; urban) 2. Firms, academia and public agencies are cooperating as a strategy to achieve competitiveness Value chains with potential for cluster development prioritised 50 Number of clusters with triple helix membership Number of firms cooperating within a cluster 20 80% ISCP-Tz reports ISCP-Tz is able to identify value chains with most potential to for cluster formation ISCP-TZ is able to identify and train cluster facilitators CI reports Firms are willing and engage in cluster activities Regular monitoring data & reports Competent facilitators are available Cluster evaluation case studies 3. Firms recognise the need to utilise new technologies or techniques and are able to do so effectively Change in work practices and procedures of firms following various training (including for example introduction of appropriate record keeping) Firms utilising technologies and techniques which lead to increases in quality and production 90% Cluster reports 200% ISCP-Tz monitoring and evaluation reports 100% of those that need it Cluster evaluation case studies Increase in yields/ production/ productivity as a result of new technologies or techniques New technologies and techniques are effective Firms continue to use new technologies and techniques Firms able to raise additional investment capital and working capital requirements Firms can find additional capital required both for capital equipment and working capital Value of new investment by cluster firms 4. Firms/ cluster leadership are marketing effectively demonstrated by increased profitable sales to up markets domestically and abroad Capacity building leads to improvements in cluster firms’ attitudes and behaviour which leads to improved quality, better management, better understanding of what makes a successful firm without significantly raising costs Products (or services) certified to appropriate quality standards 100% Firms financial reports Firms in clusters co-operate Firms/ clusters preparing appropriate marketing strategies 100% Regular cluster reports Participating firms (and cluster leadership) are able to market effectively Increase in sales (volume and value) ISCP-Tz reports Cluster evaluation case studies Improved quality and productivity increases demand Firms are consistently able to meet quality, volume and delivery requirements Prices are competitive Costs can be contained such that profitability increases 5. Research on innovation systems is influencing policies Published cluster reports Xx ISCP-TZ national conferences reports ISCP-EA regional conferences reports ISCP-Tz participate in policy review meetings 6. ISCP-Tz has laid the foundations for the continuation of sustainable cluster initiatives Cluster information disseminated to wider community Cluster progress reports Involvement in appropriate networks, alliances, partnerships Means of sustaining future ISCP-Tz services identified National Steering Committee is providing effective governance Yy ISCP-Tz reports NSC meetings’ minutes Public sector and agencies willing to use research findings and or engage with ISCP-Tz

BIBLIOGRAPHY % % % % % Chisawillo, Peter (2007), Monitoring and Evaluation of Cluster Initiatives: Presentation to the National Steering Committee and the International Team, February 27 th, 2007 Co. ET, VINNOVA, SIDA, CLUSTER NAVIGATORS (2006) Innovation Systems & Clusters Handbook: Training notes, Kunduchi Beach Hotel, Dar es Salaam, Tanzania, 28 th August-1 st September 2006 Diyamett, Bitrina (2007) Cluster Initiatives Monitoring and Evaluation Indicators: Proposed Framework for monitoring and Evaluating Pilot Cluster Initiatives Project in Tanzania (draft for discussion only) University of Dar es Salaam, Prospective College of Engineering and Technology (September 2005), Monitoring and Evaluation (M&E) System: National Business Incubator Programme Mwamilla B. L. M and Temu A. K (eds) (2005) “Innovation Systems and Clusters Programme in Tanzania”, Proceedings of National Stakeholders Conference, Bagamoyo, Tanzania, January 24 -25 Mwamilla B. L. M and Temu A. K (eds) (2006) “Innovation Systems and Clusters Programme in Tanzania”, Proceedings of the Launching workshop on Pilot Cluster Initiatives, Dar es Salaam, Tanzania, February 9 Mwamilla B. L. M and Temu A. K (eds) (2006) “Innovation Systems and Clusters Programme in Tanzania”, Proceedings of the 3 rd Regional Conference on Innovation Systems and Innovative Clusters in Africa, Dar es Salaam, Tanzania, September 3 -7 Enterprise Development Resources (2007), www. microlinks. org/usaidraf/seminar notes: USAID Value Chain Training Rural Livelihood Development Company (2008) “RLDC makes market systems work better for rural producers to improve their welfare”, Workshop Handout, Golden Tulip, Dar es Saalam, Tanzania, June 26 th 2008

www. diligentconsulting. TZ. com