Indicators, Monitoring, Evaluation
Project Cycle Management Basic PCM Training Course
The Logframe Matrix Overall objective Project Descriptio n Verifiable Indicators Sources Assumption of s Verificatio n Project purpose Outputs Activities means costs Pre-conditions 3
Levels of the project description Overall objective Project purpose Results Activities G Relate to the wider sectoral or national objective, to which the project will contribute; G The sustainable flow of benefits to the project’s target groups(s); G The goods and services delivered by the project; G What the project does to produce the results (work program/tasks). 4
Indicators – What are they? • An indicator is a unit of measurement • E. g. kilometres of road, population numbers • An indicator tells us something about changes in a system • e. g. Financial indicators (stock markets up / down) • An indicator helps to quantify and simplify phenomena and helps us understand complex realities • E. g. Blue Flag for bathing water quality An Indicator is not a Target! 5
Characteristics of a good indicator The Objectively Verifiable Indicators (OVI) should be: • Specific : including (where appropriate) clear targets; • Measurable: in terms of quality and/or quantity; • Available : at an acceptable cost; • Relevant : to what they are supposed to measure (horizontal logic); • Timely : produced in due time to be useful for project 6
Sources of Verification (I) The logical framework requires that one spells out how the indicators will be collected : G G G How/through which methods ? By whom? When/how often? Admin records Activities, resources and costs required to collect this information must be included in the project design Project managers should focus on verifying the delivery of results and their contribution to the purpose 7
Sources of Verification (II) Cost Specialised surveys Interviews with beneficiaries Management Administrative reports Adapted available statistics Available statistics Complexity 8
Describing Indicators help to define: Q Q T P Quality Quantity Time Place 9
Defining an Objective: Define objective: Educational status improved Specify quality: Number of graduates completing degree courses increased Specify quantity: Number of graduates completing degree courses increased from 500 to 1000 Specify timeframe: Number of graduates completing degree courses increased from 500 to 1000 per year by the year 2007 Specify Place: Number of graduates from Tartu Technical College completing degree courses increased from 500 to 1000 per year by 10
Indicators, Projects and Programmes Global objectives are expressed in terms of impacts (e. g. a reduction in unemployment among the previously long term unemployed). Specific objectives are expressed in terms of results (e. g. the improvement, through training, of the employability of the long term unemployed); Operational objectives are expressed in terms of outputs (e. g. the provision of training courses to the 1 1
Note! • Indicators require good data, based on monitoring • Indicators are not targets! • Indicators can help to reduce uncertainty, but not eliminate it • Sets of indicators evolve over time • Sets of indicators are seldom, if ever, complete • Indicators can help influence behaviour. • Indicators must be able to take into account different locations, people, cultures and institutions • Different people from different places have different values 12
Criteria for Indicator Selection For Policy relevance and utility for users an indicator : • should provide a representative picture • should be simple, easy to interpret and able to show trends over time; • must be responsive to changes • can provide a basis for international, regional comparisons; • must have a comparative threshold or reference value/Baseline against to assess the significance of the values. 13
Criteria for Indicator Selection Analytical soundness • be theoretically well founded in technical and scientific terms; • be based on international standards and international consensus about its validity; • lend itself to being linked to economic models, forecasting and information systems. Measurability The data required to support the indicator should be: • readily available or made available at a reasonable cost/benefit ratio; • adequately documented and of known quality; • updated at regular intervals in accordance with reliable 14
Setting up a system of indicators G Baseline G Context Indicators G Monitoring Indicators Example: Scottish Executive ESF & ERDF Indicators 15
Indicators for ESF Projects - Scotland Project Level Core Indicators Number of beneficiaries of ESF assistance, show separately: • % of persons aged 16 -24 receiving assistance • % of persons in this group unemployed for less than 6 months before assistance began • % of persons aged 25 and over receiving assistance • % of this group unemployed for less than 12 months before assistance began • % of those who complete their course • % of those leaving ESF funded training for positive outcomes, split by end destination: G into full or part-time employment G self employment G full time FE/training/other Government training schemes • % of leavers who gain a qualification or part qualification • % of beneficiaries who receive assistance specifically geared towards 16 self employment
Indicators for ERDF projects - Scotland Projects in Scotland; Project Level Core Indicators Indicator Name Assistance to Business • Number of businesses assisted and instances of assistance, showing SMEs separately. • Increase in sales in assisted businesses. • Hectares of serviced land created or enhanced, split by greenfield / brownfield sites. • M 2 of business space created or enhanced - occupied after 18 mths/3 yrs. • Private sector finance levered in by new assisted projects. • Value of investment in R&D by assisted SMEs. • Number of patents/intellectual property rights registrations by assisted SMEs. 17 • Number of organisations taking up e-commerce trading.
Indicators ERDF Projects - Scotland Jobs and Employment G Gross new jobs created. G Number of jobless people securing employment. G Number of young people under 25 years of age placed in employment. G Number of gross jobs created through self-employment. G Gross number of jobs safeguarded. Social Inclusion and Equal Opportunities G Gross number of assisted organisations introducing active equal opportunity policies. G Number of childcare facilities and places created. G Number of organisations achieving recognised quality awards. Environment G K/W of new renewable energy infrastructure installed. G Number of SMEs undertaking environmental management. G Number of strategic environmental/forestry partnerships funded. G Hectares of natural habitat under management. 18
Indicators Progress Chart Scottish Executive Environment Group Indicators of Sustainable Development for Scotland: Progress Report 2004 February 2004 Paper 2004/3 19
Headline Indicators National Assessment UK Regional Quality of Life Counts – 2002 20
21
Monitoring & Evaluation
Some Principles of M&E G Keep the needs of the information user clearly in mind. Who is this information for? How will they use it? G More information is not better information G A balance of quantitative and qualitative information is usually best G Collecting and using information requires time and resources – allow for this in the project design 23
Monitoring vs Evaluation Monitoring Evaluation Who? Internal management responsibility – all levels Usually incorporates external inputs (objectivity) When? Ongoing Periodic – mid-term, completion, ex-post Why? Check progress, take remedial action, update plans Learn broad lessons applicable to other programs/projects, policy review, etc Focus on Inputs, activities, outputs Results, purpose, overall objective 24
Definition of Monitoring A regular reporting mechanism, preparing and submitting analytical assessment to project management and other involved parties 25
Monitoring of Projects & Programmes Monitoring needs a formal structure which sets out: • • • What is to be monitored? How will it be done? Who will do it? When will it be carried out? Where and how is information to be recorded? Baselines data and project Indicators
Why Monitor ? • To keep track of where you are • To identify any significant changes in the project • It assists in communications • It is good management practice! • It is, in some cases, an EU requirement 27
Indicators & Project Responsibilities 28
Project Monitoring and Indicators 29
Activity Planning 30
Using Activity Plan to Monitor 31
What is to be monitored 1? Examples in a Project G Beneficiary details G Staffing details G Project activity progress / milestones / deadlines vs plan G Quality checks G Attendance records G Finance – income G Finance – costs vs budget 32
What is to be monitored 2? Examples in a Programme G Outputs • No of people trained • No. courses completed / % passing approved test • No of people on study tours / placements G Results • Higher quality of project fiches • Improved contracting rate G Impact • Improved absorption of EU funds 33
Data needed G G Inputs Monitoring of the project or programme Outputs Monitoring of the project or programme Results Evaluation of the project or programme Impact Evaluation of the project or programme 34
Identifying Indicators for Monitoring How to measure what we get is what we ordered and pay for? G Project duration G Deliverables G Activity plan / • milestones / phases • Activities • Reporting incl handover certificates etc G Indicators and sources of verification • Baseline indicators • Performance indicators for managing the project G Monitoring • • Staff Recording Reporting Communicating 35
Practical exercise to identify the information required to monitor a project 36
Monitoring data exercise A project is working directly with young people with a disability to improve confidence, assertiveness and presentation. Each young person will have a mentor – an older person with a disability who is successful in their chosen employment. At the same time the project is raising awareness with employers of the value of employing young people with a disability. The ultimate aim is for the young people to get jobs in the open labour market. The project is funded by the ESF and in order to be eligible all the beneficiaries must be aged between 16 and 24, be unemployed and have a disability. 37
Evaluation
Definition of Evaluation? A time-bound assessment that systematically and objectively assesses the relevance, performance, success and effectiveness of ongoing and completed programmes and projects. 39
Why Evaluate? Reporting of results to: • • • Funders Project promoter Local partners Peer organisations Transnational audiences Beneficiaries 40
Project Evaluation Project evaluation is a structured process comprising: G G Collection of relevant information Assessment of this information against agreed criteria Reporting of the evaluation results Incorporation of these results into future planning 41
Project Evaluation Gathering information Method Record keeping Participant observation Self administered questionnaires Individual interviews Group discussion Quantitative Qualitative ***** * **** 42
Relationship to the Logframe Level of Logframe Type of information Timing of collection Measurement concept Overall Objectives Substantive development change to specific sector Project completion Sustainability and ex-post Impact Project purpose Realistic and sustainable change to target group specific to project intervention Mid-term and project completion Relevance Effectiveness Results Quantitative and qualitative measures of physical progress Quarterly/yearly Efficiency Costs Measures based upon project budget Weekly/monthly Economy 43
Programme Evaluation G Ex Ante (Before) • Can the programme achieve the results set out in the plan? G Mid Term (Mid way) • How are we doing and what changes are needed? G Ex Post (After) • What did we achieve vs plan? 44
Evaluation and Indicators G Relevance G Efficiency G Effectiveness G Impact G Sustainability 45
Evaluation and Indicators Need s Socio-economic problems Programme Objectiv es Relevanc e Inputs Operatio ns Efficiency Impact Result s Output s Effectiveness Sustainabilit y Evaluation 46
One for the road! It is not necessarily a crime to be running late………… ……. . It is always a crime not to know! You can only manage the work that remains to be done! Getting people to think ahead is 50% of the benefit of project planning A successful project is one that where outputs are delivered on time, on budget, at the required quality……. and no one died of a heart attack 47
Скачать презентацию Indicators Monitoring Evaluation Project Cycle Management Basic
5c627a94ed1b850b805f6b525984e411.ppt