TA 4581: Developing a National Result-Based M&E System For Key Projects TA 4581: 建设国家重大项目以结果为导向的 监测与评估系统 15 December 2005年 12月15日
Result-based Monitoring and Evaluation Systems 以结果为导向的监测与评估系统 • Results 结果 – The outputs, outcomes and impacts of a project. • Monitoring 监测 – Systematic collection of data to provide stakeholders with indications of progress towards objectives and in the use of allocated funds. • Evaluation 评价 – The systematic and objective assessment of an inprogress or completed project. Includes assessment of its design, implementation and results.
Results-based Monitoring Model 以结果为导向的监测模型 Strategic goal 战略目标 Need 需求 Inputs 输入 Impact 长期效应 Outcome 直接结果 Activities 活动 Outputs 提交物
Results-based Monitoring Model 以结果为导向的监测模型 Strategic goal 战略目标 Need 需求 Inputs 输入 Impact 长期效应 Outcome 直接结果 Activities 活动 Outputs 提交物 Project Implementation Phase
How the Results-based Model fits in with the Project Cycle Strategic goal 战略目标 Concept Operation Need 需求 Feasibility Design Impact 长期效应 Inputs 输入 Outcome 直接结果 Activities 活动 Outputs 提交物 Project Implementation Phase
Benefits of Results-Based Monitoring 以结果为导向监测的益处 1. Optimize project outcome performance优选项目直接结果 绩效 2. 3. Avoid mistakes from the past避免以往的错误 Prioritize future project proposals优化今后的项目提案 4. 5. Greater public accountability公共责任更大化 Can demonstrate good governance用以验证优秀的管理 6. 7. 8. Have a performance culture绩效文化 Minimize project waste项目浪费最小化 Set higher standards using Benchmarks 以基准来建立更高的标准
What leaders want to know at concept and feasibility phases of the project • • • is the project worth doing? is the budget affordable? how will it be financed? can the project be financially justified? can the project achieve the intended results?
What leaders want to know during construction of the project • is the project construction on-time? • is it within budget? • will the final completion be on time and within budget? • are the outputs meeting quality standards? • is the project free from misuse of funds and corruption? • is the project likely to achieve the intended results?
What leaders want to know 1 -3 years after construction of the project • is the finished project being used as intended? • is the operating performance as planned or do we need to make some modifications to make it work better? • is the project achieving the intended results?
What leaders want to know 3 -5 years after construction of the project • did the project achieve the intended operating results? • did the project achieve the intended impact? • should we do similar projects in future? • what lessons have we learned from this project that we can apply to future projects?
How we use Results 我们如何运用结果 State Council Finance 国务院财政司 NDRC 国家发展与改革委员会 Ministry 国家机关 Critically evaluate benefits 严格评估收益 Prioritize future projects 优化未来的项目 Good governance 良好的管理 Ministry 国家机关 Department 管理部门 KPIO 重大项目稽查特派员办公室 Project sponsor 项目发起人 Project manager 项目经理 Contractor 合同人 Costs 成本 Efficiency 有效 Quality 质量 Timeliness 实效 Funding 基金 Corruption 贪污 Public accountability 公共义务 Impact 长期效应 Outcome 直接结果 Outputs 提交物 Ministry 国家机关 Department 管理部门 KPIO 重大项目稽查特派员办公室 Project sponsor 项目发起人 Optimize performance 表现最优化 Minimize waste 浪费最小化 Public accountability 公共义务 Good governance 良好的管理 Performance culture 表现内涵 Benchmarking 基准
Performance indicators must be measured during every project phase: • • Concept development Feasibility study Design Project implementation – – – – tendering right-of-way acquisition construction practical completion commissioning final completion acceptance hand-over to operating entity • Operation
Concept Model of Nakproses
Results-Based Development 基于结果的发展 TA (add on) Strategic goal 战略目标 Need 需求 Inputs 输入 Impact 长期效应 Outcomes 直接结果 Activities 活动 Outputs 提交物 Nakproses (now)
Implementing Case Studies 案例实施
• Case Studies – Objectives 案例研究----目标 Objectives: 1. Pilot Result-Based performance indicators, to bring international best practices into project monitoring and evaluation. (试点结果为导向的绩效指标，把国际上最 好的时间经验引入项目监测与评价) 2. Find practical ways of implementing them into the current computer system. (找到可行的把他们运用于现有 计算机系统的方法) 3. Develop cases and materials for promotion of and training in Result-Based Monitoring & Evaluation(开 发案例和文档用于推广和培训以结果为导向的监测与评价) 4. Provide learning experience with Result-Based performance indicators for KPIO, KPIO provincial level and executing agencies(提供以结果为导向的绩效 标准的学习经验给KPIO, 省一级KPIO和执行处)
Steps步骤 1. Choose case study projects. 2. Gather documentation on justification for projects (project summaries). 3. Develop monitoring framework for each project. 4. Define results-based indicators for each project. 5. Collect data for result-based indicators for each project. 6. Assess indicators, change if necessary. 7. Use case studies to develop training materials.
Impact 长期效应 Outcomes 直接结果 The Framework Strategic goal 战略目标 Impact 长期效应 Outputs 提交物 Need 需求 Outcome 直接结果 Activities Inputs 活动 输入 Activities 活动 Outputs 提交物 Inputs 输入
The monitoring framework Impact 长期效应 Outcomes 直接结果 Outputs 提交物 Activities 活动 Inputs 输入 The framework summarizes how the project will achieve results by converting inputs to outputs, and how outputs are expected to give the desired results.
What we have to find in the project documents Impact 长期效应 Outcomes 直接结果 Statement of the long-term or strategic objectives. 对项目战略目标或长期结果的描述。 Information on the expected direct results of the project being completed and put into operation. 预期项目完成并投入运营后可带来的直接结果。 Outputs 提交物 Physical deliverables of the project. 项目的直接产出或提交物。 Activities 活动 Major activities carried out to create the project deliverables. 为完成项目提交物所要进行的主要活动 Inputs 输入 Project budget. 项目预算
Possible indicators 可能的指标 Impacts: 长期效应 Economic benefits Cost effectiveness Distribution of benefits (social effects) Outcomes: 直接结果 Utilization Immediate benefits Financial benefits Sustainability Social effect Environmental effect Inputs, Activities & Outputs: 输入、活动及提交物 Timeliness Quality Costs – price Costs – waste; Quantities 2~5 5~10 ~100 (Nakproses)
Outputs, Activities & Inputs Monitoring • Through Nakproses – monitoring & early warning systems. • Financial • Quantities • Time • Safety • Quality
Reporting 报告 “Traffic light” Project monitoring CIB Project name Project location 2 month ago 1 month ago Current month Current Issues to discuss and resolve 1. ABC Power station Chengdu OK 2. XYZ Road Yunnan Project 30 weeks behind due to shortage of raw materials 3. Olympic stadium Nanning Steel fabrication cost/price is well above standard prices 4. ABC Water project Fujian Project 2 weeks behind due to project team members disagreeing 5. XYZ Irrigation project Shandong Project 14 weeks behind due to implementation of new computer tolling system 6. QWE Electricity generator Henan OK Make decisions quickly, concentrate on problem projects.
The traffic light system is based on two simple ideas: 1. Leaders want to see only the amount of information necessary for them to be able to judge whether a project is going well or not, and whether they need to take action. 2. When the project appears to be in difficulty, leaders need to be able to expand the information quickly to focus on what is going wrong and what needs to be done.
Outcome & Impact Indicators Only a few Outcome indicators during project construction, for example employment increase in construction phase, local economic effect of project, re-settlement. No Impact indicators during project construction.
Outcome Indicator example.
Outcome indicator example • What does this tell us about the project?
Outcome indicator: Utilization
Outcome indicator: Utilization • What does this tell us about the project?
Outcome indicator: Utilization
Outcome indicator: Utilization • What does this tell us about the project?
Outcome indicator: Operating Revenue
Outcome indicator: Operating Revenue • What does this tell us about the project?
Outcome indicator: Operating profit
Outcome indicator: Operating profit • What does this tell us about the project?
Pilot projects (试点项目） Project （项目名称） Sector （领域） Core team members （核心团队成员） Site team members （试点现场团队成员） Yes / No (red = priority Beijing You An Hospital Health Beijing KPIO (Yang) (not started yet) yes Beijing University of Technology Gymnasium Sport & Recreation Beijing KPIO(Yang) (not started yet) yes Guangxi Longtan Hydroelectric power station Energy Li, Zhou (will finish soon) Jiangsu Sutong Bridge Transport Sun, Zhang (in process, will finish in 2008) Fujian Changle Airport (desk study) Transport (completed in 200? ) Sun, Li (finished) Yes Hebei Expressway & Roads projects Transport CC (finished) Yes Sichuan Zipingpu Water Conservancy Water Song, Zhou (almost finished) yes Zhejiang Zheneng Leqing Power station Energy Pei (just started) Yes Yang Guangxiu yes Xu Yang yes
Next Tasks（下一步 作安排） Task 任务 Who 参加人员 When 时间要求 KPIO team 9 December KPIO team + Domestic Consultants 16 December Build monitoring framework for each project. International + Domestic Consultants December Define key indicators for each project. International + Domestic Consultants December KPIO team + Domestic Consultants 31 January 2006 Get project summary documents. Find key information on each project. Search for data for all key indicators.
END
M 7 Expressway （M 7高速公路） Benefits • Safer and more efficient road transport for both passenger vehicles and freight in western Sydney. • Good access to employment opportunities for the people of western Sydney by linking existing / future industrial and residential areas. • Stronger economic growth within western Sydney by encouraging further investment in the area due to potential savings in transport costs. • 1300 jobs during construction alone and encouraging further employment opportunities in western Sydney. • Reduced numbers of heavy vehicles using local roads, resulting in better air quality and less noise in key residential areas. • Improved access to other growing cities and regions, including the Central Coast, Newcastle, Canberra and the Illawarra. • Faster travel times between key western Sydney suburbs (up to 65 minutes savings).
M 7 Expressway
Example: Sydney M 7 Expressway Impact 长期效应 Economic development in West Sydney. More employment for West Sydney people. Safer road transport. More efficient road transport. Better air quality and less noise. Outcomes 直接结果 Traffic. Toll revenue. Faster travel times. Jobs during construction. Outputs 提交物 40 km expressway roads. 17 interchanges. 38 over/under-passes. Separate cycle-way. Electronic tolling system. Activities 活动 Inputs 输入
Measurement drives decisionmaking
Measurement is nothing new
Targets are needed.
Why measure ? • We can monitor progress and control our project. • We can compare our performance with other projects. • We can learn from mistakes and avoid them in future. • We can see opportunities for improvement. • We can recognise achievements. • We can publicise our success.
What gets measured gets done. So… measure what’s important! So… what should we measure? • • • Inputs? Activities? Outputs? Outcomes? Impact?
What types of measures should we consider? • • Quantity? Quality? Financial? Timeliness?
Example: Sydney M 7 Expressway Description Indicators Economic development in West Sydney. More employment for West Sydney people. Safer road transport. More efficient road transport. Better air quality and less noise. %economic growth & change in employment road safety statistics air quality measurements - expressway corridor vs rest of Sydney. Outcomes 直接结果 Traffic. Toll revenue. Faster travel times. Jobs during construction. vehicles/day using expressway. toll $M/yr average travel times at nominated times of day number of jobs Outputs 提交物 40 km expressway roads. 17 interchanges. 38 over/under-passes. Separate cycle-way. Electronic tolling system. Outputs: • quantity • quality • cost • on-time performance Impact 长期效应 Activities 活动 Inputs 输入
Inputs 输入 • Quantity – number of person-hours spent on project – materials consumed • Quality – % of supplied materials passing specification tests • Financial – cost of supplied materials – price variation of purchased goods • Timeliness – % on-time delivery of supplied materials
Activities 活动 • Quantity – number of work packages completed • Quality – (usually measured by the quality of the outputs) – activities are being carried out according to project plan and technical specifications (as shown by audit reports) • Financial – cost of activities or tasks – variations • Timeliness – (usually measure on outputs achieved on-time/late)
Outputs 提交物 • Quantity – number of buildings – km of road built • Quality – % of quality tests passed – strength of concrete – load/deflection test compared to specification • Financial – cost per km of road built – variance against contract • Timeliness – % of milestones achieved inside schedule – specific major outputs delivered … months late or … months early compared to schedule
• Quantity Outcomes 直接结果 – utilisation rate, e. g. Average number of passengers per month travelling by road increased by …% – MWh provided to industry increased by … – Number of patients served by local hospitals; proportion of population able to access primary care health services compared with before-project number. • Quality – Average travel time between xx and yy by road, across different vehicle types reduced – No load shedding, xx MW peak load network maintained – durability of the assets created by the project • Financial – revenue from the asset usage (calculated as pay-back period or rate of return) – Average vehicle operating costs reduced
Impact 长期效应 • Quantity – net additional employment attributable to the project in the long term • Quality – mortality rates of specific diseases reduced – infectious diseases incident rates (e. g. 4 Infant Mortality Rate in project area, improved by at least xx per cent in every district compared to before-project values • Financial – economic benefit brought by the project, for example: • Aggregate Gross Domestic Product (GDP) increased for affected city districts and counties along the expressway; • Aggregate direct investment in affected city districts and counties along the expressway increased. • Increased average household income per annum.
Worksheet Indicator development worksheet Indicator Impact: 长 期效应 Outcomes: 直接结 果 Outputs: 提交物 Activities: 活动 Inputs: 输 入 Definition Data sources Before-project measurement Target
Objectives & Agenda 目标及议程 • Objective目标: – Introduce new KPIO participants to the TA & invite their commitment & involvement. – Discuss pilot projects study. • Agenda 日程 – – Briefing on TA. Introduce Results-based monitoring & evaluation. Agree responsibilities, tasks & schedule. Plan next steps.
Introduction “TA 4581 - Developing a National Result-Based M&E System For Key Projects” A technical assistance funded by Asian Development Bank. 亚行技术援助项目介绍 • Objective 目标： – Supplement KPIO key project performance indicators system with “results-based” indicators. • Approach 项目路径: – Test results-based indicators through selected project case studies. – Train KPIO staff, using case studies. – Integrate selected indicators into KPIO system. • Personnel参加人员: – – KPIO staff Domestic consultants (including IEC) International consultants ADB
Results Monitoring Model 结果监测模型 Develop industry 发展 业 Good transport access 良好的运输途径 Materials材料 Equipment设备 Manpower人力 Industry grows 业增长 Road usage, Speedy, Smooth 公路高速平稳使用 Construct roads 建设公路 New road 新公路
Results Monitoring Indicators 结果监测指标 Industry grows 业增长 Road is used, Fast, Smooth 公路高速平稳使用 New road 新公路
Results Monitoring Indicators 结果监测指标 Industry grows 业增长 Road is used, Fast, Smooth 公路高速平稳使用 New road 新公路
Results Monitoring Indicators 结果监测指标 Industry grows 业增长 Road is used, Speedy, Smooth 公路高速平稳使用 New road 新公路
Keep the number of indicators manageable…
Nakproses Dynamic Monitoring and Early Warning Sub-system — Index Tool for Inputs, Activities & Outputs System Manager shall use index management tool to 1. Establish monitoring index system 2. Set up early warning parameters 3. Modify index system and ity ibil em flex syst the en gth tren S e f th ty o ili tab sui
Framework框架 • We use the project documentation to develop a framework for the results-based indicators for each project. • It is called the “Design and Monitoring Framework”.
Gathering documents（文挡收集） • Need documents that show this information for each project: – Long-term development goal. – Benefits analysis: • • • utilization rates; revenue forecasts; financial rate of return; socioeconomic benefits; economic rate of return; etc. • Should be available in KPIO or NDRC records, in project summary.
Possible traffic light logic If all lights in the group of indicators are green, then the summary light is green. = If two-thirds or more of the lights are green, the rest of the lights are yellow and there are no red lights, then the light is green. = If more than one-third of the lights are yellow and there are no red lights, the light is yellow. = If there is just one red light in a group then the summary light is yellow. = If more than one-third of the lights are red, the summary light is red. = An indicator can be pre-set with a high priority so if it is red, then the group summary must be red, no matter what the colour of the other indicator lights. = High priority
Скачать презентацию TA 4581 Developing a National Result-Based M E System
a16f663db89d8de9a2dd930e3afed264.ppt