The DEA assessment procedure guideline Assessment of development

The DEA assessment procedure guideline Assessment of development impacts from energy interventions Example: Wind power system development ≠ the ”physical” wind power system development = improvements among energy users from access to the services enabled by the wind power system improvement = e. g. swifter/cheaper production or consumption, higher incomes, reduced drudgery, higher livingstandards, better health

development ”impact” formalized (for now) Agents (region, village, households, individuals) Behaviors and practices over a period of time The difference between outcomes is the impact we seek. Intervention/ program/project Agents (other/later) Outcome for the agent(s) or other agent(s) Modified behaviors and practices over a period of time Modified outcome for the agent(s) or other agent(s) Impact

Impact from hypothetical wind powered irrigation project Agent: Rural villagers Practice: Outcome: Rainfed agriculture Harvest size/quality/ diversification The difference in harvest size/quality/diversification is an impact Wind powered irrigation project Agent: Rural villagers (other/later) Modified practice: Modified outcome: Irrigated agriculture Greater/improved/more diversified harvests Impact

Mediating circumstances Agent: Rural village E. g. location- or time-specific rainfall, temperatures, ínfrastructure functionality, customs/traditions, prices Practice: Outcome: Rainfed agriculture Harvest size/quality/ diversification Harvest sizes/quality/diversification affected by mediating circumstances? How assess the impact from the intervention? Attribution problem Agent: Rural village (other/later Wind powered irrigation project Modified practice: Modified outcome: Irrigated agriculture Greater/improved/more Mediating circumstances diversified harvests Impact

General ambition Empirically assess a causal linkage from the technological input of a given energy intervention to observable development results via the utlization of the (improved) energy services provided by the intervention while keeping an eye on mediating processes

Causality challenges (1) • Energy does not in itself e. g. quench, feed, house, or clothe people Energy services may be a vital input into meeting those needs => long and complex chain of causality • Energy services often bring about improvements in many aspects of life – difficult to capture or conceive of all such improvements • Impacts of energy access may take many years to become manifest

Causality challenges (2) The type/range of energy services made available varies across energy interventions (different services from wind driven water pumps, solar homes, improved stoves, solar geezers, MFPs) Households, enterprises, or institutions differ in • abilities and willingness to invest in new technologies • energy – related preferences, traditions and behaviors ÞExisting demand patterns vary geographically across regions, districts, settlements within districts, and users within settlements ÞImpacts that are observed in one context do not necessarily appear in another

Key requirements to assessment methodology One –size–fits–all approach not possible Need for • flexible approach/methodology – adaptable to different types of interventions and contexts • knowledge of ”realities on the ground” – energy services needed and used among the specific end-users affected by the intervention • systemize interventions into inputs brought in and observable results – causes, effects, and circumstances that intervene Þ Draw on methodology used for interventions in other sectors, that has been adapted to energy by a group of experts assembled by GVEP.

Assessment procedure Design stage 1. 2. 3. Focus, conceptualization, and modeling Identify the stakeholders of your project and their IA needs. Define/delimit the intervention and your focus on the intervention. Illustrate and model the project with results and chains of causality. Bridge model to the real world situation Choose observable indicators for all elements in the results chains. Specify the appropriate data collection methods for each link. Construct a research plan and discuss it with your stakeholders. 4. 5. 6. Implementation stage 7. Collect data. 8. Analyze the data. 9. Draw conclusions and write report. 10. Present the results to your stakeholders.

The implementation stage 7. choice of data collection methods – contingent on study-specific circumstances in the design stage 8. data analysis methods and types of conclusions depend on data collection methods. 9. – 10. means of presentation to stakeholders - also case specific

Where can we provide universally applicable guidance? Design stage 1. 2. 3. Focus, conceptualization and modeling Identify the stakeholders of your project and their IA needs. Define/delimit the intervention and your focus on the intervention. Illustrate and model the project with results and chains of causality. Bridge model to reality Choose indicators for all elements in the causality chains. Specify the appropriate data collection methods for each link. Construct a research plan and discuss it with your stakeholders. 4. 5. 6. Implementation stage 7. Collect data. 8. Analyze the data. 9. Draw conclusions and write report. 10. Present the results to your stakeholders.

Stakeholders Conceivable stakeholders • domestic public authorities • share holders, financial partners • donor country public authorities • beneficiaries/users/clients • project management team • women’s groups • community based organisations • researchers in academia • the impact assessor him-/herself Factors that affect the assessment design: • what kind of information different stakeholders require • the intended use of that information • how stakeholders would like the information communicated to themselves and other stakeholders

General format guidance possible Design stage 1. 2. 3. Focus, conceptualization and modeling √ Identify the stakeholders of your project and their IA needs. Define/delimit the intervention and your focus on the intervention. Illustrate and model the project with results and chains of causality. 4. 5. 6. Bridge model to reality Choose observable indicators for all elements in the causality chains. Specify the appropriate data collection methods for each link. Construct a research plan and discuss it with your stakeholders.

Intervention – define, delimit and focus Does the intervention: • provide/improve energy utilization for one or several purposes? • involve one or several types of technology? Single technology, single purpose - wind driven water pump for irrigation Single technology, multi-purposes - rural solar PV charge stations offering battery charging services for several types of equipment Multi-technology, single purpose – Solar PV and diesel powered water pumps for rural irrigation Multi-technology, multi-purposes - grid electrification: lighting, communication water access, refrigeration and entertainment for homes, agricultural sector, health sector and schools. Which are your stakeholders’ main areas of interest?

General format guidance possible Design stage 1. 2. 3. . 4. 5. 6. Focus, conceptualization and modeling √ Identify the stakeholders of your project and their IA needs. √ Define/delimit the intervention and your focus on the intervention. Illustrate and model the project with results and chains of causality. Bridge model to reality Choose observable indicators for all elements in the causality chains. Specify the appropriate data collection methods for each link. Construct a research plan and discuss it with your stakeholders.

Illustration and modeling – the “results chain” terminology • Inputs: the material, financial, human, and social resources used • Activities: actions taken in order to produce specific outputs • Outputs: resultant products, capital goods and services (domestic, collective or institutional) • Outcome: short- and medium-term effects of an intervention’s outputs • effect - a “change, intended or unintended, due - directly or indirectly - to an intervention” • Impacts: positive and/or negative long-term effects, subject to greater influence of mediating circumstances than outcomes (inverse “log–frame”)

Revised terminology Outcome: Agents Output : Irrigation services practicing irrigated agriculture Impact: : Higher agricultural income (”further downstream”) Input : Wind power equipment Mediating circumstances

Alternative application of results chain : solar home systems (focus: income generation) Input + Solar home system components + Activity Sales, distribution, and installation of SHS components Output Installed solar home systems Use of output Lighting Outcome Income generating activties in the evening Impact Income poverty reduction

General format guidance possible Design stage 1. 2. 3. Focus, conceptualization and modeling √ Identify the stakeholders of your project and their IA needs. √ Define/delimit the intervention and your focus on the intervention. √ Illustrate and model the project with results and chains of causality. 4. 5. 6. Bridge model to reality Choose observable indicators for all elements in the causality chains. Specify the appropriate data collection methods for each link. Construct a research plan and discuss it with your stakeholders.

Indicators A measurable key characteristic of the agents, their behaviour, or their circumstances, that can be related to the intervention’s desired outcome. Operational objective of the assessment: Capture differences in indicator values between agents that have experienced the intervention and those that have not.

Choosing indicators – some considerations • • • Pertinence to your project Pertinence to national or regional (aggregate) development objectives Interest for project stakeholders Ease and cost of measurement or data collection Possibilities for triangulation between sources of information Information format - not necessarily numeric. Other examples; • pictures • videos • voice recordings • change in behavior or attitudes

Solar home systems with indicators: focus on income generation Input Solar home system components - no of solar panels - no of inverters + + Activity Sales, distribution, and installation of SHS components Output Installed solar home systems - no of SHS installed - no of SHS operational Use of output Lighting - hours of lighting - no of light bulbs used Outcome Income generating activties in the evening Impact Income poverty reduction - no of SHS sold - no of SHS installed - evening working hours - goods/services production - change in household income

General format guidance possible Design stage 1. 2. 3. Focus, conceptualization and modeling √ Identify the stakeholders of your project and their IA needs. √ Define/delimit the intervention and your focus on the intervention. √ Illustrate and model the project with results and chains of causality. 4. 5. 6. Bridge model to reality √ Choose indicators for all elements in the causality chains. Specify the appropriate data collection methods for each link. Construct a research plan and discuss it with your stakeholders.

Data collection methods Assessment depends crucially on data quality! Each indicator is ideally measured using more than on data collection methods, e. g. : • Physical measurement (satellite data on forest cover) • Data extraction from public statistics (school attendance or agricultural production) • Interviews • Extraction of accounting or administrative data of a public or private organization (ESCO customer records) • Focus groups and other participative methods • Household or firm surveys Often combinations of methods!

General format guidance possible Design stage 1. 2. 3. Focus, conceptualization and modeling √ Identify the stakeholders of your project and their IA needs. √ Define/delimit the intervention and your focus on the intervention. √ Illustrate and model the project with results and chains of causality. 4. 5. 6. Bridge model to reality √ Choose indicators for all elements in the causality chains. √ Specify the appropriate data collection methods for each link. Construct a research plan and discuss it with your stakeholders.

General format guidance possible Design stage Focus, conceptualization and modeling 1. 2. 3. 4. 5. 6. √ Identify the stakeholders of your project and their IA needs. √ Define/delimit the intervention and your focus on the intervention. √ Illustrate and model the project with results and chains of causality. Bridge model to reality √ Choose indicators for all elements in the causality chains. √ Specify the appropriate data collection methods for each link. √ Construct a research plan and discuss it with your stakeholders. Implementation stage 7. Collect data. 8. Analyze the data. 9. Draw conclusions and write report. 10. Present the results to your stakeholders.

Concluding remarks Impact assessments of energy interventions are possible – but challenging!! • Huge variety in types of energy interventions • Multitude of resultant energy services • Complex demand side – difficult to trace development impacts. • Need for systemizing intervention and consequences; modelling and design requires creativity, analytical skills, and knowledge of the reality on the ground. • Collection of high quality data and dissemination – possibly the most difficult components!