Monitoring and Evaluation Information Sources and Systems

Monitoring and Evaluation: Information Sources and Systems

Session Objectives At the end of this session, participants will be able to: • Name the main information sources for PHN M&E • Describe the main strengths and weaknesses of different data sources • Discuss the main data-quality issues that need to be considered • Explain why complementary data sources are often required to monitor and evaluate health systems • Identify potential data sources that might apply in a specific program context.

Overview • Types of information • Strengths and weaknesses of selected data sources • Data quality • Linking data sources • Exercise

The Finagle’s Laws of Information The information you have is not the information you want The information you want is not the information you need And the information you need is usually not available

Definitions (1) • Data: the raw facts that are collected and form the basis for what we know. • Information: the product of transforming the data by adding order, context, and purpose • Knowledge: the product of adding meaning to information by making connections and comparisons and by exploring causes and consequences

Definitions (2) • Health system “all resources, organizations and actors that are involved in the regulation, financing, and provision of actions whose primary intent is to protect, promote or improve health. ” (WHO, 2000) • Program A set of procedures to conduct activities. The objective is normally the solution to a problem Neither a health system or program is a static phenomena. They experience a continuous process of changes due to pressure from both outside the system and from within the system.

Definitions (3) • Health Information System (HIS): A health-information system (HIS), similar to a health management information system (HMIS) “…a system that provides specific information support to the decision-making process at each level of an organization” (Hurtubise, 1984) • Data Systems “a way of talking about the whole set of M&E indicators in a performance monitoring-and-evaluation plan, and all of the data and other information that need to be gathered and understood in an orderly fashion that makes sense and help in program management and implementation”

Types of Information • Surveillance – Epidemiological – Behavioral • • Routine service reporting Special program reporting systems Administrative systems Vital registration systems Facility surveys Household surveys Censuses Research and special studies

Frequency of Data Collection • ROUTINE or continuous data collection • NON-ROUTINE or periodic data collection

Classify the previous information types by frequency of data collection • Routine • Non-Routine

Frequency of Data Collection • ROUTINE or continuous data collection – Health facility-based (patient information and service statistics) – Community-based (service-statistics) – Program-based (administrative) – Vital registration – Sentinel reporting/demographic surveillance • NON-ROUTINE or periodic data collection – – Household or facility-based surveys Population census Rapid-assessment procedures (RAP) Special studies/research

Geographic System Levels • National • Sub-national (e. g. district) • Program area

The Health Information System: Data for Planning, Monitoring and Evaluation in the PHN Sector Routine TYPE Non-Routine USE Aggregated Service Statistics Aggregated Mgmt Data Aggregated Surveillance Data Financial Data Vital Registration Systems Policy-Making Strategic Planning Program Tracking Disease Surveillance Technical & Logistical Support Aggregated Service Statistics Aggregated Mgmt Data Sentinel Sites Observation Checklist Self-Evaluation (e. g. COPE) Planning (Access) Management (Quality/Efficiency) Supervision (Performance) Disease Surveillance Facility/ Client Records Financial Records Supply Records Facility logbooks/data records Aggregated Community Data Client Mgmt and Follow-Up Health Unit Management Work Planning/Priority Setting Community Birth and Death Records School Records CBD logbooks Drug Revolving Fund records Client Mgmt and Follow-up Supplies Management Community Awareness National Level District Level Population-based surveys e. g. DHS Rapid Assessment Methods Facility-based surveys e. g. Situation Analysis, SPA Special Studies e. g. EPI cluster surveys, KAP studies, etc. Census

Data-Collection Levels l l l Policy or program Service environment Client Population Spatial/geographic

Data Sources at the Policy/Program Level l l Official documents (legislative, administrative) National budgets or other accounts data Policy inquiries Reputational rankings Program effort scores

Trends in Family-Planning Effort Score: 1972 -1999

Data Sources: Service Environment Level • Administrative records – Service statistics – Management information – Financial data • Service-delivery point – Routine service statistics – Audits/inventories – Facility surveys • Agent, staff or provider – Performance, competency – Training records

EXAMPLE: SERVICE ENVIRONMENT LEVEL DATA

HEALTH MANAGEMENT INFORMATION SYSTEMS (HMIS) • Note: An important way of monitoring routine data over time is through a Health Management Information System. An HMIS is a system for ongoing (routine) collection and reporting of data about client-service delivery. In many countries, this system operates at the national level. Ideally, these routine data are collected from a comprehensive set of service delivery points, and should cover topics such as: – Costs – Stockouts – Births – Mortality – Morbidity – Numbers of clients seen, referred (inpatient; outpatient) – Numbers of clients by types of service

Data Sources: Client • Client-exit interviews • Case surveillance – Epidemiology of disease • Provider-client observation – Management of the sick child – Vendor-client interaction • Contact or visit registers • Customer record

Data Sources: Population § § Census Vital registration system Sample household surveys Special population surveys - Demographic (elderly, youth) Risk groups (CSWs, MSMs, IDUs) Occupational (farmer, skilled labor) Area-based (catastrophe-affected) § Biomarkers

Spatial/Geographic Data Sources § § § § Satellite imagery Aerial photography Digital line graphs Digital elevation models Cadastral maps Global Positioning System data PLACE (site-based surveys)

Integrated GIS Database Aerial Photography Digital Elevation Models Satellite Imagery Cadastral Data Digital Line Graphs GPS Data Chris Betz Christian Carl Chris Mc. Afee Dale Legere Donna Black 1757 Millbrook Ln 28226 1761 Millbrook Ln 28226 1765 Millbrook Ln 28226 1776 Millbrook Ln 28226 1780 Millbrook Ln 28226 Y 2 Y 1 Y 2 N 6 Y 2 Demographic Data Differential GPS

High Transmission of HIV Guguletu, Cape Town South Africa Carolina Population Center University of North Carolina at Chapel Hill Neighborhood Statistics Showing High Transmission Sites within 500 Meters Air Photo Showing Potential High Transmission Individual Structures Sites in Guguletu Can be Identified 1 - 13 14 - 25 26 - 37 38 - 49 50 - 61 No Data

Strengths and Weaknesses of Selected Data Sources

Focus M&E Data Sources • • Facility-based routine information systems Facility surveys Population-based surveys Program records/administrative data

Facility-Based RHIS: Types of Information Generated • Service statistics • Outcomes of health interventions if individual patient records kept • Not coverage (but can be estimated in some cases with other data) • Not incidence (except nosocomial infections) • Not prevalence

What is Wrong with Existing RHIS? • Irrelevance of information gathered • Poor data quality • Duplication and waste among parallel health information systems • Lack of timely reporting and feedback • Poor use of information • Centralization of information management without feedback to lower levels

Strengths of Routine Health Information Systems • Continuously collected – suitable for frequent reporting • Existing system – no new data collection; builds local capacity; sustainability • Typically available at lowest administrative levels (e. g. district, facility) • Integral part of health system – direct link to health system actions

Common Problems With Facility. Based RHIS • Variation in quality and completeness of reporting • Timeliness of reporting • Difficulty of providing coverage estimates • Indicators may not be exactly what you want in a particular context • May only cover government facilities • Double-counting

Facility Surveys: Types of Information • Readiness to provide services (inventory) – Infrastructure, staffing, hours of operation, fees • Health worker knowledge – Provider interviews • Quality of Care – Client-provider observation • Client satisfaction – Exit interviews

Strengths of Facility Surveys • Can cover both public and private health facilities • More detailed information than is typically available in routine systems • Can be tailored to specific program needs • Timing can coincide with program implementation • Can combine with population survey for outcome monitoring and impact evaluation • Quality control may be easier than in routine systems

Limitations of Facility Surveys • Survey sampling design and analysis may be complex • Expensive, time-consuming • Stand-alone – sustainability concerns; less connected to ongoing program decision-making • Information rapidly outdated, unless repeated – not available regularly • Coverage/sample size constraints – National vs. sub-national – By type of facility • Small client sample sizes for some services (e. g. FP, STIs)

Facility Survey Initiatives and Tools • Service Provision Assessment (SPA/HSPA) – DHS • Service Availability Mapping (SAM) – WHO • Quick Investigation of Quality (QIQ) – M/Eval (FP only) • Situation Analysis (SA) – Population Council (FP only) • JICA facility surveys and mapping

Population-based surveys: Types of information collected • Knowledge and attitudes • Practices • Coverage

Strengths of population-based surveys • Representative of the general population – no selection bias • Wide range of outcome-level indicators can be collected • Program coverage • Well-tested instruments; quality control

Limitations of population-based surveys • Coverage; national versus sub-national – not suitable for district-level estimates • Frequency; typically only conducted every 35 years. • Cannot detect small changes or changes over short periods of time without large sample sizes (expensive) • Not suitable for some types of information (e. g. retrospective attitudes – recall bias)

Household survey programs (national) • Demographic and Health Surveys (DHS) • CDC Reproductive Health Surveys (RHS) • UNICEF Multiple Indicator Cluster Surveys (MICS) • PVO Knowledge Practices and Coverage Survey (KPC) (not national) • CDC Young Adult Reproductive Health Surveys (YARHS)

Class Activity (1) How to improve facility-based routine information systems in developing countries?

Class Activity (2) What are the determinants of health information systems performance?

Data Quality

Overview • Types of information • Strengths and weaknesses of selected data sources • Data quality • Linking data sources • Exercise

Data Quality Issues l l l l Coverage Completeness (census, sample) Accuracy – measurement error; biases; double-counting Frequency of collection Reporting flow Reporting schedule Public accessibility Supervision

• Hierholzer (Am. J. Med 1991; 21 -26) has called data the Researcher’s (M&E expert) sand. A lens maker takes sand, refines it, melts it, and through a long process of grinding and smoothing, fashions a lens with which to see the world more clearly. Similarly, a M&E expert takes data, refines it and smoothes it until a clearer picture of nature is revealed. If the sand is dirty or impure, the lens will be cloudy and distorted. If data is unreliable or invalid, the M&E expert’s understanding of nature will be clouded and distorted.

• By paying close attention to the data collection process from the conception of the data collection document through the editing of the data after it is collected, the M&E expert help keep his “sand” pure so that, in the end, nature may be viewed with much clarity and possible • No amount of sophisticated analysis can salvage either a poorly designed or a badly carried out study

Linking data

Linking data • Data can be linked from different sources, across different levels, or over time • Linking data appropriately requires planning, preferably prior to data collection • Understanding linked data can provide depth and continuity to enrich otherwise discrete points of information

Linking Data Why link? • Survey data sets (e. g. , household and facility information) can be linked to compare services available and health outcomes across geographical units • Geographical and survey data can be linked to examine the effects of physical attributes on service utilization • Time series and panel data can help build causal explanations of program or project effects Why not link? • May not be necessary for a given program in a given context • Improper methodology can confuse issues more than explain them

Linking Data Examples • Population and facility data can be linked to ascertain health outcomes correlated with service availability, training, or quality of care (e. g. % of live births in catchment area attended by a trained personnel or % of women exclusively breastfeeding until 6 months among women going to facilities where provider training took place. ) • Facility and client data can be linked to learn about program expenditures per new family planning acceptor • Facility and staff data can be combined to provide information about the proportion of clients per provider or the proportion of doctors per facility

Population-based data highlight limitations in HIS

Facility-based data provide additional explanations

Comparing HIS and population-based data: confirming trends. . .

…. . and raising questions

What do you need for evaluating program impact? • A specific question: Are program inputs X influencing behavior (outcome) Y? , If yes, by how much? • A conceptual framework • Appropriate Data • An empirical model and estimation procedure

Linking Inputs to Outcomes for Evaluating Program Impact ……………. . Individual/ Household Characteristics Health Service Supply and Community Infrastructure USAID Program Healthy Practice Service Utilization Other donors, NGOs, and the Government Healthy Outcome

Linking Inputs to Outcomes for Evaluating Program Impact (sources of data) Individual/ Household Characteristics Health Service Supply and Community Infrastructure USAID Program Healthy Practice Service Utilization Facility Surveys Community survey Other donors, NGOs, and the Government Compilation of data on relevant interventions Healthy Outcome Household Survey

Levels of factors influencing health behavior and outcomes Other Community Characteristics Health Service Supply Community Based Clinic Based community household individual

Data Sources Community Survey Facility Survey Other Community Characteristics Health Service Supply Community Based Clinic Based Household Survey

Data System needed for Program Impact Evaluation • Population - based measures of individual demographic and health behaviors and outcomes, and household characteristics Household survey • Objective measures of health service supply and program inputs Facility Survey • Objective measures of other community characteristics Community Survey • Key: Areal Linkage of Surveys

Unlinked Program Inputs to Outcomes Individual/ Household Characteristics Health Service Supply and Community Infrastructure USAID Program Healthy Practice Service Utilization Other donors, NGOs, and the Government Healthy Outcome

Facility Survey linked to Household Survey Health facilities selected on basis of household survey clusters Objectives: Describe women’s accessibility to health services Describe communities’ health service supply environment Example accessibility Indicators: “% women who have a SDP with FP within 10 kms. ” “% women who have a physician within 5 kms. ” Sampling strategies: Service Availability Module (SAM) Family Life Survey Living Standard Measurement Survey Concentric clusters

Linked Surveys: Service Availability Module (SAM) Identify facilities by interviewing key community respondents with Community Questionnaire, then visit the closest facility of each type if distance <= 30 km. Key issue: Access to services Pros: • Facility indicators valid for “Average” woman or household (access indicators) • Easy and cheap to implement • Linked to household survey, so impact evaluation possible Cons: • It does not necessarily give estimates for “universe” of facilities (but, see Hermalin, A. et. al. , 1996 paper) • Partial description of communities’ health service supply environment, likely

Service Availability Module (SAM) X 30 km. X contd. X DHS cluster Health facility

Linked Facility Surveys: Family Life Surveys (FLS) Identify facilities by asking household survey respondents on sources of services, compile list, and then visit facilities most frequently mentioned (up o a quota and < 45 minutes driving) Key Issue: Community characteristics, not only Health supply Pros: • Facility indicators valid for “Average” woman or household (access indicators) • Linked to household survey, so impact evaluation possible • More complete picture of supply of services / objective measures? Cons: • It does not necessarily give estimates for “universe” of facilities (but, see Rand IFLS documentation) • Less easy to implement (close coordination with HS)

Linked Facility Surveys: Living Standard Measurement Survey (LSMS) Identify facilities by interviewing key community respondents with Community Questionnaire, then visit the closest facility of each type Key issue: Basic Community characteristics, access to services Pros: • Facility indicators valid for “Average” woman or household (access indicators) • Easy to implement • Linked to household survey, so impact evaluation possible Cons: - It does not necessarily give estimates for “universe” of facilities - Partial description of communities’ health service supply environment, likely

Linked Facility Survey: Contiguous Clusters Approach Steps: 1 • Define or two rings of clusters around the DHS cluster 2 • Canvass the DHS cluster and the surrounding ring of clusters to compile list of facilities 3 • Conduct interview in all facilities in that area 4 • Collect measure of size of clusters (population size) to calculate sample weights

Linked Facility Survey: Contiguous Clusters Approach X X X Cluster boundary Household Survey cluster Health facility X X

Linked Facility Survey: Contiguous Clusters Approach Reliability of Estimates: Provides unbiased estimates of facility characteristics, but less efficient Analytical Utility: Provides proxy of health service supply environment, It makes evaluation of program impact possible, without limiting monitoring Cost: It is a variation of Area Frame Sample (1 st. stage areas are DHS clusters) so, cost should be similar to stand-alone FS with area sampling and, lower than FS with list sampling (of same sample size). Cost also reduced if coordination of DHS and FS operations Practical Feasibility: Needs careful planning since survey preparation location of clusters and their population size information required for sample selection and weights

Tanzania Reproductive and Child Health Facility Survey, 1999 • Objective: 1. To collect information on availability and characteristics of reproductive and child health services 2. To provide data for evaluating program impact • Sampling Design: linked to TRCH Household Survey, contiguous clusters approach Clusters: 146 (mainland) Government NGO/Private Total Hospitals 77 Health Centers 11 40 88 22 62 Dispensaries 138 117 255 UMATI / MS /oth 5 35 40 Total 260 185 445 Pharmacies 306

Tanzania Reproductive and Child Facility Survey, 1999

WRAP UP

Supplemental Slide RHIS: Issues • Do we trust HIS data? – Reliability and validity of routine data – Timeliness and relevance of data – Costs of data collection and sustainability – Rational process and parsimonious selection of indicators • Is there an “information culture” with clear purposes of data collected for managerial and strategic decisions? – Environment that facilitates the direct use of routine data • Does HIS meet data need of decentralized health services? • Does HIS monitor referral systems effectively? • Does HIS monitor private sector activities that are of public health interest (what are the incentives, if any? )

Group Work: Case study 1 • Instructions: • You have been asked to advise the following two programs on their M&E plan. Identify the potential data sources that you might explore to provide information for the M&E plan. List the factors that you consider as you assess the suitability of different data sources for each program.

• Program A is an NGO-run RH/MCH program operating in 3 districts in a country. The program aims to improve use of MCH services such as immunization, ANC, and family planning use in the districts in which it works. It provides training to staff in MOH clinics in the districts to improve the quality of services provided. Private sector health services are limited in the program areas so most people use government sector services. The program also undertakes community mobilization through community health workers and local radio spots to promote use of services. The program wishes to use some of the M&E plan data for ongoing program management and will be required to report to its donor annually on its performance as well as at the end of the project on its overall results.

Program B is a national AIDS prevention program. The program includes a mass media campaign on the ABCs aimed at reducing risk behaviors in the general population, the initiation of a PMTCT program and the expansion of its VCT program. The PMTCT and VCT program activities include training of health workers to provide quality VCT and PMTCT services, strengthening logistics systems to provide reliable supplies of HIV test kits to PMTCT and VCT sites as well as ARVs to PMTCT sites, opening new sites to increase the physical accessibility of these services to the population, and community mobilization to use VCT and PMTCT services through local media and community-based activities in areas where sites are located. In addition, new data collection forms will be added to the RHIS for PMTCT and VCT sites to collect service statistics on the new services, and sites will receive regular supervisory visits during their first few years of operation. The program wishes to use the M&E plan data for ongoing program management and annual reporting, as well as to fulfil relevant UNGASS and donor reporting requirements.

Group Work: Case study 2 • Instructions: • • Identify for each indicator the appropriate source of data needed in its calculation. Present and discuss your work in a plenary session.

1. Proportion of households with at least one ITN 2. Proportion of children under 5 years old who slept under an ITN the previous night 3. Proportion of children under 5 years old with fever in last 2 weeks who received antimalarial treatment according to national policy within 24 hours from onset of fever 4. Proportion of pregnant women who slept under an ITN the previous night 5. Proportion of women who received intermittent preventive treatment for malaria during their last pregnancy 6. HIV seroprevalence among all TB patients

7. Number of health facilities involved in DOTS with sufficient drug and laboratory supplies 8. Number of health facilities and laboratories involved in DOTS with sufficient capacity for DOTS 9. Number of health facilities where TB and HIV services are both available 10. Number of project staff trained 11. % of project budget spent on health infrastructure 12. % of project beneficiaries (patients) who are accurately referred

13. Number of networks/partnerships involved in project 14. Number of project service deliverers trained in M&E 15. % of overall project budget spent on M&E 16. Number of project services deliverers trained in procurement and supply management 17. % of project service delivery points with sufficient drug supplies 18. Unit cost(s) of project drug(s) and commodities

19. Number of people reached by the services 20. Number of service points supported by the funding 21. Number of providers trained in the service 22. HIV-infected pregnant women receiving a complete course of antiretroviral prophylaxis to reduce the risk of MTCT (number and percentage) 23. Districts with access to donor recruitment and blood transfusion 24. Transfused blood units screened for HIV 25. People with advanced HIV infection receiving antiretroviral combination therapy (number and percentage)