Building New Institutional Capacity in M E The Experience

Building New Institutional Capacity in M&E: The Experience of National AIDS Coordinating Authority V F Kemerer Getting To Results IDEAS Global Assembly Johannesburg, South Africa 18 March 2009

Overview § Background § Use of Assessment Tools for Strategic Planning § NACA Specific Capacity Building Interventions § Measurement and Determination of Effectiveness and Outcomes

Background System - Assessments, development of operational plans, and implementation of selected interventions Organization Reinforce key management systems and use of proven leadership practices through a cadre of tools Individual - On-the-job mentoring & coaching, workshops, M&E Master’s Programs, distance learning

Misc Bi- & Multi. Laterals USG & PEPFAR UN Family GFATM National Agencies Background NAC M&E UNIT Coordinate the National HIV DMRS Lead National M&E System Development CB through TA using various tools & strategies to establish and/or improve performance

Strategic Vision: Build effective and sustainable M&E systems to provide comprehensive, highquality information Poor understanding of existing data Organizational cultural Technical capacity Reduce constraints to data use and increase ownership of information

Use of Assessment Tools for Strategic Planning § SI Assessment § MESST § DQA/RDQA

SI Assessment Activities § Map national HIV output indicators and identify data flow § Outline challenges in the DMRS § Assess M&E structures § Stakeholder Analysis – National M&E Framework § Stakeholder Analysis –HIV researchers, organisations, and research

Linking SI Capacity and SI Planning with SI Products Timely, high quality strategic information products Unified SI Strategic Planning and Coordinated SI Support Ongoing Monitoring & Feedback Building of National SI Systems Assessments

Use of Assessment Tools for Strategic Planning § SI Assessment § MESST § DQA/RDQA

A new approach to M&E Assessments FROM … TO … Donor-driven Country-driven Focused exclusively on capacities of specific silos Expanded to integration in the wider M&E Systems General approach Strictly Assessment Tool 10 More systematic approach (based on a checklist of items) Broader Assessment, Management, Planning & Capacity Building Tool

How does the M&E Strengthening Tool work? Step 1: Diagnosis through completion of a Checklist: ■ Answers to Statements/ Questions ■ Provision of Comments/Expla nations 11 Step 2: Analysis A Summary Dashboard will be automatically generated: ■ Displays the overall distribution of answers for each of the checklists’ sections ■ Provides a more detailed analysis of the specific gaps that helps respondents complete the ACTION PLAN Step 3: Action Plan Development of a Costed Action plan with: ■ Identification of Strengths and Weaknesses ■ Defining Strengthening Measures: - Responsibility - Timeline - Funding Implication - T/A Needs

Expected Benefits ■ Better identify M&E capacity gaps and corresponding strengthening measures, including through Technical Assistance (TA) ■ Guide investments in M&E (within the recommended range of 5%-10% of the overall Budget) ■ Ensure that such investments contribute to the strengthening of the National Systems (avoiding parallel reporting systems) ■ Enhance the quality of programmatic data to improve program management 12

Use of Assessment Tools for Strategic Planning § SI Assessment § DDIU Tools § MESST § DQA/RDQA

DQA METHODOLOGY – 2 Protocols ■ The methodology for the DQA includes two (2) protocols: 1 Data Verifications (Protocol 1) Quantitative comparison of recounted to reported data and review of timeliness, completeness and availability of reports. 2 Assessment of Data Management Systems (Protocol 2) 14 Qualitative assessment of the strengths and weaknesses of the data-collection and reporting system.

PROTOCOL 2: Assessment of Data Management and Reporting Systems M&E Management Unit Service Delivery Sites / Organizations Intermediate Aggregation levels (eg. District, Region) 3. Assess Data Management and Reporting Systems (Part 1) ■ PURPOSE: Identify potential risks to data quality created by the datamanagement and reporting systems at: - the M&E Management Unit; - the Service Delivery Points; - any Intermediary Aggregation Level (District or Region). ■ The DQA assesses both (1) the design; and (2) the implementation of the datamanagement and reporting systems. ■ The assessment covers 8 functional areas (HR, Training, Data Management Processes , etc. ) 15

Objective of the Data Quality Assessment Tool The Data-Quality Assessment (DQA) Tool is designed to: u verify the quality of reported data for key indicators at selected sites; and v assess the ability of data-management systems to collect manage, and report quality data. ! 16 The DQA Tool is not intended to assess the entire M&E system; it relates and focuses more in depth into component 10 (i. e. Supportive supervision and data auditing) of the “Organizing Framework for a Functional National HIV M&E System”.

NACA Specific Capacity Building Interventions MESST RDQA TA for DQ SI Assessment M&E Road Map 1/4 ly Mentoring (SHAPMo. S)

CB INTERVENTIONS

SI Assessment M&E Road Map Implementation § Incorporation of Data Flow Findings into SHAPMo. S § Incorporation of the Institutional Inventory List into: § National Programmatic Databased at the NACA § Supporting primary efforts of WHO’s P-SAM Exercise § Participation in the Launch of the National HIV M&E Strategy

SI Assessment M&E Road Map Implementation § To. T M&E Basic Concepts § 4 quarterly mentoring & coaching visits to Regional M&E Officers § Support to the NACA to produce QSCR

Institutional Capacity Strengthened – Regional M&E Officers § Internal monitoring systems developed and in place § Trace and verification conducted in each Region at SDPs § Regional QSCRs produced § Information products developed for feedback to SDPs

CB INTERVENTIONS

Timeline: TA to Malawi 2007 2008 01 02 03 04 05 06 07 08 09 10 11 12 01 02 03 04 05 Discussions with NAC: Next Steps Plan for MESST M&E Plan Revision MESST: Malaria MESST: HIV Plan for DQ T/A Data Quality

RDQA Tool To Operationalize Statement of Work §Support finalization of the Draft Data Quality Protocol §Assist to roll-out the data quality protocol §Conduct data quality training(s) for key implementing agencies Draft Services & Indicators Matrix Draft register(s) to be used by all CBO SDPs Draft clear instructions Draft DQ policies Draft data steps performed at each level of the DMRS Draft guidelines for quality controls Draft written back-up procedures Data Quality Training in 5 Zones

Institutional Capacity to Support Data Quality Developed § Data quality plan written that includes guidance, reporting forms, and indicator matrix § Key stakeholders at national and sub-national levels oriented to – and reviewed – new data quality documents § NACA M&E Officers trained to train on data quality § Key staff in all 5 Zones have been trained by NACA on data quality

Conclusions § NACA M&E Unit institutional capacity strengthened at National and Sub-National levels § Data reporting more timely, with increasingly accurate, reliable and complete data § Foundation for effective data use laid § Documented systems establish institutional memory

MEASURE Evaluation is funded by the U. S. Agency for International Development (USAID) through Cooperative Agreement GPO-A-00 -0300003 -00 and is implemented by the Carolina Population Center at the University of North Carolina in partnership with Futures Group, John Snow, Inc. , Management Sciences for Health, ORC Macro International, and Tulane University. Visit us online at http: //www. cpc. unc. edu/measure.