Historical and emerging theoretical foundations for Monitoring and

Historical and emerging theoretical foundations for Monitoring and Evaluation in South Africa: Implications for practice. Dr Mark Abrahams, University of Cape Town, March 2007

Introduction • Monitoring and evaluation (M&E) relatively new (Louw, 1998; Potter, 1999; Potter and Kruger, 2001; Babbie and Mouton, 2001). • Historical development of programme evaluation in South Africa • The emerging theoretical foundations • Implications for practice.

Analytic Framework • Shadish, Cook and Leviton (1991); Shadish (2005) • Social programmes and policies aimed at improving the welfare of individuals, organisations and society. • Theory - connotes a body of knowledge that organizes, categorizes, describes, predicts, explains, and otherwise aids in understanding and controlling a topic

Questions • What good is a fine evaluation of an important attempt to solve a puny problem? • What good is a fine evaluation of a puny attempt to solve a serious problem? • What good is a fine evaluation of a program that solves a serious problem if the results are not stored and used to ameliorate the problem?

Point (s) of departure • Evaluation is just one part of a complex, interdependent, non-linear set of problemsolving activities • Evaluation research is more than the application of methods, more than an isolated academic concern and it operates in the context of the social policy and public administration movements.

Five ‘fundamental issues’ • • • Social programming: the ways that social programs and policies develop, improve, and change, especially in regards to social problems. Knowledge construction: the ways researchers learn about social action. Valuing: the ways values can be attached to program descriptions.

Cont. • • Knowledge use: the ways social science information is used to modify programs and policies. Evaluation practice: the tactics and strategies evaluators follow in their professional work, especially given the constraints they face.

Three periods in S. A. • Pre - 1994 – Anti-apartheid, anti-government, liberation • Post – 1994 – Reconstruction and development • 1999 to date – Governance, sustainability, accountability

The Drivers • The Democratic state – – The Presidential address Power and accountability Benchmarking Legislation – Performance management • International and regional agreements – – Millennium goals NEPAD Sustainable Development International Exchange/ globalisation

Drivers (cont. ) • • • Economic growth Sustainable development Unemployment Infrastructure and spatial development HIV/AIDS and other health issues Equity and redress Poverty alleviation Safety and security Skills development Climate change and environmental concerns Civil and ‘uncivil’ society Free press

Emerging theories of social programmes • Programmes are policy-connected interventions underpinned by the Constitution and Bill of Rights • They are constructed with clear objectives, timelines and action plans and resources are allocated to achieve targets • Integrated approach emphasised, recognising the multi-faceted nature of social problems. • Social engineering

Theories of knowledge construction • Multi-methodological approach with emphasis on reliable data • Preference for quantitative data • Audit and compliance strategies • Monitoring vs evaluation

Theories of valuing • Stakeholder dilemma – Who are the ‘holders’? – What is the stake ‘steak’ for each holder? – What is at stake for each holder? – Who holds what? – Who suffers? – Who gains – What are the consequences of non-delivery?

Theories of Use • Centralising structures (GWM&E) to ensure and enable use. • Advocacy purposes • Early warning systems • Performance measurement • Policy development

Theories of practice • Objectivity through indicators • Elements of the following – Manipulable solution theory – reducing uncertainty about programme effects – Generalisable explanations theory – knowledge of complex interrelationships among multiple causal determinants – Stakeholder service theory - tailored to information needs of stakeholders

Implications for M&E in S. A.

Theory of Social Programmes • Anatomy of social programmes (Pawson, 2006) – – – – Intervention as theory Interventions as active Intervention chains thickly populated Intervention chains non-linear can go into reverse Interventions embedded in multiple social systems Interventions leaky (change over time) Interventions are open systems and change the conditions that make them work in the first place.

Theory of Knowledge • Ontology, Epistemology and methodology essential topics • Knowledge of causation and generalisation • All methods are potentially fallible • The limitations and potential of the single case study – multiple studies • Parameters that influence the choice of methods

Theory of Valuing • Prescriptive and descriptive valuing • Agreed upon principles • Standards of performance and measuring performance • Synthesizing results? • Stakeholder interest vs criteria of merit

Theory of Use • • The challenges of instrumental use Promoting use Facilitating use Influencing use Conceptual use of findings Communicating results Agreement about use Multiple stakeholders

End here today • Thank you

Theory of Practice

Historical overview • In 1999, Potter reported that since the 1970 s, an estimated R 6 billion of overseas and local funding had been used by various NGOs to engage in development projects in various sectors of society, and only a small fraction of these programmes had been evaluated.

History cont. • Potter and Kruger (2001) illustrate with a 1998 Psych. LIT database (Silverplatter International N. V. ) search for references from 1974 to 1997 that produced only “fifteen articles and one book chapter that were indexed under ‘programme evaluation’ and ‘Africa’, out of a total of 4 721 articles and books that were indexed under ‘programme evaluation’ (p. 192). ” Eleven of the articles, including the book chapter were from South Africa, representing less than half a percent of the total available on that database.

History cont. • Louw’s (1998) ‘informal’ survey of programme evaluation and the types of evaluations produced in South Africa revealed that qualitative approaches to programme evaluation was the dominant one and experimental, including quasiexperimental designs were present but not in great numbers. He found that most of the work involved formative rather than summative evaluation.

Historical Elements of P. E. • Pre 1994 – – – – Project Driven Donor requirements Mostly formative Personality driven Vague objectives Liberation rhetoric Dwindling resources • Post 1994 – – – Programme focused Planning requirements Clear objectives Logical frameworks Policy frameworks Accountability principles – International agencies

Post 1999 • • Policy - programme relationship Integration at conceptual level Accountability structures and frameworks Need for sophisticated information management systems Managing development implementation Large scale implementation International treaties and agreements Sustainable development

Historical theoretical foundations • Theory of social programmes – Focused projects can bring about desired change – Desired change can be brought about within specified time – Participants involved and eager for the same reasons – Clear lines of contestation/ conflict – Local knows best, as long as people are involved

Historical theoretical foundations • Theory of use – Instrumental intent