Monitoring and evaluation for social and behavioural change

Monitoring and evaluation for social and behavioural change communication Dr Renay Weiner Soul City: Institute for Health and Development Communication

Outline Soul City: background What do we measure? Selecting indicators Approach to evaluation Issues and challenges

Soul City: mission Mission is to make an improvement in people's health and quality of life by: • Harnessing the power of the mass media • Developing high quality education material through thorough research and evaluation involving the communities we serve • While remaining an organisation not for gain, ensure the sustainability of our effort through sound business principles and practices • Being strong advocates for health and development • Measuring impact on health and development

SC model of SBCC • Effect change at individual, community and societal levels • Methods: MEDIA; SOCIAL MOBILISATION and ADVOCACY • Learning and Reflection; self-efficacy • Draw on different theories of SBCC

5 Pillars of Health Promotion(Ottawa Charter): 1. Create enabling environments 2. Develop personal skills 3. Advocate for healthy public policy 4. Re-orientate services 5. Stimulate community action

Social Cognitive Theory Social Learningis. Theory by : A person’s behaviour influenced • Personal Modelling factors (cognitive, affective, biological events) • Environmental factors Self efficacy In many instances, without social change, individual behaviour change, People learn by observing behaviour of others and the outcomes is almost impossible thereof Soul City aims to impact on individual, community and societal levels People adopt those things that help behaviour change, and decrease to enhance behaviours if they believe they are capable. those things that hinder. Important theory specifically for edutainment because it includes concept of traveling on a journey with characters and seeing the results of risky / negative behaviour and can role model positive behaviours and outcomes)

Changing behaviour is a complex endeavour In many instances, without social change, individual behaviour change, is almost impossible Soul City aims to impact on individual, community and societal levels to enhance those things that help behaviour change, and decrease those things that hinder.

Soul City Institute for Health and Development Communication 2008 -2012, Soul City is tackling two important social and public health issues in South Africa: HIV/AIDS and violence. Campaign approach • Onelove campaign: SA and regional • Phuza. Wize: SA

example of regional level OL

Soul City: interventions and activities MEDIA • Soul City TV drama: 13 episodes • Soul Buddyz TV drama • Soul City radio drama: translated into all languages • Other: Kwanda; Love Stories, BOM • Print booklets SOCIAL MOBILISATION • Soul Buddyz programme • Adult training and community dialogues ADVOCACY

What do we monitor and evaluate? Need to consider objectives of programme/project REACH OF Onelove REACH numbers and % by age and sex geographical location province and geotype

Selecting indicators: deciding what to measure MEDIA SOCIAL MOBILISATION ADVOCACY Social Knowledge Community---Intervention --- Attitudes ---Behaviours---- Prevalence Individual Self-efficacy Incidence INPUT OUTCOME IMPACT PROCESS OUTCOME IMPACT OUTPUT

Selecting indicators Example: HIV prevention Intervention (output) Knowledge Attitudes Self efficacy % of adults exposed % of adults with to the TV/radio/print accurate HIV intervention prevention knowledge % of respondents who agree that a woman can refuse to have unprotected sex with her husband if he has an STI Behaviours Prevalence/Incidence % of respondents who used a condom at last sex HIV prevalence amongst pregnant women aged 15 -24 years

Approach to evaluation: quantitative and qualitative approaches Quantitative questions Qualitative questions Relate to `quantities’ Count or measure phenomena How many? What proportion? How often? Relate to `quality’ of variations in experience or the meaning of experience for different people Classify events or phenomena How? Why? eg % of youth that abstained in past year Reasons for abstaining---lack of opportunities versus choice to self-protect

What do we monitor and evaluate? PROCESS EVALUATION • How was the intervention implemented? • Was the intervention implemented according to plan? • Under what conditions was implementation successful?

What do we monitor and evaluate? IMPACT/OUTCOMES Individual level knowledge attitudes/beliefs self-efficacy behaviours Community level -Social norms -Community cohesion/social capital -Civic action -Sustainability

Evaluation: quantitative and qualitative approaches Quantitative • Aims to quantify research findings using numbers, %, ratios etc • Quantify frequency of disease/health events, risks, measures associations • Standardised procedures decided in advance • Samples used to approximate status in the population • Measures are estimates, with defined levels of certainty eg HIV prevalence: 10% (95% CI: 8 -12%) • Random sampling allows generalisability to population • Sample sizes large enough to ensure precision of estimates

Impact evaluations of SBCC require household surveys

Evaluation: quantitative and qualitative approaches Qualitative • Concepts presented in written words and images • Good for explaining how and why; good for exploring beliefs and behaviours • Methods and Qs can change as data collected; can measure impacts that have not been predicted and generate new ideas • Sampling is purposive---data not statistically generalisable • Sample sizes small • Analysis: extract themes, organise data into coherent findings • “Yes, Desert Soul has made a difference to my life because I used to have many boyfriends. When I read this book I have changed. That means that I have dumped all my boyfriends because I read that it is dangerous to have many partners that increases the risk of being infected by HIV”. (Rural female, 26 -35; FG 8)

Results: Impact - Behaviour

Results: Impact - Behaviour • “I have realised the importance of protected sex and have since refrained from sex without a condom. The booklet also influenced me to go for voluntary counselling and find out my HIV status” – (Lesotho) • “I never used to like condoms but now with my girlfriend we buy and use” (Zambia)

Quantitative approaches: RCT and evaluating mass media intervention • Cannot allocate intervention- implemented via national public media • Cannot restrict intervention-aimed for maximum reach of largest possible audience • Intervention tested in development phase through target audience research • Recent systematic review effect of communication programs on HIV/AIDS—no randomised studies for full coverage programmes (Bertrand et al)

Evaluation methodologies: quantitative approaches What study design to use to evaluate impact of health communication interventions? Observational Cross-sectional, case-control, cohort Intervention Quasi-experimental (pre-post), RCT (individual/cluster) `GOLD STANDARD’ randomisation

Evaluation methodologies: quantitative approaches What study design to use to evaluate impact of health communication interventions? Consider: level of implementation of the intervention SC intervention Level of implementation MEDIA- TV Population MEDIA-radio Population MEDIA-print Individual (support other media) SOCMOB-Soul Buddyz; community training Individual/family/schools ADVOCACY Population

Quantitative approaches: alternative study designs 1. Cross-sectional analytic exposure to SC measured impact compared in exposed and non exposed; dose response relationships measured reverse causality, need to control confounders 2. Cohort follow-up same group over time impact compared in exposed and non exposed 3. Pre-post/quasi-experimental baseline for comparison

Evaluation: qualitative methods examples • Focus Group Discussions: facilitator guides group; participants have similar characteristics • Interviews in-depth: guided by the respondent semi-structured: loose set of questions • Ethnographic observation: immersion in another’s culture • Most Significant Change methodology

Participatory M and E AIMS • To generate data on activities being carried out at the local community level • To increase beneficiaries commitment and understanding in planning and implementing community based development projects Examples: Community Dialogues

Evaluation: triangulation Refers to the use of two or more different research methods in combination, principally as a check on validity SC series 7 and 8 • Quantitative data HIV/AIDS is the result of sinning Multimedia exposure None Low High Agree/strongly agree 47. 7% 28. 2% 19. 1% • Qualitative data `To me, to watch SC changed my whole life because I was affected with HIV and AIDS. I was able to teach others to disclose to their partners…. ’ (female KZN, rural 16 -25)

Evaluation: triangulation Routine data: 1. National media monitoring Television Audience Measurement Surveys 2. Helpline data AIDS helpline

Community responses Letters and newspaper articles

Issues and challenges Multilevel intervention—multiple sources of data New activities---new indicators: dynamic M and E plan Reach • Media monitoring: data limited especially radio; cannot tell if same or new viewers • Survey data better but infrequent

Monitoring media: special issues • Need ongoing monitoring data to estimate reach • Two sources of mass media monitoring data: SAARF media monitoring Response to advice lines • Print media monitoring: distribution data

Media surveys: SAARF South African Advertising Research Foundation 1. AMPS: All Media Products Survey Annual national household survey; F 2 F interviews sample of 25 000 adults TV viewing information: past 4 weeks; past 7 days, yesterday 2. TAMS: TV Audience measurement survey `Peoplemeters’ attached to TV equipment of sampled households to monitor what is being watched Every household member must log onto the system when viewing Measures viewership patterns at home only Daily reports provided

Issues and challenges REACH • Print distribution as a proxy for booklet reach —survey data better • Training data: from partners —m and e support/quality audits; feedback IMPACT • For mass media cannot do RCT • Costs of surveys

Conclusion • Monitoring and evaluating social change communication complex • Quantitative and qualitative approaches complementary • Alternative study designs to RCT needed for evaluating media interventions