Methods for Research Synthesis The logic and practice

Methods for Research Synthesis The logic and practice of research synthesis' — 8 th July 2005 — David Gough EPPI-Centre, Social Science Research Unit, Institute of Education, University of London

Today’s seminar 1. Welcome and introduction 2. Purpose, principles and potential 3. Small group work: exploring selected review tools and procedures 4. Ways forward 16 June 2005 (2)

Aims of this seminar • have a critical understanding of the value of systematic methods for synthesizing research evidence; • have an understanding of the importance of reviews being question-led; • have discussed the diversity of approaches to synthesis and identified principles and decision points central to all; • considered the potential for systematic approaches to your own review work. 16 June 2005 (3)

Social Science Research Unit Five streams of work: – – Childhood Studies Evaluation of Social Interventions Sexual Health, Reproduction and Social Exclusion Evidence for Policy and Practice Information and Co-ordinating Centre – Perspectives, Participation and Research http: //www. ioe. ac. uk/ssru/ 16 June 2005 (4)

EPPI-Centre personnel David Gough (Director), Ann Oakley (Founding Director) Angela Harden and Sandy Oliver (Co-directors HP & PH) Jo Garcia (User involvement, Education) Jon Ashton Mukdarut Bangpan Jackie Barry Ginny Brunton Jeff Brunton Helen Burchett Esther Coren Kelly Dickson Adam Fletcher Zoe Garrett Nicholas Houghton Josephine Kavanagh 16 June 2005 (5) Eva Moran Mark Newman Chloe Powell Rebecca Rees Abigail Rowe Katy Sutcliffe James Thomas Carole Torgerson Alex Trouton Helen Tucker Lisa Underwood

EPPI-Centre vision and mission statement Developing and promoting participatory and user-friendly systematic reviews addressing important questions in different domains of policy, practice and research in the public interest 16 June 2005 (6)

EPPI-Centre external Review Groups (RGs) and teams • 16 funded by the Department for Education and Skills (Df. ES) on different aspects of teaching and learning (some with joint funding – e. g. Nuffield Foundation, NUT (HEFCE), ESRC TLRP) • 8 funded by the Teacher Training Agency on different aspects of teacher education • 1 on transitions of students with disabilities from school to work funded by (based in Colorado in the US) • 1 in higher education funded by the Learning Skills Development Agency • New collaborations with SCIE and DWP 16 June 2005 (7)

EPPI-Centre ‘in-house’ reviews • The effect of travel modes on children’s mental health, cognitive, social development (DETR) • The effects of personal development profiling for improving student learning (LTSN) • Support for pupils with emotional and behavioural difficulties (NFER) • Effect of secondary school size on outcomes (Df. ES) • Series of reviews on health promotion and public health (HP & PH) topics funded primarily by the Department of Health (DH) (see next slide) 16 June 2005 (8)

Health promotion & public health review series 1993– 1996 1997– 1999 Young people and smoking Workplace health promotion Peer-delivered health promotion for young people Older people and accidents Young people and sexual health 1999– 2001 Men who have sex with men (MSM) and sexual health Young people and: …mental health …physical activity …healthy eating 2004– 2007 2001– 2004 Incentives for young people Active transport Teenage pregnancy and parenting Risk behaviour and accidental injury Children and: …physical activity …healthy eating HIV-health promotion for MSM 16 June 2005 (9)

Purpose, principles and potential • Why use systematic approaches to reviewing? • What are the key principles of systematic research synthesis (SRS)? 16 June 2005 (10)

Imaginary scenario • You’re in a PTA AGM. The head teacher is speaking about homework. In the room are teachers, parents (one of whom is also a researcher) and school governors. • Someone says “I’ve read a literature review that says homework is bad for children”. • What might it be useful to know about this review? • Nature of variation between summaries of research 16 June 2005 (11)

Key features of SRS • Explicit research question • Explicit and transparent method: protocol up front • A piece of research: methods to avoid bias • Explicit reporting: accountable, replicable, updateable • Judgement of study quality and relevance: for question • Relevant: need for involvement of research users 16 June 2005 (12)

Variation between reviews 6 reviews of older people and accident prevention Total studies included 137 Common to at least two reviews Common to all six reviews 2 Treated consistently in all reviews 33 1 From Oliver S, Peersman G, Harden A and Oakley A (1999) Discrepancies in findings from effectiveness reviews: the case of health promotion for older people in accident and injury prevention. Health Ed J, 58: 66 -77. 16 June 2005 (13)

How some findings evade us Profusion of published and unpublished material n Much hidden: only 50% abstracts presented at conferences are later published in full n Constraints of own expertise and disciplinary affiliations n 16 June 2005 (14)

Some evasion is systematic Some types of studies are harder to find than others* n E. g. statistically significant, ‘positive’ results more likely to be: n. Published, published rapidly n. Published in English n. Published more than once n. Cited by others n *work cited in Egger M et al (2003). How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. Health Technology Assessment 2003; Vol. 7(1) 16 June 2005 (15)

Are systematic searches important? Some, limited evidence from health about how this affects estimates of effect: Unpublished trials show less beneficial effects than published trials n. Non-English language trials and non-Medline indexed trials tend to show larger treatment effects n. Trials that are difficult to locate tend to be smaller and of lower methodological quality n *Egger et al (2003) op cit. 16 June 2005 (16)

Different searches find different studies (1) How are decisions made about the entry of people aged 65+ to care services? SSCI Medline CINAHL Caredata Unique articles retrieved 237 182 27 16 Unique relevant articles 116 73 24 15 Taylor B, Dempster M and Donnelly M (2003) Hidden Gems: systematically Searching Electronic Databases for Research Publications for Social Work and Social Care. C J Social Work, 33: 423 -429. 16 June 2005 (17)

Different searches find different studies (2) Detail of search strategy Total number of citations Number of relevant studies Sensitivity % 1 31 terms 1048 72 100 2 11 terms 669 64 89 3 7 terms 385 47 65 *Adapted from: Harden A, Peersman G, Oliver S, Oakley A (1999) Identifying primary research on electronic databases to inform decision-making in health promotion: the case of sexual health promotion. Health Education Journal 58: 290– 301. 16 June 2005 (18)

A matter of degree? Discuss. . Table 1. Systematic and traditional reviews compared Pettigrew, M. (2001) Systematic reviews BMJ, 322, 98 -101 Systematic Traditional Review question Often not stated Searching for studies Attempt to find all relevant No attempt to find all relevant Inclusion/exclusion criteria Explicit Implicit Assessing quality Included Not included Synthesis 16 June 2005 (19) Stated Based on sound studies only Includes all studies

What is a systematic review? A systematic review develops an evidencebased statement in a particular area of interest. It is a piece of research in its own right using explicit and transparent methods following a standard set of procedures. This means it can be replicated. It judges the quality and relevance of evidence for a given question. 16 June 2005 (20)

Research evidence for policy and practice • “…… policy makers and practitioners who intervene in the lives of other people not infrequently do more harm than good” Chalmers I (2003) Trying to do more good than harm in policy and practice: the role of rigorous, transparent, up to date, replicable evaluation. Paper commissioned for the Annals of the American Academy of Political and Social Science. 16 June 2005 (21)

16 June 2005 (22)

16 June 2005 (23)

‘Reduce the Risk’ Campaign in the early 1990 s in the UK “The risk of cot death is reduced if babies are NOT put on the tummy to sleep. Place your baby on the back to sleep…. …. . Healthy babies placed on their backs are not more likely to choke. ” 16 June 2005 (24)

16 June 2005 (25)

Research for practice “Teaching is not at present a researchbased profession” Hargreaves D (1996) Teaching as a researchbased profession: possibilities and prospects. Teacher Training Agency (TTA) Annual Lecture. London: TTA. Procedural (craft / apprentice) Vs. declarative (for e. g. research based) knowledge 16 June 2005 (26)

Access to research “. . (which is) presented in a form or medium which is largely inaccessible to a non-academic audience; and lack(s) interpretation for a policy-making or practitioner audience”. Hillage J, Pearson R, Anderson A, Tamkin P (1998) Excellence in Research in Schools. London: Department for Education and Employment/Institute of Employment Studies. 16 June 2005 (27)

Public debate “We are, through the media, as ordinary citizens, confronted daily with controversy and debate across a whole spectrum of public policy issues. But typically, we have no access to any form of a systematic ‘evidence base’ - and therefore no means of participating in the debate in a mature and informed manner”. AFM Smith (1996) Mad cows and ecstasy: chance and choice in an evidence-based society. Journal of the Royal Statistical Society 159(3): 367 -383. 16 June 2005 (28)

Facilitating the use of research for policy and practice By preparing systematic reviews of the results of relevant, reliable research Additional research, IF systematic reviews of existing research show that this is needed 16 June 2005 (29)

3 Key stages and variation • What key stages do the different models of systematic research synthesis (SRS) have in common? • How do the different models differ? • In what ways do review questions influence review methods? 16 June 2005 (30)

Key decision-making stages in SRS Form review team Formulate review question and develop protocol Search for and screen studies (search strategy) Describe studies (systematic map of research) Assess study quality (and relevance) Synthesise findings (answering review question) Communicate and engage 16 June 2005 (31)

Reviewing as a team activity • Using more than one reviewer at key decision-making stages – Independent work, followed by consensus – Quality assurance of samples • Research user participation – In setting question to increase relevance – Could mean service users, practitioners, policy makers… 16 June 2005 (32)

Formulating the review question • Provides framework for all other stages of review • Specify/clarify conceptual framework – Type of question ? – Study types ? – Settings / populations / phenomena ? • Issues to consider – Why is the review needed ? – Who will use the review ? 16 June 2005 (33)

Developing the review protocol • Why do we want one? • Protocol contains – – – Review question Definitions of key concepts Explicit inclusion criteria Details of review methods for each stage Details of who’s involved/ how to participate • Guidance on developing protocols is available – EPPI-Centre, Cochrane, Campbell 16 June 2005 (34)

What is a systematic search? • • Thoughtfully planned Carefully executed Accurately recorded Made explicit • Open to scrutiny • Can be replicated and extended 16 June 2005 (35)

Systematic and comprehensive searching • Based on conceptual framework and resultant inclusion and exclusion criteria • Techniques of searching and retrieval aim to maximise the yield of relevant studies while minimizing the number not relevant to the review • Searching and screening stages are generally kept separate • Both stages are documented 16 June 2005 (36)

Sensitivity and specificity • Sensitivity – Expresses ability to locate all studies of interest – Calculated as the number of relevant studies located as a proportion of all that exist – Low sensitivity means that many relevant studies missed by search • Specificity – Expresses the accuracy of the search strategy in identifying studies of interest – Calculated as the proportion of the total number of studies identified by search which are deemed ‘relevant’ – Low specificity means that the search identified many studies that were not relevant to the review. TRADE OFF BETWEEN SENSITIVITY AND SPECIFICITY 16 June 2005 (37)

Screening then Describing studies • Screen to check that studies found through search strategy do meet the inclusions criteria • Describe the studies = systematic map 16 June 2005 (38)

Systematic maps and systematic synthesis • Map: What has been done? – maps out research activity (for e. g. broader question with multiple designs) – provides context for synthesis – research designs in primary studies part of that context – Allows potential for narrowing question (inclusion criteria) • Synthesis: What is known from what has been done? 16 June 2005 (39)

16 June 2005 (40)

Quality assessment of studies • To make (and record) judgements about study quality • To inform the synthesis – Only use findings from studies judged to be of high quality to inform recommendations for policy or practice – Qualify findings (‘quality framework’) – To examine differences in findings according to quality 16 June 2005 (41)

Methods for synthesis Tend to build on techniques used in primary research • Statistical meta-analysis – E. g. numerical effect sizes are combined to create a pooled and weighted overall effect size • Narrative ‘empirical’ – Conceptual framework used to construct and organise a narrative to combine the evidence from each study (evidence could be numerical or textual) • Conceptual – E. g. Qualitative techniques used to organise and combine conceptual or textual data 16 June 2005 (42)

Dimensions of difference in synthesis models • • • 16 June 2005 (43) Review/research questions (impact, process, need) Research designs considered relevant Types of data - numerical or textual Quality assessment of different designs Breadth of designs and study focus Variation in contribution of each study to the systematic synthesis

Review research questions • What are the questions that users want answered? – – – What do people want? (Needs assessment) What works? (Impact/effectiveness) Why/how does it it work? (Process/explanation) What is happening? (Implementation levels) Others…? • Different research designs best for addressing different questions (fit for purpose) 16 June 2005 (44)

Organisations supporting systematic reviewing • The Cochrane Collaboration – healthcare interventions www. cochrane. org • The Campbell Collaboration – interventions in social work, criminal justice and education www. campbellcollaboration. org • The EPPI-Centre – full range of empirical research in education (all phases), health promotion, social care and employment www. eppi. ioe. ac. uk 16 June 2005 (45)

Quality-assessment methods • SYNTHESIS 1 • Quantitative (Trials) • Provision of pre- and post- data on outcomes • Provision of data on all outcomes measured • Employment of equivalent control/comparison group • Resulted in ‘high’, ‘medium’ and ‘not sound’/ ‘low’ trials 16 June 2005 (46) • SYNTHESIS 2 • Qualitative (‘Views’) • Quality of reporting (5 items) • Sufficiency of strategies for reliability/validity (4 items) • Extent to which study findings were rooted in children’s own perspectives (3 items)

5 Ways forward • What else do you want to know about the principles and methods discussed today? • If planning to use some SRS methods, what issues (drivers, resources, methodology) might be important? 16 June 2005 (47)

Scoping, mapping and reviewing in-depth • Can select from these approaches for a systematic approach to the following kinds of review: – What is the size of the literature? – What kinds of literature exist? – What can this literature tell us reliably? 16 June 2005 (48)

Examples of other systematic review sources • The ESRC UK Centre for Evidence Based Policy and Practice Evidence Network – http: //www. evidencenetwork. org/ • The NHS Centre for Reviews and Dissemination, University of York: Database of Abstracts of Effectiveness (DARE) – http: //agatha. york. ac. uk/ • Bibliographic databases (try combining your topic with ‘meta-analysis’ or ‘systematic review’ or ‘research synthesis’) 16 June 2005 (49)

EPPI-Centre Dissemination • Web site for all EPPI-Centre reviews: http: //eppi. ioe. ac. uk/ • Research Evidence in Education Library (REEL) – Web-page accessibility – Review Group details – Protocols – Reviews and user perspectives – Primary data underlying reviews • Bibliographic information • Detailed descriptions and quality assessment 16 June 2005 (50)

SRS: not just medicine and effectiveness • Campbell R, Pound P, Pope C, Britten N, Pill R, Morgan M, Donovan J (2003) Evaluating meta-ethnography: a synthesis of qualitative research on lay experiences of diabetes and diabetes care. Social Science and Medicine. • Gilbert R, Salanti G, Harden M, See S (2005) Infant sleeping position and the sudden infant death syndrome: systematic review of observational studies and historical review of recommendations from 1940 to 2002. Int J Epidemiol. doi: 10. 1093/ije/dyi 088 • Petrosino A, Turpin-Petrosino, J Buehler J (2005) "Scared Straight" and other juvenile awareness programs for preventing juvenile delinquency. The Cochrane Database of Systematic Reviews 2005 Issue 2 16 June 2005 (51)

Contact us • David Gough d. gough@ioe. ac. uk • Rebecca Rees r. rees@ioe. ac. uk 16 June 2005 (52)

• EPPI-Reviewer • The EPPI-Centre’s in-house reviewing software 16 June 2005 (53)

16 June 2005 (54)

Methodological and conceptual challenges • Nature of question and data for review and synthesis • Exclusion criteria: (i) breadth; (ii) primary research designs • Quality assessment of studies • Contribution of studies to synthesis • Methods of synthesis and interpretation • Synthesis of non-empirical and interpretative qualitative research • Operationalising user involvement 16 June 2005 (55)

Technical/structural/financial challenges • Lack of infrastructure to support systematic research synthesis – training, funding, sustainability • Time and costs of systematic reviews – EPPI-Centre Df. ES education funded at £ 2. 5 m for initial 5 year support for EPPI Centre infrastructure and ‘seed corn’ support for 4 new RGs per year 16 June 2005 (56)

Ideological/political challenges • Fear of central political control of research rather than democratization of the research process • Confusion between experimental methodologies and systematic research synthesis • Polarization of ‘quantitative’ and ‘qualitative’ paradigms • Shown to be useful processes and products • National and international culture change in research, policy, practice and public involvement in research 16 June 2005 (57)