GSSR Research Methodology and Methods of Social Inquiry

GSSR Research Methodology and Methods of Social Inquiry socialinquiry. wordpress. com January 17, 2012 I. Mixed Methods Research II. Evaluation Research

MULITPLE-METHODS APPROACH Triangulation: applying 2 or more dissimilar measures and/or methods (research strategies) to investigating a certain problem. Why do it: increased confidence in findings

Key: what you want to study (i. e. the nature of the research question, the phenomena considered) should determine your research strategy/methods! - the relative strengths & weaknesses of alternative approaches should be weighted in deciding which methods to select, and how best to combine them when possible. Table 12. 2 in Singleton and Straights, p. 399

Multiple methods can also be used within a single approach: - allows exploiting the strengths & weaknesses of complementary methods. Ex: One approach (survey method), but mail questionnaire to probability sample, & face-to-face interviews on smaller sample of non-respondents, to estimate non-response bias. Vignette experimental designs in survey research; Use of archival records to identify groups for field research …

II. EVALUATION RESEARCH www. socialresearchmethods. net/kb/intreval. php http: //ec. europa. eu/regional_policy/sources/docgener/evaluation/evalsed/sourc ebooks/method_techniques/index_en. htm Application of social research methods for: (a) assessing social intervention programs & policies instituted to solve social problems; (b) in the private sector: assess policy, personnel, products. Major goal of evaluation: Influence decision-making/policy formulation through providing empirically-driven feedback.

Evaluation takes place within a political & organizational context, where researchers face multiple stakeholders. Stakeholders: - individuals/ groups/ or organizations that have a significant interest in how well a program/product functions/performs. Ex: Program sponsor (actor who initiates & funds the program/product) Evaluation sponsor (who mandates & funds the evaluation) Policymaker/decision maker who determines the fate of the program/product, …

Outcome of evaluation: Detailed technical report that describes the research design, methods and results. Plus: executive summaries, memos, oral reports geared to the needs of specific stakeholders.

Evaluation Strategies Scientific-experimental models (see socialresearchmethods. net/kb/intreval. php) Take values & methods from the social sciences; - prioritize on the desirability of impartiality, accuracy, objectivity & the validity of the information generated. Ex: - experimental & quasi-experimental designs; - objectives-based research that comes from education; - econometrically-oriented perspectives including costeffectiveness and cost-benefit analysis; - theory-driven evaluation.

Management-oriented systems models - emphasize comprehensiveness in evaluation, placing evaluation within a larger framework of organizational activities. The Program Evaluation and Review Technique (PERT) The Critical Path Method (CPM). The Logical Framework -- "Logframe" model developed at U. S. Agency for International Development Units Treatments Observing Observations Settings (UTOS); Context Input Process Product (CIPP)

Qualitative/anthropological models Emphasize: - the importance of observation; - the need to retain the phenomenological quality of the evaluation context - the value of subjective human interpretation in the evaluation process. Ex: naturalistic or 'Fourth Generation' evaluation; the various qualitative schools; critical theory & art criticism approaches; and, the 'grounded theory' approach of Glaser and Strauss among others.

Participant-oriented models Emphasize the central importance of the evaluation participants, especially clients & users of the program or technology. Ex: Client-centered and stakeholder approaches; consumeroriented evaluation systems.

Types of Evaluation Formative Evaluation (Product): Needs assessment: who needs the program? How great the is the need? & What might work to meet the need? Evaluability assessment: is an evaluation feasible & how can stakeholders help shape its usefulness? Structured conceptualization: helps stakeholders define the program/ technology, the target population, & the possible outcomes Implementation evaluation: monitors the fidelity of the program or technology delivery Process evaluation investigates the process of delivering the program or technology, including alternative delivery procedures

Summative evaluation (Effects/Outcome): Outcome evaluations: did the program/technology produce demonstrable effects on specifically defined target outcomes? (effect assessment) Impact evaluation: broader; assesses overall/ net effects -intended or unintended -- of the program/ technology as a whole Cost-effectiveness & cost-benefit analysis address questions of efficiency by standardizing outcomes in terms of their dollar costs & values Secondary analysis: reexamines existing data to address new questions or use methods not previously employed Meta-analysis integrates the outcome estimates from multiple studies to arrive at an overall/ summary judgement on an evaluation question

Methodological Issues in Evaluation Research Effect Assessment: did the program/technology caused demonstrable effects? The ‘black box’ paradigm We can observe: - what goes into the ‘black box’ – the inputs (here, the program/product/intervention) and - what comes out of the box – the output (certain effects). Theory as guide to research

Research Design & Internal Validity Ideal strategy for effect assessment: experiment, with units of analysis randomly assigned to at least 2 conditions (one with intervention present, one without).

Measurement Validity - need good conceptualization - reliable and valid measures of cause (treatment program) and effect (expected outcome). Issues with creating valid indicators of program outcomes. Timing of outcome measurement: - lagged effect of the program; continuous/gradual effects, vs. instantaneous effects. To increase measurement validity: multiple measurement (independent measures) & different points in time.

External Validity - Random sample, or ‘true’ experiments are most often not feasible non-probability sample Selection biases: - self-selection into the treatment group; - selection of program participants because they are likely to generate positive results / are available; Social context of evaluation may threaten external validity