Скачать презентацию Evaluating Media Assistance Programs What We Have Done Скачать презентацию Evaluating Media Assistance Programs What We Have Done

4a96518533222d40f00f03d4bda68cc7.ppt

  • Количество слайдов: 42

Evaluating Media Assistance Programs: What We Have Done and What We Have Learned Lee Evaluating Media Assistance Programs: What We Have Done and What We Have Learned Lee B. Becker & Tudor Vlad

Background • University of Georgia • Grady College of Journalism & Mass Communication • Background • University of Georgia • Grady College of Journalism & Mass Communication • James M. Cox Jr. Center for International Mass Communication Training and Research – Conducts training programs – Focuses on evaluation Athens and the university in 1840 as depicted in the painting by George Cooke. 2

My Research Interests • Media effects – Opinion and learning • Organizational behavior and My Research Interests • Media effects – Opinion and learning • Organizational behavior and message construction – Characteristics of workers, including educational credentials • Sociology of education and labor markets – Needs assessment – Evaluation of impact 3

Evaluation of the Knight International Press Fellowship Program 1. Interviewed 531 persons in 8 Evaluation of the Knight International Press Fellowship Program 1. Interviewed 531 persons in 8 European and 3 Latin American Countries 2. At 31 people in each of the 11 countries 3. Field work completed in 1999. 4

Basic Distinction • Assessing and monitoring program process. • Measuring and monitoring program outcomes. Basic Distinction • Assessing and monitoring program process. • Measuring and monitoring program outcomes. 5

Locus of Impact • On journalists – Attitudes – Behaviors • On organizations • Locus of Impact • On journalists – Attitudes – Behaviors • On organizations • On media system • On society overall 6

Problems with Self-Reports • Timing is important – Immediately after training is too soon Problems with Self-Reports • Timing is important – Immediately after training is too soon – Duration of effect is variable • Likely to overestimate effect – To satisfy the program provider – To justify own investment of time and effort • Lacks reference – Most participants in training on projectile to change • Participant may not know the answer 7

Journalists in Philippines • Overestimated the amount of coverage they gave to the issues Journalists in Philippines • Overestimated the amount of coverage they gave to the issues behind the conflict. • Underestimated the amount of coverage of the Manila media to issues other than the conflict. 8

Measurement Solutions • Go from the general to the specific – Now I want Measurement Solutions • Go from the general to the specific – Now I want to start with a very general question. Please tell me some of the things that you learned from your participation in the XXX Fellowship that you consider at this point to have been most important to your work as a journalist. Of course, if you didn’t learn anything just tell me that. – OK. Now I’m going to ask about some specifics. You may already have touched on some of these, but I want to make sure I cover everything. Did your participation in the XXX Fellowship help you develop your skills in findings sources for stories you have reported since you returned? • Use what we can a “jab and probe” questioning technique. – Are there specific sources that you came into contact with through the XX Fellowship that you still use today? • IF YES: Who are some of those? 9

Validation Measures • Interviews others who also can attest to attitudes, behaviors and changes Validation Measures • Interviews others who also can attest to attitudes, behaviors and changes in both – Colleagues – Supervisors • Examples – Since you completed the XXX Fellowship, have you held any “brown bag” or other information discussions about your experiences in the program with other journalists? – As far as you know, has (NAME OF FELLOW) made any efforts to share her/his experiences at the Fellowship with others at your organization since she/he returned? 10

Design Solutions • Treat timing as a variable – We have compared short-term programs Design Solutions • Treat timing as a variable – We have compared short-term programs conducted across time • Timing matters—effect several years • But topic matters too and can offset lag • Develop a “control” group – True experimental design isn’t often possible – Imperfect “control” better than none • Develop before and after design 11

Some Control Examples • Control group made up of matched respondents – Successful applicants Some Control Examples • Control group made up of matched respondents – Successful applicants for subsequent ongoing programs – Those who work in similar settings • Can match based on lists • Can get respondent to help create a match • Control groups made up of matched organizations • Best control is the group itself – Before and after has limitation of systemic change 12

Post-Employment Training Common Part of Media Assistance • When evaluating journalism training component of Post-Employment Training Common Part of Media Assistance • When evaluating journalism training component of media assistance, keep in mind – Measurement problems – Design problems • Use tactics to overcome them 13

Central Concerns of Media Assistance • Concern with Media Freedom or media independence • Central Concerns of Media Assistance • Concern with Media Freedom or media independence • Concern with media performance • Concern with the connection between Media Freedom and media performance • Concern with the relationship between media performance and democracy 14

Mapping Media Assistance • • Effort by Monroe Price and colleagues Our purpose was Mapping Media Assistance • • Effort by Monroe Price and colleagues Our purpose was to expand that Shifted to narrower short-term goal Rough estimate: $1 billion year spent on media assistance each year • Ultimately, valuable to understand media assistance this way 15

Two Actor Groups • Media assistance community – Donors – Media assistance providers – Two Actor Groups • Media assistance community – Donors – Media assistance providers – Targets of media assistance – Media monitors • Academic community – Political scientists – Communication scientists 16

Academic Community • Political scientists – Extensive literature on democratization – Extensive debate on Academic Community • Political scientists – Extensive literature on democratization – Extensive debate on meaning of democracy • Communication scientists – Extensive literature on media freedom – Extensive debate on meaning of media freedom • Relatively little empirical testing of contribution of media freedom to democracy or democratization 17

Different Perspectives of Advocates of Media Assistance and Political and Communication Scientists • Former Different Perspectives of Advocates of Media Assistance and Political and Communication Scientists • Former believe media assistance leads to democratization – Role of the research is to document this • Scientists think all of these questions are open to examination 18

Media Assistance Trained, Skilled, Motivated Journalists Model of Impact of Media Assistance Media Organizations Media Assistance Trained, Skilled, Motivated Journalists Model of Impact of Media Assistance Media Organizations that Facilitate and Distribute the Work of Journalists Independent Media System Information Needed for Functioning of Democracy, Open Economy Functioning Democracy, Economy ? Civil Society Assistance Institutions of Civil Society 19

Media Assistance Trained, Skilled, Motivated Journalists Model of Impact of Media Assistance Media Organizations Media Assistance Trained, Skilled, Motivated Journalists Model of Impact of Media Assistance Media Organizations that Facilitate and Distribute the Work of Journalists Independent Media System Information Needed for Functioning of Democracy, Open Economy Functioning Democracy, Economy ? Civil Society Assistance Institutions of Civil Society 20

Does Media Assistance Lead to Trained, Skilled, Motivated Journalists? – Yes, probably • Evaluation Does Media Assistance Lead to Trained, Skilled, Motivated Journalists? – Yes, probably • Evaluation is spotty – Process evaluation rather than impact evaluation • Often rely on self-reports of impact • Control groups are rarely used • Evaluations often not independent of funder, media assistance provider 21

Media Assistance Trained, Skilled, Motivated Journalists Model of Impact of Media Assistance Media Organizations Media Assistance Trained, Skilled, Motivated Journalists Model of Impact of Media Assistance Media Organizations that Facilitate and Distribute the Work of Journalists Independent Media System Information Needed for Functioning of Democracy, Open Economy Functioning Democracy, Economy ? Civil Society Assistance Institutions of Civil Society 22

Does Media Assistance Lead to Facilitative Media Organizations? • Maybe – Most evidence is Does Media Assistance Lead to Facilitative Media Organizations? • Maybe – Most evidence is indirect, based on observations of participants – Designs are inadequate – Many examples of failed investments – Problems of sustainability paramount 23

Media Assistance Trained, Skilled, Motivated Journalists Model of Impact of Media Assistance Media Organizations Media Assistance Trained, Skilled, Motivated Journalists Model of Impact of Media Assistance Media Organizations that Facilitate and Distribute the Work of Journalists Independent Media System Information Needed for Functioning of Democracy, Open Economy Functioning Democracy, Economy ? Civil Society Assistance Institutions of Civil Society 24

Does Media Assistance Produce Independent Media Systems? • Some limited concrete evidence it does Does Media Assistance Produce Independent Media Systems? • Some limited concrete evidence it does • Comparative study by Steven Finkel and colleagues for USAID – Found that USAID Media investments across 165 countries led to gains in media freedom – Used Freedom House indicators • Considerable controversy over what is meant by media freedom 25

Media Assistance Trained, Skilled, Motivated Journalists Model of Impact of Media Assistance Media Organizations Media Assistance Trained, Skilled, Motivated Journalists Model of Impact of Media Assistance Media Organizations that Facilitate and Distribute the Work of Journalists Independent Media System Information Needed for Functioning of Democracy, Open Economy Functioning Democracy, Economy ? Civil Society Assistance Institutions of Civil Society 26

Do Independent Media Produce Information Needed for Democracy? • Almost no evidence either way Do Independent Media Produce Information Needed for Democracy? • Almost no evidence either way – No real explication of what information is needed for democracy 27

Media Assistance Trained, Skilled, Motivated Journalists Model of Impact of Media Assistance Media Organizations Media Assistance Trained, Skilled, Motivated Journalists Model of Impact of Media Assistance Media Organizations that Facilitate and Distribute the Work of Journalists Independent Media System Information Needed for Functioning of Democracy, Open Economy Functioning Democracy, Economy ? Civil Society Assistance Institutions of Civil Society 28

Do Independent Media Produce Civil Society Institutions? • No one knows • Mixed evidence Do Independent Media Produce Civil Society Institutions? • No one knows • Mixed evidence of the effectiveness of civil society assistance as well 29

Media Assistance Trained, Skilled, Motivated Journalists Model of Impact of Media Assistance Media Organizations Media Assistance Trained, Skilled, Motivated Journalists Model of Impact of Media Assistance Media Organizations that Facilitate and Distribute the Work of Journalists Independent Media System Information Needed for Functioning of Democracy, Open Economy Functioning Democracy, Economy ? Civil Society Assistance Institutions of Civil Society 30

Do Independent Media Foster Democratization? • Surprisingly little empirical evidence has been gathered – Do Independent Media Foster Democratization? • Surprisingly little empirical evidence has been gathered – No understanding of what conditions might be placed on this relationship 31

Six Hypotheses 1. The media-supremacist position, which holds that media freedom and independence produce Six Hypotheses 1. The media-supremacist position, which holds that media freedom and independence produce democracy. 2. The democracy-supremacist position, which holds that democratic reform determines and produces media freedom and independence. 3. The media-freedoms-are-an-element-of-democracy position, which argues that media freedoms are a part of democracy and, as such, have no causal force leading to democracy. 4. The null-effect position, which holds that there is no relationship between media freedom and democracy. 5. A media-freedom-hinders-democracy argument. 6. A democratization-hinders-media-freedoms stance. 32

Rozumilowicz Stages of Media Reform Linked to Stage Theory of Democracy 33 Rozumilowicz Stages of Media Reform Linked to Stage Theory of Democracy 33

Media Democracy Link Year of Press Freedom Political Rights +1 Political Liberty +1 Political Media Democracy Link Year of Press Freedom Political Rights +1 Political Liberty +1 Political Rights 1 Political Liberty -1 1981 0. 79 0. 82 0. 79 0. 81 1982 0. 80 0. 84 0. 82 0. 85 1983 0. 79 0. 82 0. 81 0. 85 1984 0. 80 0. 84 0. 82 0. 87 1985 0. 82 0. 83 0. 82 0. 87 1986 0. 79 0. 83 0. 82 0. 87 1987 0. 77 0. 81 0. 86 1988 0. 84 0. 80 0. 83 0. 86 1989 0. 82 0. 77 0. 93 0. 89 1990 0. 83 0. 77 0. 94 0. 88 1991 0. 81 0. 77 0. 93 0. 91 1992 0. 84 0. 90 0. 88 1993 0. 81 0. 89 0. 85 1994 0. 86 0. 88 0. 90 1995 0. 88 0. 91 1996 0. 88 0. 90 0. 91 1997 0. 88 0. 90 0. 92 1998 0. 89 0. 91 0. 92 1999 0. 89 0. 91 0. 90 0. 93 2000 0. 89 0. 90 0. 91 0. 93 2001 0. 92 0. 90 0. 93 2002 0. 94 0. 93 0. 94 2003 0. 94 0. 95 0. 93 0. 94 2004 34

What We Know About Media Freedom Measures? • Strong evidence of reliability across time What We Know About Media Freedom Measures? • Strong evidence of reliability across time • Strong evidence of reliability across measures • Some evidence of validity – Criterion • FH measures reflected changes in Warsaw Pact countries – Construct • IREX MSI related in predicted way in analysis of impact of hypercompetition on press performance 35

36 36

Relationship between Market Competition and Press Performance 37 Relationship between Market Competition and Press Performance 37

Limitations of Media Freedom Measures • Focus on structure of media system • Limited Limitations of Media Freedom Measures • Focus on structure of media system • Limited evidence of performance • Limited notion of media independence – Focus on independence from government – Do not recognize pressures of commercial dependence • Largely ignore audience • Largely ignore “needs” of democracy 38

Additional Monitoring • Develop checklist of needs of democracy – A democratic state is Additional Monitoring • Develop checklist of needs of democracy – A democratic state is a state in which all citizens have access to information about how the state operates. – A democratic state is one in which citizens have the ability to communicate to each other. – A democratic state is one in which conflicts are managed without resort to violence. – A democratic state is one in which representations of members of the state are presented in a way that foster appreciation and understanding. 39

Relationship between Press Freedom and Confidence in Media 40 Relationship between Press Freedom and Confidence in Media 40

Media Assistance Trained, Skilled, Motivated Journalists Media Organizations that Facilitate and Distribute the Work Media Assistance Trained, Skilled, Motivated Journalists Media Organizations that Facilitate and Distribute the Work of Journalists Information Needed for Functioning of Democracy Independent Media System Functioning Democracy ? Civil Society Assistance Institutions of Civil Society Model of Impact of Media Assistance 41

42 42