MSc ASR SR 06 Session 9 Quantitative methods

MSc ASR, SR 06 Session 9 Quantitative methods of social research for cross-national comparisons Paul Lambert, 5. 2. 02 http: //staff. stir. ac. uk/paul. lambert/teaching. htm 1

Quantitative cross-national social research 1) Introduction 2) Three traditions in Qn cross-national research 3) Seven themes in Qn cross-national research Case study 1: Secondary analysis of cross-national surveys 2

Introduction: Formats of Quantitative Cross-National research Aside: cross-national between country cross-national comparative q But in quantitative methods, ‘XN’ & ‘comparative’ often used interchangeably 3

QDA: Analysis of patterns of relationships between variables in the variable-by-case matrix [Low # of vars; stats / graphical summaries] Cases 1 2 3 4 1 1 2 2. . 17 18. Variables 1. 73 A. 1. 85 B. 1. 60 C. 1. 69 A. . . . N 4

A convenient distinction Macro-social data Micro-social data Work and/or report at level of aggregated unit constituent unit (eg (country) individuals) Micro-macro distinction isn’t always important (& can be confusing). But is widely used, & tends to be associated with different research fields. 5

a) Macro-Social Qn. XR Each case represents country, & aggregate statistics are compared Denmark Ireland Italy Portugal UK Ideal family size 1979 1989 Religiosity ‘ 81 2. 31 2. 13 2. 06 3. 62 2. 79 3. 42 2. 11 2. 20 2. 90 2. 29 2. 23 2. 66 2. 29 2. 14 2. 33 (eg from Coleman 1996: 39) 6

b) Micro-social Qn. XR Cases (eg people) are grouped by country Case id Country Indv. vars 1 1 17 1. 73 A 2 3 4 5 6. 1 1 2 2 3. 18 17 18 18 19. 1. 85 1. 60 1. 69 1. 65 1. 84. B C A C B. Natl. var 56. 2 50. 8 260. 3. 7

Data Analytical techniques Same core data analysis techniques as for other social science applications, eg : v. Critical issue is ‘level of measurement’ v. Univariate, bivariate, multivariate v. Description v’s inference v. Survey methodology issues v. A few advanced extensions, eg ‘mixed models’ to cater for hierarchical effects. 8

Key feature of Qn. XR: Country as a categorical factor Analyse within countries then compare outcomes (‘case oriented’) V’s Analyse data pooled between countries, use countries / country level factors as explanations (‘variable oriented’) 9

Country as a categorical factor Often criticised: • Appears to be overly simplistic However • Same as other QDA factors, eg gender, . . • Critics forget qualified interpretations that good QDA makes: [these patterns] are associated with categories, all other things being equal. • Bad QDA: forget controls for relevant other things 10

Quantitative cross-national social research 1) Introduction 2) Three traditions in Qn cross-national research 3) Seven themes in Qn cross-national research Case study 1: Secondary analysis of cross-national surveys 11

A typology of quantitative crossnational research designs? • Bryman 2001(p 53): 4 types of cross-cultural research • Ragin 1987: 2 analytical orientations, one mainly Qn, the other mainly Ql; proposed resolution with Qn-style summaries of Ql research Ø No typology is perfect – there is much overlap and ambiguity in methods – but it can be useful to classify patterns of modern social research… 12

A popular two-stage story: Early quantitative researchers naively attempted to measure national differences as single variables. They badly misclassified or ignored important national level differences. Much more thoughtful considerations of complex national contexts are needed, & often these are more suited to qualitative research methods. Eg: Hantrais and Mangen 96: moves to interpretive methods; Ragin 87: variable v’s case oriented approaches 13

This inaccurate simplification implies a false Qn/Ql division: • Doesn’t reflect variety of current practice in Qn. XR (& indeed past practice) • Doesn’t acknowledge multivariate Qn. XR • Doesn’t do justice to many carefully conducted / reported Qn. XR projects • Tends to over-estimate Ql. XR capactity 14

A picture of Qn. XR under this typology: Crude variable oriented Case oriented 15

Multitude of contemporary social research examples don’t fit this • There a great many quantitative caseoriented designs • It is unfair to describe all variable-oriented designs as inadequate • . . though to be fair, many variable-oriented projects are genuinely weak! 16

A fairer typology of Qn. XR Crude variable-oriented Case oriented Sophisticated variable oriented 17

Crude variable oriented Early or recent, micro- and macro- research making claims over country level differences, with: • Insufficient exploration of relevant explanatory factors • Limited or poor quality variable operationalisations & discussions • Relevant national contexts not appreciated • False assumptions of good harmonisation Example: see the illustrated analysis using the ESS 18

Sophisticated variable oriented Early or recent micro- and macro- research making claims over country level differences, with: • Sufficient exploration of relevant explanatory factors • Good quality variable operationalisations and discussions • Relevant national contexts suitably described • Accurate assumptions of good harmonisation Example: more applications than is often realised… 19

Case oriented q Qn analyses within countries, then outcomes evaluated between countries by authors / readers • Doesn’t require strong assumptions of data harmonisation • Expertise of report writer covers national context Examples: Edited books; centrally coordinated projects; end user reviews; … 20

Sophisticated variable oriented • Attractive method: – offers parsimony of XN summary – uses large scale resources • Methodology for good conduct necessary – Reliability, validity, implementation, translation – Sample design – Reporting strategy and claims • Boundary to crude research subjective / contested • Existence often denied by anti-Qn sociologists… 21

Why not be over-cautious? • Case oriented Qn. XR seems a safe bet? ØDoesn’t make claims not justified ØBut doesn’t make much impact either • Remains need for good variable oriented: Ø Offers a parsimonious summary of national differences Ø Govt / media with utilise regardless 22

Quantitative cross-national social research 1) Introduction 2) Three traditions in Qn cross-national research 3) Seven themes in Qn cross-national research Case study 1: Secondary analysis of cross-national surveys 23

3. 1) Data availability • Massive increases in data resources accessible to social researchers – Secondary survey datasets – Official statistics resources – Internet provision / communications • • Many data resources under-exploited Most data originates from survey sources - but some exceptions 24

3. 2) Dataset complexity • Secondary surveys tend to feature – – – • Many variables and cases Complex variable operationalisation choices Complex structuring (eg multiple hierarchies) Complex weighting / sampling information Data analysis & management software needs Aggregate statistics’ features – Difficulty understanding source derivation 25

3. 3) Variable operationalisation • Single biggest issue in most Qn. XR conduct – – – • Survey design Dataset analysis Result reporting Models of comparability – – – Exact equivalence of measures Relativistic equivalence of meanings Wide literature on ‘reliability’, ‘validity’ of X-N variable measures and aggregate statistics 26

Variable harmonisation ctd • Choices over key variables allow use of previous literatures (eg H-Z & Wolf 2003). Eg measures of income; occupation; ethnic group; education; region; crime; health; . . • Choices over specific analytical variables require new efforts Eg, attitude harmonisations of Inglehart. 27

3. 4) Survey design Harkness et al 2003: Ex post facto harmonisation (more widespread, eg Eurostat, IPUMS, LIS) v’s Coordinated design, sampling, & implementation (big money projects, eg ESS, ISSP) Latter as preferable – but whilst many projects attempt this model, far fewer succeed. . . 28

3. 5) Conduct and logistics • • • High costs of coordinated surveys Considerable efforts, and many errors, in ex post facto harmonisation Issues of cooperating with colleagues / diverging academic traditions, eg – – – different views data access / confidentiality Technical / software compatibility different organisations involved in survey production Qn. XR can be very slow process 29

3. 6) Temptation Cross-national datasets nearly always look simpler than they really are dangerous temptation to rush into uncritical variable analysis 30

3. 7) Prejudice • Prejudices against quantitative methods pronounced in European sociology, especially wrt cross-national comparisons – – • Qn. XR evidence often ignored Qn. XR researchers portrayed as simplistic Prejudices favouring quantitative methods often seen in governmental and media organisations – Mainly: uncritical acceptance of harmonisations 31

Quantitative cross-national social research 1) Introduction 2) Three traditions in Qn cross-national research 3) Seven themes in Qn cross-national research Case study 1: Secondary analysis of cross-national surveys 32

Some leading secondary surveys: (see handout for internet links) ESS ISSP IPUMS LIS / LES / LWS ECHP / CHER / PACO WVS / EVS Eurobarometer Education: PISA / TIMSS Social Stratification: CASMIN / CCAP / … Health /Welfare: eg SVH 33

European Social Survey • New annual attitudes / values / social circumstances cross-sections, 2002 • Equivalence of design and survey implementation between countries • Extensive methodological resources • Free access to data 34

Analysis (see SPSS syntax eg) • Opens harmonised files from 15 countries in 2002 • Select variables measuring attitudes, age, gender and educational levels • Generate tables of patterns split by countries • Use regression models to evaluate contribution of mulitiple explanatory factors: – Country specific ‘structural breaks’ – Country effects as variables / interactions 35

Liberal attitudes to homosexuality and their associations with educational level (national average and Cramer’s V to educ) % Switzerland Czech Rep Spain Finland UK Greece Hungary Ireland 81 58 70 62 75 51 48 82 CV 10 11 20 14 7 23 5 8 % Israel Netherlands Norway Poland Portugal Sweden Slovenia 59 88 77 46 71 82 52 CV 20 6 13 16 15 12 18 36

Log-regression prediction of liberalism to homosexuality for ESS adults (value & significance of coefficient estimate) Age-squared -1. 72** Interactions: Low educ -0. 31** Low educ*NW 0. 19** High educ 0. 35** Female*NW 0. 35** Female 0. 21** Female*South -0. 15* North West 1. 07** Contrast: medium education Southern male from eastern European 0. 56** country. 37

This is ‘crude’ variable oriented • Didn’t try out sufficient relevant explanatory factors • Didn’t check variable choices extensively • Merged variable categories for convenience • Didn’t use survey weights • Didn’t contextualise reporting with sufficient substantive national background and crossexaminations of data sources and measures 38

. . but it could have been sophisticated variable oriented • • Could have evaluated variable meanings Could have studied backgrounds Could have added more explanatory factors Could have reported more carefully • . . Research consumption = understanding how well the results were prepared 39

Summary on Quantitative crossnational research Ø Quant methods contribute to both ‘variable’ & ‘case’ oriented comparisons Ø Crude variable oriented widely criticised, and many bad examples persist Ø Sophisticated variable oriented research can be found, and represents most attractive format of Qn. XR 40