Quantitative methods Questionnaire Design Research Stages n

Quantitative methods Questionnaire Design

Research Stages n n n n Stage 1: Research aims Stage 2: Literature Stage 3: Research design Stage 4: Instrumentation Stage 5: Piloting Stage 6: Data collection Stage 7: Data cleaning and Data analysis Stage 8: Research report 2

What is a questionnaire? n n A questionnaire is a structured instrument for collecting primary data on populations of interest in applied or theory-based research. A well-designed questionnaire facilitates the respondents to provide complete and accurate information. 3

Main types of questionnaires n n Mail/Online questionnaires Structured interview schedules 4

Methods of administering Interview: face to face or telephone; n Mail or other distribution method n Computer-based n 5

Interview/telephone n Sampling implication ¨ Time n of interview, sample selection Minimise interviewer effect ¨ Standardise interview schedule, scripted ¨ Example: Year_After_9 -11. doc 6

Mail or Distribution Sampling implication n Not supervised n ¨ Clarity of questions ¨ Complexity versus simplicity ¨ Room for comments, problems with the questionnaire 7

Online or computer administered Sampling implication n Not supervised n ¨ Clarity of questions ¨ Can incorporate complex pathways of questionnaire items n If answer is yes, go to Q 5. If answer is no, go to Q 7, etc ¨ Room for comments, problems with the questionnaire ¨ Missing responses checked 8

Steps Establish a table of specifications, panel and revise if necessary Write the questions 1. 2. 3. 4. Prepare the questionnaire layout for printing 3. 1. 2. 3. 4. Determine the general question content needed to obtain each of the desired information Determine the form of response for each of the questions Choose the exact question wording. Panel and revise the questions if necessary Arrange the questions into an effective sequence. Specify the physical characteristics of the questionnaire (paper type, number of questions per page, etc. ) Panel and revise the questions and the whole questionnaire Pre-testing and Pilot the questionnaire. Analyse and revise the questions and the whole questionnaire if needed 9

Establish a table of specifications n Steps 1. 2. 3. 4. 5. 6. 7. Identify the program objectives for which the questionnaire is being developed Operationalise the objectives Identify the population to be addressed Identify the methods of administration Establish the link between research questions, information needed, source of information and methods of data collection Decide on how to measure each variable Establish a table of specifications 10

Identify the program objectives for which the questionnaire is being developed What are the general program objectives or research questions? n What are the specific research questions? n What are hypotheses? (draw a diagram) n 11

Example PISA Contextual Questionnaire PISA_Questionnaire_Technical. Report. pdf 12

PISA Research Themes Table 3. 1 n Examples: n ¨ Student engagement with mathematics Mathematics self-efficacy n Mathematics self-concept n Mathematics anxiety n Interest in and enjoyment of mathematics n Instrumental motivation to learn mathematics n Study time in mathematics n 13

Example Report: n Student engagement in schools Student. Engagement. OECDWillms. pdf 14

Determine specific information needed List dimensions, variables that you should measure n List specific information you hope to collect n 15

Establishing the link between information needed, source of information and methods of data collection Information needed / variables to be measured Source of information Methods of data collection 16

Identify the population to be addressed Source of information n Who is appropriate to provide the necessary information n Characteristics of the target population n 17

Identify the methods of administration n Think in advance how the questionnaire should be administered? 18

How to measure each variable Decide if the variable is directly observable or latent n Is the variable measured by one item? n Is the variable a composite of a number of items (indicators)? 19

Example: Student engagement in schools (see Willms report) Define student engagement. n Student engagement is measured by two components: n ¨ Sense of belonging ¨ Participation n What are indicators of Sense of belonging and Participation? 20

How to measure each variable Latent variable (Construct) n Is the variable measured through a set of indicators? n If yes, what are the possible dimensions and/or indicators n How is each indicator measured? n Produce a table showing how to measure each indicator 21

Questionnaire design Latent Variables and Indicators 1 2 3 Latent Variable Indicators 4 5 6 Not directly observable Directly observable 22

Questionnaire design Construct development Step 1: Define a meaning for your construct. It will be of narrow focus, capable of sustaining precise measurement. Step 2: Develop appropriate items for this construct. Step 3: Test the hypothesis that the items do indeed imply the meaning of the construct as defined. Step 4: Revise the items (Barrett, 2002) 23

How to develop indicators (items) 1. 2. Draft the first items Panel the indicators 24

Example: How to measure Sense of belonging – draft items n n n n A. I feel like an outsider (or left out of things) B. I make friends easily C. I feel like I belong D. I feel awkward and out of place E. Other students seem to like me F. I feel lonely G. I do not want to go to school H. I often feel bored. 25

Example: How to measure participation n Measured by the frequency of absence, class-skipping and late arrival at school during the two weeks prior to the PISA 2000 survey. 26

Panel and Review the table of specifications for the program/ project n n Do the variables cover all the information needed for the program? Do the indicators cover all the dimensions of the variable measured? 27

Data analysis to check the appropriateness of the indicators n Example on student engagement in schools: (p 64, Willms) n A factor analysis of the responses found two factors ¨ one that is based on the first six items and describes whether students feel accepted and included by their classmates, ¨ The second is based primarily on the last two items and describes whether students like school and find it interesting. ¨ The analysis also revealed that the six belonging items contributed almost equally to the first factor. n Therefore, the measure of sense of belonging used in this report is based on a Rasch scaling of the first six items 28

Reliability and Validity 29

Reliability and Validity Low Reliability (A) High Validity High Reliability (B) Reliable but low validity (C) 30

Validity is the ability of an instrument to measure what is designed to measure (Smith, 1991) n Validity refers to the extent to which an empirical measure adequately reflects the real meaning of the concept under consideration (Babbie, 1990: 33) n 31

Example: Student engagement in schools (p. 18, Willms) n n Participation is measured by the frequency of absence, class-skipping and late arrival at school during the two weeks prior to the PISA 2000 survey. There are two issues concerning the validity of the participation measure. ¨ One issue is that the measure of participation could be more extensive. It was measured in this study with a rather narrow focus on student absenteeism. ¨ The second issue pertains to how participation is measured. A number of students may have missed school because of illness or for other legitimate reasons. 32

Reliability - 1 Reliability is concerned with how much error is included in the evidence. n If there is no error in the measurement, the same measurement should be consistent over time and context. n The reliability of a measure refers to the consistency of measurement for repeated measurements of the same phenomenon. (Willms, p 65) n 33

Reliability - 2 Internal consistency reliability Description Concerned with how well the items act together to elicit a consistent type of response. Often referred to as Coefficient a Limitations Requires statistical procedures to estimate reliability. Does not capture sources of error such as variation over time. Assumes all items tap into one single dimension. Usages Important to establish when designing a scale 34

Reliability example Willms, p 65. n The measures of sense of belonging and participation are highly reliable at the country level: the reliability coefficients are 0. 99 for both sense of belonging and participation. n 35

Writing questions

Questionnaire design Design the items Issues to be considered for each item n n n What information do I want to get? Is this factual or non-factual? How to ask? What kind of responses do I want to get? /How do I want the respondents to answer my questions (format) How will I code this item? Will I include the coding in the item format? 37

Questionnaire design Measurement Types n n n Nominal Ordinal Interval 38

Questionnaire design Question types n n n To facilitate question writing, it is important to know types of questions. There are two ways of classification of questions: Classification by response format Classification by types of information n Factual. Non-factual (e. g. , attitude) 39

Questionnaire design Question types - classification by response format n n Closed questions Open-ended questions 40

Questionnaire design Open-ended questions n Open response types Explain why you left school? What were your reasons for leaving school? 41

Questionnaire design Open-ended questions - advantages n n People can express their exact opinions and feelings Do not limit the range of possible answers Potentially produce responses which draw attentions to an unanticipated situation or outcome when constructing the questionnaire Useful for testing hypotheses about ideas or awareness 42

Questionnaire design Open-ended questions - Disadvantages Difficult and time consuming to answer (require much effort from respondents) n Difficult and time consuming to analyse n 43

Questionnaire design Closed questions Alternative answers are provided and respondents are asked to choose from a list of provided answers 44

Questionnaire design Types of closed questions Checklists n Two-way questions n Multiple choice questions n Ranking Scales n Scaling questions n 45

Questionnaire design Checklists n Is used to verify the presence or absence of some phenomenon 46

Questionnaire design Checklists Example n Which of these materials did you use? n Which of these activities did you engage in? n Which of these are the steps of conducting the project? 47

Questionnaire design What is a good checklist? Contains all the relevant options n It is helpful to provide the option “other” for respondents to fill in at the end of a checklist. n 48

Questionnaire design Two-way questions n n Measure a dichotomous variables Respondents are asked to choose one from two alternatives: • Yes/No; • Agree/disagree; • For/Against; • Good/Bad; • Like/Dislike; • Approve/Disapprove; 49

Questionnaire design MCQ questions MCQ is useful when there are several possible responses and you want to ensure that the respondents is aware of all the possibilities. n Alternatives in MCQ should be mutually exclusive categories. n 50

Questionnaire design Ranking Scales This format gives you an indication of how a respondent ranks a number of things. n It is useful when there a limited number of things you would like to have ranked. n 51

Questionnaire design Scaling questions n Questions with ratings on a latent scale 52

Questionnaire design Advantages of closed questions Compared to open questions, this type of questions is quicker and easier to answer n More questions can be asked in a given length of time n Can deal with a large number of respondents n Low cost n Make group comparison easy n Avoid interviewer training n 53

Questionnaire design Disadvantages of closed questions Loss of spontaneous responses n May introduce bias by forcing respondents to choose between given alternatives n May irritate respondents n Relatively difficult to design n 54

Questionnaire design Types of questions - Classification by types of information Factual questions n Non-factual questions (Attitudes, stereotypes, beliefs, awareness) n 55

Questionnaire design Factual questions Can be verified n Single variable n Relatively easy to design n 56

Questionnaire design Non-factual questions Difficult to verify n Latent variable n Relatively difficult to design n 57

Questionnaire design Issues to consider when writing the factual questions Do the respondents have the necessary information to answer the question? – Knowledge, memory. n Will the respondents provide the information willingly? – Sensitive issues. n 58

Questionnaire design Question Wordings n Use Simple Words “the catalogue system is too difficult for most readers to master “ vs “I can never find the books I want” (more direct, more appealing) n n Avoid acronyms, abbreviations, jargon and technical terms Avoid ambiguous words or the words with many meanings Have you ever assessed your colleagues’ teaching? n Avoid leading questions You haven’t skipped any lessons in this semester, have you? 59

Questionnaire design Question wordings n Avoid double-barrelled questions Do you buy weekly and monthly magazines / newspapers? n Avoid implicit assumptions 1. 2. n When did you last borrow a video tape? Did your sibling’s decision to leave school influence your decision to leave school? Don’t overtax the respondents’ memory. 60

Questionnaire design Question Wordings Avoid proverbs or well-known sayings n Avoid loaded words (heavily value laden terms) n Do you think union bosses should be allowed so much power? n Attitude statements are good if the respondents recognise the statements which force them to think 61

Questionnaire design Selection of types of questions n n n The number of respondents The amount and types of information needed The characteristics of respondents (knowledge, age, culture, religions) The amount of time you have to process and interpret the data Your knowledge of the issues (the extent to which you can anticipate the range of possible answers). Your methods of data analysis 62

Preparing questionnaire layout Panelling, pre-testing and piloting 63

Questionnaire design Spacing n n Allocating sufficient space for answers Space requirements should be considered for : Open-ended questions ¨ Scaling questions ¨ Coding ¨ 64

Questionnaire design Instructions n n n General Section Question 65

Questionnaire design General Instruction n n n Reason(s) for the questionnaire A statement about anonymity The sample design - to indicate how the respondent was chosen How to return the questionnaire - if it is mailed A contact person What will happen to the results Thanks 66

Questionnaire design Question instructions n n How to answer the questions Make sure that the instructions and the questions correspond 67

Questionnaire design Order of the questions n n Very important There is no correct order 68

Questionnaire design Suggestions 1. 2. 3. 4. 5. 6. 7. Begin with easy and non-threatening questions Do not begin with open-ended questions Arrange questions from general to specific Group questions into sections or topics Use filter questions to ensure that the respondents are answering relevant questions Attitude statements are suggested to be arranged in more or less random order Keep the questionnaire as short as possible 69

Questionnaire design Consistency of questionnaire layout n n Try to use similar format for questions Distinguish different instruction levels 70

Questionnaire design Panelling and Reviewing n n n Relevance of questions to the topic (check against the table of specification) Wording (Instructions, questions) Layout ü Spacing Instructions Order ü Consistency ü ü 71

Questionnaire design Pre-testing n Test questionnaire ¨ Do the respondents understand the questions? ¨ Are there any difficulties? ¨ Are there any sensitive questions? ¨ Is the question order appropriate ¨ Does the researcher understand the respondent's response 72

Questionnaire design Who will be involved in Pre-testing n Very small sample of the population targeted 73

Questionnaire design How to conduct Pre-testing n n Step 1: Brief the respondents about the questionnaire Step 2: Researchers record the respondents’ process of completing the questionnaire: ¨ through observation, video recording or audio recording to find out signs of difficulties or distractions and timing 74

Questionnaire design How to conduct Pre-testing n ü ü ü n Step 3: Debrief the respondents about the questions in the questionnaire Any difficulties? Why? What are the easy questions? Why? Any suggestions for improvement? Step 4: Revise the questions if needed 75

Questionnaire design Pilot n ü ü ü Test the whole process Questionnaire Methods of administration and collecting the questionnaires Response rate/missing data Item analysis Data analysis 76

Questionnaire design Analysis n n Look at frequency of options in each question Too many “uncertain”, “don’t know” responses, too many skipped or omitted items are bad signs in a pilot study. Reliability of scale constructed Decision of removing or replacing items of scales (for measuring latent variables) 77

Questionnaire design Revise and prepare the final version 78