Joe Finnerty

QUESTION-BASED DATA COLLECTION METHODS

Contents:

  1. question based data-collection methods – general introduction

  2. main types of question-based methods

  3. postal surveys

  4. response rates

  5. some advantages of the face to face/personal interview

  6. questionnaire construction – a rough guide

  7. other issues

SECTION 1. QUESTION-BASED DATA COLLECTION METHODS

‘questions and answers’

In these kinds of research, the researcher collects data by asking people who have experienced certain phenomena to reconstruct their experiences, describe their behaviour and express their preferences/attitudes. S/he may also ask about their knowledge of events.

 

Keeping in mind budget restrictions (time, money, personnel), the researcher must choose the data collection method that will elicit the most complete responses from a sample of individuals / organisations relevant to the research topic / objectives / questions.

 The responses obtained are the data on which answers to the research questions are based.

NB: distinguish between

your general research questions and objectives, which you will have spelt out in your research proposal, and

 the specific questions you put to respondents in  postal questionnaires and personal interviews.

SECTION 2. MAIN TYPES OF QUESTION-BASED DATA COLLECTION METHOD

readings

Fowler (2002), Survey Research Methods 3rd edition ch. 5 – ‘Designing Questions to be Good Measures’;

Blaikie Designing Social Research pp. 232-5;

 

1.     Quantitative:

Questionnaire (self-administered), often postal.

Structured personal interview

2.     Qualitative

Focused personal interview

 

‘Questionnaires’ are used in (non-face to face) postal or telephone surveys and in (face-to-face) structured personal interviews. The looser set of question headings used in the qualitative interviews are often called ‘interview schedules’.

 ‘Multi-method approach’ = using more than one method e.g. combination of field research and a survey. Different kinds of questions i.e. questions that produce both quantitative and qualitative forms of data,  may be used in the same interview or quetionnaire.

 

SECTION 3. POSTAL SURVEYS

 

= an ‘impersonal’ method of data collection: whether this is desirable or otherwise depends on the kind of research strategy and research topic adopted

this involves a self-administered questionnaire in which access and return is through the post.

 

advantages of the postal survey

  1. low cost per person questioned you don’t need a trained staff of interviewers; what is required is planning, sampling, photocopying, posting, and providing SAEs. Data processing and analysis is usually cheaper. Costs relating to access are also much lower, especially where the population of interest is geographically dispersed.

  2. reduction in biasing error           since the postal questionnaire is impersonal (there’s no face-to-face interaction involved), there are no biasing errors that might result from the personal characteristics of the interviewers and the variability in their skills

  3. greater anonymity         the absence of an interviewer also provides greater anonymity for the respondent. This is particularly important when the research deals with sensitive topics e.g. sexual behaviour.

  4. greater time to respond by contrast with personal interviews, postal questionnaires allow greater time to consider a question or to consult documents.

 

Disadvantages of the postal survey

1.      requires simple questions                       a postal questionnaire can be used to gather data only when the questions are straightforward enough to be understood and answered on the basis of printed instructions and definitions. In particular, the questions tend to be ‘closed’ and quantitative.

2.      no opportunity for probing        there is no opportunity to probe beyond the given answer, to clarify ambiguous answers, perhaps to appraise the non-verbal behaviour of respondents

3.      no control over who fills out the questionnaire/ no control over the interview situation                  there is no way of knowing who has filled out the questionnaire: someone other than the intended respondent may have done so.

4. low response rate                 it is often difficult to obtain a satisfactory response rate ( = the % in the sample who return completed questionnaires). Typical response rates for postal questionnaires without follow-up are 20% - 40%. The problem here is there is no way of knowing if non-respondents are different from respondents in ways that affect the findings.

SECTION 4. FACTORS AFFECTING THE RESPONSE RATE / STRATEGIES TO INCREASE RESPONSE RATES

 note:   

in the discussion below of personal interviews and postal questionnaires, it’s by and large assumed that the potential respondents are individuals to be interviewed or mailed in their own homes.

 

sponsorship      providing information on who has commissioned the questionnaire often motivates respondents to complete and return them. (This is usually done in the ‘accompanying / covering letter’.) This information may convince respondents  of

1.      the legitimacy of the research

2.       the usefulness of the research.

 

With officially sponsored research,

3.      the perceived sanctions of failure to reply may also be a factor – these may be real legal sanctions, or they may be wrongly anticipated on the part of respondents. It is of course undesirable,  and in general unethical, to suggest sanctions for failure to respond

 

inducements to respond           

appealing to respondents goodwill e.g. a student research project might mention the importance for their marks of a good response rate!

offering a reward e.g. a recent research project into drug use amongst teenagers in Northern England offered a £10 record voucher. However, this kind of inducement may backfire – if the ‘reward’ is in the form of a voucher, it is invevitably going to be small, and respondents may be indignant that researchers place such a low value on their time and effort.

The offer of entry into a draw overcomes this problem, but perhaps at the loss of any likely  ‘payoff’ for participation.

Mark et research is more likely to offer inducements of this kind.

Appeal to norms of common good / public spiritedness  e.g. ‘making things better’

 

Questionnaire design and methods of mailing this included the font used, colour, length of questionnaire, legibility of questionnaire. Even the colour of the envelope may be important – the use of brown paper envelopes may reduce the response rate!

 

Covering letter and Pre-publicity. The letter must succeed in persuading the reader to fill out the questionnaire and post it back. Thus, it should identify the sponsor, explain the purpose of the research, say why its important, and assure them of the confidentiality of the response (where appropriate).

 

Sometimes a semi-formal (i.e. more personalised) style of letter is used to encourage responses.

 

Ads in newspapers, publicity in newspapers, radio and television interviews, may also encourage potential respondents to participate in the research.

 

Pre-paid addressed envelope

 It’s unreasonable to expect the respondent not only to spend time and effort to fill out the postal questionnaire but also to buy a stamp and write out the address on the envelope! 

 

Timing of mailing / interviewing

Holiday periods are bad times to send out a postal questionnaire. The time of the day that a personal interview is arranged for is important, depending on who in the household it’s intended to question.

 

Follow-up

the most common strategy for postal questionnaires  is to first send a reminder postcard to those who haven’t replied. Then perhaps a reminder, along with a substitute questionnaire.

 

However, sending a reminder may raise the question of the anonymity of the responses – as an alternative, the reminder could take the form of an announcement in the newspaper or other media.

 

SECTION 5. SOME ADVANTAGES OF THE PERSONAL / FACE-TO-FACE INTERVIEW ( NOTE THIS ENCOMPASSES BOTH THE STRUCTURED AND THE FOCUSED OR IN-DEPTH VARIETIES).

1.       Flexibility in the questioning process

Depending on the research problem and research strategy, the interviews can vary from highly structured [in the DEDUCTIVE RESEARCH STRATEGY] to highly unstructured [ABDUCTIVE RESEARCH STRATEGY]. For focused and nondirective interviews, the interviewer can clarify questions for the interviewee, probe for specific issues raised in the course of the interview etc.

2.       Control of the interview situation

The interview gives the researcher much greater ‘control’ of the interview situation.

This isn’t as authoritarian as it sounds! -

·        Sequence of questions:   it may be desirable to ensure that questions are asked in a particular sequence. If in a survey about facilities (childcare, community centre, credit union etc.) in an area e.g. if two research questions relate to

1.     knowledge / awareness of the existence of the facility (you want to find out how aware a respondent is about the existence of facilities in their area, perhaps to gauge the success or otherwise of a publicity campaign)

2.     their use of and attitude towards the facility

-         you need to ask them question 1 before presenting them with a list of such facilities and asking them to comment on if or under what circumstances they would use the facility

·        getting the right respondent     ensuring that the respondent him/herself has answered the questions, rather than a relative – example of MS survey.

·        Timing       it may be important to fix a particular time when the data was collected: this can allows more accurate interpretation of the research results, especially when an event occurring around the time of the interview could have influenced the respondent’s answers.

3.       High response rate.

Response rates to personal interviews are usually higher than for postal questionnaires. Respondents who mightn’t bother filling out and returning a postal questionnaire will often cooperate with a request for a personal interview – there may be less effort involved on their part, e.g. if they have reading or writing difficulties.

 

SECTION 6. QUESTIONNAIRE CONSTRUCTION (structured and semi-structured)

The central component of the data collection instrument in the postal questionnaire or the structured personal interview, is the question.

The answers to the questions asked = the data on which answers to the research questions and objectives are built. The questions are the variables, which can be correlated to look for associations between variables.

 4 aspects to questions:

·        Content

·        Structure

·        Format

·        Sequence

 [1]   CONTENT OF QUESTIONS

Different classifications possible

-         Objective/factual & subjective (‘objective’ = questions that generate demographic economic data on gender, age, nos. in household, occupation etc. & questions relating to possessions and behaviour e.g. car ownership, mode of transport; ‘subjective’ = attitudes & opinions & beliefs e.g. ‘do you favour greater investment in public transport?’)

-         Behaviour & knowledge & beliefs & attitudes

 [2]  STRUCTURE OF QUESTIONS

Closed-Ended & Open-Ended Questions

Contingency & Filter Questions

 

Closed- and open-ended questions

Closed ended: respondents are offered a set of answers and asked to choose the one that most closely fits their situation/ views/etc. The set of categories will depend on the research questions.

Quick & easy to answer

Easy to analyse

May introduce bias, by limiting the replies given or by offering alternatives the respondent wouldn’t otherwise have thought of

 Open-ended: these questions aren’t followed by a set menu of choices. Rather the respondent talks or writes in their own words.

Greater flexibility & a higher response rate may be introduced by letting each individual option cover a range (of years, income etc)

 

Please indicate your age/ What age are you?

[   ]16 – 18                     [   ]19-24                       [   ] 25-35

 

[   ]36-45                       [   ]46-64                       [   ]64 or over

 

(spot the deliberate error!)

 

Don’t have overlapping possibilities in your response categories

Clearly indicate if respondents are allowed one or many responses i.e. ‘one only’ or ‘all that apply’

 

Contingency & filter questions

Some questions may be irrelevant for a particular respondent. E.g. if the respondent has no children, a question on age or level of schooling of children doesn’t apply. The questionnaire may cover a number of types of respondents, so rather than doing out as many separate questionnaires as there are respondent types, you use one flexible, ‘signposted’ questionnaire.

filter questions are the signposts for detours

The contingency questions are the optional detours taken. Asking/answering them is contingent on the response to the preceding filter question.

 

Reminder: any given questionnaire (= the data collection instrument) may use a combination of question types (= mix of closed and open ended questions).

 

 [3]  FORMAT

This relates to the structuring of the response categories of closed-ended questions.

The general format is to present all possible answers and have respondents choose between the different categories.

What if the the respondents response doesn’t fit into any of the answers supplied? = importance of the OTHER option + a ‘please specify’ instruction.

 

For ease of data analysis, use ‘tick boxes’ or ‘circle numbers’ rather than ‘tick blanks’. Sometimes blanks can be confusing, both for the respondent and the data analyst.

   

·        Note that the response categories offered should relate to the type of data being sought. Take the following example of a question to obtain data about respondent age:

 

Please indicate your age/ What age are you?

[   ]16 – 18                     [   ]19-24                       [   ] 25-34

[   ]35-44                       [   ]45-64                       [   ]65 or over

 

If the research objectives include description and exploration of the experiences of older people, then ’65 or over’ as above, may be too broad a category; instead you may want to merge some of the other response categories (as being data irrelevant to the research questions):

[   ]16 – 24                     [   ] 25-44

 

[   ]45-64                       [   ]65-74              [   ]75 or over

 

 

Ranking

A ranking (or ordinal) scale may be used to structure responses to a question

e.g.

Q. People found in possession of cannabis for personal use should not receive jail sentences. Please indicate whether you:

  1. Agree strongly

  2. Agree

  3. Neither agree nor disagree

  4. Disagree

  5. Disagree strongly

 

Other common response categories are:

  1. Too little

  2. About right

  3. Too much

 

    1. More

    2. Same

    3. Less

 

[4]   SEQUENCE OF QUESTIONS

two general patterns:

A. FUNNEL (going from the general question to the specific question).

  1. what are some of the most important problems facing Ireland today? (list all mentioned)

  2. Of all the problems mentioned, which do you think is the most important?

  3. Where have you obtained most of your information about this problem?

  4. What newspapers do you read?

Each successive question is related to the previous question and has a progressively narrower scope.

 allows interviewers to avoid imposing a frame of reference before obtaining a response. E.g. if a research objective was to explore whether there was a link between respondents’ outlook on social & economic problems and the newspapers they read, the sequence might be as follows:

Requires respondents to be motivated to respond (since you’re starting off with ‘opinion’ questions

 

B. INVERTED FUNNEL

goes from the narrower question (eg aimed at establishing specific facts) to the more general question (eg aimed at getting the respondents overall judgement about these facts)

used where the survey topic isn’t of great interest to respondents or where the researcher isn’t very knowledgeable about the situation/context

 

This sequence structure could apply to the entire questionnaire, or to specific topics / sections within the questionnaire.

 

In general, it’s advisable to start off with ‘objective’ descriptive and neutral/non-threatening questions and then move on to more ‘subjective’ and controversial questions.

   

SECTION 7. OTHER ISSUES

Clarity of questions asked

The answers to the questions asked = the data on which answers to the research questions and objectives are built.

 

A fundamental requirement of the question is that it’s meaning should be clear to both the researcher and the respondent, at the data analysis stage and when the findings are being interpreted. If the questions - or any particular question - asked is unclear, then the validity and reliability of the research is suspect.

 

Example of an unclear question: Does it seem possible or does it seem impossible to you that the Nazi extermination of the Jews never happened?

 

This was a question asked by the Roper polling organisation in the United States . On the basis of responses obtained, Roper concluded that one in five Americans thought the Holocaust never happened! (A subsequent survey carried out by Gallup found that about 3% of Americans doubted the Holocaust happened).

   

Access to interviewees and Ethics of research

note the frequent use of a covering letter asking for access, giving appropriate assurances of confidentiality and/or anonymity (many methods textbooks will supply a specimen letter)

letting respondents see a draft of the report/interview

furnishing respondents / key informants with a final Report