The debate about open-end versus close-end questions in surveys research is an old one. Few market researchers today argue for one or the other exclusively. There is general agreement in the field of market research that there is room for both - and reason for both. Experts do agree that certain Alistair Campbell, Professor of Psychology at James Cook University, argues that "each methodology has a place, and that none should claim to be 'better' than all the rest."
Frequently, the methodology chosen for a market research project is surveys research. Survey construction is increasingly moving to online and mobile venues, making the construction of surveys a much easier task than ever before. A market researcher who uses surveys research software to construct a survey will find enhanced flexibility, and will be faced with numerous options for question item configuration. A very basic decision will be whether to use open-end or close-end question items.
Scientific Bias Toward Quantitative Research
Most surveys research software uses close-end question items. This predilection for close-end questions is partly due to the halo-effect quantitative research has in popular and scientific worlds. Dr. Campbell suggests that the bias toward a particular type of evidence-based practice,
"reflects the very limited, and very limiting, effect of the current conceptualization of what constitutes 'good' research…there is a range of sources of information that can be considered evidentiary. But there continues to be a dominant belief in a hierarchy of evidence, the 'gold standard' or 'royal road' to scientific 'truths…There is no 'royal road' to the production of knowledge through an empirical method. There are as many byways, detours, and woodland tracks as there are freeways, highways, and expressways between the questions we want to ask and the information that we use to find answers."
Source: Australian and New Zealand Journal of Family Therapy, Volume 24 Number 1 2003 pp. 51-52
When Is an Open-End Question Preferred?
With the legitimacy of the qualitative survey methods established, what is left is to determine when to use a qualitative approach and how to configure the survey when a qualitative method will be employed. The key indicators that open-end questions will provide higher quality data than close-end questions are considered to be:
- Too many categories would need to be listed
- Unforeseen categories are likely to emerge or be suggested
- Spontaneous and uninfluenced responses are desired
- Under reporting is likely to occur with close-end questions, particularly when questions deal with sensitive issues that require self-disclosure
- Rapport-building is indicated, for instance following an extended series of close-end questions that may frustrate the respondent because of limited opportunity to express themselves more fully or explain responses that could be unclear
- When parameters and dynamics of an issue are being explored or pilot-tested prior to diverting to only close-end questions at another time
How Can Response-Order Effect Be Avoided in Surveys Research?
One major benefit of open-end questions is that they simply avoid the common response-order effect that a series of close-end questions can elicit from respondents. A drawback of close-end questions is that they are constructed a priori and are based on assumptions made by the market researcher. While expert opinion is an important component of survey construction, it may result in a set of response alternatives that do not provide meaningful wording or content to respondents. Even when the question and answer choice wording is clear to a respondent, close-end questions often establish a condition in which the respondent must deliberate between response choices that are too similar or too different. In other words, a 7-point Likert scale may require that the respondent make discriminations that seem too fine.
Responding to close-end questions and open-end questions requires different types of mental processing. To accurately express their thoughts using close-end questions requires careful and systematic thinking. This process is quite different from the type of thinking that open-end questions require, which is decidedly more extemporaneous and generally doesn't require making fine discriminations between alternatives. If asked, respondents may say that answering close-end questions is faster and easier than answering open-end questions. And they mean that - literally. That is precisely why the phenomenon known as the response-order effect exists. Respondents, faced with the difficult or intense mental processes of responding to close-end questions may simply take the expedient route and stick with a point level or rating for most, if not all, of the close-end questions. Further, when the pattern of responses becomes salient to the respondent, they may switch to another point level or rating just to "change it up."
Does Advance Notice of Survey Follow-Up Help?
In order to guard against the race to the end of the survey that frequently overcomes even the most earnest of respondents, market researchers use various strategies to ensure that the answer intent of respondents is represented. A strategy used by ethnographic researchers is the member check. Using a member check process, the market researcher asks the respondent if the summarization of their responses correctly reflects the respondents' communicative intent. Member checks are frequently done with interviews and it can be adapted to survey responses as well. To establish the ability to fall back to a member check, surveys will often close by asking respondents if they can contact them for further information. The problem here, of course, is that a respondent who recognizes that they have haphazardly responded is likely to decline a member check. This brings the reader full circle - there are very good reasons for including open-end questions in a survey.
Next up, we will take a look at options for coding and analyzing open-end responses in surveys and interviews.