Office of Survey Methods Research

Abstract

Christine Rho and Roberta L. Sangster (2003) "How Much Can you Trust the Answers You Get Using Cognitive Interviews?"

Cognitive interviews has been widely used and studied as a research tool to reduce sources of response error and improve survey questions. Cognitive interviews are based on the assumption that respondents are reasonably reliable informants about how they comprehend and answer questions. We recently conducted a study using a greater than average number of cognitive interviews to test question comprehension (n=40) in a housing survey. The study was conducted in two parts: 1) survey completion and 2) retrospective probing. In the first part, respondents answered questions from the housing survey with the option of indicating when they did not know the answer or understand the questions. In the second part, respondents were interviewed and probed about the clarity of questions, their subjective experience of answering the survey questions and how they arrived at the answers. This study yielded both quantitative and qualitative data, which allowed a comparison of actual survey responses and information respondents reported about clarity and comprehensibility of questions. This comparison showed that while respondents would say a question was clear and that they understood it, their actual responses to survey indicated a lack of comprehension. This study will show data from the housing study that indicate this discrepancy and will discuss the limits of relying on respondents as informants of their thought processes.

Last Modified Date: July 19, 2008

Recommend this page using: