We address two long-standing survey research problems: measuring complicated concepts, such as political freedom or efficacy, that researchers define best with reference to examples; and what to do when respondents interpret identical questions in different ways. Scholars have long addressed these problems with approaches to reduce incomparability, such as writing more concrete questions -- with uneven success. Our alternative is to measure directly response category incomparability and to correct for it. We measure incomparability via respondents' assessments, on the same scale as the self-assessments to be corrected, of hypothetical individuals described in short vignettes. Since actual levels of the vignettes are invariant over respondents, variability in vignette answers reveals incomparability. Our corrections require either simple recodes or a statistical model designed to save survey administration costs. With analysis, simulations, and cross-national surveys, we show how response incomparability can drastically mislead survey researchers and how our approach can fix them.
You may also be interested in the Anchoring Vignettes web site which includes information about conferences on the subject, a FAQ, software, example vignettes, and other materials. (Website: Anchoring Vignettes) Also see related research.
*This article originally appeared with printing errors in the American Political Science Review (Vol. 97, No. 4, December, 2003, 567-584). The citation above is for the full corrected article, as originally intended.
This paper has also been reprinted in