Previous: Why will anchoring vignettes

Up: Survey Instrumentation

Next: Do I need one

Is there a simpler way of asking questions so we can avoid any statistical analysis?

Direct measurement, that is without statistical analysis, is preferable when possible. We have tried a variety of simpler strategies in a diverse array of national surveys, but none seem to do remotely as well as anchoring vignettes. For example, we tried asking which of a set of vignettes the respondent is most like, but we found that respondents had a difficult time remembering them all at the same time. Another possibility is to ask if the respondent has a higher or lower level of health/efficacy/etc than the first vignette, and then the second, etc. This is better, but it also does not fully correct for DIF in our experience, and in any event would require assuming that DIF is fully corrected rather than allowing empirical verification.

Another possibility of course is the usual strategy of trying to make the self-assessment question even more concrete. This is always a good strategy, but no matter how obvious and unambiguous a survey question appears to the researcher, respondents always seem to surprise us in their ability to interpret questions in different ways than intended. This surprise of course is not revealed unless researchers debrief respondents in post-interview debriefing sessions. Researchers who are sure that their survey questions have no DIF need to verify this at first with these interviews, but ideally also with at least pretesting with anchoring vignettes.