What has to go wrong for anchoring vignette corrections to bias my results?

Here are several ways to think about this issue:

1. First, for simplicity and since statistical methods can deal with it in fairly straightforward ways, imagine that random perceptual and measurement error were nonexistent. Then what needs to happen for all the problems to be fixed is that respondents differ in their interpretation of the vignettes only due to DIF (differential item functioning, or interpersonal incomparability), whereas the responses to the self-assessments must differ due to DIF and the actual values (A) on the concept of interest. In addition, since it is the same person answering both questions, we assume that the nature of the DIF is the same for both. The goal is to estimate A. Self-assessments are misleading by themselves because they give DIFA , and vignettes can correct since they give us a measure of DIF, and so the correction is DIFADIFA . But that's when everything works perfectly.

   Now, the fact that the vignettes are subject to DIF and are interpreted in different ways in different cultures by different people is not a problem in and of itself. In fact, the technique relies on vignettes having DIF too. What would be a problem is if the nature of the DIF differs for the vignettes and the self-assessments. In that case, suppose we have DIFv for the vignettes but DIFs for the self-assessments, and so our correction would be DIFsADIFv , which is the same as DIFsDIFvA . So the ultimate question is not whether the vignettes have DIF, but rather whether DIFsDIFv or DIFs is closer to zero. For precisely the reason that it is the same person with the same biases answering both questions, DIFsDIFv will normally be closer to zero than DIFs. This is the reason why we find that this technique usually is an improvement over self-assessments alone and why only in rare situations does the correction make things worse.

2. The basic assumption is that a respondent uses the same thresholds to translate their perceptions into a categorical response for their self-assessment as for the vignette assessment. An exception would be Rodney (``I never get any respect'') Dangerfield. If he and others with inferiority complexes rank themselves lower than (even hypothetical) vignettes solely because of this complex, and if the pattern of under-ranking themselves is related to other variables of interest but not controlled for in our analysis, then our approach would be biased. The opposite bias may also be possible, whereby individuals rate themselves more favorably than they do hypotheticals (e.g. because of optimism or wishful thinking). We think, however, that these are extreme situations and that the biases would have to be unrealistically large before an unadjusted approach would do better than our adjustment, even with some degree of bias.

3. See the penultimate section of King, Murray, Salomon, and Tandon for additional disadvantages of the model.