When do anchoring vignettes make the most difference?

Any or all of these items will improve the efficacy of the approach.

1. Use highly concrete vignettes. The technique makes the most difference for concepts where self-assessment questions are unavoidably vague but vignettes can be concrete.
2. Design vignettes to be roughly equally spaced through the distribution of self-assessment answers. Those too close to each other will provide repetitive information; those too extreme, will provide little or no information.
3. Carefully pretest the survey instrument, analyze the data with our methods and diagnostics to verify that respondents believe the order of the vignettes is as you intended, and remove vignettes that statistical analyses with chopit indicate are too variable incorrectly ordered or provide little information.
4. Make sure the vignettes are tapping only a single unidimensional concept. We find that the process of writing anchoring vignettes often reveals new concepts or dimensions better than writing self-assessment questions alone. Discovering new dimensions makes it possible to narrow the current concept, hence making it more concrete, and possibly to add another self-assessment question and corresponding vignettes for the new dimension.
5. Ask vignettes for every self-assessment question if possible, although our model allows vignettes to be asked that correspond to only one of the self-assessment questions, and to still use the information in the others if all the self-assessment questions are measuring the same concept.
6. Include variables in the survey that will help predict the threshold values. The better the information content in these variables, the better problems with DIF can be detected and corrected with chopit.
7. If you are unable to find variables that can predict threshold variation, then the nonparametric version of the model can be used to correct DIF, but it only works for those respondents who have both self-assessments and vignette answers.
8. Use chopit with a random effect and our conditional predictive method. This tends to work considerably better than unconditional predictions, especially when good variables to predict the thresholds are not available.
9. If you ask all the respondents both the self-assessment and all the vignette questions, you can use chopit with a random effect and then condition on both responses, which can improve the efficacy of the approach even further.
10. Follow all the usual rules and advice given by survey researchers over the last half century. That is, be careful of question wording, question order, accurate translation of the meaning of different items, sampling design, interview length, social background of the interviewer and respondent, etc.