Any or all of these items will improve the efficacy of the approach.
Use highly concrete vignettes. The technique makes the most
difference for concepts where self-assessment questions are
unavoidably vague but vignettes can be concrete.
Design vignettes to be roughly equally spaced through the
distribution of self-assessment answers. Those too close to each
other will provide repetitive information; those too extreme, will
provide little or no information.
Carefully pretest the survey instrument, analyze the data with
our methods and diagnostics to verify that respondents believe the
order of the vignettes is as you intended, and remove vignettes that
statistical analyses with chopit indicate are too variable
incorrectly ordered or provide little information.
Make sure the vignettes are tapping only a single unidimensional
concept. We find that the process of writing anchoring vignettes
often reveals new concepts or dimensions better than writing
self-assessment questions alone. Discovering new dimensions makes
it possible to narrow the current concept, hence making it more
concrete, and possibly to add another self-assessment question and
corresponding vignettes for the new dimension.
Ask vignettes for every self-assessment question if possible,
although our model allows vignettes to be asked that correspond to
only one of the self-assessment questions, and to still use the
information in the others if all the self-assessment questions are
measuring the same concept.
Include variables in the survey that will help predict the
threshold values. The better the information content in these
variables, the better problems with DIF can be detected and
corrected with chopit.
If you are unable to find variables that can predict threshold
variation, then the nonparametric version of the model can be used
to correct DIF, but it only works for those respondents who have
both self-assessments and vignette answers.
Use chopit with a random effect and our conditional predictive
method. This tends to work considerably better than unconditional
predictions, especially when good variables to predict the
thresholds are not available.
If you ask all the respondents both the self-assessment and all
the vignette questions, you can use chopit with a random effect and
then condition on both responses, which can improve the efficacy of
the approach even further.
Follow all the usual rules and advice given by survey
researchers over the last half century. That is, be careful of
question wording, question order, accurate translation of the
meaning of different items, sampling design, interview length,
social background of the interviewer and respondent, etc.
