We recommend asking the self-assessment early in the survey and
the vignettes some time later.
The vignettes should not be ordered in the survey
according to your understanding of their actual value. One problem
with this approach is that respondents will tend to try to make
their answers consistent over the set of responses, or may use
simple heuristics such as placing one vignette in each response
category. Both of these outcomes would compromise the requirement of
response consistency between vignette ratings and self-assessments.
Further, respondents may have different abilities to remember all
the previous vignettes, and those who pick the wrong or an unusual
value for the first vignette may feel locked in for the rest. The
result will be essentially constant responses for the rest that do
not discriminate well and so provide little information.
We find that vignettes are best presented to the respondent in
randomized or mixed order. In addition, if you have more than one
set of vignettes, it is helpful to shuffle the two sets together.
Since a separate question follows each vignette, this does not cause
respondents to have any additional trouble in understanding the
survey instrument.
Why not ask the self-assessment question after the vignettes?
This would lead to an undesirable priming effect, and a different
one depending on the order in which the vignettes were presented.
Although we might like respondents to read, internalize, and
remember the set of vignettes prior to being asked the
self-assessment question, this is infeasible for most respondents.
