Previous: What empirical evidence do

Up: Survey Instrumentation

Next: Should I ask vignettes

How many anchoring vignettes should I ask for each concept I want to measure?

Here are the considerations:

1. Chopit, our parametric statistical model, requires, at a minimum, only one vignette (with two or more response categories). (The logic as to why this is sufficient is the same as that in using logistic regression with a dichotomous dependent variable to estimate a continuous probability.) Our nonparametric method can also work with as few as one vignette. In practice, however, since it is sufficiently difficult to write survey questions, we recommend multiple vignettes. This follows the same advice that survey researchers have given in measuring any concept. The use of multiple vignettes would also be required for certain extensions to the standard chopit model such as the addition of random effects in the threshold equations.

2. More important than how many vignettes are asked is designing vignettes that provide discriminatory power. Thus, the best anchoring vignettes are those which are equally spaced through the distribution of self-assessment answers. (For example, asking how mobile a person is who can run 500 miles in a day is obviously of no use in assessing mobility.) The statistical procedures are most powerful when a vignette is asked near to (and preferably on each side of) each respondent's self-assessment answer. The implication is that the more diverse your respondents in terms of their actual levels and their threshold variations, the more vignettes should be asked.

3. In our research with WHO, we have usually used 5-7, or sometimes as many as 12, vignettes, but our applications involve a large fraction of the world's population. Surveys of less diverse populations, such as within a single culture, may be possible to do with many fewer vignettes. When possible, we recommend asking more vignettes during the pretest, and then studying how much information is lost by examining the stability of the $\gamma$ parameters when dropping subsets of vignettes. Monte Carlo experiments can also be helpful.

4. We're still doing research on the subject but our current (optimistic) guess is that four vignettes asked of 1/4 of the respondents each may be sufficient when you know where the respondents self-assessments roughly are, and you have good covariates to predict the thresholds. (If so, this would add the equivalent of only one item in terms of time on the survey per self-assessment question and so would not be very expensive to administer.)

5. Roughly speaking, the amount of information the data provide about the actual levels increases at most by $2J+1$ in the number of vignettes $J$ . This maximum speed is achieved when answers to the vignettes are most equally spaced through the distribution of self-assessment answers.

6. If you are interested in having higher resolution in measurement at some point in the scale (such as the bottom), then it pays to include more vignettes in this region.