Is Survey Instability Due to Respondents Who Don't Understand Politics or Researchers Who Don't Understand Respondents? (Caltech)

For over 75 years, survey researchers have observed disturbingly large proportions of respondents changing answers when asked the same question again later, even if no material changes have taken place. This “survey instability” is central to substantive debates in many scholarly fields and, more generally, for choosing the data generation process underlying all survey data analysis methods. By building on developments in neuroscience, cognitive psychology, and statistical measurement, we construct an encompassing model of the survey response, narrow competing hypotheses to a single data generation process, and validate it with extensive observational and experimental data. We conclude that (a) human beings have stable preferences, changing only with reason, and error-prone survey responses, (b) most of the variability leading to instability is due to an inherent human characteristic, not bugs in respondent attentiveness or biases in our survey instruments, and (c) the ``causes of the effects’’ of instability can be traced backward to respondents’ time on task, mind wandering, competition among brain networks, and possibly evolutionary optimization favoring randomness.