Who's to Blame for Survey Instability: Respondents with Nonexistent Preferences or Researchers with Flawed Measures?

Neither. For at least 75 years, survey researchers have found that about 25% of respondents give different answers when asked the same question twice (even if no material changes occur and respondents do not remember being asked the first time). This “survey instability” problem casts doubt on a vast research enterprise spanning large areas of academia and industry, is core to many ongoing substantive debates, and requires a resolution for proper survey design and analysis methods. We collect a wide variety of observational and experimental evidence, including 59 unique surveys. We first show that instability barely drops after accounting for both existing explanations, i.e., when respondents have fixed knowledge of their preferences and researchers use high quality, unbiased survey instruments. We trace a large component of survey instability to a different source recognized only in fields with non-survey measurement instruments — intrinsic human stochasticity. We then trace the decision making, cognitive, psychological, and individual characteristic precursors of this stochasticity and reveal their wide ranging implications for understanding respondents, avoiding inattention, designing surveys, and building statistical analysis methods.