Correcting Measurement Error Bias in Conjoint Survey Experiments (Harvard Experiments Working Group)

Conjoint survey designs are spreading across the social sciences due to their unusual capacity to estimate many causal effects from a single randomized experiment. Unfortunately, by their ability to mirror complicated real-world choices, these designs often generate substantial measurement error and thus bias. We first present a simplified statistical framework for conjoint designs that also enables researchers to study a wider array of substantive questions. We then replicate both the data collection and analysis from eight prominent conjoint studies, all of which closely reproduce published results, and show that a large amount of observed variation in answers to conjoint questions is effectively random noise. We then discover a common empirical pattern in how measurement error appears in conjoint studies and, with it, we introduce an easy-to-use statistical method to correct the bias.

Based on joint work available at GaryKing.org/conjointE by Katherine Clayton, Yusaku Horiuchi, Gary King, Aaron Kaufman, and Mayya Komisarchik.