Correcting Measurement Error Bias in Conjoint Survey Experiments

Conjoint survey designs are spreading across the social sciences due to their unusual capacity to estimate many causal effects from a single randomized experiment. Unfortunately, by their ability to mirror complicated real-world choices, these designs often generate substantial measurement error and thus bias. We first present a simplified statistical framework for conjoint designs that also enables researchers to study a wider array of substantive questions. We then replicate both the data collection and analysis from eight prominent conjoint studies, all of which closely reproduce published results, and show that a large amount of observed variation in answers to conjoint questions is effectively random noise. We then discover a common empirical pattern in how measurement error appears in conjoint studies and, with it, we introduce an easy-to-use statistical method to correct the bias.

Based on joint work available at GaryKing.org/conjointE by Katherine Clayton, Yusaku Horiuchi, Gary King, Aaron Kaufman, and Mayya Komisarchik.