%0 Journal Article %J American Journal of Political Science %D In Press %T The Balance-Sample Size Frontier in Matching Methods for Causal Inference %A Gary King %A Christopher Lucas %A Richard Nielsen %X
We propose a simplified approach to matching for causal inference that simultaneously optimizes balance (similarity between the treated and control groups) and matched sample size. Existing approaches either fix the matched sample size and maximize balance or fix balance and maximize sample size, leaving analysts to settle for suboptimal solutions or attempt manual optimization by iteratively tweaking their matching method and rechecking balance. To jointly maximize balance and sample size, we introduce the matching frontier, the set of matching solutions with maximum possible balance for each sample size. Rather than iterating, researchers can choose matching solutions from the frontier for analysis in one step. We derive fast algorithms that calculate the matching frontier for several commonly used balance metrics. We demonstrate with analyses of the effect of sex on judging and job training programs that show how the methods we introduce can extract new knowledge from existing data sets.
Easy to use, open source, software is available here to implement all methods in the paper.%B American Journal of Political Science %8 2016 %G eng %0 Journal Article %J American Journal of Political Science %D In Press %T Computer-Assisted Keyword and Document Set Discovery from Unstructured Text %A Gary King %A Patrick Lam %A Margaret Roberts %X
The (unheralded) first step in many applications of automated text analysis involves selecting keywords to choose documents from a large text corpus for further study. Although all substantive results depend on this choice, researchers usually pick keywords in ad hoc ways that are far from optimal and usually biased. Paradoxically, this often means that the validity of the most sophisticated text analysis methods depend in practice on the inadequate keyword counting or matching methods they are designed to replace. Improved methods of keyword selection would also be valuable in many other areas, such as following conversations that rapidly innovate language to evade authorities, seek political advantage, or express creativity; generic web searching; eDiscovery; look-alike modeling; intelligence analysis; and sentiment and topic analysis. We develop a computer-assisted (as opposed to fully automated) statistical approach that suggests keywords from available text without needing structured data as inputs. This framing poses the statistical problem in a new way, which leads to a widely applicable algorithm. Our specific approach is based on training classifiers, extracting information from (rather than correcting) their mistakes, and summarizing results with Boolean search strings. We illustrate how the technique works with analyses of English texts about the Boston Marathon Bombings, Chinese social media posts designed to evade censorship, among others.%B American Journal of Political Science %8 2017 %G eng %0 Journal Article %J IEEE Transactions on Visualization and Computer Graphics %D Forthcoming %T booc.io: An Education System with Hierarchical Concept Maps %A Michail Schwab %A Hendrik Strobelt %A James Tompkin %A Colin Fredericks %A Huff, Connor %A Dana Higgins %A Anton Strezhnev %A Mayya Komisarchik %A Gary King %A Pfister, Hanspeter %X
Information hierarchies are difficult to express when real-world space or time constraints force traversing the hierarchy in linear presentations, such as in educational books and classroom courses. We present booc.io, which allows linear and non-linear presentation and navigation of educational concepts and material. To support a breadth of material for each concept, booc.io is Web based, which allows adding material such as lecture slides, book chapters, videos, and LTIs. A visual interface assists the creation of the needed hierarchical structures. The goals of our system were formed in expert interviews, and we explain how our design meets these goals. We adapt a real-world course into booc.io, and perform introductory qualitative evaluation with students.%B IEEE Transactions on Visualization and Computer Graphics %G eng %0 Journal Article %J American Political Science Review %D Forthcoming %T How the Chinese Government Fabricates Social Media Posts for Strategic Distraction, not Engaged Argument %A Gary King %A Jennifer Pan %A Margaret E. Roberts %X
The Chinese government has long been suspected of hiring as many as 2,000,000 people to surreptitiously insert huge numbers of pseudonymous and other deceptive writings into the stream of real social media posts, as if they were the genuine opinions of ordinary people. Many academics, and most journalists and activists, claim that these so-called "50c party" posts vociferously argue for the government's side in political and policy debates. As we show, this is also true of the vast majority of posts openly accused on social media of being 50c. Yet, almost no systematic empirical evidence exists for this claim, or, more importantly, for the Chinese regime's strategic objective in pursuing this activity. In the first large scale empirical analysis of this operation, we show how to identify the secretive authors of these posts, the posts written by them, and their content. We estimate that the government fabricates and posts about 448 million social media comments a year. In contrast to prior claims, we show that the Chinese regime's strategy is to avoid arguing with skeptics of the party and the government, and to not even discuss controversial issues. We infer that the goal of this massive secretive operation is instead to regularly distract the public and change the subject, as most of the these posts involve cheerleading for China, the revolutionary history of the Communist Party, or other symbols of the regime. We discuss how these results fit with what is known about the Chinese censorship program, and suggest how they may change our broader theoretical understanding of "common knowledge" and information control in authoritarian regimes.
This paper follows up on our articles in Science, “Reverse-Engineering Censorship In China: Randomized Experimentation And Participant Observation”, and the American Political Science Review, “How Censorship In China Allows Government Criticism But Silences Collective Expression”.%B American Political Science Review %8 2017 %G eng %0 Journal Article %J Trials %D 2016 %T Effectiveness of the WHO Safe Childbirth Checklist Program in Reducing Severe Maternal, Fetal, and Newborn Harm: Study Protocol for a Matched-Pair, Cluster Randomized Controlled Trial in Uttar Pradesh, India %A Semrau, Katherine %A Lisa R. Hirschhorn %A Bhala Kodkany %A Jonathan Spector %A Danielle E. Tuller %A Gary King %A Stuart Lisptiz %A Narender Sharma %A Vinay P. Singh %A Bharath Kumar %A Neelam Dhingra-Kumar %A Rebecca Firestone %A Vishwajeet Kumar %A Gawande, Atul %X
Background: Effective, scalable strategies to improve maternal, fetal, and newborn health and reduce preventable morbidity and mortality are urgently needed in low- and middle-income countries. Building on the successes of previous checklist-based programs, the World Health Organization (WHO) and partners led the development of the Safe Childbirth Checklist (SCC), a 28-item list of evidence-based practices linked with improved maternal and newborn outcomes. Pilot-testing of the Checklist in Southern India demonstrated dramatic improvements in adherence by health workers to essential childbirth-related practices (EBPs). The BetterBirth Trial seeks to measure the effectiveness of SCC impact on EBPs, deaths, and complications at a larger scale.
Methods: This matched-pair, cluster-randomized controlled, adaptive trial will be conducted in 120 facilities across 24 districts in Uttar Pradesh, India. Study sites, identified according to predefined eligibility criteria, were matched by measured covariates before randomization. The intervention, the SCC embedded in a quality improvement program, consists of leadership engagement, a 2-day educational launch of the SCC, and support through placement of a trained peer “coach” to provide supportive supervision and real-time data feedback over an 8-month period with decreasing intensity. A facility-based childbirth quality coordinator is trained and supported to drive sustained behavior change after the BetterBirth team leaves the facility. Study participants are birth attendants and women and their newborns who present to the study facilities for childbirth at 60 intervention and 60 control sites. The primary outcome is a composite measure including maternal death, maternal severe morbidity, stillbirth, and newborn death, occurring within 7 days after birth. The sample size (n = 171,964) was calculated to detect a 15% reduction in the primary outcome. Adherence by health workers to EBPs will be measured in a subset of births (n = 6000). The trial will be conducted in close collaboration with key partners including the Governments of India and Uttar Pradesh, the World Health Organization, an expert Scientific Advisory Committee, an experienced local implementing organization (Population Services International, PSI), and frontline facility leaders and workers
Discussion: If effective, the WHO Safe Childbirth Checklist program could be a powerful health facilitystrengthening intervention to improve quality of care and reduce preventable harm to women and newborns, with millions of potential beneficiaries.
Trial registration: BetterBirth Study Protocol dated: 13 February 2014; ClinicalTrials.gov: NCT02148952; Universal Trial Number: U1111-1131-5647.%B Trials %V 576 %P 1-10 %G eng %N 17 %0 Journal Article %J Journal of Economic Perspectives %D 2016 %T Scoring Social Security Proposals: Response from Kashin, King, and Soneji %A Konstantin Kashin %A Gary King %A Samir Soneji %X
This is a response to Peter Diamond's comment on two paragraph comment on a passage in our article, Konstantin Kashin, Gary King, and Samir Soneji. 2015. “Systematic Bias and Nontransparency in US Social Security Administration Forecasts.” Journal of Economic Perspectives, 2, 29: 239-258.%B Journal of Economic Perspectives %V 30 %P 245-248 %8 Spring 2016 %G eng %U http://www.ingentaconnect.com/contentone/aea/jep/2016/00000030/00000002/art00012 %N 2 %0 Journal Article %J Science %D 2016 %T Comment on 'Estimating the Reproducibility of Psychological Science' %A Gilbert, Daniel %A Gary King %A Stephen Pettigrew %A Timothy Wilson %X
A recent article by the Open Science Collaboration (a group of 270 coauthors) gained considerable academic and public attention due to its sensational conclusion that the replicability of psychological science is surprisingly low. Science magazine lauded this article as one of the top 10 scientific breakthroughs of the year across all fields of science, reports of which appeared on the front pages of newspapers worldwide. We show that OSC's article contains three major statistical errors and, when corrected, provides no evidence of a replication crisis. Indeed, the evidence is consistent with the opposite conclusion -- that the reproducibility of psychological science is quite high and, in fact, statistically indistinguishable from 100%. (Of course, that doesn't mean that the replicability is 100%, only that the evidence is insufficient to reliably estimate replicability.) The moral of the story is that meta-science must follow the rules of science.
Replication data is available in this dataverse archive. See also the full web site for this article and related materials, and one of the news articles written about it.%B Science %V 351 %P 1037a-1038a %G eng %U http://science.sciencemag.org/content/351/6277/1037.2 %N 6277 %0 Journal Article %J Nature %D 2015 %T Urban observatories: City data can inform decision theory %A Aristides A. N. Patrinos %A Hannah Bayer %A Paul W. Glimcher %A Steven Koonin %A Miyoung Chun %A Gary King %X
Data are being collected on human behaviour in cities such as London, New York, Singapore and Shanghai, with a view to meeting city dwellers' needs more effectively. Incorporating decision-making theory into analyses of the data from these 'urban observatories' would yield further valuable information.%B Nature %V 519 %P 291 %G eng %U http://www.nature.com/nature/journal/v519/n7543/full/519291b.html %0 Journal Article %J Journal of Economic Perspectives %D 2015 %T Systematic Bias and Nontransparency in US Social Security Administration Forecasts %A Konstantin Kashin %A Gary King %A Samir Soneji %X
The financial stability of four of the five largest U.S. federal entitlement programs, strategic decision making in several industries, and many academic publications all depend on the accuracy of demographic and financial forecasts made by the Social Security Administration (SSA). Although the SSA has performed these forecasts since 1942, no systematic and comprehensive evaluation of their accuracy has ever been published by SSA or anyone else. The absence of a systematic evaluation of forecasts is a concern because the SSA relies on informal procedures that are potentially subject to inadvertent biases and does not share with the public, the scientific community, or other parts of SSA sufficient data or information necessary to replicate or improve its forecasts. These issues result in SSA holding a monopoly position in policy debates as the sole supplier of fully independent forecasts and evaluations of proposals to change Social Security. To assist with the forecasting evaluation problem, we collect all SSA forecasts for years that have passed and discover error patterns that could have been---and could now be---used to improve future forecasts. Specifically, we find that after 2000, SSA forecasting errors grew considerably larger and most of these errors made the Social Security Trust Funds look more financially secure than they actually were. In addition, SSA's reported uncertainty intervals are overconfident and increasingly so after 2000. We discuss the implications of these systematic forecasting biases for public policy.%B Journal of Economic Perspectives %V 29 %P 239-258 %G eng %U https://www.aeaweb.org/articles.php?doi=10.1257/jep.29.2.239 %N 2 %0 Journal Article %J Political Analysis %D 2015 %T Explaining Systematic Bias and Nontransparency in US Social Security Administration Forecasts %A Konstantin Kashin %A Gary King %A Samir Soneji %X
The accuracy of U.S. Social Security Administration (SSA) demographic and financial forecasts is crucial for the solvency of its Trust Funds, other government programs, industry decision making, and the evidence base of many scholarly articles. Because SSA makes public little replication information and uses qualitative and antiquated statistical forecasting methods, fully independent alternative forecasts (and the ability to score policy proposals to change the system) are nonexistent. Yet, no systematic evaluation of SSA forecasts has ever been published by SSA or anyone else --- until a companion paper to this one (King, Kashin, and Soneji, 2015a). We show that SSA's forecasting errors were approximately unbiased until about 2000, but then began to grow quickly, with increasingly overconfident uncertainty intervals. Moreover, the errors are all in the same potentially dangerous direction, making the Social Security Trust Funds look healthier than they actually are. We extend and then attempt to explain these findings with evidence from a large number of interviews we conducted with participants at every level of the forecasting and policy processes. We show that SSA's forecasting procedures meet all the conditions the modern social-psychology and statistical literatures demonstrate make bias likely. When those conditions mixed with potent new political forces trying to change Social Security, SSA's actuaries hunkered down trying hard to insulate their forecasts from strong political pressures. Unfortunately, this otherwise laudable resistance to undue influence, along with their ad hoc qualitative forecasting models, led the actuaries to miss important changes in the input data. Retirees began living longer lives and drawing benefits longer than predicted by simple extrapolations. We also show that the solution to this problem involves SSA or Congress implementing in government two of the central projects of political science over the last quarter century:  promoting transparency in data and methods and  replacing with formal statistical models large numbers of qualitative decisions too complex for unaided humans to make optimally.%B Political Analysis %V 23 %P 336-362 %G eng %U http://pan.oxfordjournals.org/lookup/doi/10.1093/pan/mpv011 %N 3 %0 Journal Article %J ANNALS of the American Academy of Political and Social Science %D 2015 %T Automating Open Science for Big Data %A Merce Crosas %A James Honaker %A Gary King %A Latanya Sweeney %X
The vast majority of social science research presently uses small (MB or GB scale) data sets. These fixed-scale data sets are commonly downloaded to the researcher's computer where the analysis is performed locally, and are often shared and cited with well-established technologies, such as the Dataverse Project (see Dataverse.org), to support the published results. The trend towards Big Data -- including large scale streaming data -- is starting to transform research and has the potential to impact policy-making and our understanding of the social, economic, and political problems that affect human societies. However, this research poses new challenges in execution, accountability, preservation, reuse, and reproducibility. Downloading these data sets to a researcher’s computer is infeasible or not practical; hence, analyses take place in the cloud, require unusual expertise, and benefit from collaborative teamwork and novel tool development. The advantage of these data sets in how informative they are also means that they are much more likely to contain highly sensitive personally identifiable information. In this paper, we discuss solutions to these new challenges so that the social sciences can realize the potential of Big Data.%B ANNALS of the American Academy of Political and Social Science %V 659 %P 260-273 %G eng %U http://ann.sagepub.com.ezp-prod1.hul.harvard.edu/content/659/1/260.full.pdf+html %N 1 %0 Journal Article %J Sociological Methods and Research %D 2015 %T A Unified Approach to Measurement Error and Missing Data: Details and Extensions %A Matthew Blackwell %A James Honaker %A Gary King %X
We extend a unified and easy-to-use approach to measurement error and missing data. In our companion article, Blackwell, Honaker, and King give an intuitive overview of the new technique, along with practical suggestions and empirical applications. Here, we offer more precise technical details, more sophisticated measurement error model specifications and estimation procedures, and analyses to assess the approach’s robustness to correlated measurement errors and to errors in categorical variables. These results support using the technique to reduce bias and increase efficiency in a wide variety of empirical research.%B Sociological Methods and Research %P 1-28 %G eng %U http://smr.sagepub.com/cgi/reprint/0049124115589052v1.pdf?ijkey=LHcsHlSL2eONpws&keytype=finite %0 Journal Article %J Sociological Methods and Research %D 2015 %T A Unified Approach to Measurement Error and Missing Data: Overview and Applications %A Matthew Blackwell %A James Honaker %A Gary King %X
Although social scientists devote considerable effort to mitigating measurement error during data collection, they often ignore the issue during data analysis. And although many statistical methods have been proposed for reducing measurement error-induced biases, few have been widely used because of implausible assumptions, high levels of model dependence, difficult computation, or inapplicability with multiple mismeasured variables. We develop an easy-to-use alternative without these problems; it generalizes the popular multiple imputation (MI) framework by treating missing data problems as a limiting special case of extreme measurement error, and corrects for both. Like MI, the proposed framework is a simple two-step procedure, so that in the second step researchers can use whatever statistical method they would have if there had been no problem in the first place. We also offer empirical illustrations, open source software that implements all the methods described herein, and a companion paper with technical details and extensions (Blackwell, Honaker, and King, 2014b).%B Sociological Methods and Research %P 1-39 %G eng %U http://smr.sagepub.com/content/early/2015/06/24/0049124115585360.full.pdf?ijkey=bcHBxBGUEzoqjk5&keytype=finite %0 Journal Article %J Political Analysis %D 2015 %T How Robust Standard Errors Expose Methodological Problems They Do Not Fix, and What to Do About It %A Gary King %A Margaret E. Roberts %X
"Robust standard errors" are used in a vast array of scholarship to correct standard errors for model misspecification. However, when misspecification is bad enough to make classical and robust standard errors diverge, assuming that it is nevertheless not so bad as to bias everything else requires considerable optimism. And even if the optimism is warranted, settling for a misspecified model, with or without robust standard errors, will still bias estimators of all but a few quantities of interest. The resulting cavernous gap between theory and practice suggests that considerable gains in applied statistics may be possible. We seek to help researchers realize these gains via a more productive way to understand and use robust standard errors; a new general and easier-to-use "generalized information matrix test" statistic that can formally assess misspecification (based on differences between robust and classical variance estimates); and practical illustrations via simulations and real examples from published research. How robust standard errors are used needs to change, but instead of jettisoning this popular tool we show how to use it to provide effective clues about model misspecification, likely biases, and a guide to considerably more reliable, and defensible, inferences. Accompanying this article [soon!] is software that implements the methods we describe.%B Political Analysis %V 23 %P 159–179 %G eng %U http://pan.oxfordjournals.org/content/23/2/159 %N 2 %0 Journal Article %J Science %D 2014 %T Reverse-engineering censorship in China: Randomized experimentation and participant observation %A Gary King %A Jennifer Pan %A Margaret E. Roberts %X
Existing research on the extensive Chinese censorship organization uses observational methods with well-known limitations. We conducted the first large-scale experimental study of censorship by creating accounts on numerous social media sites, randomly submitting different texts, and observing from a worldwide network of computers which texts were censored and which were not. We also supplemented interviews with confidential sources by creating our own social media site, contracting with Chinese firms to install the same censoring technologies as existing sites, and—with their software, documentation, and even customer support—reverse-engineering how it all works. Our results offer rigorous support for the recent hypothesis that criticisms of the state, its leaders, and their policies are published, whereas posts about real-world events with collective action potential are censored.%B Science %V 345 %P 1-10 %G eng %U http://www.sciencemag.org/content/345/6199/1251722.abstract %N 6199 %0 Journal Article %J Science %D 2014 %T
The Parable of Google Flu: Traps in Big Data Analysis%A Lazer, David %A Ryan Kennedy %A Gary King %A Alessandro Vespignani %X Large errors in flu prediction were largely avoidable, which offers lessons for the use of big data.
In February 2013, Google Flu Trends (GFT) made headlines but not for a reason that Google executives or the creators of the flu tracking system would have hoped. Nature reported that GFT was predicting more than double the proportion of doctor visits for influenza-like illness (ILI) than the Centers for Disease Control and Prevention (CDC), which bases its estimates on surveillance reports from laboratories across the United States ( 1, 2). This happened despite the fact that GFT was built to predict CDC reports. Given that GFT is often held up as an exemplary use of big data ( 3, 4), what lessons can we draw from this error? %B Science %V 343 %P 1203-1205 %G eng %N 14 March %0 Journal Article %J PS: Political Science and Politics %D 2014 %T Restructuring the Social Sciences: Reflections from Harvard's Institute for Quantitative Social Science %A Gary King %X
The social sciences are undergoing a dramatic transformation from studying problems to solving them; from making do with a small number of sparse data sets to analyzing increasing quantities of diverse, highly informative data; from isolated scholars toiling away on their own to larger scale, collaborative, interdisciplinary, lab-style research teams; and from a purely academic pursuit to having a major impact on the world. To facilitate these important developments, universities, funding agencies, and governments need to shore up and adapt the infrastructure that supports social science research. We discuss some of these developments here, as well as a new type of organization we created at Harvard to help encourage them -- the Institute for Quantitative Social Science. An increasing number of universities are beginning efforts to respond with similar institutions. This paper provides some suggestions for how individual universities might respond and how we might work together to advance social science more generally.%B PS: Political Science and Politics %V 47 %P 165-172 %G eng %U http://journals.cambridge.org/repo_A9100Nlq %N 1 %0 Journal Article %J PS: Political Science and Politics %D 2013 %T The Troubled Future of Colleges and Universities (with comments from five scholar-administrators) %A Gary King %A Maya Sen %X
The American system of higher education is under attack by political, economic, and educational forces that threaten to undermine its business model, governmental support, and operating mission. The potential changes are considerably more dramatic and disruptive than what we've already experienced. Traditional colleges and universities urgently need a coherent, thought-out response. Their central role in ensuring the creation, preservation, and distribution of knowledge may be at risk and, as a consequence, so too may be the spectacular progress across fields we have come to expect as a result.
Symposium contributors include Henry E. Brady, John Mark Hansen, Gary King, Nannerl O. Keohane, Michael Laver, Virginia Sapiro, and Maya Sen.%B PS: Political Science and Politics %V 46 %P 81--113 %G eng %N 1 %0 Journal Article %J PS: Political Science and Politics %D 2013 %T How Social Science Research Can Improve Teaching %A Gary King %A Maya Sen %X
We marshal discoveries about human behavior and learning from social science research and show how they can be used to improve teaching and learning. The discoveries are easily stated as three social science generalizations: (1) social connections motivate, (2) teaching teaches the teacher, and (3) instant feedback improves learning. We show how to apply these generalizations via innovations in modern information technology inside, outside, and across university classrooms. We also give concrete examples of these ideas from innovations we have experimented with in our own teaching.
See also a video presentation of this talk before the Harvard Board of Overseers%B PS: Political Science and Politics %V 46 %P 621-629 %G eng %N 3 %0 Journal Article %J American Political Science Review %D 2013 %T How Censorship in China Allows Government Criticism but Silences Collective Expression %A Gary King %A Jennifer Pan %A Margaret E. Roberts %X
We offer the first large scale, multiple source analysis of the outcome of what may be the most extensive effort to selectively censor human expression ever implemented. To do this, we have devised a system to locate, download, and analyze the content of millions of social media posts originating from nearly 1,400 different social media services all over China before the Chinese government is able to find, evaluate, and censor (i.e., remove from the Internet) the large subset they deem objectionable. Using modern computer-assisted text analytic methods that we adapt to and validate in the Chinese language, we compare the substantive content of posts censored to those not censored over time in each of 85 topic areas. Contrary to previous understandings, posts with negative, even vitriolic, criticism of the state, its leaders, and its policies are not more likely to be censored. Instead, we show that the censorship program is aimed at curtailing collective action by silencing comments that represent, reinforce, or spur social mobilization, regardless of content. Censorship is oriented toward attempting to forestall collective activities that are occurring now or may occur in the future --- and, as such, seem to clearly expose government intent.%B American Political Science Review %V 107 %P 1-18 %G eng %N 2 (May) %0 Journal Article %J Statistics, Politics, and Policy %D 2012 %T Estimating Partisan Bias of the Electoral College Under Proposed Changes in Elector Apportionment %A A. C. Thomas %A Andrew Gelman %A Gary King %A Jonathan N. Katz %X
In the election for President of the United States, the Electoral College is the body whose members vote to elect the President directly. Each state sends a number of delegates equal to its total number of representatives and senators in Congress; all but two states (Nebraska and Maine) assign electors pledged to the candidate that wins the state's plurality vote. We investigate the effect on presidential elections if states were to assign their electoral votes according to results in each congressional district,and conclude that the direct popular vote and the current electoral college are both substantially fairer compared to those alternatives where states would have divided their electoral votes by congressional district.%B Statistics, Politics, and Policy %P 1-13 %G eng %U http://www.degruyter.com/view/j/spp.ahead-of-print/spp-2012-0001/spp-2012-0001.xml?format=INT %0 Journal Article %J New England Journal of Medicine %D 2012 %T Letter to the Editor on the "Medicare Health Support Pilot Program" (by McCall and Cromwell) %A Gary King %A Richard Nielsen %A Aaron Wells %B New England Journal of Medicine %V 366 %P 667 %G eng %U http://www.nejm.org/doi/full/10.1056/NEJMc1114006 %N 7 %0 Journal Article %J Demography %D 2012 %T Statistical Security for Social Security %A Samir Soneji %A Gary King %X
The financial viability of Social Security, the single largest U.S. Government program, depends on accurate forecasts of the solvency of its intergenerational trust fund. We begin by detailing information necessary for replicating the Social Security Administration’s (SSA’s) forecasting procedures, which until now has been unavailable in the public domain. We then offer a way to improve the quality of these procedures due to age-and sex-specific mortality forecasts. The most recent SSA mortality forecasts were based on the best available technology at the time, which was a combination of linear extrapolation and qualitative judgments. Unfortunately, linear extrapolation excludes known risk factors and is inconsistent with long-standing demographic patterns such as the smoothness of age profiles. Modern statistical methods typically outperform even the best qualitative judgments in these contexts. We show how to use such methods here, enabling researchers to forecast using far more information, such as the known risk factors of smoking and obesity and known demographic patterns. Including this extra information makes a sub¬stantial difference: For example, by only improving mortality forecasting methods, we predict three fewer years of net surplus, $730 billion less in Social Security trust funds, and program costs that are 0.66% greater of projected taxable payroll compared to SSA projections by 2031. More important than specific numerical estimates are the advantages of transparency, replicability, reduction of uncertainty, and what may be the resulting lower vulnerability to the politicization of program forecasts. In addition, by offering with this paper software and detailed replication information, we hope to marshal the efforts of the research community to include ever more informative inputs and to continue to reduce the uncertainties in Social Security forecasts.
This work builds on our article that provides forecasts of US Mortality rates (see King and Soneji, The Future of Death in America), a book developing improved methods for forecasting mortality (Girosi and King, Demographic Forecasting), all data we used (King and Soneji, replication data sets), and open source software that implements the methods (Girosi and King, YourCast). Also available is a New York Times Op-Ed based on this work (King and Soneji, Social Security: It’s Worse Than You Think), and a replication data set for the Op-Ed (King and Soneji, replication data set).%B Demography %V 49 %P 1037-1060 %G eng %U http://link.springer.com/article/10.1007%2Fs13524-012-0106-z %N 3 %0 Journal Article %J Journal of Statistical Software %D 2011 %T Amelia II: A Program for Missing Data %A James Honaker %A Gary King %A Matthew Blackwell %X
Amelia II is a complete R package for multiple imputation of missing data. The package implements a new expectation-maximization with bootstrapping algorithm that works faster, with larger numbers of variables, and is far easier to use, than various Markov chain Monte Carlo approaches, but gives essentially the same answers. The program also improves imputation models by allowing researchers to put Bayesian priors on individual cell values, thereby including a great deal of potentially valuable and extensive information. It also includes features to accurately impute cross-sectional datasets, individual time series, or sets of time series for diﬀerent cross-sections. A full set of graphical diagnostics are also available. The program is easy to use, and the simplicity of the algorithm makes it far more robust; both a simple command line and extensive graphical user interface are included.
Amelia II software web site%B Journal of Statistical Software %V 45 %P 1-47 %G eng %N 7 %0 Journal Article %J PLoS ONE %D 2011 %T Estimating Incidence Curves of Several Infections Using Symptom Surveillance Data %A Edward Goldstein %A Benjamin J. Cowling %A Allison E. Aiello %A Saki Takahashi %A Gary King %A Ying Lu %A Marc Lipsitch %X
We introduce a method for estimating incidence curves of several co-circulating infectious pathogens, where each infection has its own probabilities of particular symptom profiles. Our deconvolution method utilizes weekly surveillance data on symptoms from a defined population as well as additional data on symptoms from a sample of virologically confirmed infectious episodes. We illustrate this method by numerical simulations and by using data from a survey conducted on the University of Michigan campus. Last, we describe the data needs to make such estimates accurate.
Link to PLoS version%B PLoS ONE %V 6 %P e23380 %G eng %N 8 %0 Journal Article %J Science %D 2011 %T Ensuring the Data Rich Future of the Social Sciences %A Gary King %X
Massive increases in the availability of informative social science data are making dramatic progress possible in analyzing, understanding, and addressing many major societal problems. Yet the same forces pose severe challenges to the scientific infrastructure supporting data sharing, data management, informatics, statistical methodology, and research ethics and policy, and these are collectively holding back progress. I address these changes and challenges and suggest what can be done.%B Science %V 331 %P 719-721 %8 2011 %G eng %N 11 February %0 Journal Article %J Proceedings of the National Academy of Sciences %D 2011 %T General Purpose Computer-Assisted Clustering and Conceptualization %A Justin Grimmer %A Gary King %X
We develop a computer-assisted method for the discovery of insightful conceptualizations, in the form of clusterings (i.e., partitions) of input objects. Each of the numerous fully automated methods of cluster analysis proposed in statistics, computer science, and biology optimize a different objective function. Almost all are well defined, but how to determine before the fact which one, if any, will partition a given set of objects in an "insightful" or "useful" way for a given user is unknown and difficult, if not logically impossible. We develop a metric space of partitions from all existing cluster analysis methods applied to a given data set (along with millions of other solutions we add based on combinations of existing clusterings), and enable a user to explore and interact with it, and quickly reveal or prompt useful or insightful conceptualizations. In addition, although uncommon in unsupervised learning problems, we offer and implement evaluation designs that make our computer-assisted approach vulnerable to being proven suboptimal in specific data types. We demonstrate that our approach facilitates more efficient and insightful discovery of useful information than either expert human coders or many existing fully automated methods.%B Proceedings of the National Academy of Sciences %G eng %U http://www.pnas.org/content/early/2011/01/31/1018067108.abstract %0 Journal Article %J Population Health Management %D 2011 %T Avoiding Randomization Failure in Program Evaluation %A Gary King %A Richard Nielsen %A Carter Coberley %A James E. Pope %A Aaron Wells %X
We highlight common problems in the application of random treatment assignment in large scale program evaluation. Random assignment is the defining feature of modern experimental design. Yet, errors in design, implementation, and analysis often result in real world applications not benefiting from the advantages of randomization. The errors we highlight cover the control of variability, levels of randomization, size of treatment arms, and power to detect causal effects, as well as the many problems that commonly lead to post-treatment bias. We illustrate with an application to the Medicare Health Support evaluation, including recommendations for improving the design and analysis of this and other large scale randomized experiments.%B Population Health Management %V 14 %P S11-S22 %8 2011 %G eng %N 1 %0 Journal Article %J Journal of Statistical Software %D 2011 %T Anchors: Software for Anchoring Vignettes Data %A Jonathan Wand %A Gary King %A Olivia Lau %X
When respondents use the ordinal response categories of standard survey questions in different ways, the validity of analyses based on the resulting data can be biased. Anchoring vignettes is a survey design technique intended to correct for some of these problems. The anchors package in R includes methods for evaluating and choosing anchoring vignettes, and for analyzing the resulting data.%B Journal of Statistical Software %V 42 %P 1--25 %G eng %U http://www.jstatsoft.org/v42/i03/ %N 3 %0 Journal Article %J Journal of Statistical Software %D 2011 %T MatchIt: Nonparametric Preprocessing for Parametric Causal Inference %A Daniel E. Ho %A Kosuke Imai %A Gary King %A Elizabeth A. Stuart %X MatchIt implements the suggestions of Ho, Imai, King, and Stuart (2007) for improving parametric statistical models by preprocessing data with nonparametric matching methods. MatchIt implements a wide range of sophisticated matching methods, making it possible to greatly reduce the dependence of causal inferences on hard-to-justify, but commonly made, statistical modeling assumptions. The software also easily ts into existing research practices since, after preprocessing data with MatchIt, researchers can use whatever parametric model they would have used without MatchIt, but produce inferences with substantially more robustness and less sensitivity to modeling assumptions. MatchIt is an R program, and also works seamlessly with Zelig. %B Journal of Statistical Software %V 42 %G eng %U http://www.jstatsoft.org/v42/i08 %N 8 %0 Journal Article %J Journal of the American Statistical Association %D 2011 %T Multivariate Matching Methods That are Monotonic Imbalance Bounding %A Stefano M. Iacus %A Gary King %A Giuseppe Porro %X
We introduce a new "Monotonic Imbalance Bounding" (MIB) class of matching methods for causal inference with a surprisingly large number of attractive statistical properties. MIB generalizes and extends in several new directions the only existing class, "Equal Percent Bias Reducing" (EPBR), which is designed to satisfy weaker properties and only in expectation. We also offer strategies to obtain specific members of the MIB class, and analyze in more detail a member of this class, called Coarsened Exact Matching, whose properties we analyze from this new perspective. We offer a variety of analytical results and numerical simulations that demonstrate how members of the MIB class can dramatically improve inferences relative to EPBR-based matching methods.%B Journal of the American Statistical Association %V 106 %P 345-361 %8 2011 %G eng %N 493 %0 Journal Article %J Political Analysis %D 2011 %T Causal Inference Without Balance Checking: Coarsened Exact Matching %A Stefano M. Iacus %A Gary King %A Giuseppe Porro %X
We discuss a method for improving causal inferences called "Coarsened Exact Matching'' (CEM), and the new "Monotonic Imbalance Bounding'' (MIB) class of matching methods from which CEM is derived. We summarize what is known about CEM and MIB, derive and illustrate several new desirable statistical properties of CEM, and then propose a variety of useful extensions. We show that CEM possesses a wide range of desirable statistical properties not available in most other matching methods, but is at the same time exceptionally easy to comprehend and use. We focus on the connection between theoretical properties and practical applications. We also make available easy-to-use open source software for R and Stata which implement all our suggestions.
Political Analysis version
An Explanation of CEM Weights%B Political Analysis %G eng %0 Journal Article %J Demographic Research %D 2011 %T The Future of Death in America %A Gary King %A Samir Soneji %X
Population mortality forecasts are widely used for allocating public health expenditures, setting research priorities, and evaluating the viability of public pensions, private pensions, and health care financing systems. In part because existing methods seem to forecast worse when based on more information, most forecasts are still based on simple linear extrapolations that ignore known biological risk factors and other prior information. We adapt a Bayesian hierarchical forecasting model capable of including more known health and demographic information than has previously been possible. This leads to the first age- and sex-specific forecasts of American mortality that simultaneously incorporate, in a formal statistical model, the effects of the recent rapid increase in obesity, the steady decline in tobacco consumption, and the well known patterns of smooth mortality age profiles and time trends. Formally including new information in forecasts can matter a great deal. For example, we estimate an increase in male life expectancy at birth from 76.2 years in 2010 to 79.9 years in 2030, which is 1.8 years greater than the U.S. Social Security Administration projection and 1.5 years more than U.S. Census projection. For females, we estimate more modest gains in life expectancy at birth over the next twenty years from 80.5 years to 81.9 years, which is virtually identical to the Social Security Administration projection and 2.0 years less than U.S. Census projections. We show that these patterns are also likely to greatly affect the aging American population structure. We offer an easy-to-use approach so that researchers can include other sources of information and potentially improve on our forecasts too.%B Demographic Research %V 25 %P 1--38 %G eng %U http://www.demographic-research.org/volumes/vol25/1/ %N 1 %0 Journal Article %J Population Health Metrics %D 2010 %T Designing Verbal Autopsy Studies %A Gary King %A Ying Lu %A Kenji Shibuya %X Background: Verbal autopsy analyses are widely used for estimating cause-specific mortality rates (CSMR) in the vast majority of the world without high quality medical death registration. Verbal autopsies -- survey interviews with the caretakers of imminent decedents -- stand in for medical examinations or physical autopsies, which are infeasible or culturally prohibited. Methods and Findings: We introduce methods, simulations, and interpretations that can improve the design of automated, data-derived estimates of CSMRs, building on a new approach by King and Lu (2008). Our results generate advice for choosing symptom questions and sample sizes that is easier to satisfy than existing practices. For example, most prior effort has been devoted to searching for symptoms with high sensitivity and specificity, which has rarely if ever succeeded with multiple causes of death. In contrast, our approach makes this search irrelevant because it can produce unbiased estimates even with symptoms that have very low sensitivity and specificity. In addition, the new method is optimized for survey questions caretakers can easily answer rather than questions physicians would ask themselves. We also offer an automated method of weeding out biased symptom questions and advice on how to choose the number of causes of death, symptom questions to ask, and observations to collect, among others. Conclusions: With the advice offered here, researchers should be able to design verbal autopsy surveys and conduct analyses with greatly reduced statistical biases and research costs. %B Population Health Metrics %V 8 %G eng %N 19 %0 Journal Article %J American Journal of Political Science %D 2010 %T A Method of Automated Nonparametric Content Analysis for Social Science %A Daniel Hopkins %A Gary King %X
The increasing availability of digitized text presents enormous opportunities for social scientists. Yet hand coding many blogs, speeches, government records, newspapers, or other sources of unstructured text is infeasible. Although computer scientists have methods for automated content analysis, most are optimized to classify individual documents, whereas social scientists instead want generalizations about the population of documents, such as the proportion in a given category. Unfortunately, even a method with a high percent of individual documents correctly classified can be hugely biased when estimating category proportions. By directly optimizing for this social science goal, we develop a method that gives approximately unbiased estimates of category proportions even when the optimal classifier performs poorly. We illustrate with diverse data sets, including the daily expressed opinions of thousands of people about the U.S. presidency. We also make available software that implements our methods and large corpora of text for further analysis.
This article led to the formation of Crimson Hexagon%B American Journal of Political Science %V 54 %P 229–247 %8 01/2010 %G eng %N 1 %0 Journal Article %J Population Health Metrics %D 2010 %T Deaths From Heart Failure: Using Coarsened Exact Matching to Correct Cause of Death Statistics %A Gretchen Stevens %A Gary King %A Kenji Shibuya %X
Background: Incomplete information on death certificates makes recorded cause of death data less useful for public health monitoring and planning. Certifying physicians sometimes list only the mode of death (and in particular, list heart failure) without indicating the underlying disease(s) that gave rise to the death. This can prevent valid epidemiologic comparisons across countries and over time. Methods and Results: We propose that coarsened exact matching be used to infer the underlying causes of death where only the mode of death is known; we focus on the case of heart failure in U.S., Mexican and Brazilian death records. Redistribution algorithms derived using this method assign the largest proportion of heart failure deaths to ischemic heart disease in all three countries (53%, 26% and 22%), with larger proportions assigned to hypertensive heart disease and diabetes in Mexico and Brazil (16% and 23% vs. 7% for hypertensive heart disease and 13% and 9% vs. 6% for diabetes). Reassigning these heart failure deaths increases US ischemic heart disease mortality rates by 6%.Conclusions: The frequency with which physicians list heart failure in the causal chain for various underlying causes of death allows for inference about how physicians use heart failure on the death certificate in different settings. This easy-to-use method has the potential to reduce bias and increase comparability in cause-of-death data, thereby improving the public health utility of death records. Key Words: vital statistics, heart failure, population health, mortality, epidemiology%B Population Health Metrics %V 8 %G eng %N 6 %0 Journal Article %J Public Opinion Quarterly %D 2010 %T Improving Anchoring Vignettes: Designing Surveys to Correct Interpersonal Incomparability %A Daniel Hopkins %A Gary King %X
We report the results of several randomized survey experiments designed to evaluate two intended improvements to anchoring vignettes, an increasingly common technique used to achieve interpersonal comparability in survey research. This technique asks for respondent self-assessments followed by assessments of hypothetical people described in vignettes. Variation in assessments of the vignettes across respondents reveals interpersonal incomparability and allows researchers to make responses more comparable by rescaling them. Our experiments show, first, that switching the question order so that self-assessments follow the vignettes primes respondents to define the response scale in a common way. In this case, priming is not a bias to avoid but a means of better communicating the question’s meaning. We then demonstrate that combining vignettes and self-assessments in a single direct comparison induces inconsistent and less informative responses. Since similar combined strategies are widely employed for related purposes, our results indicate that anchoring vignettes could reduce measurement error in many applications where they are not currently used. Data for our experiments come from a national telephone survey and a separate on-line survey.%B Public Opinion Quarterly %P 1-22 %G eng %0 Journal Article %J American Journal of Political Science %D 2010 %T What to do About Missing Values in Time Series Cross-Section Data %A James Honaker %A Gary King %X
Applications of modern methods for analyzing data with missing values, based primarily on multiple imputation, have in the last half-decade become common in American politics and political behavior. Scholars in these fields have thus increasingly avoided the biases and inefficiencies caused by ad hoc methods like listwise deletion and best guess imputation. However, researchers in much of comparative politics and international relations, and others with similar data, have been unable to do the same because the best available imputation methods work poorly with the time-series cross-section data structures common in these fields. We attempt to rectify this situation. First, we build a multiple imputation model that allows smooth time trends, shifts across cross-sectional units, and correlations over time and space, resulting in far more accurate imputations. Second, we build nonignorable missingness models by enabling analysts to incorporate knowledge from area studies experts via priors on individual missing cell values, rather than on difficult-to-interpret model parameters. Third, since these tasks could not be accomplished within existing imputation algorithms, in that they cannot handle as many variables as needed even in the simpler cross-sectional data for which they were designed, we also develop a new algorithm that substantially expands the range of computationally feasible data types and sizes for which multiple imputation can be used. These developments also made it possible to implement the methods introduced here in freely available open source software that is considerably more reliable than existing strategies.%B American Journal of Political Science %V 54 %P 561-581 %8 2010 %G eng %U http://gking.harvard.edu/files/abs/pr-abs.shtml %N 3 %0 Journal Article %J Political Analysis %D 2009 %T Empirical versus Theoretical Claims about Extreme Counterfactuals: A Response %A Gary King %A Langche Zeng %X
In response to the data-based measures of model dependence proposed in King and Zeng (2006), Sambanis and Michaelides (2008) propose alternative measures that rely upon assumptions untestable in observational data. If these assumptions are correct, then their measures are appropriate and ours, based solely on the empirical data, may be too conservative. If instead and as is usually the case, the researcher is not certain of the precise functional form of the data generating process, the distribution from which the data are drawn, and the applicability of these modeling assumptions to new counterfactuals, then the data-based measures proposed in King and Zeng (2006) are much preferred. After all, the point of model dependence checks is to verify empirically, rather than to stipulate by assumption, the effects of modeling assumptions on counterfactual inferences.%B Political Analysis %V 17 %P 107-112 %G eng %0 Journal Article %J The Stata Journal %D 2009 %T CEM: Coarsened Exact Matching in Stata %A Matthew Blackwell %A Stefano Iacus %A Gary King %A Giuseppe Porro %X In this article, we introduce a Stata implementation of coarsened exact matching, a new method for improving the estimation of causal effects by reducing imbalance in covariates between treated and control groups. Coarsened exact matching is faster, is easier to use and understand, requires fewer assumptions, is more easily automated, and possesses more attractive statistical properties for many applications than do existing matching methods. In coarsened exact matching, users temporarily coarsen their data, exact match on these coarsened data, and then run their analysis on the uncoarsened, matched data. Coarsened exact matching bounds the degree of model dependence and causal effect estimation error by ex ante user choice, is monotonic imbalance bounding (so that reducing the maximum imbalance on one variable has no effect on others), does not require a separate procedure to restrict data to common support, meets the congruence principle, is approximately invariant to measurement error, balances all nonlinearities and interactions in sample (i.e., not merely in expectation), and works with multiply imputed datasets. Other matching methods inherit many of the coarsened exact matching method’s properties when applied to further match data preprocessed by coarsened exact matching. The cem command implements the coarsened exact matching algorithm in Stata. %B The Stata Journal %V 9 %P 524–546 %G eng %0 Journal Article %J Historical Social Research %D 2009 %T Preserving Quantitative Research-Elicited Data for Longitudinal Analysis. New Developments in Archiving Survey Data in the U.S. %A Mark Abrahamson %A Kenneth A. Bollen %A Gutmann, Myron P. %A Gary King %A Amy Pienta %X
Social science data collected in the United States, both historically and at present, have often not been placed in any public archive -- even when the data collection was supported by government grants. The availability of the data for future use is, therefore, in jeopardy. Enforcing archiving norms may be the only way to increase data preservation and availability in the future.%B Historical Social Research %V 34 %P 51-59 %G eng %N 3 %0 Journal Article %J Statistical Science %D 2009 %T Matched Pairs and the Future of Cluster-Randomized Experiments: A Rejoinder %A Kosuke Imai %A Gary King %A Clayton Nall %X
A basic feature of many field experiments is that investigators are only able to randomize clusters of individuals–-such as households, communities, firms, medical practices, schools, or classrooms–-even when the individual is the unit of interest. To recoup the resulting efficiency loss, some studies pair similar clusters and randomize treatment within pairs. However, many other studies avoid pairing, in part because of claims in the literature, echoed by clinical trials standards organizations, that this matched-pair, cluster-randomization design has serious problems. We argue that all such claims are unfounded. We also prove that the estimator recommended for this design in the literature is unbiased only in situations when matching is unnecessary and and its standard error is also invalid. To overcome this problem without modeling assumptions, we develop a simple design-based estimator with much improved statistical properties. We also propose a model-based approach that includes some of the benefits of our design-based estimator as well as the estimator in the literature. Our methods also address individual-level noncompliance, which is common in applications but not allowed for in most existing methods. We show that from the perspective of bias, efficiency, power, robustness, or research costs, and in large or small samples, pairing should be used in cluster-randomized experiments whenever feasible and failing to do so is equivalent to discarding a considerable fraction of one’s data. We develop these techniques in the context of a randomized evaluation we are conducting of the Mexican Universal Health Insurance Program.%B Statistical Science %V 24 %P 64–72 %G eng %0 Journal Article %J Journal of Statistical Software %D 2009 %T CEM: Software for Coarsened Exact Matching %A Stefano M. Iacus %A Gary King %A Giuseppe Porro %X
This program is designed to improve causal inference via a method of matching that is widely applicable in observational data and easy to understand and use (if you understand how to draw a histogram, you will understand this method). The program implements the coarsened exact matching (CEM) algorithm, described below. CEM may be used alone or in combination with any existing matching method. This algorithm, and its statistical properties, are described in Iacus, King, and Porro (2008).%B Journal of Statistical Software %V 30 %G eng %U http://gking.harvard.edu/cem %0 Journal Article %J Statistical Science %D 2009 %T The Essential Role of Pair Matching in Cluster-Randomized Experiments, with Application to the Mexican Universal Health Insurance Evaluation %A Kosuke Imai %A Gary King %A Clayton Nall %X A basic feature of many field experiments is that investigators are only able to randomize clusters of individuals–-such as households, communities, firms, medical practices, schools, or classrooms–-even when the individual is the unit of interest. To recoup the resulting efficiency loss, some studies pair similar clusters and randomize treatment within pairs. However, many other studies avoid pairing, in part because of claims in the literature, echoed by clinical trials standards organizations, that this matched-pair, cluster-randomization design has serious problems. We argue that all such claims are unfounded. We also prove that the estimator recommended for this design in the literature is unbiased only in situations when matching is unnecessary and and its standard error is also invalid. To overcome this problem without modeling assumptions, we develop a simple design-based estimator with much improved statistical properties. We also propose a model-based approach that includes some of the benefits of our design-based estimator as well as the estimator in the literature. Our methods also address individual-level noncompliance, which is common in applications but not allowed for in most existing methods. We show that from the perspective of bias, efficiency, power, robustness, or research costs, and in large or small samples, pairing should be used in cluster-randomized experiments whenever feasible and failing to do so is equivalent to discarding a considerable fraction of one’s data. We develop these techniques in the context of a randomized evaluation we are conducting of the Mexican Universal Health Insurance Program. %B Statistical Science %V 24 %P 29–53 %G eng %0 Journal Article %J Science %D 2009 %T Computational Social Science %A Lazer, David %A Pentland, Alex %A Adamic, Lada %A Aral, Sinan %A Barabasi, Albert-Laszlo %A Brewer, Devon %A Christakis, Nicholas %A Contractor, Noshir %A Fowler, James %A Myron Gutmann %A Jebara, Tony %A Gary King %A Macy, Michael %A Roy, Deb %A Van Alstyne, Marshall %X
A field is emerging that leverages the capacity to collect and analyze data at a scale that may reveal patterns of individual and group behaviors.%B Science %V 323 %P 721-723 %G eng %0 Journal Article %J Library Trends %D 2009 %T
From Preserving the Past to Preserving the Future: The Data-PASS Project and the Challenges of Preserving Digital Social Science Data%A Gutmann, Myron P. %A Mark Abrahamson %A Margaret O. Adams %A Micah Altman %A Caroline Arms %A Kenneth Bollen %A Michael Carlson %A Jonathan Crabtree %A Darrell Donakowski %A Gary King %A Jaret Lyle %A Marc Maynard %A Amy Pienta %A Richard Rockwell %A Lois Rocms-Ferrara %A Copeland H. Young %X
Social science data are an unusual part of the past, present, and future of digital preservation. They are both an unqualified success, due to long-lived and sustainable archival organizations, and in need of further development because not all digital content is being preserved. This article is about the Data Preservation Alliance for Social Sciences (Data-PASS), a project supported by the National Digital Information Infrastructure and Preservation Program (NDIIPP), which is a partnership of five major U.S. social science data archives. Broadly speaking, Data-PASS has the goal of ensuring that at-risk social science data are identified, acquired, and preserved, and that we have a future-oriented organization that could collaborate on those preservation tasks for the future. Throughout the life of the Data-PASS project we have worked to identify digital materials that have never been systematically archived, and to appraise and acquire them. As the project has progressed, however, it has increasingly turned its attention from identifying and acquiring legacy and at-risk social science data to identifying on going and future research projects that will produce data. This article is about the project's history, with an emphasis on the issues that underlay the transition from looking backward to looking forward.%B Library Trends %V 57 %P 315–337 %G eng %0 Journal Article %J The Lancet %D 2009 %T Public Policy for the Poor? A Randomised Assessment of the Mexican Universal Health Insurance Programme %A Gary King %A Emmanuela Gakidou %A Kosuke Imai %A Jason Lakin %A Ryan T. Moore %A Clayton Nall %A Nirmala Ravishankar %A Manett Vargas %A Martha María Téllez-Rojo %A Juan Eugenio Hernández Ávila %A Mauricio Hernández Ávila %A Héctor Hernández Llamas %X
Background: We assessed aspects of Seguro Popular, a programme aimed to deliver health insurance, regular and preventive medical care, medicines, and health facilities to 50 million uninsured Mexicans. Methods: We randomly assigned treatment within 74 matched pairs of health clusters–-i.e., health facility catchment areas–-representing 118,569 households in seven Mexican states, and measured outcomes in a 2005 baseline survey (August 2005, to September 2005) and follow-up survey 10 months later (July 2006, to August 2006) in 50 pairs (n=32 515). The treatment consisted of encouragement to enrol in a health-insurance programme and upgraded medical facilities. Participant states also received funds to improve health facilities and to provide medications for services in treated clusters. We estimated intention to treat and complier average causal effects non-parametrically. Findings: Intention-to-treat estimates indicated a 23% reduction from baseline in catastrophic expenditures (1·9% points and 95% CI 0·14-3·66). The effect in poor households was 3·0% points (0·46-5·54) and in experimental compliers was 6·5% points (1·65-11·28), 30% and 59% reductions, respectively. The intention-to-treat effect on health spending in poor households was 426 pesos (39-812), and the complier average causal effect was 915 pesos (147-1684). Contrary to expectations and previous observational research, we found no effects on medication spending, health outcomes, or utilisation. Interpretation: Programme resources reached the poor. However, the programme did not show some other effects, possibly due to the short duration of treatment (10 months). Although Seguro Popular seems to be successful at this early stage, further experiments and follow-up studies, with longer assessment periods, are needed to ascertain the long-term effects of the programme.%B The Lancet %V 373 %P 1447-1454 %8 04/2009 %G eng %0 Journal Article %J Journal of Economic History %D 2008 %T Ordinary Economic Voting Behavior in the Extraordinary Election of Adolf Hitler %A Gary King %A Ori Rosen %A Martin Tanner %A Alexander Wagner %X
The enormous Nazi voting literature rarely builds on modern statistical or economic research. By adding these approaches, we find that the most widely accepted existing theories of this era cannot distinguish the Weimar elections from almost any others in any country. Via a retrospective voting account, we show that voters most hurt by the depression, and most likely to oppose the government, fall into separate groups with divergent interests. This explains why some turned to the Nazis and others turned away. The consequences of Hitler's election were extraordinary, but the voting behavior that led to it was not.%B Journal of Economic History %V 68 %P 996 %8 12/2008 %G eng %N 4 %& 951 %0 Journal Article %J Election Law Journal %D 2008 %T The Future of Partisan Symmetry as a Judicial Test for Partisan Gerrymandering after LULAC v. Perry %A Bernard Grofman %A Gary King %X
While the Supreme Court in Bandemer v. Davis found partisan gerrymandering to be justiciable, no challenged redistricting plan in the subsequent 20 years has been held unconstitutional on partisan grounds. Then, in Vieth v. Jubilerer, five justices concluded that some standard might be adopted in a future case, if a manageable rule could be found. When gerrymandering next came before the Court, in LULAC v. Perry, we along with our colleagues filed an Amicus Brief (King et al., 2005), proposing the test be based in part on the partisan symmetry standard. Although the issue was not resolved, our proposal was discussed and positively evaluated in three of the opinions, including the plurality judgment, and for the first time for any proposal the Court gave a clear indication that a future legal test for partisan gerrymandering will likely include partisan symmetry. A majority of Justices now appear to endorse the view that the measurement of partisan symmetry may be used in partisan gerrymandering claims as “a helpful (though certainly not talismanic) tool” (Justice Stevens, joined by Justice Breyer), provided one recognizes that “asymmetry alone is not a reliable measure of unconstitutional partisanship” and possibly that the standard would be applied only after at least one election has been held under the redistricting plan at issue (Justice Kennedy, joined by Justices Souter and Ginsburg). We use this essay to respond to the request of Justices Souter and Ginsburg that “further attention … be devoted to the administrability of such a criterion at all levels of redistricting and its review.” Building on our previous scholarly work, our Amicus Brief, the observations of these five Justices, and a supporting consensus in the academic literature, we offer here a social science perspective on the conceptualization and measurement of partisan gerrymandering and the development of relevant legal rules based on what is effectively the Supreme Court’s open invitation to lower courts to revisit these issues in the light of LULAC v. Perry.%B Election Law Journal %V 6 %P 2-35 %8 01/2007 %G eng %N 1 %& 2 %0 Journal Article %J PLoS Medicine %D 2008 %T The Effects of International Monetary Fund Loans on Health Outcomes %A Megan Murray %A Gary King %X A "Perspective" article that discusses an article by David Stuckler and colleagues showing that, in Eastern European and former Soviet countries, participation in International Monetary Fund economic programs have been associated with higher mortality rates from tuberculosis. %B PLoS Medicine %V 5 %8 June %G eng %0 Journal Article %J Journal of the Royal Statistical Society, Series A %D 2008 %T Misunderstandings Among Experimentalists and Observationalists about Causal Inference %A Kosuke Imai %A Gary King %A Elizabeth Stuart %X
We attempt to clarify, and suggest how to avoid, several serious misunderstandings about and fallacies of causal inference in experimental and observational research. These issues concern some of the most basic advantages and disadvantages of each basic research design. Problems include improper use of hypothesis tests for covariate balance between the treated and control groups, and the consequences of using randomization, blocking before randomization, and matching after treatment assignment to achieve covariate balance. Applied researchers in a wide range of scientific disciplines seem to fall prey to one or more of these fallacies, and as a result make suboptimal design or analysis choices. To clarify these points, we derive a new four-part decomposition of the key estimation errors in making causal inferences. We then show how this decomposition can help scholars from different experimental and observational research traditions better understand each other’s inferential problems and attempted solutions.%B Journal of the Royal Statistical Society, Series A %V 171, part 2 %P 481–502 %G eng %0 Journal Article %J Journal of Computational Graphics and Statistics %D 2008 %T Toward A Common Framework for Statistical Analysis and Development %A Kosuke Imai %A Gary King %A Olivia Lau %X We describe some progress toward a common framework for statistical analysis and software development built on and within the R language, including R’s numerous existing packages. The framework we have developed offers a simple unified structure and syntax that can encompass a large fraction of statistical procedures already implemented in R, without requiring any changes in existing approaches. We conjecture that it can be used to encompass and present simply a vast majority of existing statistical methods, regardless of the theory of inference on which they are based, notation with which they were developed, and programming syntax with which they have been implemented. This development enabled us, and should enable others, to design statistical software with a single, simple, and unified user interface that helps overcome the conflicting notation, syntax, jargon, and statistical methods existing across the methods subfields of numerous academic disciplines. The approach also enables one to build a graphical user interface that automatically includes any method encompassed within the framework. We hope that the result of this line of research will greatly reduce the time from the creation of a new statistical innovation to its widespread use by applied researchers whether or not they use or program in R. %B Journal of Computational Graphics and Statistics %V 17 %P 1–22 %G eng %0 Journal Article %J Statistical Science %D 2008 %T Verbal Autopsy Methods with Multiple Causes of Death %A Gary King %A Ying Lu %X Verbal autopsy procedures are widely used for estimating cause-specific mortality in areas without medical death certification. Data on symptoms reported by caregivers along with the cause of death are collected from a medical facility, and the cause-of-death distribution is estimated in the population where only symptom data are available. Current approaches analyze only one cause at a time, involve assumptions judged difficult or impossible to satisfy, and require expensive, time consuming, or unreliable physician reviews, expert algorithms, or parametric statistical models. By generalizing current approaches to analyze multiple causes, we show how most of the difficult assumptions underlying existing methods can be dropped. These generalizations also make physician review, expert algorithms, and parametric statistical assumptions unnecessary. With theoretical results, and empirical analyses in data from China and Tanzania, we illustrate the accuracy of this approach. While no method of analyzing verbal autopsy data, including the more computationally intensive approach offered here, can give accurate estimates in all circumstances, the procedure offered is conceptually simpler, less expensive, more general, as or more replicable, and easier to use in practice than existing approaches. We also show how our focus on estimating aggregate proportions, which are the quantities of primary interest in verbal autopsy studies, may also greatly reduce the assumptions necessary, and thus improve the performance of, many individual classifiers in this and other areas. As a companion to this paper, we also offer easy-to-use software that implements the methods discussed herein. %B Statistical Science %V 23 %P 78–91 %G eng %0 Journal Article %J Sociological Methods and Research %D 2007 %T An Introduction to the Dataverse Network as an Infrastructure for Data Sharing %A Gary King %X
We introduce a set of integrated developments in web application software, networking, data citation standards, and statistical methods designed to put some of the universe of data and data sharing practices on somewhat firmer ground. We have focused on social science data, but aspects of what we have developed may apply more widely. The idea is to facilitate the public distribution of persistent, authorized, and verifiable data, with powerful but easy-to-use technology, even when the data are confidential or proprietary. We intend to solve some of the sociological problems of data sharing via technological means, with the result intended to benefit both the scientific community and the sometimes apparently contradictory goals of individual researchers.%B Sociological Methods and Research %V 36 %P 173–199 %G eng %0 Journal Article %J Political Analysis %D 2007 %T Matching as Nonparametric Preprocessing for Reducing Model Dependence in Parametric Causal Inference %A Daniel Ho %A Kosuke Imai %A Gary King %A Elizabeth Stuart %X
Although published works rarely include causal estimates from more than a few model specifications, authors usually choose the presented estimates from numerous trial runs readers never see. Given the often large variation in estimates across choices of control variables, functional forms, and other modeling assumptions, how can researchers ensure that the few estimates presented are accurate or representative? How do readers know that publications are not merely demonstrations that it is possible to find a specification that fits the author’s favorite hypothesis? And how do we evaluate or even define statistical properties like unbiasedness or mean squared error when no unique model or estimator even exists? Matching methods, which offer the promise of causal inference with fewer assumptions, constitute one possible way forward, but crucial results in this fast-growing methodological literature are often grossly misinterpreted. We explain how to avoid these misinterpretations and propose a unified approach that makes it possible for researchers to preprocess data with matching (such as with the easy-to-use software we offer) and then to apply the best parametric techniques they would have used anyway. This procedure makes parametric models produce more accurate and considerably less model-dependent causal inferences.%B Political Analysis %V 15 %P 199–236 %G eng %0 Journal Article %J Political Analysis %D 2007 %T Comparing Incomparable Survey Responses: New Tools for Anchoring Vignettes %A Gary King %A Jonathan Wand %X When respondents use the ordinal response categories of standard survey questions in different ways, the validity of analyses based on the resulting data can be biased. Anchoring vignettes is a survey design technique, introduced by King, Murray, Salomon, and Tandon (2004), intended to correct for some of these problems. We develop new methods both for evaluating and choosing anchoring vignettes, and for analyzing the resulting data. With surveys on a diverse range of topics in a range of countries, we illustrate how our proposed methods can improve the ability of anchoring vignettes to extract information from survey data, as well as saving in survey administration costs. %B Political Analysis %V 15 %P 46-66 %8 Winter %G eng %0 Journal Article %J International Studies Quarterly %D 2007 %T When Can History Be Our Guide? The Pitfalls of Counterfactual Inference %A Gary King %A Langche Zeng %X Inferences about counterfactuals are essential for prediction, answering "what if" questions, and estimating causal effects. However, when the counterfactuals posed are too far from the data at hand, conclusions drawn from well-specified statistical analyses become based on speculation and convenient but indefensible model assumptions rather than empirical evidence. Unfortunately, standard statistical approaches assume the veracity of the model rather than revealing the degree of model-dependence, and so this problem can be hard to detect. We develop easy-to-apply methods to evaluate counterfactuals that do not require sensitivity testing over specified classes of models. If an analysis fails the tests we offer, then we know that substantive results are sensitive to at least some modeling choices that are not based on empirical evidence. We use these methods to evaluate the extensive scholarly literatures on the effects of changes in the degree of democracy in a country (on any dependent variable) and separate analyses of the effects of UN peacebuilding efforts. We find evidence that many scholars are inadvertently drawing conclusions based more on modeling hypotheses than on their data. For some research questions, history contains insufficient information to be our guide. %B International Studies Quarterly %P 183-210 %8 March %G eng %0 Journal Article %J International Studies Quarterly %D 2007 %T Detecting Model Dependence in Statistical Inference: A Response %A Gary King %A Langche Zeng %X
Inferences about counterfactuals are essential for prediction, answering "what if" questions, and estimating causal effects. However, when the counterfactuals posed are too far from the data at hand, conclusions drawn from well-specified statistical analyses become based on speculation and convenient but indefensible model assumptions rather than empirical evidence. Unfortunately, standard statistical approaches assume the veracity of the model rather than revealing the degree of model-dependence, and so this problem can be hard to detect. We develop easy-to-apply methods to evaluate counterfactuals that do not require sensitivity testing over specified classes of models. If an analysis fails the tests we offer, then we know that substantive results are sensitive to at least some modeling choices that are not based on empirical evidence. We use these methods to evaluate the extensive scholarly literatures on the effects of changes in the degree of democracy in a country (on any dependent variable) and separate analyses of the effects of UN peacebuilding efforts. We find evidence that many scholars are inadvertently drawing conclusions based more on modeling hypotheses than on their data. For some research questions, history contains insufficient information to be our guide.%B International Studies Quarterly %V 51 %P 231-241 %8 March %G eng %0 Journal Article %J D-Lib Magazine %D 2007 %T
A Proposed Standard for the Scholarly Citation of Quantitative Data%A Micah Altman %A Gary King %X
An essential aspect of science is a community of scholars cooperating and competing in the pursuit of common goals. A critical component of this community is the common language of and the universal standards for scholarly citation, credit attribution, and the location and retrieval of articles and books. We propose a similar universal standard for citing quantitative data that retains the advantages of print citations, adds other components made possible by, and needed due to, the digital form and systematic nature of quantitative data sets, and is consistent with most existing subfield-specific approaches. Although the digital library field includes numerous creative ideas, we limit ourselves to only those elements that appear ready for easy practical use by scientists, journal editors, publishers, librarians, and archivists.%B D-Lib Magazine %V 13 %8 March / April %G eng %U http://www.dlib.org/dlib/march07/altman/03altman.html %0 Journal Article %J Journal of Policy Analysis and Management %D 2007 %T A "Politically Robust" Experimental Design for Public Policy Evaluation, with Application to the Mexican Universal Health Insurance Program %A Gary King %A Emmanuela Gakidou %A Nirmala Ravishankar %A Ryan T. Moore %A Jason Lakin %A Manett Vargas %A Martha María Téllez-Rojo %A Juan Eugenio Hernández Ávila %A Mauricio Hernández Ávila %A Héctor Hernández Llamas %X
We develop an approach to conducting large scale randomized public policy experiments intended to be more robust to the political interventions that have ruined some or all parts of many similar previous efforts. Our proposed design is insulated from selection bias in some circumstances even if we lose observations and our inferences can still be unbiased even if politics disrupts any two of the three steps in our analytical procedures and and other empirical checks are available to validate the overall design. We illustrate with a design and empirical validation of an evaluation of the Mexican Seguro Popular de Salud (Universal Health Insurance) program we are conducting. Seguro Popular, which is intended to grow to provide medical care, drugs, preventative services, and financial health protection to the 50 million Mexicans without health insurance, is one of the largest health reforms of any country in the last two decades. The evaluation is also large scale, constituting one of the largest policy experiments to date and what may be the largest randomized health policy experiment ever.%B Journal of Policy Analysis and Management %V 26 %P 479-506 %G eng %0 Journal Article %J Political Analysis %D 2006 %T The Dangers of Extreme Counterfactuals %A Gary King %A Langche Zeng %X We address the problem that occurs when inferences about counterfactuals – predictions, "what if" questions, and causal effects – are attempted far from the available data. The danger of these extreme counterfactuals is that substantive conclusions drawn from statistical models that fit the data well turn out to be based largely on speculation hidden in convenient modeling assumptions that few would be willing to defend. Yet existing statistical strategies provide few reliable means of identifying extreme counterfactuals. We offer a proof that inferences farther from the data are more model-dependent, and then develop easy-to-apply methods to evaluate how model-dependent our answers would be to specified counterfactuals. These methods require neither sensitivity testing over specified classes of models nor evaluating any specific modeling assumptions. If an analysis fails the simple tests we offer, then we know that substantive results are sensitive to at least some modeling choices that are not based on empirical evidence. %B Political Analysis %V 14 %P 131–159 %G eng %0 Journal Article %J Demography %D 2006 %T Death by Survey: Estimating Adult Mortality without Selection Bias from Sibling Survival Data %A Emmanuela Gakidou %A Gary King %X The widely used methods for estimating adult mortality rates from sample survey responses about the survival of siblings, parents, spouses, and others depend crucially on an assumption that we demonstrate does not hold in real data. We show that when this assumption is violated – so that the mortality rate varies with sibship size – mortality estimates can be massively biased. By using insights from work on the statistical analysis of selection bias, survey weighting, and extrapolation problems, we propose a new and relatively simple method of recovering the mortality rate with both greatly reduced potential for bias and increased clarity about the source of necessary assumptions. %B Demography %V 43 %P 569–585 %8 August %G eng %0 Journal Article %J PS: Political Science and Politics %D 2006 %T Publication, Publication %A Gary King %X
I show herein how to write a publishable paper by beginning with the replication of a published article. This strategy seems to work well for class projects in producing papers that ultimately get published, helping to professionalize students into the discipline, and teaching them the scientific norms of the free exchange of academic information. I begin by briefly revisiting the prominent debate on replication our discipline had a decade ago and some of the progress made in data sharing since.%B PS: Political Science and Politics %V 39 %P 119–125 %8 January %G eng %U http://gking.harvard.edu/papers %0 Journal Article %J Journal of Statistical Software %D 2005 %T WhatIf: Software for Evaluating Counterfactuals %A Heather Stoll %A Gary King %A Langchee Zeng %B Journal of Statistical Software %V 15 %G eng %U http://gking.harvard.edu/whatif %0 Journal Article %J New York University Law Review %D 2005 %T The Supreme Court During Crisis: How War Affects only Non-War Cases %A Lee Epstein %A Daniel E. Ho %A Gary King %A Jeffrey A. Segal %X Does the U.S. Supreme Court curtail rights and liberties when the nation’s security is under threat? In hundreds of articles and books, and with renewed fervor since September 11, 2001, members of the legal community have warred over this question. Yet, not a single large-scale, quantitative study exists on the subject. Using the best data available on the causes and outcomes of every civil rights and liberties case decided by the Supreme Court over the past six decades and employing methods chosen and tuned especially for this problem, our analyses demonstrate that when crises threaten the nation’s security, the justices are substantially more likely to curtail rights and liberties than when peace prevails. Yet paradoxically, and in contradiction to virtually every theory of crisis jurisprudence, war appears to affect only cases that are unrelated to the war. For these cases, the effect of war and other international crises is so substantial, persistent, and consistent that it may surprise even those commentators who long have argued that the Court rallies around the flag in times of crisis. On the other hand, we find no evidence that cases most directly related to the war are affected. We attempt to explain this seemingly paradoxical evidence with one unifying conjecture: Instead of balancing rights and security in high stakes cases directly related to the war, the Justices retreat to ensuring the institutional checks of the democratic branches. Since rights-oriented and process-oriented dimensions seem to operate in different domains and at different times, and often suggest different outcomes, the predictive factors that work for cases unrelated to the war fail for cases related to the war. If this conjecture is correct, federal judges should consider giving less weight to legal principles outside of wartime but established during wartime, and attorneys should see it as their responsibility to distinguish cases along these lines. %B New York University Law Review %V 80 %P 1–116 %8 April %G eng %0 Journal Article %J Journal of the Royal Statistical Society %D 2004 %T Finding New Information for Ecological Inference Models: A Comment on Jon Wakefield, 'Ecological Inference in 2X2 Tables' %A Gary King %B Journal of the Royal Statistical Society %V 167 %P 437 %G eng %0 Journal Article %J Journal of Statistical Software %D 2004 %T EI: A Program for Ecological Inference %A Gary King %B Journal of Statistical Software %V 11 %G eng %U http://www.jstatsoft.org/index.php?vol=11 %0 Journal Article %J American Political Science Review %D 2004 %T Enhancing the Validity and Cross-cultural Comparability of Measurement in Survey Research %A Gary King %A Christopher J.L. Murray %A Joshua A. Salomon %A Ajay Tandon %X We address two long-standing survey research problems: measuring complicated concepts, such as political freedom or efficacy, that researchers define best with reference to examples and and what to do when respondents interpret identical questions in different ways. Scholars have long addressed these problems with approaches to reduce incomparability, such as writing more concrete questions – with uneven success. Our alternative is to measure directly response category incomparability and to correct for it. We measure incomparability via respondents’ assessments, on the same scale as the self-assessments to be corrected, of hypothetical individuals described in short vignettes. Since actual levels of the vignettes are invariant over respondents, variability in vignette answers reveals incomparability. Our corrections require either simple recodes or a statistical model designed to save survey administration costs. With analysis, simulations, and cross-national surveys, we show how response incomparability can drastically mislead survey researchers and how our approach can fix them. %B American Political Science Review %V 98 %P 191–207 %8 February %G eng %0 Journal Article %D 2004 %T Theory and Evidence in International Conflict: A Response to de Marchi, Gelpi, and Grynaviski %A Nathaniel Beck %A Gary King %A Langche Zeng %X We thank Scott de Marchi, Christopher Gelpi, and Jeffrey Grynaviski (2003 and hereinafter dGG) for their careful attention to our work (Beck, King, and Zeng, 2000 and hereinafter BKZ) and for raising some important methodological issues that we agree deserve readers’ attention. We are pleased that dGG’s analyses are consistent with the theoretical conjecture about international conflict put forward in BKZ –- "The causes of conflict, theorized to be important but often found to be small or ephemeral, are indeed tiny for the vast majority of dyads, but they are large stable and replicable whenever the ex ante probability of conflict is large" (BKZ, p.21) –- and that dGG agree with our main methodological point that out-of-sample forecasting performance should always be one of the standards used to judge studies of international conflict, and indeed most other areas of political science. However, dGG frequently err when they draw methodological conclusions. Their central claim involves the superiority of logit over neural network models for international conflict data, as judged by forecasting performance and other properties such as ease of use and interpretation ("neural networks hold few unambiguous advantages... and carry significant costs" relative to logit and dGG, p.14). We show here that this claim, which would be regarded as stunning in any of the diverse fields in which both methods are more commonly used, is false. We also show that dGG’s methodological errors and the restrictive model they favor cause them to miss and mischaracterize crucial patterns in the causes of international conflict. We begin in the next section by summarizing the growing support for our conjecture about international conflict. The second section discusses the theoretical reasons why neural networks dominate logistic regression, correcting a number of methodological errors. The third section then demonstrates empirically, in the same data as used in BKZ and dGG, that neural networks substantially outperform dGG’s logit model. We show that neural networks improve on the forecasts from logit as much as logit improves on a model with no theoretical variables. We also show how dGG’s logit analysis assumed, rather than estimated, the answer to the central question about the literature’s most important finding, the effect of democracy on war. Since this and other substantive assumptions underlying their logit model are wrong, their substantive conclusion about the democratic peace is also wrong. The neural network models we used in BKZ not only avoid these difficulties, but they, or one of the other methods available that do not make highly restrictive assumptions about the exact functional form, are just what is called for to study the observable implications of our conjecture. %V 98 %P 379-389 %8 May %G eng %0 Journal Article %J Sociological Methods and Research %D 2004 %T What to do When Your Hessian is Not Invertible: Alternatives to Model Respecification in Nonlinear Estimation %A Jeff Gill %A Gary King %X What should a researcher do when statistical analysis software terminates before completion with a message that the Hessian is not invertable? The standard textbook advice is to respecify the model, but this is another way of saying that the researcher should change the question being asked. Obviously, however, computer programs should not be in the business of deciding what questions are worthy of study. Although noninvertable Hessians are sometimes signals of poorly posed questions, nonsensical models, or inappropriate estimators, they also frequently occur when information about the quantities of interest exists in the data, through the likelihood function. We explain the problem in some detail and lay out two preliminary proposals for ways of dealing with noninvertable Hessians without changing the question asked. %B Sociological Methods and Research %V 32 %P 54-87 %8 August %G eng %0 Journal Article %J Perspectives on Politics %D 2004 %T Did Illegal Overseas Absentee Ballots Decide the 2000 U.S. Presidential Election? %A Kosuke Imai %A Gary King %X
Although not widely known until much later, Al Gore received 202 more votes than George W. Bush on election day in Florida. George W. Bush is president because he overcame his election day deficit with overseas absentee ballots that arrived and were counted after election day. In the final official tally, Bush received 537 more votes than Gore. These numbers are taken from the official results released by the Florida Secretary of State's office and so do not reflect overvotes, undervotes, unsuccessful litigation, butterfly ballot problems, recounts that might have been allowed but were not, or any other hypothetical divergence between voter preferences and counted votes. After the election, the New York Times conducted a six month long investigation and found that 680 of the overseas absentee ballots were illegally counted, and no partisan, pundit, or academic has publicly disagreed with their assessment. In this paper, we describe the statistical procedures we developed and implemented for the Times to ascertain whether disqualifying these 680 ballots would have changed the outcome of the election. The methods involve adding formal Bayesian model averaging procedures to King's (1997) ecological inference model. Formal Bayesian model averaging has not been used in political science but is especially useful when substantive conclusions depend heavily on apparently minor but indefensible model choices, when model generalization is not feasible, and when potential critics are more partisan than academic. We show how we derived the results for the Times so that other scholars can use these methods to make ecological inferences for other purposes. We also present a variety of new empirical results that delineate the precise conditions under which Al Gore would have been elected president, and offer new evidence of the striking effectiveness of the Republican effort to convince local election officials to count invalid ballots in Bush counties and not count them in Gore counties.%B Perspectives on Politics %V 2 %P 537–549 %8 September %G eng %0 Journal Article %J International Organization %D 2003 %T An Automated Information Extraction Tool For International Conflict Data with Performance as Good as Human Coders: A Rare Events Evaluation Design %A Gary King %A Will Lowe %X Despite widespread recognition that aggregated summary statistics on international conflict and cooperation miss most of the complex interactions among nations, the vast majority of scholars continue to employ annual, quarterly, or occasionally monthly observations. Daily events data, coded from some of the huge volume of news stories produced by journalists, have not been used much for the last two decades. We offer some reason to change this practice, which we feel should lead to considerably increased use of these data. We address advances in event categorization schemes and software programs that automatically produce data by "reading" news stories without human coders. We design a method that makes it feasible for the first time to evaluate these programs when they are applied in areas with the particular characteristics of international conflict and cooperation data, namely event categories with highly unequal prevalences, and where rare events (such as highly conflictual actions) are of special interest. We use this rare events design to evaluate one existing program, and find it to be as good as trained human coders, but obviously far less expensive to use. For large scale data collections, the program dominates human coding. Our new evaluative method should be of use in international relations, as well as more generally in the field of computational linguistics, for evaluating other automated information extraction tools. We believe that the data created by programs similar to the one we evaluated should see dramatically increased use in international relations research. To facilitate this process, we are releasing with this article data on 4.3 million international events, covering the entire world for the last decade. %B International Organization %V 57 %P 617-642 %8 July %G eng %0 Journal Article %J Journal of Legal Education %D 2003 %T Building An Infrastructure for Empirical Research in the Law %A Lee Epstein %A Gary King %X In every discipline in which "empirical research" has become commonplace, scholars have formed a subfield devoted to solving the methodological problems unique to that discipline’s data and theoretical questions. Although students of economics, political science, psychology, sociology, business, education, medicine, public health, and so on primarily focus on specific substantive questions, they cannot wait for those in other fields to solve their methoodological problems or to teach them "new" methods, wherever they were initially developed. In "The Rules of Inference," we argued for the creation of an analogous methodological subfield devoted to legal scholarship. We also had two other objectives: (1) to adapt the rules of inference used in the natural and social sciences, which apply equally to quantitative and qualitative research, to the special needs, theories, and data in legal scholarship, and (2) to offer recommendations on how the infrastructure of teaching and research at law schools might be reorganized so that it could better support the creation of first-rate quantitative and qualitative empirical research without compromising other important objectives. Published commentaries on our paper, along with citations to it, have focused largely on the first-our application of the rules of inference to legal scholarship. Until now, discussions of our second goal-suggestions for the improvement of legal scholarship, as well as our argument for the creation of a group that would focus on methodological problems unique to law-have been relegated to less public forums, even though, judging from the volume of correspondence we have received, they seem to be no less extensive. %B Journal of Legal Education %V 53 %P 311–320 %G eng %0 Journal Article %J International Studies Perspectives %D 2003 %T The Future of Replication %A Gary King %X
Since the replication standard was proposed for political science research, more journals have required or encouraged authors to make data available, and more authors have shared their data. The calls for continuing this trend are more persistent than ever, and the agreement among journal editors in this Symposium continues this trend. In this article, I offer a vision of a possible future of the replication movement. The plan is to implement this vision via the Virtual Data Center project, which – by automating the process of finding, sharing, archiving, subsetting, converting, analyzing, and distributing data – may greatly facilitate adherence to the replication standard.%B International Studies Perspectives %V 4 %P 443–499 %8 February %G eng %0 Journal Article %J Journal of Statistical Software %D 2003 %T ReLogit: Rare Events Logistic Regression %A Michael Tomz %A Gary King %A Langche Zeng %B Journal of Statistical Software %V 8 %G eng %U http://gking.harvard.edu/relogit %0 Journal Article %J Political Analysis %D 2003 %T Analyzing Second Stage Ecological Regressions %A Christopher Adolph %A Gary King %B Political Analysis %V 11 %P 65-76 %8 Winter %G eng %0 Journal Article %J Political Analysis %D 2003 %T A Consensus on Second Stage Analyses in Ecological Inference Models %A Christopher Adolph %A Gary King %A Kenneth W. Shotts %A Michael C. Herron %X Since Herron and Shotts (2003a and hereinafter HS), Adolph and King (2003 andhereinafter AK), and Herron and Shotts (2003b and hereinafter HS2), the four of us have iterated many more times, learned a great deal, and arrived at a consensus on this issue. This paper describes our joint recommendations for how to run second-stage ecological regressions, and provides detailed analyses to back up our claims. %B Political Analysis %V 11 %P 86–94 %8 Winter %G eng %0 Journal Article %J Proceedings of the 10th Conference of the European Chapter of the Association for Computational Linguistics %D 2003 %T Some Statistical Methods for Evaluating Information Extraction Systems %A Will Lowe %A Gary King %X
We present new statistical methods for evaluating information extraction systems. The methods were developed to evaluate a system used by political scientists to extract event information from news leads about international politics. The nature of this data presents two problems for evaluators: 1) the frequency distribution of event types in international event data is strongly skewed, so a random sample of newsleads will typically fail to contain any low frequency events. 2) Manual information extraction necessary to create evaluation sets is costly, and most effort is wasted coding high frequency categories . We present an evaluation scheme that overcomes these problems with considerably less manual effort than traditional methods, and also allows us to interpret an information extraction system as an estimator (in the statistical sense) and to estimate its bias.%B Proceedings of the 10th Conference of the European Chapter of the Association for Computational Linguistics %P 19-26 %G eng %0 Journal Article %J Statistics in Medicine %D 2002 %T Estimating Risk and Rate Levels, Ratios, and Differences in Case-Control Studies %A Gary King %A Langche Zeng %X Classic (or "cumulative") case-control sampling designs do not admit inferences about quantities of interest other than risk ratios, and then only by making the rare events assumption. Probabilities, risk differences, and other quantities cannot be computed without knowledge of the population incidence fraction. Similarly, density (or "risk set") case-control sampling designs do not allow inferences about quantities other than the rate ratio. Rates, rate differences, cumulative rates, risks, and other quantities cannot be estimated unless auxiliary information about the underlying cohort such as the number of controls in each full risk set is available. Most scholars who have considered the issue recommend reporting more than just the relative risks and rates, but auxiliary population information needed to do this is not usually available. We address this problem by developing methods that allow valid inferences about all relevant quantities of interest from either type of case-control study when completely ignorant of or only partially knowledgeable about relevant auxiliary population information. %B Statistics in Medicine %V 21 %P 1409–1427 %G eng %0 Journal Article %J BioMed Central: International Journal for Equity in Health %D 2002 %T Measuring Total Health Inequality: Adding Individual Variation to Group-Level Differences %A Emmanuela Gakidou %A Gary King %X Background: Studies have revealed large variations in average health status across social, economic, and other groups. No study exists on the distribution of the risk of ill-health across individuals, either within groups or across all people in a society, and as such a crucial piece of total health inequality has been overlooked. Some of the reason for this neglect has been that the risk of death, which forms the basis for most measures, is impossible to observe directly and difficult to estimate. Methods: We develop a measure of total health inequality – encompassing all inequalities among people in a society, including variation between and within groups – by adapting a beta-binomial regression model. We apply it to children under age two in 50 low- and middle-income countries. Our method has been adopted by the World Health Organization and is being implemented in surveys around the world and preliminary estimates have appeared in the World Health Report (2000). Results: Countries with similar average child mortality differ considerably in total health inequality. Liberia and Mozambique have the largest inequalities in child survival, while Colombia, the Philippines and Kazakhstan have the lowest levels among the countries measured. Conclusions: Total health inequality estimates should be routinely reported alongside average levels of health in populations and groups, as they reveal important policy-related information not otherwise knowable. This approach enables meaningful comparisons of inequality across countries and future analyses of the determinants of inequality. %B BioMed Central: International Journal for Equity in Health %V 1 %8 August %G eng %0 Journal Article %J BMJ (British Medical Journal) %D 2002 %T Armed Conflict as a Public Health Problem %A Christopher J.L. Murray %A Gary King %A Alan D. Lopez %A Niels Tomijima %A Etienne Krug %X Armed conflict is a major cause of injury and death worldwide, but we need much better methods of quantification before we can accurately assess its effect. Armed conflict between warring states and groups within states have been major causes of ill health and mortality for most of human history. Conflict obviously causes deaths and injuries on the battlefield, but also health consequences from the displacement of populations, the breakdown of health and social services, and the heightened risk of disease transmission. Despite the size of the health consequences, military conflict has not received the same attention from public health research and policy as many other causes of illness and death. In contrast, political scientists have long studied the causes of war but have primarily been interested in the decision of elite groups to go to war, not in human death and misery. We review the limited knowledge on the health consequences of conflict, suggest ways to improve measurement, and discuss the potential for risk assessment and for preventing and ameliorating the consequences of conflict. %B BMJ (British Medical Journal) %V 324 %P 346–349 %8 February 9 %G eng %0 Journal Article %J Political Analysis %D 2002 %T Isolating Spatial Autocorrelation, Aggregation Bias, and Distributional Violations in Ecological Inference %A Gary King %X This is an invited response to an article by Anselin and Cho. I make two main points: The numerical results in this article violate no conclusions from prior literature, and the absence of the deterministic information from the bounds in the article’s analyses invalidates its theoretical discussion of spatial autocorrelation and all of its actual simulation results. An appendix shows how to draw simulations correctly. %B Political Analysis %V 10 %P 298–300 %8 Summer %G eng %0 Journal Article %J Political Science Quarterly %D 2002 %T Rethinking Human Security %A Gary King %A Christopher J.L. Murray %X In the last two decades, the international community has begun to conclude that attempts to ensure the territorial security of nation-states through military power have failed to improve the human condition. Despite astronomical levels of military spending, deaths due to military conflict have not declined. Moreover, even when the borders of some states are secure from foreign threats, the people within those states do not necessarily have freedom from crime, enough food, proper health care, education, or political freedom. In response to these developments, the international community has gradually moved to combine economic development with military security and other basic human rights to form a new concept of "human security". Unfortunately, by common assent the concept lacks both a clear definition, consistent with the aims of the international community, and any agreed upon measure of it. In this paper, we propose a simple, rigorous, and measurable definition of human security: the expected number of years of future life spent outside the state of "generalized poverty". Generalized poverty occurs when an individual falls below the threshold in any key domain of human well-being. We consider improvements in data collection and methods of forecasting that are necessary to measure human security and then introduce an agenda for research and action to enhance human security that follows logically in the areas of risk assessment, prevention, protection, and compensation. %B Political Science Quarterly %V 116 %P 585–610 %8 Winter %G eng %0 Journal Article %J University of Chicago Law Review %D 2002 %T The Rules of Inference %A Lee Epstein %A Gary King %X
Although the term "empirical research" has become commonplace in legal scholarship over the past two decades, law professors have, in fact, been conducting research that is empirical – that is, learning about the world using quantitative data or qualitative information – for almost as long as they have been conducting research. For just as long, however, they have been proceeding with little awareness of, much less compliance with, the rules of inference, and without paying heed to the key lessons of the revolution in empirical analysis that has been taking place over the last century in other disciplines. The tradition of including some articles devoted to exclusively to the methododology of empirical analysis – so well represented in journals in traditional academic fields – is virtually nonexistent in the nation’s law reviews. As a result, readers learn considerably less accurate information about the empirical world than the studies’ stridently stated, but overconfident, conclusions suggest. To remedy this situation both for the producers and consumers of empirical work, this Article adapts the rules of inference used in the natural and social sciences to the special needs, theories, and data in legal scholarship, and explicate them with extensive illustrations from existing research. The Article also offers suggestions for how the infrastructure of teaching and research at law schools might be reorganized so that it can better support the creation of first-rate empirical research without compromising other important objectives.%B University of Chicago Law Review %V 69 %P 1–209 %8 Winter %G eng %0 Journal Article %J University of Chicago Law Review %D 2002 %T Empirical Research and The Goals of Legal Scholarship: A Response %A Lee Epstein %A Gary King %X Although the term "empirical research" has become commonplace in legal scholarship over the past two decades, law professors have, in fact, been conducting research that is empirical – that is, learning about the world using quantitative data or qualitative information – for almost as long as they have been conducting research. For just as long, however, they have been proceeding with little awareness of, much less compliance with, the rules of inference, and without paying heed to the key lessons of the revolution in empirical analysis that has been taking place over the last century in other disciplines. The tradition of including some articles devoted to exclusively to the methododology of empirical analysis – so well represented in journals in traditional academic fields – is virtually nonexistent in the nation’s law reviews. As a result, readers learn considerably less accurate information about the empirical world than the studies’ stridently stated, but overconfident, conclusions suggest. To remedy this situation both for the producers and consumers of empirical work, this Article adapts the rules of inference used in the natural and social sciences to the special needs, theories, and data in legal scholarship, and explicate them with extensive illustrations from existing research. The Article also offers suggestions for how the infrastructure of teaching and research at law schools might be reorganized so that it can better support the creation of first-rate empirical research without compromising other important objectives. %B University of Chicago Law Review %V 69 %P 1–209 %8 Winter %G eng %0 Journal Article %J Political Analysis %D 2002 %T A Fast, Easy, and Efficient Estimator for Multiparty Electoral Data %A James Honaker %A Gary King %A Jonathan N. Katz %X Katz and King (1999) develop a model for predicting or explaining aggregate electoral results in multiparty democracies. This model is, in principle, analogous to what least squares regression provides American politics researchers in that two-party system. Katz and King applied this model to three-party elections in England and revealed a variety of new features of incumbency advantage and where each party pulls support from. Although the mathematics of their statistical model covers any number of political parties, it is computationally very demanding, and hence slow and numerically imprecise, with more than three. The original goal of our work was to produce an approximate method that works quicker in practice with many parties without making too many theoretical compromises. As it turns out, the method we offer here improves on Katz and King’s (in bias, variance, numerical stability, and computational speed) even when the latter is computationally feasible. We also offer easy-to-use software that implements our suggestions. %B Political Analysis %V 10 %P 84–100 %8 Winter %G eng %0 Journal Article %J Proceedings of The First ACM+IEEE Joint Conference on Digital Libraries %D 2001 %T An Introduction to the Virtual Data Center Project and Software %A Micah Altman %A Leonid Andreev %A Mark Diggory %A Gary King %A Elizabeth Kolster %A Krot, M. %A Sidney Verba %A Daniel L. Kiskis %B Proceedings of The First ACM+IEEE Joint Conference on Digital Libraries %I ACM Press %P 203–204 %G eng %0 Journal Article %J American Political Science Review %D 2001 %T Analyzing Incomplete Political Science Data: An Alternative Algorithm for Multiple Imputation %A Gary King %A James Honaker %A Anne Joseph %A Kenneth Scheve %X
We propose a remedy for the discrepancy between the way political scientists analyze data with missing values and the recommendations of the statistics community. Methodologists and statisticians agree that "multiple imputation" is a superior approach to the problem of missing data scattered through one’s explanatory and dependent variables than the methods currently used in applied data analysis. The discrepancy occurs because the computational algorithms used to apply the best multiple imputation models have been slow, difficult to implement, impossible to run with existing commercial statistical packages, and have demanded considerable expertise. We adapt an algorithm and use it to implement a general-purpose, multiple imputation model for missing data. This algorithm is considerably easier to use than the leading method recommended in statistics literature. We also quantify the risks of current missing data practices, illustrate how to use the new procedure, and evaluate this alternative through simulated data as well as actual empirical examples. Finally, we offer easy-to-use that implements our suggested methods. (Software: AMELIA)%B American Political Science Review %V 95 %P 49–69 %8 March %G eng %0 Journal Article %J Statistica Neerlandica %D 2001 %T Bayesian and Frequentist Inference for Ecological Inference: The RxC Case %A Ori Rosen %A Wenxin Jiang %A Gary King %A Martin A. Tanner %X In this paper we propose Bayesian and frequentist approaches to ecological inference, based on R x C contingency tables, including a covariate. The proposed Bayesian model extends the binomial-beta hierarchical model developed by King, Rosen and Tanner (1999) from the 2 x 2 case to the R x C case, the inferential procedure employs Markov chain Monte Carlo (MCMC) methods. As such the resulting MCMC analysis is rich but computationally intensive. The frequentist approach, based on first moments rather than on the entire likelihood, provides quick inference via nonlinear least-squares, while retaining good frequentist properties. The two approaches are illustrated with simulated data, as well as with real data on voting patterns in Weimar Germany. In the final section of the paper we provide an overview of a range of alternative inferential approaches which trade-off computational intensity for statistical efficiency. %B Statistica Neerlandica %V 55 %P 134–156 %G eng %0 Journal Article %J Political Analysis %D 2001 %T Logistic Regression in Rare Events Data %A Gary King %A Langche Zeng %X We study rare events data, binary dependent variables with dozens to thousands of times fewer ones (events, such as wars, vetoes, cases of political activism, or epidemiological infections) than zeros ("nonevents"). In many literatures, these variables have proven difficult to explain and predict, a problem that seems to have at least two sources. First, popular statistical procedures, such as logistic regression, can sharply underestimate the probability of rare events. We recommend corrections that outperform existing methods and change the estimates of absolute and relative risks by as much as some estimated effects reported in the literature. Second, commonly used data collection strategies are grossly inefficient for rare events data. The fear of collecting data with too few events has led to data collections with huge numbers of observations but relatively few, and poorly measured, explanatory variables, such as in international conflict data with more than a quarter-million dyads, only a few of which are at war. As it turns out, more efficient sampling designs exist for making valid inferences, such as sampling all variable events (e.g., wars) and a tiny fraction of nonevents (peace). This enables scholars to save as much as 99% of their (nonfixed) data collection costs or to collect much more meaningful explanatory variables. We provide methods that link these two results, enabling both types of corrections to work simultaneously, and software that implements the methods developed. %B Political Analysis %V 9 %P 137–163 %8 Spring %G eng %0 Journal Article %J JCDL ’01: First Joint Conference on Digital Libraries %D 2001 %T An Overview of the Virtual Data Center Project and Software %A Micah Altman %A Leonid Andreev %A Mark Diggory %A Gary King %A Daniel L. Kiskis %A Elizabeth Kolster %A Michael Krot %A Sidney Verba %X In this paper, we present an overview of the Virtual Data Center (VDC) software, an open-source digital library system for the management and dissemination of distributed collections of quantitative data. (see http://TheData.org). The VDC functionality provides everything necessary to maintain and disseminate an individual collection of research studies, including facilities for the storage, archiving, cataloging, translation, and on-line analysis of a particular collection. Moreover, the system provides extensive support for distributed and federated collections including: location-independent naming of objects, distributed authentication and access control, federated metadata harvesting, remote repository caching, and distributed "virtual" collections of remote objects. %B JCDL ’01: First Joint Conference on Digital Libraries %P 203-204 %G eng %0 Journal Article %J International Organization %D 2001 %T Proper Nouns and Methodological Propriety: Pooling Dyads in International Relations Data %A Gary King %X The intellectual stakes at issue in this symposium are very high: Green, Kim, and Yoon (2000 and hereinafter GKY) apply their proposed methodological prescriptions and conclude that they key findings in the field is wrong and democracy "has no effect on militarized disputes." GKY are mainly interested in convincing scholars about their methodological points and see themselves as having no stake in the resulting substantive conclusions. However, their methodological points are also high stakes claims: if correct, the vast majority of statistical analyses of military conflict ever conducted would be invalidated. GKY say they "make no attempt to break new ground statistically," but, as we will see, this both understates their methodological contribution to the field and misses some unique features of their application and data in international relations. On the ltter, GKY’s critics are united: Oneal and Russett (2000) conclude that GKY’s method "produces distorted results," and show even in GKY’s framework how democracy’s effect can be reinstated. Beck and Katz (2000) are even more unambiguous: "GKY’s conclusion, in table 3, that variables such as democracy have no pacific impact, is simply nonsense...GKY’s (methodological) proposal...is NEVER a good idea." My given task is to sort out and clarify these conflicting claims and counterclaims. The procedure I followed was to engage in extensive discussions with the participants that included joint reanalyses provoked by our discussions and passing computer program code (mostly with Monte Carlo simulations) back and forth to ensure we were all talking about the same methods and agreed with the factual results. I learned a great deal from this process and believe that the positions of the participants are now a lot closer than it may seem from their written statements. Indeed, I believe that all the participants now agree with what I have written here, even though they would each have different emphases (and although my believing there is agreement is not the same as there actually being agreement!). %B International Organization %V 55 %P 497–507 %8 Fall %G eng %0 Journal Article %J Political Analysis %D 2001 %T Aggregation Among Binary, Count, and Duration Models: Estimating the Same Quantities from Different Levels of Data %A James E. Alt %A Gary King %A Signorino, Curtis %X Binary, count and duration data all code discrete events occurring at points in time. Although a single data generation process can produce all of these three data types, the statistical literature is not very helpful in providing methods to estimate parameters of the same process from each. In fact, only single theoretical process exists for which know statistical methods can estimate the same parameters - and it is generally used only for count and duration data. The result is that seemingly trivial decisions abut which level of data to use can have important consequences for substantive interpretations. We describe the theoretical event process for which results exist, based on time independence. We also derive a set of models for a time-dependent process and compare their predictions to those of a commonly used model. Any hope of understanding and avoiding the more serious problems of aggregation bias in events data is contingent on first deriving a much wider arsenal of statistical models and theoretical processes that are not constrained by the particular forms of data that happen to be available. We discuss these issues and suggest an agenda for political methodologists interested in this very large class of aggregation problems. %B Political Analysis %V 9 %P 21–44 %8 Winter %G eng %0 Journal Article %J Social Science Computer Review %D 2001 %T A Digital Library for the Dissemination and Replication of Quantitative Social Science Research %A Micah Altman %A Leonid Andreev %A Mark Diggory %A Gary King %A Daniel L. Kiskis %A Elizabeth Kolster %A Michael Krot %A Sidney Verba %X The Virtual Data Center (VDC) software is an open-source, digital library system for quantitative data. We discuss what the software does, and how it provides an infrastructure for the management and dissemination of disturbed collections of quantitative data, and the replication of results derived from this data. %B Social Science Computer Review %V 19 %P 458–470 %8 Winter %G eng %0 Journal Article %J International Organization %D 2001 %T Explaining Rare Events in International Relations %A Gary King %A Langche Zeng %X Some of the most important phenomena in international conflict are coded s "rare events data," binary dependent variables with dozens to thousands of times fewer events, such as wars, coups, etc., than "nonevents". Unfortunately, rare events data are difficult to explain and predict, a problem that seems to have at least two sources. First, and most importantly, the data collection strategies used in international conflict are grossly inefficient. The fear of collecting data with too few events has led to data collections with huge numbers of observations but relatively few, and poorly measured, explanatory variables. As it turns out, more efficient sampling designs exist for making valid inferences, such as sampling all available events (e.g., wars) and a tiny fraction of non-events (peace). This enables scholars to save as much as 99% of their (non-fixed) data collection costs, or to collect much more meaningful explanatory variables. Second, logistic regression, and other commonly used statistical procedures, can underestimate the probability of rare events. We introduce some corrections that outperform existing methods and change the estimates of absolute and relative risks by as much as some estimated effects reported in the literature. We also provide easy-to-use methods and software that link these two results, enabling both types of corrections to work simultaneously. %B International Organization %V 55 %P 693–715 %8 Summer %G eng %0 Journal Article %J World Politics %D 2001 %T Improving Forecasts of State Failure %A Gary King %A Langche Zeng %X
We offer the first independent scholarly evaluation of the claims, forecasts, and causal inferences of the State Failure Task Force and their efforts to forecast when states will fail. State failure refers to the collapse of the authority of the central government to impose order, as in civil wars, revolutionary wars, genocides, politicides, and adverse or disruptive regime transitions. This task force, set up at the behest of Vice President Gore in 1994, has been led by a group of distinguished academics working as consultants to the U.S. Central Intelligence Agency. State Failure Task Force reports and publications have received attention in the media, in academia, and from public policy decision-makers. In this article, we identify several methodological errors in the task force work that cause their reported forecast probabilities of conflict to be too large, their causal inferences to be biased in unpredictable directions, and their claims of forecasting performance to be exaggerated. However, we also find that the task force has amassed the best and most carefully collected data on state failure in existence, and the required corrections which we provide, although very large in effect, are easy to implement. We also reanalyze their data with better statistical procedures and demonstrate how to improve forecasting performance to levels significantly greater than even corrected versions of their models. Although still a highly uncertain endeavor, we are as a consequence able to offer the first accurate forecasts of state failure, along with procedures and results that may be of practical use in informing foreign policy decision making. We also describe a number of strong empirical regularities that may help in ascertaining the causes of state failure.%B World Politics %V 53 %P 623–658 %8 July %G eng %0 Journal Article %J Annals of the Association of American Geographers %D 2000 %T Geography, Statistics, and Ecological Inference %A Gary King %X I am grateful for such thoughtful review from these three distinguished geographers. Fotheringham provides an excellent summary of the approach offered, including how it combines the two methods that have dominated applications (and methodological analysis) for nearly half a century– the method of bounds (Duncan and Davis, 1953) and Goodman’s (1953) least squares regression. Since Goodman’s regression is the only method of ecological inference "widely used in Geography" (O’Loughlin), adding information that is known to be true from the method of bounds (for each observation) would seem to have the chance to improve a lot of research in this field. The other addition that EI provides is estimates at the lowest level of geography available, making it possible to map results, instead of giving only single summary numbers for the entire geographic region. Whether one considers the combined method offered "the" solution (as some reviewers and commentators have portrayed it), "a" solution (as I tried to describe it), or, perhaps better and more simply, as an improved method of ecological inference, is not importatnt. The point is that more data are better, and this method incorporates more. I am gratified that all three reviewers seem to support these basic points. In this response, I clarify a few points, correct some misunderstandings, and present additional evidence. I conclude with some possible directions for future research. %B Annals of the Association of American Geographers %V 90 %P 601–606 %8 September %G eng %0 Journal Article %J American Political Science Review %D 2000 %T Improving Quantitative Studies of International Conflict: A Conjecture %A Nathaniel Beck %A Gary King %A Langche Zeng %X We address a well-known but infrequently discussed problem in the quantitative study of international conflict: Despite immense data collections, prestigious journals, and sophisticated analyses, empirical findings in the literature on international conflict are often unsatisfying. Many statistical results change from article to article and specification to specification. Accurate forecasts are nonexistant. In this article we offer a conjecture about one source of this problem: The causes of conflict, theorized to be important but often found to be small or ephemeral, are indeed tiny for the vast majority of dyads, but they are large, stable, and replicable wherever the ex ante probability of conflict is large. This simple idea has an unexpectedly rich array of observable implications, all consistent with the literature. We directly test our conjecture by formulating a statistical model that includes critical features. Our approach, a version of a "neural network" model, uncovers some interesting structural features of international conflict, and as one evaluative measure, forecasts substantially better than any previous effort. Moreover, this improvement comes at little cost, and it is easy to evaluate whether the model is a statistical improvement over the simpler models commonly used. %B American Political Science Review %V 94 %P 21–36 %8 March %G eng %0 Journal Article %J American Journal of Political Science %D 2000 %T Making the Most of Statistical Analyses: Improving Interpretation and Presentation %A Gary King %A Michael Tomz %A Jason Wittenberg %X Social Scientists rarely take full advantage of the information available in their statistical results. As a consequence, they miss opportunities to present quantities that are of greatest substantive interest for their research and express the appropriate degree of certainty about these quantities. In this article, we offer an approach, built on the technique of statistical simulation, to extract the currently overlooked information from any statistical method and to interpret and present it in a reader-friendly manner. Using this technique requires some expertise, which we try to provide herein, but its application should make the results of quantitative articles more informative and transparent. To illustrate our recommendations, we replicate the results of several published works, showing in each case how the authors’ own conclusions can be expressed more sharply and informatively, and, without changing any data or statistical assumptions, how our approach reveals important new information about the research questions at hand. We also offer very easy-to-use Clarify software that implements our suggestions. %B American Journal of Political Science %V 44 %P 341–355 %8 April %G eng %U http://gking.harvard.edu/files/abs/making-abs.shtml %0 Journal Article %J Sociological Methods and Research %D 1999 %T Binomial-Beta Hierarchical Models for Ecological Inference %A Gary King %A Ori Rosen %A Martin A. Tanner %X The authors develop binomial-beta hierarchical models for ecological inference using insights from the literature on hierarchical models based on Markov chain Monte Carlo algorithms and King’s ecological inference model. The new approach reveals some features of the data that King’s approach does not, can easily be generalized to more complicated problems such as general R x C tables, allows the data analyst to adjust for covariates, and provides a formal evaluation of the significance of the covariates. It may also be better suited to cases in which the observed aggregate cells are estimated from very few observations or have some forms of measurement error. This article also provides an example of a hierarchical model in which the statistical idea of "borrowing strength" is used not merely to increase the efficiency of the estimates but to enable the data analyst to obtain estimates. %B Sociological Methods and Research %V 28 %P 61–90 %8 August %G eng %0 Journal Article %J Journal of the American Statistical Association %D 1999 %T The Future of Ecological Inference Research: A Reply to Freedman et al. %A Gary King %X I appreciate the editor’s invitation to reply to Freedman et al.’s (1998) review of "A Solution to the Ecological Inference Problem: Reconstructing Individual Behavior from Aggregate Data" (Princeton University Press.) I welcome this scholarly critique and JASA’s decision to publish in this field. Ecological inference is a large and very important area for applications that is especially rich with open statistical questions. I hope this discussion stimulates much new scholarship. Freedman et al. raise several interesting issues, but also misrepresent or misunderstand the prior literature, my approach, and their own empirical analyses, and compound the problem, by refusing requests from me and the editor to make their data and software available for this note. Some clarification is thus in order. %B Journal of the American Statistical Association %V 94 %P 352-355 %8 March %G eng %0 Journal Article %J Electoral Studies %D 1999 %T Many Publications, but Still No Evidence %A Gary King %A Michael Laver %X In 1990, Budge and Hofferbert (B&H) claimed that they had found solid evidence that party platforms cause U.S. budgetary priorities, and thus concluded that mandate theory applies in the United States as strongly as it does elsewhere. The implications of this stunning conclusion would mean that virtually every observer of the American party system in this century has been wrong. King and Laver (1993) reanalyzed B&H’s data and demonstrated in two ways that there exists no evidence for a causal relationship. First, accepting their entire statistical model, and correcting only an algebraic error (a mistake in how they computed their standard errors), we showed that their hypothesized relationship holds up in fewer than half the tests they reported. Second, we showed that their statistical model includes a slightly hidden but politically implausible assumption that a new party achieves every budgetary desire immediately upon taking office. We then specified a model without this unrealistic assumption and we found that the assumption was not supported, and that all evidence in the data for platforms causing government budgets evaporated. In their published response to our article, B&H withdrew their key claim and said they were now (in 1993) merely interested in an association and not causation. That is how it was left in 1993—a perfectly amicable resolution as far as we were concerned—since we have no objection to the claim that there is a non-causal or chance association between any two variables. Of course, we see little reason to be interested in non-causal associations in this area any more than in the chance correlation that exists between the winner of the baseball World Series and the party winning the U.S. presidency. Since party mandate theory only makes sense as a causal theory, the conventional wisdom about America’s porous, non-mandate party system stands. %B Electoral Studies %V 18 %P 597–598 %8 December %G eng %0 Journal Article %J Political Analysis %D 1999 %T No Evidence on Directional vs. Proximity Voting %A Jeffrey Lewis %A Gary King %X The directional and proximity models offer dramatically different theories for how voters make decisions and fundamentally divergent views of the supposed microfoundations on which vast bodies of literature in theoretical rational choice and empirical political behavior have been built. We demonstrate here that the empirical tests in the large and growing body of literature on this subject amount to theoretical debates about which statistical assumption is right. The key statistical assumptions have not been empirically tested and, indeed, turn out to be effectively untestable with exiting methods and data. Unfortunately, these assumptions are also crucial since changing them leads to different conclusions about voter processes. %B Political Analysis %V 8 %P 21–33 %8 August %G eng %0 Journal Article %J American Political Science Review %D 1999 %T A Statistical Model for Multiparty Electoral Data %A Jonathan Katz %A Gary King %X We propose a comprehensive statistical model for analyzing multiparty, district-level elections. This model, which provides a tool for comparative politics research analagous to that which regression analysis provides in the American two-party context, can be used to explain or predict how geographic distributions of electoral results depend upon economic conditions, neighborhood ethnic compositions, campaign spending, and other features of the election campaign or aggregate areas. We also provide new graphical representations for data exploration, model evaluation, and substantive interpretation. We illustrate the use of this model by attempting to resolve a controversy over the size of and trend in electoral advantage of incumbency in Britain. Contrary to previous analyses, all based on measures now known to be biased, we demonstrate that the advantage is small but meaningful, varies substantially across the parties, and is not growing. Finally, we show how to estimate the party from which each party’s advantage is predominantly drawn. %B American Political Science Review %V 93 %P 15–32 %8 March %G eng %0 Journal Article %J Journal of the American Statistical Association %D 1999 %T Not Asked and Not Answered: Multiple Imputation for Multiple Surveys %A Andrew Gelman %A Gary King %A Chuanhai Liu %X We present a method of analyzing a series of independent cross-sectional surveys in which some questions are not answered in some surveys and some respondents do not answer some of the questions posed. The method is also applicable to a single survey in which different questions are asked or different sampling methods are used in different strata or clusters. Our method involves multiply imputing the missing items and questions by adding to existing methods of imputation designed for single surveys a hierarchical regression model that allows covariates at the individual and survey levels. Information from survey weights is exploited by including in the analysis the variables on which the weights are based, and then reweighting individual responses (observed and imputed) to estimate population quantities. We also develop diagnostics for checking the fit of the imputation model based on comparing imputed data to nonimputed data. We illustrate with the example that motivated this project: a study of pre-election public opinion polls in which not all the questions of interest are asked in all the surveys, so that it is infeasible to impute within each survey separately. %B Journal of the American Statistical Association %V 93 %P 846–857 %8 September %G eng %0 Journal Article %J Sociological Methods and Research %D 1998 %T The Record of American Democracy, 1984-1990 %A Gary King %A Bradley Palmquist %B Sociological Methods and Research %V 26 %P 424–427 %8 February %G eng %U http://www.hmdc.harvard.edu/ROAD %0 Journal Article %J Journal of the American Statistical Association %D 1998 %T Estimating the Probability of Events that Have Never Occurred: When Is Your Vote Decisive? %A Andrew Gelman %A Gary King %A John Boscardin %X Researchers sometimes argue that statisticians have little to contribute when few realizations of the process being estimated are observed. We show that this argument is incorrect even in the extreme situation of estimating the probabilities of events so rare that they have never occurred. We show how statistical forecasting models allow us to use empirical data to improve inferences about the probabilities of these events. Our application is estimating the probability that your vote will be decisive in a U.S. presidential election, a problem that has been studied by political scientists for more than two decades. The exact value of this probability is of only minor interest, but the number has important implications for understanding the optimal allocation of campaign resources, whether states and voter groups receive their fair share of attention from prospective presidents, and how formal "rational choice" models of voter behavior might be able to explain why people vote at all. We show how the probability of a decisive vote can be estimated empirically from state-level forecasts of the presidential election and illustrate with the example of 1992. Based on generalizations of standard political science forecasting models, we estimate the (prospective) probability of a single vote being decisive as about 1 in 10 million for close national elections such as 1992, varying by about a factor of 10 among states. Our results support the argument that subjective probabilities of many types are best obtained through empirically based statistical prediction models rather than solely through mathematical reasoning. We discuss the implications of our findings for the types of decision analyses used in public choice studies. %B Journal of the American Statistical Association %V 93 %P 1–9 %8 March %G eng %0 Journal Article %J Political Analysis %D 1996 %T The Generalization in the Generalized Event Count Model, With Comments on Achen, Amato, and Londregan %A Gary King %A Curtis S. Signorino %X We use an analogy with the normal distribution and linear regression to demonstrate the need for the Generalize Event Count (GEC) model. We then show how the GEC provides a unified framework within which to understand a diversity of distributions used to model event counts, and how to express the model in one simple equation. Finally, we address the points made by Christopher Achen, Timothy Amato, and John Londregan. Amato's and Londregan's arguments are consistent with ours and provide additional interesting information and explanations. Unfortunately, the foundation on which Achen built his paper turns out to be incorrect, rendering all his novel claims about the GEC false (or in some cases irrelevant). %B Political Analysis %V 6 %P 225–252 %G eng %0 Journal Article %J Social Science Computer Review %D 1996 %T A Preview of EI and EzI: Programs for Ecological Inference %A Kenneth Benoit %A Gary King %X Ecological inference, as traditionally defined, is the process of using aggregate (i.e., "ecological") data to infer discrete individual-level relationships of interest when individual-level data are not available. Existing methods of ecological inference generate very inaccurate conclusions about the empirical world- which thus gives rise to the ecological inference problem. Most scholars who analyze aggregate data routinely encounter some form of this problem. EI (by Gary King) and EzI (by Kenneth Benoit and Gary King) are freely available software that implement the statistical and graphical methods detailed in Gary King’s book A Solution to the Ecological Inference Problem. These methods make it possible to infer the attributes of individual behavior from aggregate data. EI works within the statistics program Gauss and will run on any computer hardware and operating system that runs Gauss (the Gauss module, CML, or constrained maximum likelihood- by Ronald J. Schoenberg- is also required). EzI is a menu-oriented stand-alone version of the program that runs under MS-DOS (and soon Windows 95, OS/2, and HP-UNIX). EI allows users to make ecological inferences as part of the powerful and open Gauss statistical environment. In contrast, EzI requires no additional software, and provides an attractive menu-based user interface for non-Gauss users, although it lacks the flexibility afforded by the Gauss version. Both programs presume that the user has read or is familiar with A Solution to the Ecological Inference Problem. %B Social Science Computer Review %V 14 %P 433–438 %8 Winter %G eng %0 Journal Article %J Political Geography %D 1996 %T Why Context Should Not Count %A Gary King %X
This paper is an invited comment on a paper by John Agnew. I largely agree with Agnew’s comments and thus focus on remaining areas wehre an alternative perspective might be useful. My argument is that political geographers should not be so concerned with demonstrating that context matters. My reasoning is based on three arguments. First, in fact context rarely counts (Section 1) and, second, the most productive practical goal for political researchers should be to show that it does not count (Section 2). Finally, a disproportionate focus on ‘context counting’ can lead, and has led, to some seriosu problems in practical research situations, such as attempting to give theoretical answers to empirical questions (Section 3) and empirical answers to theoretical questions (Section 4).%B Political Geography %V 15 %P 159–164 %G eng %0 Journal Article %J Political Analysis %D 1995 %T A Correction for an Underdispersed Event Count Probability Distribution %A Rainer Winkelmann %A Signorino, Curtis %A Gary King %X We demonstrate that the expected value and variance commonly given for a well-known probability distribution are incorrect. We also provide corrected versions and report changes in a computer program to account for the known practical uses of this distribution. %B Political Analysis %P 215–228 %G eng %0 Journal Article %J American Political Science Review %D 1995 %T The Importance of Research Design in Political Science %A Gary King %A Robert O. Keohane %A Sidney Verba %X Receiving five serious reviews in this symposium is gratifying and confirms our belief that research design should be a priority for our discipline. We are pleased that our five distinguished reviewers appear to agree with our unified approach to the logic of inference in the social sciences, and with our fundamental point: that good quantitative and good qualitative research designs are based fundamentally on the same logic of inference. The reviewers also raised virtually no objections to the main practical contribution of our book– our many specific procedures for avoiding bias, getting the most out of qualitative data, and making reliable inferences. However, the reviews make clear that although our book may be the latest word on research design in political science, it is surely not the last. We are taxed for failing to include important issues in our analysis and for dealing inadequately with some of what we included. Before responding to the reviewers’ more direct criticisms, let us explain what we emphasize in Designing Social Inquiry and how it relates to some of the points raised by the reviewers. %B American Political Science Review %V 89 %P 454–481 %8 June %G eng %0 Journal Article %J PS: Political Science and Politics %D 1995 %T A Revised Proposal, Proposal %A Gary King %B PS: Political Science and Politics %V XXVIII %P 494–499 %8 September %G eng %0 Journal Article %J Public Opinion Quarterly %D 1995 %T Pre-Election Survey Methodology: Details From Nine Polling Organizations, 1988 and 1992 %A D. Steven Voss %A Andrew Gelman %A Gary King %X Before every presidential election, journalists, pollsters, and politicians commission dozens of public opinion polls. Although the primary function of these surveys is to forecast the election winners, they also generate a wealth of political data valuable even after the election. These preelection polls are useful because they are conducted with such frequency that they allow researchers to study change in estimates of voter opinion within very narrow time increments (Gelman and King 1993). Additionally, so many are conducted that the cumulative sample size of these polls is large enough to construct aggregate measures of public opinion within small demographic or geographical groupings (Wright, Erikson, and McIver 1985). %B Public Opinion Quarterly %V 59 %P 98–132 %8 Spring %G eng %0 Journal Article %J PS: Political Science and Politics %D 1995 %T Replication, Replication %A Gary King %X
Political science is a community enterprise and the community of empirical political scientists need access to the body of data necessary to replicate existing studies to understand, evaluate, and especially build on this work. Unfortunately, the norms we have in place now do not encourage, or in some cases even permit, this aim. Following are suggestions that would facilitate replication and are easy to implement – by teachers, students, dissertation writers, graduate programs, authors, reviewers, funding agencies, and journal and book editors.%B PS: Political Science and Politics %V 28 %P 444-452 %8 September %G eng %0 Journal Article %J Comparative Political Studies %D 1994 %T Transfers of Governmental Power: The Meaning of Time Dependence %A James E. Alt %A Gary King %X King, Alt, Burns, and Laver (1990) proposed and estimated a unified model in which cabinet durations depended on seven explanatory variables reflecting features of the cabinets and the bargaining environments in which they formed, along with a stochastic component in which the risk of a cabinet falling was treated as a constant across its tenure. Two recent research reports take issue with one aspect of this model. Warwick and Easton replicate the earlier findings for explanatory variables but claim that the stochastic risk should be seen as rising, and at a rate which varies, across the life of the cabinet. Bienen and van de Walle, using data on the duration of leaders, allege that random risk is falling. We continue in our goal of unifying this literature by providing further estimates with both cabinet and leader duration data that confirm the original explanatory variables’ effects, showing that leaders’ durations are affected by many of the same factors that affect the durability of the cabinets they lead, demonstrating that cabinets have stochastic risk of ending that is indeed constant across the theoretically most interesting range of durations, and suggesting that stochastic risk for leaders in countries with cabinet government is, if not constant, more likely to rise than fall. %B Comparative Political Studies %V 27 %P 190–210 %8 July %G eng %0 Journal Article %J American Journal of Political Science %D 1994 %T A Unified Method of Evaluating Electoral Systems and Redistricting Plans %A Andrew Gelman %A Gary King %X We derive a unified statistical method with which one can produce substantially improved definitions and estimates of almost any feature of two-party electoral systems that can be defined based on district vote shares. Our single method enables one to calculate more efficient estimates, with more trustworthy assessments of their uncertainty, than each of the separate multifarious existing measures of partisan bias, electoral responsiveness, seats-votes curves, expected or predicted vote in each district in a legislature, the probability that a given party will win the seat in each district, the proportion of incumbents or others who will lose their seats, the proportion of women or minority candidates to be elected, the incumbency advantage and other causal effects, the likely effects on the electoral system and district votes of proposed electoral reforms, such as term limitations, campaign spending limits, and drawing majority-minority districts, and numerous others. To illustrate, we estimate the partisan bias and electoral responsiveness of the U.S. House of Representatives since 1900 and evaluate the fairness of competing redistricting plans for the 1992 Ohio state legislature. %B American Journal of Political Science %V 38 %P 514–554 %8 May %G eng %0 Journal Article %J American Political Science Review %D 1994 %T Enhancing Democracy Through Legislative Redistricting %A Andrew Gelman %A Gary King %X We demonstrate the surprising benefits of legislative redistricting (including partisan gerrymandering) for American representative democracy. In so doing, our analysis resolves two long-standing controversies in American politics. First, whereas some scholars believe that redistricting reduces electoral responsiveness by protecting incumbents, others, that the relationship is spurious, we demonstrate that both sides are wrong: redistricting increases responsiveness. Second, while some researchers believe that gerrymandering dramatically increases partisan bias and others deny this effect, we show both sides are in a sense correct. Gerrymandering biases electoral systems in favor of the party that controls the redistricting as compared to what would have happened if the other party controlled it, but any type of redistricting reduces partisan bias as compared to an electoral system without redistricting. Incorrect conclusions in both literatures resulted from misjudging the enormous uncertainties present during redistricting periods, making simplified assumptions about the redistricters’ goals, and using inferior statistical methods. %B American Political Science Review %V 88 %P 541–559 %8 September %G eng %0 Journal Article %J American Political Science Review %D 1993 %T On Party Platforms, Mandates, and Government Spending %A Gary King %A Michael Laver %X
In their 1990 Review article, Ian Budge and Richard Hofferbert analyzed the relationship between party platform emphases, control of the White House, and national government spending priorities, reporting strong evidence of a "party mandate" connection between them. Gary King and Michael Laver successfully replicate the original analysis, critique the interpretation of the causal effects, and present a reanalysis showing that platforms have small or nonexistent effects on spending. In response, Budge, Hofferbert, and Michael McDonald agree that their language was somewhat inconsistent on both interactions and causality but defend their conceptualization of "mandates" as involving only an association, not necessarily a causal connection, between party commitments and government policy. Hence, while the causes of government policy are of interest, noncausal associations are sufficient as evidence of party mandates in American politics.%B American Political Science Review %V 87 %P 744–750 %8 September %G eng %0 Journal Article %J PS: Political Science and Politics %D 1993 %T The Science of Political Science Graduate Admissions %A Gary King %A John M. Bruce %A Michael Gilligan %X
As political scientists, we spend much time teaching and doing scholarly research, and more time than we may wish to remember on university committees. However, just as many of us believe that teaching and research are not fundamentally different activities, we also need not use fundamentally different standards of inference when studying government, policy, and politics than when participating in the governance of departments and universities. In this article, we describe our attempts to bring somewhat more systematic methods to the process and policies of graduate admissions.%B PS: Political Science and Politics %V XXVI %P 772–778 %8 December %G eng %0 Journal Article %J British Journal of Political Science %D 1993 %T Why are American Presidential Election Campaign Polls so Variable when Votes are so Predictable? %A Andrew Gelman %A Gary King %X As most political scientists know, the outcome of the U.S. Presidential election can be predicted within a few percentage points (in the popular vote), based on information available months before the election. Thus, the general election campaign for president seems irrelevant to the outcome (except in very close elections), despite all the media coverage of campaign strategy. However, it is also well known that the pre-election opinion polls can vary wildly over the campaign, and this variation is generally attributed to events in the campaign. How can campaign events affect people’s opinions on whom they plan to vote for, and yet not affect the outcome of the election? For that matter, why do voters consistently increase their support for a candidate during his nominating convention, even though the conventions are almost entirely predictable events whose effects can be rationally forecast? In this exploratory study, we consider several intuitively appealing, but ultimately wrong, resolutions to this puzzle, and discuss our current understanding of what causes opinion polls to fluctuate and yet reach a predictable outcome. Our evidence is based on graphical presentation and analysis of over 67,000 individual-level responses from forty-nine commercial polls during the 1988 campaign and many other aggregate poll results from the 1952–1992 campaigns. We show that responses to pollsters during the campaign are not generally informed or even, in a sense we describe, "rational." In contrast, voters decide which candidate to eventually support based on their enlightened preferences, as formed by the information they have learned during the campaign, as well as basic political cues such as ideology and party identification. We cannot prove this conclusion, but we do show that it is consistent with the aggregate forecasts and individual-level opinion poll responses. Based on the enlightened preferences hypothesis, we conclude that the news media have an important effect on the outcome of Presidential elections–-not due to misleading advertisements, sound bites, or spin doctors, but rather by conveying candidates’ positions on important issues. %B British Journal of Political Science %V 23 %P 409–451 %8 October %G eng %0 Journal Article %J Early Childhood Research Quarterly %D 1993 %T Good Research and Bad Research: Extending Zimile’s Criticism %A Gary King %A Daniel J. Walsh %X Herbert Zimiles has written a provocative article on quantitative research. Because his specific critiques of research on infant day care are nominal examples of his much broader arguments, we focus only on his general methodological perspectives in this brief comment. We write as methodologists, a qualitative researcher with a quantitative background (Walsh) and a quantitative researcher completing a book on qualitative research (King and see King, Keohane & Verba, in preparation). %B Early Childhood Research Quarterly %V 8 %P 397–401 %8 September %G eng %0 Journal Article %J Political Analysis %D 1991 %T Stochastic Variation: A Comment on Lewis-Beck and Skalaban’s ’The R-Square’ %A Gary King %X In an interesting and provocative article, Michael Lewis-Beck and Andrew Skalaban make an important contribution by emphasizing several philosophical issues in political methodology that have received too little attention from methodologists and quantitative researchers. These issues involve the role of systematic, and especially stochastic, variation in statistical models. After briefly discussing a few points of disagreement, hoping to reduce them to points of clarification, I turn to the philosophical issues. Examples with real data follow. %B Political Analysis %V 2 %P 185–200 %G eng %0 Journal Article %J The Political Methodologist %D 1991 %T Calculating Standard Errors of Predicted Values based on Nonlinear Functional Forms %A Gary King %B The Political Methodologist %V 4 %8 Fall %G eng %0 Journal Article %J British Journal of Political Science %D 1991 %T Constituency Service and Incumbency Advantage %A Gary King %X This Note addresses the long-standing discrepancy between scholarly support for the effect of constituency service on incumbency advantage and a large body of contradictory empirical evidence. I show first that many of the methodological problems noticed in past research reduce to a single methodological problem that is readily resolved. The core of this Note then provides among the first systematic empirical evidence for the constituency service hypothesis. Specifically, an extra $10,000 added to the budget of the average state legislator gives this incumbent an additional 1.54 percentage points in the next election (with a 95% confidence interval of 1.14 to 1.94 percentage points). %B British Journal of Political Science %V 21 %P 119–128 %8 January %G eng %0 Journal Article %J Political Analysis %D 1991 %T On Political Methodology %A Gary King %X "Politimetrics" (Gurr 1972), "polimetrics" (Alker 1975), "politometrics" (Hilton 1976), "political arithmetic" (Petty  1971), "quantitative Political Science (QPS)," "governmetrics," "posopolitics" (Papayanopoulos 1973), "political science statistics (Rai and Blydenburgh 1973), "political statistics" (Rice 1926). These are some of the names that scholars have used to describe the field we now call "political methodology." The history of political methodology has been quite fragmented until recently, as reflected by this patchwork of names. The field has begun to coalesce during the past decade and we are developing persistent organizations, a growing body of scholarly literature, and an emerging consensus about important problems that need to be solved. I make one main point in this article: If political methodology is to play an important role in the future of political science, scholars will need to find ways of representing more interesting political contexts in quantitative analyses. This does not mean that scholars should just build more and more complicated statistical models. Instead, we need to represent more of the essence of political phenomena in our models. The advantage of formal and quantitative approaches is that they are abstract representations of the political world and are, thus, much clearer. We need methods that enable us to abstract the right parts of the phenomenon we are studying and exclude everything superfluous. Despite the fragmented history of quantitative political analysis, a version of this goal has been voiced frequently by both quantitative researchers and their critics (Sec. 2). However, while recognizing this shortcoming, earlier scholars were not in the position to rectify it, lacking the mathematical and statistical tools and, early on, the data. Since political methodologists have made great progress in these and other areas in recent years, I argue that we are now capable of realizing this goal. In section 3, I suggest specific approaches to this problem. Finally, in section 4, I provide two modern examples, ecological inference and models of spatial autocorrelation, to illustrate these points. %B Political Analysis %V 2 %P 1–30 %G eng %0 Journal Article %J American Journal of Political Science %D 1991 %T Systemic Consequences of Incumbency Advantage in the U.S. House %A Gary King %A Andrew Gelman %X The dramatic increase in the electoral advantage of incumbency has sparked widespread interest among congressional researchers over the last 15 years. Although many scholars have studied the advantages of incumbency for incumbents, few have analyzed its effects on the underlying electoral system. We examine the influence of the incumbency advantage on two features of the electoral system in the U.S. House elections: electoral responsiveness and partisan bias. Using a district-level seats-votes model of House elections, we are able to distinguish systematic changes from unique, election-specific variations. Our results confirm the significant drop in responsiveness, and even steeper decline outside the South, over the past 40 years. Contrary to expectations, we find that increased incumbency advantage explains less than a third of this trend, indicating that some other unknown factor is responsible. Moreover, our analysis also reveals another dramatic pattern, largely overlooked in the congressional literature: in the 1940’s and 1950’s the electoral system was severely biased in favor of the Republican party. The system shifted incrementally from this severe Republican bias over the next several decades to a moderate Democratic bias by the mid-1980’s. Interestingly, changes in incumbency advantage explain virtually all of this trend in partisan bias since the 1940’s. By removing incumbency advantage and the existing configuration of incumbents and challengers analytically, our analysis reveals an underlying electoral system that remains consistently biased in favor of the Republican party. Thus, our results indicate that incumbency advantage affects the underlying electoral system, but contrary to conventional wisdom, this changes the trend in partisan bias more than electoral responsiveness. %B American Journal of Political Science %V 35 %P 110–138 %8 February %G eng %0 Journal Article %J American Journal of Political Science %D 1991 %T 'Truth' is Stranger than Prediction, More Questionable Than Causal Inference %A Gary King %X Robert Luskin’s article in this issue provides a useful service by appropriately qualifying several points I made in my 1986 American Journal of Political Science article. Whereas I focused on how to avoid common mistakes in quantitative political sciences, Luskin clarifies ways to extract some useful information from usually problematic statistics: correlation coefficients, standardized coefficients, and especially R2. Since these three statistics are very closely related (and indeed deterministic functions of one another in some cases), I focus in this discussion primarily on R2, the most widely used and abused. Luskin also widens the discussion to various kinds of specification tests, a general issue I also address. In fact, as Beck (1991) reports, a large number of formal specification tests are just functions of R2, with differences among them primarily due to how much each statistic penalizes one for including extra parameters and fewer observations. Quantitative political scientists often worry about model selection and specification, asking questions about parameter identification, autocorrelated or heteroscedastic disturbances, parameter constancy, variable choice, measurement error, endogeneity, functional forms, stochastic assumptions, and selection bias, among numerous others. These model specification questions are all important, but we may have forgotten why we pose them. Political scientists commonly give three reasons: (1) finding the "true" model, or the "full" explanation and (2) prediction and and (3) estimating specific causal effects. I argue here that (1) is used the most but useful the least and (2) is very useful but not usually in political science where forecasting is not often a central concern and and (3) correctly represents the goals of political scientists and should form the basis of most of our quantitative empirical work. %B American Journal of Political Science %V 35 %P 1047–1053 %8 November %G eng %0 Journal Article %J American Journal of Political Science %D 1990 %T A Unified Model of Cabinet Dissolution in Parliamentary Democracies %A Gary King %A James Alt %A Nancy Burns %A Michael Laver %X The literature on cabinet duration is split between two apparently irreconcilable positions. The attributes theorists seek to explain cabinet duration as a fixed function of measured explanatory variables, while the events process theorists model cabinet durations as a product of purely stochastic processes. In this paper we build a unified statistical model that combines the insights of these previously distinct approaches. We also generalize this unified model, and all previous models, by including (1) a stochastic component that takes into account the censoring that occurs as a result of governments lasting to the vicinity of the maximum constitutional interelection period, (2) a systematic component that precludes the possibility of negative duration predictions, and (3) a much more objective and parsimonious list of explanatory variables, the explanatory power of which would not be improved by including a list of indicator variables for individual countries. %B American Journal of Political Science %V 34 %P 846–871 %8 August %G eng %0 Journal Article %J Legislative Studies Quarterly %D 1990 %T Electoral Responsiveness and Partisan Bias in Multiparty Democracies %A Gary King %X Because the goals of local and national representation are inherently incompatible, there is an uncertain relationship between aggregates of citizen votes and the national allocation of legislative seats in almost all democracies. In particular electoral systems, this uncertainty leads to diverse configurations of electoral responsiveness and partisian bias, two fundamental concepts in empirical democratic theory. This paper unifies virtually all existing multiyear seats-votes models as special cases of a new general model. It also permits the first formalization of, and reliable method for empirically estimating, electoral responsiveness and partisian bias in electoral systems with any number of political parties. I apply this model to data from nine democratic countries, revealing clear patterns in responsiveness and bias across different types of electoral rules. %B Legislative Studies Quarterly %V XV %P 159–181 %8 May %G eng %0 Journal Article %J American Journal of Political Science %D 1990 %T Estimating Incumbency Advantage Without Bias %A Andrew Gelman %A Gary King %X In this paper we prove theoretically and demonstrate empirically that all existing measures of incumbency advantage in the congressional elections literature are biased or inconsistent. We then provide an unbiased estimator based on a very simple linear regression model. We apply this new method to congressional elections since 1900, providing the first evidence of a positive incumbency advantage in the first half of the century. %B American Journal of Political Science %V 34 %P 1142–1164 %8 November %G eng %0 Journal Article %J Journal of the American Statistical Association %D 1990 %T Estimating the Electoral Consequences of Legislative Redistricting %A Andrew Gelman %A Gary King %X We analyze the effects of redistricting as revealed in the votes received by the Democratic and Republican candidates for state legislature. We develop measures of partisan bias and the responsiveness of the composition of the legislature to changes in statewide votes. Our statistical model incorporates a mixed hierarchical Bayesian and non-Bayesian estimation, requiring simulation along the lines of Tanner and Wong (1987). This model provides reliable estimates of partisan bias and responsiveness along with measures of their variabilities from only a single year of electoral data. This allows one to distinguish systematic changes in the underlying electoral system from typical election-to-election variability. %B Journal of the American Statistical Association %V 85 %P 274–282 %8 June %G eng %0 Journal Article %J Journal of Politics %D 1990 %T Measuring the Consequences of Delegate Selection Rules in Presidential Nominations %A Stephen Ansolabehere %A Gary King %X In this paper, we formalize existing normative criteria used to judge presidential selection contests by modeling the translation of citizen votes in primaries and caucuses into delegates to the national party conventions. We use a statistical model that enables us to separate the form of electoral responsiveness in presidential selection systems, as well as the degree of bias toward each of the candidates. We find that (1) the Republican nomination system is more responsive to changes in citizen votes than the Democratic system and (2) non-PR primaries are always more responsive than PR primaries and (3) surprisingly, caucuses are more proportional than even primaries held under PR rules and (4) significant bias in favor of a candidate was a good prediction of the winner of the nomination contest. We also (5) evaluate the claims of Ronald Reagan in 1976 and Jesse Jackson in 1988 that the selection systems were substantially biased against their candidates. We find no evidence to support Reagan’s claim, but substantial evidence that Jackson was correct. %B Journal of Politics %V 52 %P 609–621 %8 May %G eng %0 Journal Article %J Proceedings of the Social Statistics Section, American Statistical Association %D 1989 %T Electoral Responsiveness in U.S. Congressional Elections, 1946-1986 %A Andrew Gelman %A Gary King %B Proceedings of the Social Statistics Section, American Statistical Association %P 208 %G eng %0 Journal Article %J International Studies Quarterly %D 1989 %T Event Count Models for International Relations: Generalizations and Applications %A Gary King %X International relations theorists tend to think in terms of continuous processes. Yet we observe only discrete events, such as wars or alliances, and summarize them in terms of the frequency of occurrence. As such, most empirical analyses in international relations are based on event count variables. Unfortunately, analysts have generally relied on statistical techniques that were designed for continuous data. This mismatch between theory and method has caused bias, inefficiency, and numerous inconsistencies in both theoretical arguments and empirical findings throughout the literature. This article develops a much more powerful approach to modeling and statistical analysis based explicity on estimating continuous processes from observed event counts. To demonstrate this class of models, I present several new statistical techniques developed for and applied to different areas of international relations. These include the influence of international alliances on the outbreak of war, the contagious process of multilateral economic sanctions, and reciprocity in superpower conflict. I also show how one can extract considerably more information from existing data and relate substantive theory to empirical analyses more explicitly with this approach. %B International Studies Quarterly %V 33 %P 123–147 %8 June %G eng %0 Journal Article %J American Journal of Political Science %D 1989 %T Representation Through Legislative Redistricting: A Stochastic Model %A Gary King %X This paper builds a stochastic model of the processes that give rise to observed patterns of representation and bias in congressional and state legislative elections. The analysis demonstrates that partisan swing and incumbency voting, concepts from the congressional elections literature, have determinate effects on representation and bias, concepts from the redistricting literature. The model shows precisely how incumbency and increased variability of partisan swing reduce the responsiveness of the electoral system and how partisan swing affects whether the system is biased toward one party or the other. Incumbency, and other causes of unresponsive representation, also reduce the effect of partisan swing on current levels of partisan bias. By relaxing the restrictive portions of the widely applied "uniform partisan swing" assumption, the theoretical analysis leads directly to an empirical model enabling one more reliably to estimate responsiveness and bias from a single year of electoral data. Applying this to data from seven elections in each of six states, the paper demonstrates that redistricting has effects in predicted directions in the short run: partisan gerrymandering biases the system in favor of the party in control and, by freeing up seats held by opposition party incumbents, increases the system’s responsiveness. Bipartisan-controlled redistricting appears to reduce bias somewhat and dramatically to reduce responsiveness. Nonpartisan redistricting processes substantially increase responsiveness but do not have as clear an effect on bias. However, after only two elections, prima facie evidence for redistricting effects evaporate in most states. Finally, across every state and type of redistricting process, responsiveness declined significantly over the course of the decade. This is clear evidence that the phenomenon of "vanishing marginals," recognized first in the U.S. Congress literature, also applies to these different types of state legislative assemblies. It also strongly suggests that redistricting could not account for this pattern. %B American Journal of Political Science %V 33 %P 787–824 %8 November %G eng %0 Journal Article %J Sociological Methods and Research %D 1989 %T A Seemingly Unrelated Poisson Regression Model %A Gary King %X This article introduces a new estimator for the analysis of two contemporaneously correlated endogenous event count variables. This seemingly unrelated Poisson regression model (SUPREME) estimator combines the efficiencies created by single equation Poisson regression model estimators and insights from "seemingly unrelated" linear regression models. %B Sociological Methods and Research %V 17 %P 235–255 %8 February %G eng %0 Journal Article %J American Journal of Political Science %D 1989 %T Variance Specification in Event Count Models: From Restrictive Assumptions to a Generalized Estimator %A Gary King %X This paper discusses the problem of variance specification in models for event count data. Event counts are dependent variables that can take on only nonnegative integer values, such as the number of wars or coups d’etat in a year. I discuss several generalizations of the Poisson regression model, presented in King (1988), to allow for substantively interesting stochastic processes that do not fit into the Poisson framework. Individual models that cope with, and help analyze, heterogeneity, contagion, and negative contagion are each shown to lead to specific statistical models for event count data. In addition, I derive a new generalized event count (GEC) model that enables researchers to extract significant amounts of new information from existing data by estimating features of these unobserved substantive processes. Applications of this model to congressional challenges of presidential vetoes and superpower conflict demonstrate the dramatic advantages of this approach. %B American Journal of Political Science %V 33 %P 762–784 %8 August %G eng %0 Journal Article %J American Journal of Political Science %D 1988 %T Statistical Models for Political Science Event Counts: Bias in Conventional Procedures and Evidence for The Exponential Poisson Regression Model %A Gary King %X This paper presents analytical, Monte Carlo, and empirical evidence on models for event count data. Event counts are dependent variables that measure the number of times some event occurs. Counts of international events are probably the most common, but numerous examples exist in every empirical field of the discipline. The results of the analysis below strongly suggest that the way event counts have been analyzed in hundreds of important political science studies have produced statistically and substantively unreliable results. Misspecification, inefficiency, bias, inconsistency, insufficiency, and other problems result from the unknowing application of two common methods that are without theoretical justification or empirical unity in this type of data. I show that the exponential Poisson regression (EPR) model provides analytically, in large samples, and empirically, in small, finite samples, a far superior model and optimal estimator. I also demonstrate the advantage of this methodology in an application to nineteenth-century party switching in the U.S. Congress. Its use by political scientists is strongly encouraged. %B American Journal of Political Science %V 32 %P 838-863 %8 August %G eng %0 Journal Article %J American Political Science Review %D 1987 %T Democratic Representation and Partisan Bias in Congressional Elections %A Gary King %A Robert X Browning %X The translation of citizen votes into legislative seats is of central importance in democratic electoral systems. It has been a longstanding concern among scholars in political science and in numerous other disciplines. Through this literature, two fundamental tenets of democratic theory, partisan bias and democratic representation, have often been confused. We develop a general statistical model of the relationship between votes and seats and separate these two important concepts theoretically and empirically. In so doing, we also solve several methodological problems with the study of seats, votes and the cube law. An application to U.S. congressional districts provides estimates of bias and representation for each state and deomonstrates the model’s utility. Results of this application show distinct types of representation coexisting in U.S. states. Although most states have small partisan biases, there are some with a substantial degree of bias. %B American Political Science Review %V 81 %P 1252–1273 %8 December %G eng %0 Journal Article %J Law and Policy %D 1987 %T Seats, Votes, and Gerrymandering: Measuring Bias and Representation in Legislative Redistricting %A Robert X Browning %A Gary King %X The Davis v. Bandemer case focused much attention on the problem of using statistical evidence to demonstrate the existence of political gerrymandering. In this paper, we evaluate the uses and limitations of measures of the seat-votes relationship in the Bandemer case. We outline a statistical method we have developed that can be used to estimate bias and the form of representation in legislative redistricting. We apply this method to Indiana State House and Senate elections for the period 1972 to 1984 and demonstrate a maximum bias 6.2% toward the Republicans in the House and a 2.8% bias in the Senate. %B Law and Policy %V 9 %P 305–322 %8 July %G eng %0 Journal Article %J American Politics Quarterly %D 1987 %T Presidential Appointments to the Supreme Court: Adding Systematic Explanation to Probabilistic Description %A Gary King %X Three articles, published in the leading journals of three disciplines over the last five decades, have each used the Poisson probability distribution to help describe the frequency with which presidents were able to appoint United States Supreme Court Justices. This work challenges these previous findings with a new model of Court appointments. The analysis demonstrates that the number of appointments a president can expect to make in a given year is a function of existing measurable variables. %B American Politics Quarterly %V 15 %P 373–386 %8 July %G eng %0 Journal Article %J American Journal of Political Science %D 1986 %T How Not to Lie With Statistics: Avoiding Common Mistakes in Quantitative Political Science %A Gary King %X This article identifies a set of serious theoretical mistakes appearing with troublingly high frequency throughout the quantitative political science literature. These mistakes are all based on faulty statistical theory or on erroneous statistical analysis. Through algebraic and interpretive proofs, some of the most commonly made mistakes are explicated and illustrated. The theoretical problem underlying each is highlighted, and suggested solutions are provided throughout. It is argued that closer attention to these problems and solutions will result in more reliable quantitative analyses and more useful theoretical contributions. %B American Journal of Political Science %V 30 %P 666–687 %8 August %G eng %0 Journal Article %J Social Science Research %D 1986 %T The Significance of Roll Calls in Voting Bodies: A Model and Statistical Estimation %A Gary King %X In the long history of legislative roll call analyses, there continues to exist a particularly troubling problem: There is no satisfactory method for measuring the relative importance or significance of individual roll calls. A measure of roll call significance would be interesting in and of itself, but many have realized that it could also substantially improve empirical research. The consequence of this situation is that hundreds of researchers risk heteroskedastic disturbances (resulting in inefficient estimates and biased standard errors and test statistics), are unable to appropriately choose the roll calls most suited to their theory (resulting in analyses that may not correctly test their theory), and often use methods that create more problems than they solve (resulting in selection bias, unrealistic weighting schemes, or relatively subjective measures). This article introduces a new method designed to meet these problems. Based on an application of Box-Tiao intervention analysis, the method extracts from observed voting participation scores the "revealed preferences" of legislators as a measure of roll call significance. Applying this method to roll calls from the U.S. Senate demonstrates the success of the method and suggests its utility in applied research. %B Social Science Research %V 15 %P 135–152 %8 June %G eng %0 Journal Article %J Social Science Quarterly %D 1986 %T The Development of Political Activists: A Model of Early Learning %A Gary King %A Richard Merelman %X An analysis of panel data reveals the unique importance of early learning to the development of political activism among Americans. A combination of two learning models– the frequently used crystallization model and the rarely analyzed sensitization model– is advanced as most appropriate for understanding political socialization and the development of political activism. The findings contribute to research on elite behavior and on the process of political socialization. %B Social Science Quarterly %V 67 %P 473–490 %8 September %G eng %0 Journal Article %J Political Psychology %D 1986 %T Political Parties and Foreign Policy: A Structuralist Approach %A Gary King %X This article introduces the theory and approach of structural anthropology and applies it to a problem in American political science. Through this approach, the "bipartisan foreign policy hypothesis" and that "two presidencies hypothesis" are reformulated and reconsidered. Until now participants in the debate over each have only rarely built on, or even cited, the other’s research. An additional problem is that the widespread conventional wisdom in support of the two hypotheses is inconsistent with systematic scholarly analyses. This paper demonstrates that the two hypotheses are drawn from the same underlying structure. Each hypothesis and the theoretical model it implies is conceptually and empirically extended to take into account the differences between congressional leaders and members. Then, historical examples and statistical analyses of House roll call data are used to demonstrate that the hypotheses, while sometimes supported for the congressional members, are far more applicable to leadership decision making. Conclusions suggest that conventional wisdom be revised to take these differences into account. %B Political Psychology %V 7 %P 83–101 %8 March %G eng %0 Journal Article %J American Political Science Review %D 1985 %T Book Review of `Forecasting Presidential Elections' %A Gary King %X
This is a book review of Steven J. Rosenstone, Forecasting Presidential Elections, New Haven: Yale University Press, 1983.%B American Political Science Review %V 79 %P 855 %G eng %N 3