Thursday, September 28, 2017
The spectacular progress the media describes as "big data" has little to do with the growth of data. Data, after all, is becoming commoditized, less expensive, and an automatic byproduct of other changes in organizations and society. More data alone doesn't generate insights; it often merely makes data analysis harder. The real revolution isn't about the data, it is about the stunning progress in the statistical and other methods of extracting insights from the data. I illustrate these points with a wide range of examples from research I've participated in, including forecasting the solvency of Social Security; reverse engineering Chinese government censorship and fabrication of social media posts; how the same methods can estimate the causes of death in developing countries and understand billions of social media posts; an educational innovation that guarantee that students will do the reading; among others.