Skip to Main Content

Machines and Society

A growing guide on the latest in data-driven research and emerging technologies at the intersection of society, information and technology.


In this section, we turn to the second facet of computational social research—research design utilizing crowdsourcing technology to extract collective intelligence. We focus on one stage in a research project, data generation, including data collection and data production processes such as text annotation.

We discuss three such types of research designs: 1) an open survey that evolves over time based on the ideas of its participants; 2) a system that distributes microtasks in the crowd, whose outputs are as reliable and valid as those from expert human readers; and 3) a software application that interfaces with crowdsourcing technology, and that automates recruiting, collecting both behavior and survey data, and providing incentives to generate responses all in one stop.

These kinds of research design have the potential to improve the scope, efficiency, cost, scalability, sampling, response rates, and convenience of social scientific projects, compared to research in the analog age. Challenges of data quality control, assessing response biases, or adjusting sampling biases can be addressed in the design phase or analysis of data.

Wiki Survey


Salganik, Matthew J., and Karen E. C. Levy. 2015. “Wiki Surveys: Open and Quantifiable Social Data Collection.” PLoS One, 10 (5):e0123483.

Distribution of Microtasks


Benoit, K., Conway, D., Lauderdale, B. E., Laver, M., & Mikhaylov, S. (2016). Crowd-sourced Text Analysis: Reproducible and Agile Production of Political Data. American Political Science Review, 110(2), 278-295.


Pennycook, G., & Rand, D. G. (2019). Fighting Misinformation on Social  Media using Crowdsourced Judgments of News Source Quality. Proceedings of the National Academy of Sciences, 116(7), 2521-2526.


Benoit, K., Munger, K., & Spirling, A. (2019). Measuring and Explaining Political Sophistication through Textual Complexity. American Journal of Political Science, 63(2), 491-508.


Porter, N. D., Verdery, A. M., & Gaddis, S. M. (2020). Enhancing Big Data in the Social Sciences with Crowdsourcing: Data Augmentation Practices, Techniques, and Opportunities. PloS One, 15(6), e0233154.

Social Media Survey App


Bail, Christopher A. (2015). “Taming Big Data Using App Technology to Study Organizational Behavior on Social Media.” Sociological Methods & Research, 46(2), 189-217.