Skip to Main Content

Machines and Society

A growing guide on the latest in data-driven research and emerging technologies at the intersection of society, information and technology.

Introduction

In this section, we turn to the second facet of computational social research—research design utilizing crowdsourcing technology to extract collective intelligence. We focus on one stage in a research project, data generation, including data collection and data production processes such as text annotation.

We discuss three such types of research designs: 1) an open survey that evolves over time based on the ideas of its participants; 2) a system that distributes microtasks in the crowd, whose outputs are as reliable and valid as those from expert human readers; and 3) a software application that interfaces with crowdsourcing technology, and that automates recruiting, collecting both behavior and survey data, and providing incentives to generate responses all in one stop.

These kinds of research design have the potential to improve the scope, efficiency, cost, scalability, sampling, response rates, and convenience of social scientific projects, compared to research in the analog age. Challenges of data quality control, assessing response biases, or adjusting sampling biases can be addressed in the design phase or analysis of data.

Wiki Survey

 

Salganik, Matthew J., and Karen E. C. Levy. 2015. “Wiki Surveys: Open and Quantifiable Social Data Collection.” PLoS One, 10 (5):e0123483. https://doi.org/10.1371/journal.pone.0123483

Distribution of Microtasks

 

Benoit, K., Conway, D., Lauderdale, B. E., Laver, M., & Mikhaylov, S. (2016). Crowd-sourced Text Analysis: Reproducible and Agile Production of Political Data. American Political Science Review, 110(2), 278-295. https://doi.org/10.1017/S0003055416000058

 

Pennycook, G., & Rand, D. G. (2019). Fighting Misinformation on Social  Media using Crowdsourced Judgments of News Source Quality. Proceedings of the National Academy of Sciences, 116(7), 2521-2526. https://doi.org/10.1073/pnas.1806781116

 

Benoit, K., Munger, K., & Spirling, A. (2019). Measuring and Explaining Political Sophistication through Textual Complexity. American Journal of Political Science, 63(2), 491-508. https://doi.org/10.1111/ajps.12423

 

Porter, N. D., Verdery, A. M., & Gaddis, S. M. (2020). Enhancing Big Data in the Social Sciences with Crowdsourcing: Data Augmentation Practices, Techniques, and Opportunities. PloS One, 15(6), e0233154. https://doi.org/10.1371/journal.pone.0233154

Social Media Survey App

 

Bail, Christopher A. (2015). “Taming Big Data Using App Technology to Study Organizational Behavior on Social Media.” Sociological Methods & Research, 46(2), 189-217.