Introduction to Social Science Experiments
Thursday, January 12th 2:00 p.m. - 6:20 p.m. (EST)
Dr. Donald P. Green
J.W. Burgess Professor of Political Science, Columbia University
In this workshop, we will discuss the logic of experimentation, its strengths and weaknesses compared to other methodologies, and the ways in which experimentation has been -- and could be -- used to investigate political, social, and economic phenomena. Emphasis will be placed on field experiments, randomized trials conducted in real-world settings. Examples will be drawn from a broad array of disciplines.
After describing the attractive statistical properties of experiments, we consider a variety of potential threats to core assumptions. In particular, we consider the complications that arise when (1) treatment and control conditions different in systematic ways other than the intended treatment, (2) treatments are not administered according to the randomly assigned plan, (3) subjects are affected by the treatments assigned or administered to others, and (4) outcome measures are not obtained for all subjects. In each case, we discuss possible statistical and design solutions. We conclude by discussing the practical and ethical issues that arise when conducting experiments in field settings.
​
​
Audience & Prerequisites
This workshop serves as a broad introduction to social science experiments. Participants are expected to come to the session with some experience designing a research study and with a question or hypothesis that they wish to investigate using experimental design.
​
​
Readings & Resources
The primary text for the course is
Gerber, Alan S., and Donald P. Green. 2012. Field Experiments: Design, Analysis, and Interpretation. New York: W.W. Norton.
This textbook (FEDAI for short) is too extensive to be covered in two sessions, but we will make our way through much of the first half, which covers core topics.
Supplementary readings are designed to illustrate a wide range of experimental applications. They will be provided to participants via a shared Dropbox folder. Further readings beyond the dropbox PDFs are available on request.
Data analysis examples will be provided in both R and Stata format, but no special programming expertise is required. If you have some experience with statistical software, you can join in when we analyze data together.
​
​
Schedule
Session 1.
What are experiments? Why conduct experimental research? Experiments and Models of Potential Outcomes. Varieties of experimental designs. What are covariates and why are they useful?
- FEDAI: Chapters 1 and 2, but skim 3 and 4.
Varieties of survey experiments:
- Hainmueller, Jens, and Daniel J. Hopkins. The hidden American immigration consensus: A conjoint analysis of attitudes toward immigrants. American Journal of Political Science 59.3 (2015): 529-548.
- Healy, Andrew, Katrina Kosec, and Cecilia Hyunjung Mo. Economic development, mobility, and political discontent: An experimental test of Tocqueville’s thesis in Pakistan. American Political Science Review 111.3 (2017): 605-621.
- Kuklinski, James H., Michael D. Cobb, and Martin Gilens. Racial attitudes and the New South. Journal of Politics 59.2 (1997): 323-349.
​
Varieties of Field experiments:
- Kalla, Joshua L., and David E. Broockman. 2015. Campaign Contributions Facilitate Access to Congressional Officials: A Randomized Field Experiment. American Journal of Political Science 60(3): 545-558.
- Bertrand, Marianne and Sendhil Mullanathan. 2004. Are Emily and Greg More Employable than Lakisha and Jamil? A Field Experiment on Labor Market Discrimination. American Economic Review 94(4): 991-1013.
​
Varieties of Lab experiments:
- Mutz, Diana C. and Byron Reeves. 2005. The New Video Malaise: Effects of Televised Incivility on Political Trust. American Political Science Review 99 (February):1-15.
- Casella, Alessandra, and Thomas R. Palfrey. 2021. “Trading votes for votes: A laboratory study.” Games and Economic Behavior 125: 1-26.
​
Session 2.
Experiments with One-sided Noncompliance (Failure-to-Treat):
- FEDAI: Chapter 5.
In addition, read the following article, which we will use in class to illustrate the analysis of experiments with one-sided noncompliance:
- Gerber, Alan S., and Donald P. Green. 2000. The Effects of Canvassing, Direct Mail, and Telephone Contact on Voter Turnout: A Field Experiment. American Political Science Review 94:653-63.
​
Experiments with Attrition:
- FEDAI: Chapter 7.
​
Implementing an Experiment and Reporting the Results:
- FEDAI: Chapter 13, Appendix A, and Appendix B.
​
Special topics: heterogeneous effects, causal mechanisms, and meta-analysis:
- FEDAI: Chapters 9, 10, and 11.
​