Overview
In this workshop, we will discuss the logic of experimentation, its strengths and weaknesses compared to other methodologies, and the ways in which experimentation has been -- and could be -- used to investigate political, social, and economic phenomena. Examples of field, lab, and survey experiments will be drawn from a broad array of political science subfields, as well as from other social science disciplines.
After describing the attractive statistical properties of experiments, we consider a variety of potential threats to core assumptions. In particular, we consider the complications that arise when (1) treatment and control conditions different in systematic ways other than the intended treatment, (2) treatments are not administered according to the randomly assigned plan, (3) subjects are affected by the treatments assigned or administered to others, and (4) outcome measures are not obtained for all subjects. In each case, we discuss possible statistical and design solutions. We conclude by discussing the practical and ethical issues that arise when conducting experiments in field settings.
Audience & Prerequisites
This workshop serves as a broad introduction to social science experiments. Participants are expected to come to the session with some experience designing a research study and with a question or hypothesis that they wish to investigate using experimental design.
Readings & Resources
The primary text for the course is:
Gerber, Alan S., and Donald P. Green. 2012. Field Experiments: Design, Analysis, and Interpretation. New York: W.W. Norton.
This textbook (FEDAI for short) is too extensive to be covered in an afternoon, but we will make our way through much of the first half, which covers core topics.
Supplementary readings are designed to illustrate a wide range of experimental applications. They will be provided to participants as PDFs. Further readings beyond are available on request.
Data analysis examples will be provided in both R and Stata format, but no special programming expertise is required. If you have some experience with statistical software, you can join in when we analyze data together.
Schedule
What are experiments? Why conduct experimental research? Experiments and Models of Potential Outcomes. Varieties of experimental designs. What are covariates and why are they useful?
FEDAI: Chapters 1 and 2, but skim 3 and 4.
Varieties of survey experiments:
Hainmueller, Jens, and Daniel J. Hopkins. The hidden American immigration consensus: A conjoint analysis of attitudes toward immigrants. American Journal of Political Science 59.3 (2015): 529-548.
Healy, Andrew, Katrina Kosec, and Cecilia Hyunjung Mo. Economic development, mobility, and political discontent: An experimental test of Tocqueville’s thesis in Pakistan. American Political Science Review 111.3 (2017): 605-621.
Kuklinski, James H., Michael D. Cobb, and Martin Gilens. Racial attitudes and the New South. Journal of Politics59.2 (1997): 323-349.
Varieties of Field experiments:
Kalla, Joshua L., and David E. Broockman. 2015. Campaign Contributions Facilitate Access to Congressional Officials: A Randomized Field Experiment. American Journal of Political Science 60(3): 545-558.
Bertrand, Marianne and Sendhil Mullanathan. 2004. Are Emily and Greg More Employable than Lakisha and Jamil? A Field Experiment on Labor Market Discrimination. American Economic Review 94(4): 991-1013.
Varieties of Lab experiments:
Mutz, Diana C. and Byron Reeves. 2005. The New Video Malaise: Effects of Televised Incivility on Political Trust. American Political Science Review99(February):1-15.
Casella, Alessandra, and Thomas R. Palfrey. 2021. “Trading votes for votes: A laboratory study.” Games and Economic Behavior 125: 1-26.
Experiments with One-sided Noncompliance (Failure-to-Treat)
FEDAI: Chapter 5. The next two experiments (field and lab-in-the-field) illustrate experiments with one-sided noncompliance. The second employs a placebo design.
Gerber, Alan S., and Donald P. Green. 2000. The Effects of Canvassing, Direct Mail, and Telephone Contact on Voter Turnout: A Field Experiment. American Political Science Review 94:653-63.
Green, Donald P., Dylan W. Groves, Constantine Manda, Beatrice Montano, and Bardia Rahmani. A radio drama’s effects on attitudes toward early and forced marriage: Results from a field experiment in Rural Tanzania. Comparative Political Studies 56, no. 8 (2023): 1115-1155.
Experiments with Attrition
FEDAI: Chapters 7.
Implementing an Experiment and Reporting the Results.
FEDAI: Chapter 13, Appendix A, and Appendix B.
Special topics: heterogeneous effects, causal mechanisms, and meta-analysis.
FEDAI: Chapters 9, 10, and 11.