Course Instructor: Nan Zhang (Office hours by appointment)
Course Website: https://nanzhangresearch.github.io/QMIR_HWS24
Date and Time: Tuesdays from 10:15 - 11:45 in EO 162 Seminarraum (Schloss Ehrenhof Ost)
Course Aims: This tutorial will provide a collaborative and immersive research experience where students work together with the instructor to design, implement, and analyze a survey experiment to study topical questions related to international politics. Upon completion of this course, students will have first-hand experience with the entire “research process cycle” from study design and pre-analysis planning, through fieldwork and data collection, to statistical analysis and paper writing. This course will provide students substantial preparation for writing their own BA theses.
In the first few weeks of the course, students will work with the instructor to identify a common research question and research design to pursue for the rest of the semester. Afterwards, we will work together (and in smaller groups) on the following tasks:
Students will be assigned tasks depending on their own interests and abilities.
Class meetings may include some “mini-lectures” from the instructor, but the majority of our time together will be spent updating each other on our progress and mapping out group tasks to be completed by the following week.
Diligent preparation and active participation are essential for the success of this class! Of course, situations may arise when you cannot attend class. As a courtesy, please let me and your fellow students know beforehand.
The survey experiment we design will be fielded on the January wave of the German Internet Panel (GIP). We expect to receive the data from the GIP by the end of February / early March.
Thus, although we will not have the results available by the end of the semester, all of the other “elements” of the paper (including the code to analyze the data once they are in) should be complete.
Once we have received the data from the GIP, Nan will run the prepared analysis code and share the results with the class.
At this point, students can decide whether they want to continue working on the final paper. Regardless of what you decide, you will all be co-authors on the final paper.
For students who want to continue, Nan will schedule a meeting where we will discuss the findings. Based on our meeting, Nan will draft up a short discussion and conclusion to the final paper. Students have 1 week to provide feedback. Afterwards, Nan will submit the manuscript to the agreed upon journal.
Note: the following is drawn from Jessica Calarco and Jesse Stommel’s ideas on “ungrading.”
This course will graded a bit differently than what you are used to from other classes. You will still be expected to complete assignments, and you will still receive a numeric grade on your transcript at the end of the semester. However, your grade will be determined based on your own self-assessment of your learning and effort in the class, with the possibility of adjustments up or down from me.
During the semester, you will be asked to complete two self-evaluation exercises (once at midterm, and once at the end of the semester). These exercises will include a series of questions about your work in the course. Specifically, you will be asked to reflect upon:
When assigning final grades, I will strive to honor your end-of-semester assessment of your own performance and progress in this course. However, I reserve the right to alter your proposed grade as appropriate, based on my own evaluation of your performance and progress in the course as a whole. If such an alteration seems warranted, I will contact you to set up a meeting to discuss your work in the course.
Additional Assignment: to help you keep track of your progress throughout the semester, it is helpful to keep a weekly “journal”. In particular, after each class session, take 5 minutes and write down (1) what is one thing that you learned over the past week, and (2) what is something that you did not understand?
Please include this journal as an appendix to your final reflection paper.
Note: the current version of the syllabus is a work in progress.
Session 1 (3 Sept) Introduction
Nan will introduce the course, go over the syllabus, and present the broad research question. Slides are here.
Students should confirm whether they want to take the class. We will then create a class mailing list or whatsapp group.
Assignment for next week. Please read the following and answer the reading questions:
Hirschberger et al. 2022. Making sense of the past to understand the present: Attributions for historical trauma predict contemporary social and political attitudes.
Q1: why might reminding Germans of the Holocaust shape attitudes towards the current Israeli-Palestinian conflict?
Q2: in your opinion, which types of people are more (or less) likely to feel collective guilt when they are reminded of the Holocaust?
Mutz. 2011. Population-Based Survey Experiments Chapter 1, pages 1-12.
Q3: in your own words, what is a population-based survey experiment?
Q4: what does Mutz mean by “internal” and “external validity”?
Please write up your answers (max 1 page) and submit them to me via email by Sunday, 23:59.
Session 2 (10 Sept) Theory Development
We will start of with a mini-lecture on survey experiments and discuss how this methodology can be used to answer the broad research question. This lecture will also introduce students to the terminology of treatment effects and treatment heterogeneity (that is, the treatment may work differently for different types of people). Slides are here.
The bulk of our time will be spent workshopping your ideas about treatment heterogeneity (i.e. Q2 from last week’s homework). In groups, students should come up with an idea about treatment heterogeneity that they would like to explore in more depth.
Assignment for next week. Working in their groups, students should conduct a preliminary independent literature review. Your lit review should answer the following question:
Please prepare a short (10 minute max) presentation of the results of your literature review (including a bibliography) for the next class session. Your goal is to “pitch” your idea to your classmates and convince them (and me) that this is THE topic that we should pursue as a class.
Session 3 (17 Sept) Introducing the GIP and Refining / Pivoting the Research Idea
We will begin with a short information session about the German Internet Panel (GIP) led by Anne Balz.
Afterwards, students will present their pitches to class and we will vote on 1 or 2 ideas to pursue further.
Finally, Nan will give a short presentation on a similar study (by Helbling and Traunmüller, hereafter: HT) that was just fielded on the GIP this year. We will discuss how to move forward in light of this new information.
Here are the slides for this week.
Assignments for next week.
Read this short policy brief (in German). This will give you a sense of variables used in HT’s study.
Read HT’s programming instructions. This will preview how we should submit our own programming instructions later on in the semester. You can also see the exact question wordings and treatments used. Finally, pay special attention to the randomization instructions (listed under “Experimental Split”) used to divide respondents up into different “treatment” and “control” groups.
For next week’s class, we will do an exercise where we analyze HT’s data in R. If you are unfamiliar with R, please have a quick look at this excellent online material from Lion Behrens.
Session 4 (24 Sept) Analyzing Experimental Data
Nan will demonstrate how to analyze HT’s data in R.
Assignments for next week.
Complete the short problem set in R. You may work in groups (3 people max!). Please send me your R file by Sunday 23:59. Be prepared to present your problem set answers in class next week.
Think about the theory some more: does it make sense that triggering feelings of guilt about the Holocaust would affect foreign policy attitudes (vis-a-vis Israel, or more broadly)? Write down your thoughts / arguments in one paragraph, and send it to me by Sunday 23:59. This is an individual assignment.
Please read pp.83-93 and 101-106 of Mutz 2011. Your task is to come up with an effective experimental treatment (with or without deception). You could come up with an idea completely from scratch, or else adapt / copy a manipulation that other researchers have already used. In either case, think carefully about whether your treatment actually manipulates the theoretical quantity of interest. Finally, don’t forget to consider what happens to the control group! Make a short (2 slides max!) presentation demonstrating your idea. You can work individually or in groups.
Session 5 (1 Oct) Brainstorming Experimental Treatments
Here are the solutions to the problem set.
Slides for this week.
We’ll spend our time discussing possible experimental treatments and narrowing down the list of possible candidates. The goal of the class is to identify 2-3 designs that we can pursue further.
Assignments for next week.
Please read Kerzer et al. 2014. Moral Support: How Moral Values Shape Foreign Policy Attitudes to get a sense of what “militant internationalism” (MI) and “cooperative internationalism” (CI) refer to.
In addition, please finish your prototypes that you started in class this week.
We will also spend time next week developing a common set of manipulation checks and outcome measures that we will work for all of the prototypes.
Session 6 (8 Oct) Manipulation Checks and Outcome Measures
We will spend the first half of this session discussing and refining the manipulation checks and outcome measures.
In the second half, Nan will give a short presentation on how to programme a simple survey in Qualtrics.
Working in groups, you will then start to “mock up” your preferred experimental design, the manipulation checks, and the outcome measures in Qualtrics.
Assignments for next week.
Find a couple of your family and friends to test your survey. Make any changes you want, and send me your final version by Wednesday 23:59. Nan will then pilot test all of your designs with a real online sample.
Session 7 (15 Oct) Finalize the Design
We’ll look at the results from the pilot tests.
Pilot test code and rawdata p1, rawdata p2 and rawdata p3.
Assignments for next week.
Session 8 (22 Oct) Pilot Data Continued
We will continue to look at our pilot data. The goal is to decide on a draft design which we submit to the GIP.
Finally, we will briefly map out the road ahead.
Here is a link to the code and data that we will need for class.
Assignments for next week:
The GIP will provide us feedback on the coding instructions. Everyone should look over this feedback before next week’s class.
Please read sections 1 and 2 (pages 21-26) of Monogan. 2013. A Case for Registering Studies of Political Outcomes and answer the following reading questions:
Q1: what are the problems in scientific practice that pre-registration is aiming to solve?
Q2: what are some of the downsides of pre-registration?
Nan will take care of submitting the GIP questionnaire as well as the Mannheim Ethics Application.
Session 9 (29 Oct) Analyzing Experimental Data
We will discuss feedback from GIP and agree on the “final” questionnaire.
Next, we will discuss the reading from last week.
Nan will then give an applied lecture (with R code examples ) on how to analyze (and present) experimental data and interaction effects.
Assignment for next week: complete a short problem set applying the code examples to some real data. You may work in groups if you wish. Be prepared to present your problem set answers in class next week.
Session 10 (5 Nov) Finalize Analysis Code
Students present and discuss their answers to the problem set. Afterwards, we will work together to finalize the main analysis code for the survey experiment.
Assignment for next week: Please read the Introduction (pages 1-11 only) to Cohen. 1988. Statistical and Power Analysis for the Behavioral Sciences and answer the following questions:
Q1: in the context of our field experiment, what would a type I error and a type II error look like?
Q2: holding the significance level constant, what’s one way that a researcher could increase power?
Session 11 (12 Nov) Uncertainty and Power Analysis
Nan will illustrate the intuition behind power analysis with a live-coding example using simulation to answer the question: given the GIP sample size, what is the minimum effect size that can be detected at power = 0.8?
Assignment for next week: Please read Albertson and Lawrence. 2009. After the Credits Roll. Focus on the Methodology section (pages 280 - 284) and answer the following questions:
Q1: in the field experiment, some participants in the treatment group did not actually watch the program, while some participants in the control group did in fact watch it. What problems does this introduce? In particular, what happens if we compare program-watchers to program-non-watchers?
Q2: in your own words, explain how the authors deal with this compliance problem.
Q3: what would “non-compliance” look like in our survey experiment?
Also, everyone should pilot test the programmed GIP questionnaire and help fill in the GIP feedback form.
Session 12 (19 Nov) Instrumental Variables and Non-Compliance
Using the reading as a motivating example, Nan will give an applied lecture (with R code examples ) on instrumental variables analysis.
Assignment for next week: complete a short problem set applying the code examples to some real data. You may work in groups if you wish. Be prepared to present your problem set answers in class next week.
Session 13 (26 Nov) Finishing up IV
Students present and discuss their answers to the problem set. Afterwards, we will work together to finalize the IV analysis code for the survey experiment.
At the end of this session, we should have everything we need to fill in the pre-analysis plan.
Assignment for next week: Watch this video on pre-registration in OSF.
Students should then fill in this template. You may work in groups if you wish. Be prepared to present your work in class next week.
Session 14 (3 Dec) Pre-Analysis Plan
We will discuss your pre-analysis plans. Working together, we will create a final harmonized PAP.
Finally, we will spend some time wrapping up the class, taking stock of what you have learned, and discussing next steps for paper submission.
Final Self-Evaluation due by DATE TBD.