Course Instructor: Nan Zhang (Office hours by appointment)
Course Website: https://nanzhangresearch.github.io/QMIR_HWS24
Date and Time: Tuesdays from 10:15 - 11:45 in EO 162 Seminarraum (Schloss Ehrenhof Ost)
Course Aims: This tutorial will provide a collaborative and immersive research experience where students work together with the instructor to design, implement, and analyze a survey experiment to study topical questions related to international politics. Upon completion of this course, students will have first-hand experience with the entire “research process cycle” from study design and pre-analysis planning, through fieldwork and data collection, to statistical analysis and paper writing. This course will provide students substantial preparation for writing their own BA theses.
In the first few weeks of the course, students will work with the instructor to identify a common research question and research design to pursue for the rest of the semester. Afterwards, we will work together (and in smaller groups) on the following tasks:
Students will be assigned tasks depending on their own interests and abilities.
Class meetings may include some “mini-lectures” from the instructor, but the majority of our time together will be spent updating each other on our progress and mapping out group tasks to be completed by the following week.
Diligent preparation and active participation are essential for the success of this class! Of course, situations may arise when you cannot attend class. As a courtesy, please let me and your fellow students know beforehand.
The survey experiment we design will be fielded on the January wave of the German Internet Panel (GIP). We expect to receive the data from the GIP by the end of February / early March.
Thus, although we will not have the results available by the end of the semester, all of the other “elements” of the paper (including the code to analyze the data once they are in) should be complete.
Once we have received the data from the GIP, Nan will run the prepared analysis code and share the results with the class.
At this point, students can decide whether they want to continue working on the final paper. Regardless of what you decide, you will all be co-authors on the final paper.
For students who want to continue, Nan will schedule a meeting where we will discuss the findings. Based on our meeting, Nan will draft up a short discussion and conclusion to the final paper. Students have 1 week to provide feedback. Afterwards, Nan will submit the manuscript to the agreed upon journal.
Note: the following is drawn from Jessica Calarco and Jesse Stommel’s ideas on “ungrading.”
This course will graded a bit differently than what you are used to from other classes. You will still be expected to complete assignments, and you will still receive a numeric grade on your transcript at the end of the semester. However, your grade will be determined based on your own self-assessment of your learning and effort in the class, with the possibility of adjustments up or down from me.
During the semester, you will be asked to complete two self-evaluation exercises (once at midterm, and once at the end of the semester). These exercises will include a series of questions about your work in the course. Specifically, you will be asked to reflect upon:
When assigning final grades, I will strive to honor your end-of-semester assessment of your own performance and progress in this course. However, I reserve the right to alter your proposed grade as appropriate, based on my own evaluation of your performance and progress in the course as a whole. If such an alteration seems warranted, I will contact you to set up a meeting to discuss your work in the course.
Additional Assignment: to help you keep track of your progress throughout the semester, it is helpful to keep a weekly “journal”. In particular, after each class session, take 5 minutes and write down (1) what is one thing that you learned over the past week, and (2) what is something that you did not understand?
Please include this journal as an appendix to your final reflection paper.
Note: the current version of the syllabus is a work in progress.
Session 1 (3 Sept) Introduction
Nan will introduce the course, go over the syllabus, and present the broad research question. Slides are here.
Students should confirm whether they want to take the class. We will then create a class mailing list or whatsapp group.
Assignment for next week. Please read the following and answer the reading questions:
Hirschberger et al. 2022. Making sense of the past to understand the present: Attributions for historical trauma predict contemporary social and political attitudes.
Q1: why might reminding Germans of the Holocaust shape attitudes towards the current Israeli-Palestinian conflict?
Q2: in your opinion, which types of people are more (or less) likely to feel collective guilt when they are reminded of the Holocaust?
Mutz. 2011. Population-Based Survey Experiments Chapter 1, pages 1-12.
Q3: in your own words, what is a population-based survey experiment?
Q4: what does Mutz mean by “internal” and “external validity”?
Please write up your answers (max 1 page) and submit them to me via email by Sunday, 23:59.
Session 2 (10 Sept) Theory Development
We will start of with a mini-lecture on survey experiments and discuss how this methodology can be used to answer the broad research question. This lecture will also introduce students to the terminology of treatment effects and treatment heterogeneity (that is, the treatment may work differently for different types of people). Slides are here.
The bulk of our time will be spent workshopping your ideas about treatment heterogeneity (i.e. Q2 from last week’s homework). In groups, students should come up with an idea about treatment heterogeneity that they would like to explore in more depth.
Assignment for next week. Working in their groups, students should conduct a preliminary independent literature review. Your lit review should answer the following question:
Please prepare a short (10 minute max) presentation of the results of your literature review (including a bibliography) for the next class session. Your goal is to “pitch” your idea to your classmates and convince them (and me) that this is THE topic that we should pursue as a class.
Session 3 (17 Sept) Introducing the GIP and Refining / Pivoting the Research Idea
We will begin with a short information session about the German Internet Panel (GIP) led by Anne Balz.
Afterwards, students will present their pitches to class and we will vote on 1 or 2 ideas to pursue further.
Finally, Nan will give a short presentation on a similar study (by Helbling and Traunmüller, hereafter: HT) that was just fielded on the GIP this year. We will discuss how to move forward in light of this new information.
Here are the slides for this week.
Assignments for next week.
Read this short policy brief (in German). This will give you a sense of variables used in HT’s study.
Read HT’s programming instructions. This will preview how we should submit our own programming instructions later on in the semester. You can also see the exact question wordings and treatments used. Finally, pay special attention to the randomization instructions (listed under “Experimental Split”) used to divide respondents up into different “treatment” and “control” groups.
For next week’s class, we will do an exercise where we analyze HT’s data in R. If you are unfamiliar with R, please have a quick look at this excellent online material from Lion Behrens.
Session 4 (24 Sept) Analyzing Experimental Data
Nan will demonstrate how to analyze HT’s data in R.
Assignments for next week.
Complete the short problem set in R. You may work in groups (3 people max!). Please send me your R file by Sunday 23:59. Be prepared to present your problem set answers in class next week.
Think about the theory some more: does it make sense that triggering feelings of guilt about the Holocaust would affect foreign policy attitudes (vis-a-vis Israel, or more broadly)? Write down your thoughts / arguments in one paragraph, and send it to me by Sunday 23:59. This is an individual assignment.
Please read pp.83-93 and 101-106 of Mutz 2011. Your task is to come up with an effective experimental treatment (with or without deception). You could come up with an idea completely from scratch, or else adapt / copy a manipulation that other researchers have already used. In either case, think carefully about whether your treatment actually manipulates the theoretical quantity of interest. Finally, don’t forget to consider what happens to the control group! Make a short (2 slides max!) presentation demonstrating your idea. You can work individually or in groups.
Session 5 (1 Oct) Brainstorming Experimental Treatments
Here are the solutions to the problem set.
Slides for this week.
We’ll spend our time discussing possible experimental treatments and narrowing down the list of possible candidates. The goal of the class is to identify 2-3 designs that we can pursue further.
Assignments for next week.
Please read Kerzer et al. 2014. Moral Support: How Moral Values Shape Foreign Policy Attitudes to get a sense of what “militant internationalism” (MI) and “cooperative internationalism” (CI) refer to.
In addition, please finish your prototypes that you started in class this week.
We will also spend time next week developing a common set of manipulation checks and outcome measures that we will work for all of the prototypes.
Session 6 (8 Oct) Manipulation Checks and Outcome Measures
We will spend the first half of this session discussing and refining the manipulation checks and outcome measures.
In the second half, Nan will give a short presentation on how to programme a simple survey in Qualtrics.
Working in groups, you will then start to “mock up” your preferred experimental design, the manipulation checks, and the outcome measures in Qualtrics.
Assignments for next week.
Find a couple of your family and friends to test your survey. Make any changes you want, and send me your final version by Wednesday 23:59. Nan will then pilot test all of your designs with a real online sample.
Session 7 (15 Oct) Looking at Pilot Data
We’ll look at the results from the pilot tests.
Pilot test code and rawdata p1, rawdata p2 and rawdata p3.
Assignments for next week.
Session 8 (22 Oct) Pilot Data Continued
We will start by discussing your midterm evaluations on the how the class is going so far. This is also a chance for us to map out the road ahead.
Afterwards, we will continue to look at our pilot data. The goal is to decide on a draft design which we submit to the GIP.
Here is a link to the code and data that we will need for class.
Assignments for next week:
Please rewrite the bolded sentence in the articles treatment:
Die deutsche Geschichte ist untrennbar mit dem Holocaust verbunden, bei dem Millionen unschuldige Menschen aufgrund von Hass und rassistischer Ideologie verfolgt und ermordet wurden. Diese dunkle Epoche zeigt die verheerenden Folgen von Antisemitismus. Heute steht Israel vor existenziellen Bedrohungen durch regelmäßige Terrorangriffe.
Ideas were either (1) find a more general term for violence instead of (or in addition to) “Terrorangriffe”, or (2) including some information about the founding of Israel.
Look through the remainder of today’s R code, especially the section on MI and CI. Let me know whether you want to keep the MI and CI items.
Watch the first 40 minutes (up to the Q+A) of the following video on pre-registration
Sign up for a free account in OSF.
Session 9 (29 Oct) Pre-Analysis Plans
We will discuss feedback from the GIP and make the required changes.
Next, we will discuss the reading from last week about why we should pre-register our research. Here are the slides.
Nan will present the OSF template for pre-analysis plans. We will start filling in this template together.
The goal for the rest of the semester is to learn the statistical concepts + R skills to be able to complete the pre-registration template.
Assignment for next week: read chapter 7 of moderndive, and actually work through the examples in R. You are encouraged to do this in groups. There’s no need to submit anything this week.
Note: if you are new to R and don’t know what ggplot is, have a look at chapter 2 and / or this online material from Lion Behrens.
Session 10 (5 Nov) Potential Outcomes and Uncertainty
First we’ll finish a couple of small things in the OSF template.
Nan will give an applied lecture – with R code examples and toy data – to illustrate the intuition behind statistical uncertainty as applied to experimental data.
We will show how, in experiments, control variables are unnecessary to account for potential variable bias.
This is also a good opportunity to review some of the statistical concepts we touched upon in week 2, but now we will have the chance to see how things work in R with some data.
Assignment for next week:
Read chapter 9 of moderndive, and actually work through the examples in R. You are encouraged to do this in groups. There’s no need to submit anything this week.
Session 11 (12 Nov) Hypothesis Testing and Power Analysis
Nan will give an applied lecture (with R code examples) on hypothesis testing and statistical power.
We will then simulate a power analysis using our pilot data. The goal is to answer the question: given the GIP sample size, what is the minimum effect size that can be detected at power = 0.8?
Assignment for next week: Watch this video on multiple testing corrections.
Please also test the programmed questionnaire and let me know of any errors that you find.
Session 12 (19 Nov) Wrap Up, and Multiple Hypothesis Testing
We will do a power calculation using our pilot data. Here’s the R code.
Nan will give a presentation about multiple hypothesis testing corrections. Here’s the R code if you want to follow along.
Afterwards, we will finish filling in the pre-analysis plan.
We are now in a position to (i) outline the final paper and (ii) write the final code that we will run to analyze the data.
Assignment for next week: Find a published survey experimental paper (ideally, using GIP data) and read how they present the data and methods section.
Make an outline (bullet points for topic sentences of paragrahs and supporting sentences) for how we should write our own data and methods section. You may work in groups.
Also: please fill in your official course evaluations. You should have received an email from student services.
Session 13 (26 Nov) Outlining and Data and Methods
Working together in class, we will put together an outline of the data and methods sections of the paper.
Assignment for next week: Make “pen and paper prototypes” of the graphs that you would like to present in this paper. You can work in groups. We will work on coding up these graphs next week.
Session 14 (3 Dec) Data Visualization
Working together in class, we will write the code for creating the graphs in R.
Finally, we will spend some time wrapping up the class, taking stock of what you have learned, and discussing next steps for paper submission.
Final Self-Evaluation due by 23:59 on Friday, 17 January 2025.
Extra R Practice:
If you want some extra practice with R, check out R4 Data Science. There, you can find code examples and exercises for the basic things you will need to do in R (data cleaning and data visualization).