Exercise Overview
Analyst no. 6
This page sets out the whole exercise. Your work happens in the workspace you downloaded and opened in your IDE; the Browser lab runs the same exercise in a browser tab, for following a demonstration, for trying it without an installation, and as a fallback. Come back here whenever you want the overview again.
The claim you are working on
Five independent analysts reanalysed a claim made by the original author (OA) as part of the Multi100 project. The claim, as presented to all analysts, was:
“…poor economic performances … decrease positive dimensions of EU framing” (Teney 2016, p. 619)
The published coefficient – the specific number that the Task 2 constraints pointed to – is:
Unemployment → cosmopolitan EU framing: b = −0.0034, t = −4.03, p < .001 (Table 3, Model 1)
This comes from a three-level random-intercept model using pooled Eurobarometer data (16 waves, 2004–2013, 27 EU countries). Your first job is to get as close to this result as possible using a transparent, reproducible pipeline. Your second job is to deviate from it – deliberately, with a written justification – and see what that does to the estimate.
Which claim, and why
The claim on the card is one of several that the paper makes. The SCORE programme selected claims from published abstracts, and the sentence it took from this paper is the first half of a two-part hypothesis. In the paper itself, that hypothesis reads: “Poor national economic performances significantly increase the negative and decrease the positive EU-framing dimensions. The effect is stronger on the materialist dimensions of EU framing” (p. 621). The second sentence, which compares two of the four framing scales with the other two, never reached the analysts, and neither did the second hypothesis of the paper, about the gap between high- and low-educated citizens, which the individual-level module takes up. The Multi100 project then narrowed the card further in its constrained phase, fixing the cosmopolitan scale as the outcome and the unemployment rate as the predictor, which is why the coefficient above is the one you reproduce.
Each of those steps was a decision that someone other than you made, and each one shaped what “replicating the claim” could mean. Testing the first sentence needs one outcome and one predictor. Testing the second needs a grouping of the scales, and the measurement module shows that the data only half support that grouping. The education hypothesis needs the respondents rather than the country-years. Choosing which sentence to replicate is the first specification decision of any replication, and it is made before the estimand, the graph or the menu, usually by the people who wrote the card. The course keeps to the card, so that its results can be set beside those of the five analysts.
The five Multi100 analysts
Before you begin, here is what the five Multi100 analysts found when they reanalysed the same claim:
| Analyst | Tool | Statistic | N | Partial r |
|---|---|---|---|---|
| 018OL | R | t = −1.21 | 420 | −0.060 |
| C6HJR | R | t = −3.80 | 270 | −0.242 |
| KEVF1 | R | t = −5.90 | 209,826 | −0.452 |
| KQXUE | Stata | t = −7.17 | 270 | −0.419 |
| PRL47 | Stata | z = −3.59 | 394,575 | −0.006 |
All five reported a negative effect (same direction as the original). The partial r ranges from −0.006 to −0.452, a factor of nearly 80, so five competent analysts working on one claim arrived at widely different estimates. The multiverse debrief in Part 4 explains most of that gap.
The tasks
The exercise runs as five numbered tasks across four 90-minute parts. Each task says what to do in the workspace and, where it differs, on the browser lab page, which produces the same numbers.
Part 1 – Credibility and tools
Task 1 – Get yourself running (~8 minutes, and the break if you need it). The only aim is to reach the point where a line of R runs and the data are in front of you. There is no prize for finishing early, and getting this right now saves an hour of misery later. Workspace: open the workspace folder in your IDE, let the packages install, and render the report skeleton once. If it renders, everything works, and if it does not, source("R/check_setup.R") reports what is missing. Browser lab: wait for the engine to finish downloading, then run the setup-check cell and confirm the panel has 270 rows. The vocabulary warm-up on the same page – three R’s and one impostor – is there to fill the waiting.
If your installation will not cooperate, do not fight it. Open the browser lab and carry on there; nothing in the day depends on it.
Part 2 – The claim and its estimand
Task 2 – Meet the panel, name the estimand, draw the graph (~18 minutes). The task has three parts, taken in order. First, look at the data. The EU-frames dataset (EUframes_cy.csv) is the country-year panel behind the whole exercise – 270 country-years, unemployment and GDP growth alongside several EU-framing measures – and it is published in the main folder of the analyst’s OSF fork (osf.io/6zqct), while the official Multi100 archival record is at osf.io/8rtwe. Run the small-multiples figure and find the countries that move sharply around 2009, because the model uses that within-country variation.
Second, write the estimand in words: the unit-specific quantity, the target population, the aggregation. The template gives you the structure, and filling it in makes you say precisely what this claim is about.
Third, complete the causal graph. Take the # off the arrows you are willing to defend, re-run adjustmentSets(), and check which adjustment set the graph now implies. Then two sentences for your notes: what do two-way fixed effects adjust for, and what can they not fix? Workspace: edit R/dag_starter.R, where every arrow is an ordinary R formula and adopting one means deleting a #. Browser lab: the same graph with dagify() in the browser lab. The causal-graphs companion module works the material through slowly if you want it.
Part 3 – Reproduction and exploration
Task 3 – Reproduce the recorded result (~22 minutes). Fit the constrained two-way fixed-effects model – mcosmo ~ unemp_c, with country and year fixed effects – and confirm you get t = −3.804 (df = 233, N = 270). This is the reproduction step, and reaching −3.804, the value recorded for analyst C6HJR in the Multi100 dataset, shows that the same data and the same code give the same number. Workspace: run the model in your working script. Browser lab: fill the gaps in the exercise cell; there are two hints and a full solution, and using them is not cheating.
Your reproduction is one check among several, and the paper did less well on the others. It was put through all three Rs in the SCORE dossier (osf.io/h7432): the push-button reproduction failed, the source-data reproduction was recorded as “not reproduced”, and a preregistered replication on fresh data (N = 352,114) flipped sign, with b = +0.002 and p = .046.
Task 4 – Work the menu (~20 minutes). This task is exploratory. Try several specifications from the specification menu, see what each one does, and get a feel for which choices move the estimate and which barely touch it. try_spec() takes one argument per axis and every argument defaults to the baseline that you just reproduced, so changing one thing means changing one argument. my_experiments() prints everything you have tried, side by side – the small multiverse you have just built. Then pick the one you would defend if you had to, and submit_exploration() puts it on the live Multiverse chart as an exploratory result, drawn as an open circle.
Nothing in this task is preregistered. You are allowed to look, change your mind and try again, and the chart labels the result as exploratory for that reason.
Part 4 – Multiverse and preregistration
Part 4 opens with a reading of the class results: the class chart laid over the full 840-specification curve and the five published analysts, which analytical choice moves estimates most, where the sign flips come from, and your own percentile – the share of the 840 specifications more negative than the estimate you submitted. It then turns to what a curve can and cannot establish, and to the argument that once a question is fully specified there is only one correct analysis. The reading on the specification curve is the companion for this part.
Task 5 – Declare one specification, run it, report it (~20 minutes). Pick ONE analysis from the menu, one choice on each of the fourteen axes, and complete the preregistration block before you run that specification. It is a shortened but real preregistration template: what you are testing, what you already know about the data, your one specification, why those controls (grounded in your Task 2 graph), and the rule you will judge the result by. claim_align <- TRUE is the convention for the day and is applied for you. Negative framings are reverse-coded and the claim-expected sign of GDP growth is flipped, so ‘supports the claim’ reads as a negative sign for everyone.
Render the report once that block is complete, so that the rendered report, which carries the date on which it was rendered, holds your preregistration before you run the specification that it declares. On the browser lab page, copy the block into your notes instead. Then translate the declaration into model syntax yourself – build_model() accepts only code that matches what you declared – run it, and report_result() prints a one-click, pre-filled form link. Your submission appears on the chart as a declared result, drawn as a filled circle beside the open exploratory ones.
Task 4 and Task 5 run identical code. What separates them is that an exploratory specification can have been chosen because of its result and a declared one cannot, and you submit twice so that the two sets of results can be compared side by side.
What counts as a result
Your result is valid for the multiverse chart if it:
- reports the coefficient on the predictor that measures economic performance (unemployment or GDP growth, or its transformation) from a model on
EUframes_cy.csv, - is submitted to the class-results form via the
report_result()link (so that it appears on the live Multiverse chart), and - has a written justification for the choice you made (2 sentences is enough).
The whole exercise depends on the justification. Auspurg & Brüderl (2021) argue that a multiverse of unjustified specifications shows dispersion and nothing more; the stated reasons are what let it count as evidence.
Links
- Browser lab – the whole exercise and its data in a browser tab, for demonstrations and for trying it without an installation.
- Specification menu – the fourteen axes you choose one option from, grouped into the three levels of decision they belong to, with a worked example.
- Estimand choices – set the seven Level 1 choices, read the question that they define, and watch how far the answer moves when only its measurement changes.
- Reading a specification curve – the companion reading for Part 4: what a large set of results can and cannot establish.
- Cheatsheet – the one-page card to keep open on the day, with the session links and what to do if something stalls.
Two of the sixteen companion modules pick up the skills that the day sets aside and take them further, at your own pace:
- Git and GitHub for reproducible research – version control and publishing your report, the part lifted out of the core day.
- Reproducibility repositories – archives, OSF, and preregistration templates in more depth.