Exercise Overview

You are analyst no. 6

This page sets out the whole exercise. Your work happens either in the Browser lab (Route 2) or in the Positron workspace you downloaded (Route 1). The two routes are identical in what they let you do. Come back here whenever you want the overview again.

The claim you are working on

Five independent analysts reanalysed a claim made by the original author (OA) as part of the Multi100 project. The claim, as presented to all analysts, was:

“…poor economic performances … decrease positive dimensions of EU framing” (Teney 2016, p. 619)

The anchor result – the specific number that the Task 2 constraints pointed to – is:

Unemployment → cosmopolitan EU framing: b = −0.0034, t = −4.03, p < .001 (Table 3, Model 1)

This comes from a three-level random-intercept model using pooled Eurobarometer data (16 waves, 2004–2013, 27 EU countries). Your first job is to get as close to this result as possible using a transparent, reproducible pipeline. Your second job is to deviate from it – deliberately, with a written justification – and see what that does to the estimate.

Five analysts, five answers

Before you begin, here is what the five Multi100 analysts found when they reanalysed the same claim:

Analyst Tool Statistic N Partial r
018OL R t = −1.21 420 −0.060
C6HJR R t = −3.80 270 −0.242
KEVF1 R t = −5.90 209,826 −0.452
KQXUE Stata t = −7.17 270 −0.419
PRL47 Stata z = −3.59 394,575 −0.006

All five reported a negative effect (same direction as the original). The partial r ranges from −0.006 to −0.452, a factor of 75, so five competent analysts working on one claim arrived at widely different estimates. The multiverse debrief in Part 4 explains most of that gap; for now it is enough to know that the gap is there.

The day, task by task

The exercise runs as five numbered tasks across four 90-minute parts. For every task there is a route note – what to do in the Positron workspace (Route 1) and in the browser lab (Route 2). The two reach identical numbers.

Part 1 – Why any of this, and with what tools

Task 1 – Get yourself running (~8 minutes, and the break if you need it). The only aim is to reach the point where a line of R runs and the data are in front of you. There is no prize for finishing early, and getting this right now saves an hour of misery later. Route 1: open the workspace folder in Positron, let the packages install, and render the report skeleton once – if it renders, everything works; if it does not, source("R/check_setup.R") names what is missing. Route 2: wait for the engine to finish downloading, then run the setup-check cell and confirm the panel has 270 rows. The vocabulary warm-up on the same page – three R’s and one impostor – is there to fill the waiting.

If Route 1 will not cooperate, do not fight it. Switch to Route 2 and carry on; nothing in the day depends on which one you use.

Part 2 – The question underneath

Task 2 – Meet the panel, name the estimand, draw the graph (~18 minutes). Three pieces, in order. First, look at the data: the EU-frames dataset (EUframes_cy.csv) is the country-year panel behind the whole exercise – 270 country-years, unemployment and GDP growth alongside several EU-framing measures, published in the analyst’s OSF fork (osf.io/6zqct, main folder; the official Multi100 archival record is at osf.io/8rtwe). Run the small-multiples figure and find the countries that move sharply around 2009, because that within-country variation is exactly what the model will use.

Second, write the estimand in words: the unit-specific quantity, the target population, the aggregation. The template gives you the shape; the value is in being forced to be precise about this claim in particular.

Third, complete the causal graph. Take the # off the arrows you are willing to defend, re-run adjustmentSets(), and watch what the graph now demands. Then two sentences for your notes: what do two-way fixed effects adjust for, and what can they not fix? Route 1: edit R/dag_starter.R, where every arrow is an ordinary R formula and adopting one means deleting a #. Route 2: the same graph with dagify() in the browser lab. The causal-graphs companion module works the material through slowly if you want it.

Part 3 – What you can actually do

Task 3 – Reproduce the recorded result (~22 minutes). Fit the constrained two-way fixed-effects model – mcosmo ~ unemp_c, with country and year fixed effects – and confirm you get t = −3.804 (df = 233, N = 270). This is the reproduction step, and −3.804 is the value on record for analyst C6HJR: the same data and the same code give the same number. Route 1: run the model in your working script. Route 2: fill the gaps in the exercise cell; there are two hints and a full solution, and using them is not cheating.

Your reproduction is one check among several, and the paper fares differently on the others. It was put through all three Rs in the SCORE dossier (osf.io/h7432): the push-button reproduction failed, the source-data reproduction was recorded as “not reproduced”, and a preregistered replication on fresh data (N = 352,114) flipped sign, with b = +0.002 and p = .046.

Task 4 – Work the menu (~20 minutes). This task is exploration, and it is meant to be. Try several specifications from the specification menu, see what each one does, and get a feel for which choices move the estimate and which barely touch it. try_spec() takes one argument per axis and every argument already carries the baseline you just reproduced, so changing one thing means changing one argument. my_experiments() prints everything you have tried, side by side – the small multiverse you have just built. Then pick the one you would defend if you had to, and submit_exploration() puts it on the live Multiverse chart as an exploratory result, drawn as an open circle.

Nothing here commits you to anything. You are allowed to look, change your mind and try again – which is why the label matters.

Part 4 – Reading the multiverse, choosing one path

The part opens by reading what the room built: the class chart laid over the full 840-specification curve and the five published analysts, which analytical choice moves estimates most, where the sign flips come from, and your own percentile – the share of the 840 specifications more negative than the estimate you submitted. It then turns to what a curve can and cannot establish, and to the argument that once a question is fully specified there is only one correct analysis. The reading on the specification curve is the companion for this part.

Task 5 – Declare one specification, run it, report it (~20 minutes). Now you commit. Pick ONE analysis from the menu, one choice on each of the eight axes, and complete the preregistration block before you run anything. It is a shortened but real preregistration template: what you are testing, what you already know about the data, your one specification, why those controls (grounded in your Task 2 graph), and the rule you will judge the result by. claim_align <- TRUE is the day’s convention and is applied for you: negative framings are reverse-coded and the claim-expected sign of GDP growth is flipped, so ‘supports the claim’ reads as a negative sign for everyone.

Render that block (Route 1) or copy it into your notes (Route 2) so it is written down before you see the result. Then translate the declaration into model syntax yourself – build_model() accepts only code that matches what you declared – run it, and report_result() prints a one-click, pre-filled form link. Your dot joins the chart as a declared result, drawn as a filled circle beside the open exploratory ones.

Task 4 and Task 5 run identical code. What separates them is that an exploratory specification can have been chosen because of its result and a declared one cannot, and seeing the two clouds side by side is why you submit twice.

Three things make a result count

Your result is valid for the multiverse chart if it:

  • reports the claim-carrying coefficient (unemployment or GDP growth, or its transformation) from a model on EUframes_cy.csv,
  • is submitted to the class-results form via the report_result() link (so it lands on the live Multiverse chart), and
  • has a written justification for the choice you made (2 sentences is enough).

The justification is what the whole exercise depends on. Auspurg & Brüderl (2021) argue that a multiverse of unjustified specifications shows dispersion and nothing more; the stated reasons are what let it count as evidence.

Everything you need, in one place

  • Browser lab – Route 2: the whole exercise in a browser tab, nothing installed.
  • Specification menu – the eight axes you choose one option from, with a worked example.
  • Reading a specification curve – the companion reading for Part 4: what a large set of results can and cannot establish.
  • Cheatsheet – the one-page card to keep open on the day, with the in-room links and what to do if something stalls.

Two of the nine companion modules pick up the skills that the day sets aside and take them further, at your own pace: