Browser lab – the whole exercise, no installation
Route 2: everything runs in this tab
Five analysts reanalysed a claim from the EU-frames case (Teney 2016) for the Multi100 project (Aczel et al. 2026); you are analyst no. 6. Read the full task description, the claim, and the five-analyst table of findings on the Exercise overview page (the workshop slides discuss the broader context of the project).
This page is the whole exercise, runnable in the browser: five tasks that take you from a setup check to two dots on the live class chart – one explored, one declared.
Task 1 – Get yourself running
Nothing conceptual here: the job is to reach the point where a line of R runs and the data are in front of you. Press Run Code on the cell below. If it prints the three lines it should, you are set up and everything else on this page will work.
You should see 270 rows, 27 countries and 10 years, and TRUE on the last line. If any of that is missing, reload the page and wait for the engine to finish downloading before pressing Run Code again. If it still fails, say STUCK in the chat.
Warm-up – three R’s, and one impostor
Fix the vocabulary before you touch any data. Three of the four scenarios below are doing one of our three R’s – reproducibility, robustness, replicability. The fourth is doing something else entirely. Read all four, then label each one in the code cell.
Scenario 1: A reviewer reruns the authors’ Stata .do file on the authors’ data and gets a different coefficient from the one printed in the paper. This is an example of ______ even though what they tried to do failed.
Scenario 2: The analyst uses the authors’ submitted replication dataset, but they re-implement the model in R instead of Stata and centre the predictor. Is this a case of ______ or ______?
Scenario 3: A preregistered team tests the identical hypothesis on a separate, much larger dataset chosen to be theoretically equivalent, and gets results that contradict the original. Their conclusion is that the original analysis has failed to ______.
Scenario 4: Instead of the survey data used by the original authors, another researcher tests the same underlying idea on social-media texts using a machine-learning classifier method that the original never used. To their surprise, the authors’ original findings turn out to be ______, and the researcher concludes that they provide a reliable middle-range theory of the phenomenon they investigate.
Fill in the gaps in the code cell below and run it to print your answers back.
TipAnswers – and the two traps
- reproducibility – rerunning the same code on the same data. Here it failed to match, but the activity is still reproducibility, and ‘not reproduced’ is the verdict.
- arguably still reproducibility, in a grey area – re-implementing on the same data in different software is technically a re-computation – but centring the predictor nudges towards robustness. Two packages can also implement the same estimator differently, so make sure you know what the function you call is doing.
- replicability – same question, independent new data chosen to be theoretically equivalent.
- generalisability – the impostor: new data and a new method at once. It is the fourth cell of the matrix, not one of the three we test today.
Task 2 – Meet the panel
The EU-frames dataset (EUframes_cy.csv) is already loaded for you as euframes. It is the country-year panel behind the whole exercise. The copy that this page loads is byte-identical to the one published in the analyst’s fork on OSF (osf.io/6zqct – “Multi100 | Teney_EurSocioRev_2016 | Analyst_C6HJR”, main folder), and the official Multi100 archival record sits at osf.io/8rtwe. Route 1 participants download the same file into their Positron workspace; in the browser it is pre-loaded, so you start straight on the analysis.
Run the cell below to inspect the panel. Edit it freely – this is your environment.
You should see 270 rows, 27 countries, 10 years (2004–2013), and 13 columns. The outcome we care about first is mcosmo, the mean ‘cosmopolitan’ framing of the EU in each country-year, on a 0–1 scale. The predictor is unemp, the annual unemployment rate. growth is annual GDP growth, and n_cy counts the survey respondents behind each row. The codebook describes the other framings and every remaining column.
Now look at the outcome over time, one small panel per country:
Most countries drift gently. A few move sharply around the 2009 crisis. That variation – within a country, over time – is exactly what the fixed-effects model in Task 3 uses.
Task 2, continued – name the estimand, draw the graph
Before fitting anything, say what you are trying to estimate, in words. A template you can adapt:
The unit-specific quantity is the change in mean cosmopolitan EU framing within a country-year associated with a one-point change in that country’s unemployment rate. The target population is EU member states over 2004–2013. The aggregation is the average of those unit-specific effects, holding fixed each country’s time-invariant traits and each year’s EU-wide shocks.
That last clause is a causal claim, and a directed acyclic graph makes it checkable. A DAG is a picture of what causes what: variables are nodes, arrows run from cause to effect, and the rules of the graph then tell you which variables have to be adjusted for if the arrow you care about is to be read cleanly. Here is the graph that this exercise commits to.
Five arrows are drawn. unemployment -> framing is the path of interest. Above it, country is a common cause of both: stable national traits such as size, EU tenure and labour-market institutions move local unemployment and shape how people frame the EU. year does the same job for the EU-wide shocks that hit every country in the same calendar year. The three pale nodes below carry no arrows yet, because whether they belong is a decision you make rather than one that the exercise makes for you.
You build the graph yourself in the cells below, in ordinary R. dagify() takes one formula per arrow, written the way you write a model, so framing ~ unemployment is the arrow that runs into framing. Read the tilde as ‘is caused by’ rather than ‘is regressed on’ and the direction comes out right every time. Because each arrow is a line of R rather than a fragment of a string, an arrow you are unsure about can sit there commented out, and adopting it means deleting a #.
dagify() and adjustmentSets() are ready in this page already, so you can go straight to the next cell. The optional cell below adds the picture: it installs ggdag, which brings igraph, ggraph and four further dependencies with it – roughly 23 MB of download, and a noticeable pause on a slow connection. Run it if you want the plot. Skip it and everything else in Task 2 still works, because the figure above is the same graph.
Now specify the graph. The five committed arrows are live; the arguable ones are commented out, one substantive claim per line. Uncomment any you believe, then run the cell.
With the five committed arrows the minimal adjustment set is { country, year } – exactly what a two-way fixed-effects model absorbs. Country fixed effects adjust for country and year fixed effects adjust for year, so the model you will fit in Task 3 and the graph you just wrote are one identification claim in two languages.
Now uncomment the growth pair and run the cell again. The set becomes { country, growth, year }, because a slump varies within a country over time and fixed effects only remove country-constant and year-common variation. Nothing in the Task 3 model reaches it, and your specification would need growth as a co-predictor to match your beliefs – a choice waiting for you on the menu in Task 5. The bailout arrows behave differently. A mediator sits on the path you are trying to measure, so the adjustment set does not grow when you add one; adjusting for it would strip out part of the very effect you are after.
Write two sentences for your own notes. What do two-way fixed effects adjust for, and what can they not fix? (Hint: anything time-varying within a country that you have not measured.)
Task 3 – Reproduce the recorded result
This is the reproducibility check. Fit the two-way fixed-effects model that the Multi100 Task-2 constraints point to:
mcosmo ~ unemp_c with fixed effects for country and yearwhere unemp_c is the unemployment rate centred on its grand mean. The target is t = −3.804 (df = 233, N = 270) – the value on record for analyst C6HJR in the Multi100 dataset.
The cell below is an exercise. Replace the blanks (______) with the missing code, then press “Run Code”. If you get stuck, open the hints below; the solution has the full code.
NoteHint 1 – centring
To centre a variable on its grand mean, subtract the mean of the whole column:
euframes$unemp_c <- euframes$unemp - mean(euframes$unemp)With fixed effects present this does not change the t-statistic, but it is the transformation that the constrained specification asks for, so do it explicitly.
NoteHint 2 – the fixed-effects syntax
In fixest::feols, the bar | separates the regression from the fixed effects. To add country and year fixed effects, list them after the bar separated by +:
feols(mcosmo ~ unemp_c | cntry + year, data = euframes)
TipFully worked solution
euframes$unemp_c <- euframes$unemp - mean(euframes$unemp)
m1 <- feols(mcosmo ~ unemp_c | cntry + year, data = euframes)
broom::tidy(m1)You should get estimate ≈ −0.00347, std.error ≈ 0.00091, statistic ≈ −3.80 – the recorded value reproduces from the published data and the published model. feols reports IID standard errors by default when two fixed-effect dimensions are present, which is why it matches the plm value to three decimals.
NoteEveryone gets the same number, and the five analysts still disagreed
You reproduced the recorded value exactly, and so did everyone else in the room, because everyone ran the same code on the same file – which is why reproducibility is the cheap R.
The five Multi100 analysts each wrote their own code for the same claim, and came back with five different answers, one of them not significant. Their pipelines are competent, well-organised work, so the spread is not a quality problem. It is what happens when a claim stated in words leaves several genuinely defensible quantities on the table, and different analysts estimate different ones without noticing they have chosen.
Task 2 named the quantity you are after and drew the graph that your adjustment set follows from. Task 4 now shows you how far the answer moves when the other choices move.
Task 4 – Work the menu
This task is exploration, and it is meant to be. You are going to try several specifications, see what each one does, and get a feel for which choices move the estimate and which barely touch it. Nothing here is a commitment; that comes in Task 5.
try_spec() takes one argument per axis of the specification menu, and every argument already carries the baseline you just reproduced. So changing one thing means changing one argument, and everything you do not name stays at baseline.
Now change one axis at a time. Each of the four calls below moves exactly one thing; run them, and watch which ones matter.
Then try your own combinations. The full set of arguments, with the baseline as the default for each:
Everything you try is logged. Print the log to see the small multiverse you have just built – only the axes you actually varied are shown, so the contrast is readable:
When you have tried three or four, decide which one you would defend if you had to, run it once more so it is the most recent, and submit it. It lands on the live Multiverse chart as an exploratory result:
NoteWhy ‘exploratory’?
Because you chose this specification after seeing what several specifications do, it cannot carry the guarantee that a declared choice carries – no one can rule out that it might have been chosen because of its result. This is something we almost never know in the case of published papers, unless their analysis has been pre-registered and departures from the pre-registration documented. In this workshop exploration is a natural step, but you’ll also practise pre-registering your chosen analysis; you will submit twice: once here, and once in Task 5.
Task 5 – Commit before you look
The point of preregistration is that you commit to a choice before you see what it does. It stops you quietly picking the specification that happens to give the answer you wanted. Here the commitment binds mechanically as well: the session remembers the choices you declare in this cell, and every later cell reads them from here. The model builder refuses code that departs from them.
Make one choice on each axis of the specification menu, complete the dotted lines, then run the cell. Every axis is live, the outcome family and the predictor included. claim_align <- TRUE is the day’s convention, applied automatically: negative framings are reverse-coded (1 − y) and the claim-expected sign for growth is flipped, so ‘supports the claim’ reads as a negative sign for everyone.
An optional mirror form logs preregistrations for the room, and the link is shared in the session. It is not required. The cell above, copied into your notes, is your preregistration record.
Task 5, continued – build, run and report
This is the robustness check, and it is the part that your classmates cannot copy from you: together the room samples the multiverse of defensible analyses. You have declared your specification. Now translate it into model syntax yourself, filling the three gaps – the outcome, the right-hand side, and the estimator code – exactly as you preregistered them. build_model() knows your declaration and accepts nothing else, and its error messages tell you what is off. Write the log form as log(unemp), and add any co-predictor with +.
NoteHint – from menu speak to model syntax
The formula uses the column names of euframes: the outcome you preregistered on the left of ~, the predictor terms on the right. For the baseline declaration it is mcosmo ~ unemp; with the log form it would be mcosmo ~ log(unemp); with a co-predictor, mcosmo ~ unemp + growth. The estimator is the menu code itself, in quotes: "fe_twoway", "fe_country", "re" or "pooled_cl". You do not write the fixed-effects or clustering syntax here – the estimator code stands for it, and Task 3 showed what it expands to.
When the model is accepted, run it. Every specification on the menu is pre-computed in the committed grid – the same numbers that a live fit produces, without downloading the heavier estimation packages into your browser:
Read the output like an analyst: the coefficient and its claim-aligned partial r (below zero supports the claim), and your placement – what share of the canonical 840-specification grid supports the claim more strongly than your choice does. The five Multi100 analysts run from about the 16th percentile (the most strongly claim-supporting) to about the 68th (near the null), and your dot is about to join theirs.
Now send it to the room. The result travels straight from your declaration, so there is nothing to fill in or transcribe. The cell prints your submission line and a pre-filled form link; click it and press Send. Your dot joins the live Multiverse chart within seconds, this time labelled declared rather than exploratory.
If the form will not open on your machine, read the printed numbers to the facilitator and your dot still lands.
A second declared specification? Edit the preregistration cell, rerun prereg(), rebuild, rerun. The mechanics allow it, and the label does not change – so be clear with yourself about what the second one is. Your first declaration was made without knowing its result. Anything after it was not, which puts it back in the same category as Task 4, and you should say so when you report it.
Taking it home
You did the whole analysis without installing anything. Putting your own work under version control and publishing the rendered report is the natural next step. A self-study module walks through it click by click: Git and GitHub self-study.
References
Aczel, Balazs, Barnabas Szaszi, Harry T. Clelland, et al. 2026. “Investigating the Analytical Robustness of the Social and Behavioural Sciences.” Nature 652 (8108): 135–42. https://doi.org/10.1038/s41586-025-09844-9.
Teney, Céline. 2016. “Does the EU Economic Crisis Undermine Subjective Europeanization? Assessing the Dynamics of Citizens’ EU Framing Between 2004 and 2013.” European Sociological Review 32 (5): 619–33. https://doi.org/10.1093/esr/jcw008.