Browser lab – the whole exercise, no installation
Track A: everything runs in this tab
Five analysts reanalysed a claim from Teney (2016) for the Multi100 project (Aczel et al. 2026); you are repeating the exercise following similar guidelines. Read the full task description, the claim, and the five-analyst table of findings on the Exercise overview page (the workshop slides discuss the broader context of the project).
This page is the runnable version of Tasks A to D for those working entirely in the browser.
Task A – Concepts (warm-up while the engine loads)
Before touching data, fix the vocabulary. In the four scenarios below, three are doing one of our three R’s; one is doing something else entirely. Read the four scenarios, then label each in the code cell below.
Scenario 1: A reviewer reruns the authors’ Stata .do file on the authors’ data and gets a different coefficient from the one printed in the paper. This is an example of ______, even though what they tried to do failed.
Scenario 2: The analyst uses the authors’ submitted replication dataset, but they re-implement the model in R instead of Stata and centre the predictor. Is this a case of ______ or ______?
Scenario 3: A preregistered team tests the identical hypothesis on a separate, much larger dataset chosen to be theoretically equivalent, and gets results that contradict the original. Their conclusion is that the original analysis has failed to ______.
Scenario 4: Instead of the survey data used by the original authors, another researcher tests the same underlying idea on social-media texts using a machine-learning classifier method that the original never used. To their surprise, the authors’ original findings prove ______ and the researcher concludes that they provide a reliable middle-range theory of the phenomenon they investigate.
Now, if the R engine has loaded, give it a test run by filling in the gaps in the same text in the code chunk below and run the code to see the output.
TipAnswers – and the two traps
- reproducibility – rerunning the same code on the same data. Here it failed to match, but the activity is still reproducibility; “not reproduced” is the verdict.
- arguably still reproducibility (a grey area) – re-implementing on the same data in different software is technically a re-computation – but centring the predictor nudges toward robustness, not to mention that different software may have different implementations of the same estimators, so one must make sure they understand the functions they use.
- replicability – same question, independent new data chosen to be theoretically equivalent.
- generalisability – the impostor: new data and a new method at once. It is the fourth cell of the matrix, not one of the three we test today.
Task B – Get the data and look at it
The published artifact rep_data.csv is already loaded for you as rep. It comes from the analyst’s fork on OSF (osf.io/6zqct – “Multi100 | Teney_EurSocioRev_2016 | Analyst_C6HJR”); the official Multi100 archival record is at osf.io/8rtwe. Track B participants fetch it over the network with the osfr package; in the browser it is pre-loaded so you can get straight to the analysis.
Run the cell below to inspect the panel. Edit it freely – this is your environment.
You should see 270 rows, 27 countries, 10 years (2004–2013), and 11 columns. The outcome we care about first is mcosmopolitan – the mean ‘cosmopolitan’ framing of the EU in each country-year (a 0–1 scale; see the codebook). The exposure is unemp, the annual unemployment rate.
Now look at the outcome over time, one small panel per country:
Most countries drift gently; a few move sharply around the 2009 crisis. That variation – within a country, over time – is exactly what the fixed-effects model in Task D1 uses.
Task C – Estimand and DAG
Before fitting anything, say what you are trying to estimate, in words. A template you can adapt:
The unit-specific quantity is the change in a country-year’s mean cosmopolitan EU framing associated with a one-point change in that country’s unemployment rate. The target population is EU member states over 2004–2013. The aggregation is the average of those unit-specific effects, holding fixed each country’s time-invariant traits and each year’s EU-wide shocks.
That last clause is a causal claim, and a DAG makes it checkable. If the dagitty.net page below doesn’t load, navigate to that external site.
Paste the code below into the Model code field and update the graph:
dag {
unemployment [exposure]
framing [outcome]
country -> unemployment ; country -> framing
year -> unemployment ; year -> framing
unemployment -> framing
}Task D1 – Reproduce the constrained result
This is the reproducibility check. Fit the two-way fixed-effects model the Multi100 Task-2 constraints point to:
mcosmopolitan ~ unemp_c with fixed effects for country and yearwhere unemp_c is the unemployment rate centred on its grand mean. The target is t = −3.853 (df = 233, N = 270) on the published rep_data.csv.
The cell below is an exercise. Replace the blanks (______) with the missing code, then press “Run Code”. Stuck? Open the hint; want to check? Open the solution.
NoteHint 1 – centring
To centre a variable on its grand mean, subtract the mean of the whole column:
rep$unemp_c <- rep$unemp - mean(rep$unemp)With fixed effects present this does not change the t-statistic, but it is the transformation the constrained specification asks for, so do it explicitly.
NoteHint 2 – the fixed-effects syntax
In fixest::feols, the bar | separates the regression from the fixed effects. To add country and year fixed effects, list them after the bar separated by +:
feols(mcosmopolitan ~ unemp_c | cntry + year, data = rep)
TipFully worked solution
rep$unemp_c <- rep$unemp - mean(rep$unemp)
m1 <- feols(mcosmopolitan ~ unemp_c | cntry + year, data = rep)
broom::tidy(m1)You should get estimate ≈ −0.00341, std.error ≈ 0.00088, statistic ≈ −3.85. feols reports IID standard errors by default when two fixed-effect dimensions are present, which is why it matches the plm value to three decimals.
The result you got is not the result on file
Here is the twist that makes this a workshop and not a button-press. The value recorded for this analyst (C6HJR) in the Multi100 dataset is t = −3.804, not the −3.853 you just produced. Same model, same published data, two different numbers. Run the next cell to see them side by side:
ImportantWhy the discrepancy?
The result recorded for this analyst (C6HJR) in the Multi100 dataset is t = −3.804, not −3.853. Why is that?
The original data-prep script read the EB 70.1 file (ZA4819) twice by mistake: once for 2008 (correct) and again in place of EB 72.4 (ZA4994) for 2009 (incorrect). The analyst spotted this during the project and corrected it. The recorded result came from the corrected pipeline. The annotation is preserved in the analyst’s fork (osf.io/6zqct, folder moreh_rep_vAPR2025/): the script carries the note “APRIL 2025 – Mistake in dataset here; the dataset associated with ‘eb724’ should be ‘ZA4994_v3-0-0.dta’”. The official Multi100 component (osf.io/8rtwe) preserved only the earlier, uncorrected folder; the fork publishes the full trail.
Coding can go wrong - even during a reanalysis! Keeping materials open and well documented helps spot any mistakes and correct them. That’s the purpose of reproducibility.
Task D2 – Your fork in the path
This is the robustness check, and it is the part your classmates cannot copy from you. You pick one deviation from the published specification, justify it in two sentences, and report what it does to the estimate. Together the room samples the multiverse of defensible analyses.
The cell below has six choices at the top – the same axes as the full specification menu. Change one (or more), read the comments for the allowed options, and run it. A small helper fits whichever specification you describe.
Preregister, then report
The point of preregistration is that you commit to a choice before you see what it does. It stops you quietly picking the specification that happens to give the answer you wanted.
Paste the function below into the editable cell (or run it here after running your model) to get your one-click submission link. It is self-contained and works for fixest, plm, and lm models:
Click the submission link it prints and press Send. Your result appears as an orange dot on the live Multiverse chart within seconds. That is your contribution to the class multiverse.
Task E – Still want the GitHub experience?
You did the whole analysis without installing anything – but Track A participants can still finish with a published repository. Three routes, easiest first:
- Save your code. Copy the cells you edited into a plain text file on your laptop (call it
analysis.Rorreport.qmd). Everything you need is in this tab. - Make a repo by upload. On github.com, create a new public repository, then use Add file → Upload files to drag your saved file in. No git, no command line – your first repo in about three minutes.
- Edit in the browser. On any GitHub repository, press the
.key to open github.dev (a full VS Code editor in the browser), or changegithub.comtogithub.devin the URL. You can edit, commit, and push entirely online.
The Cheatsheet has the click-by-click version, including how to switch on GitHub Pages so your report gets a live URL.
References
Aczel, Balazs, Barnabas Szaszi, Harry T. Clelland, et al. 2026. “Investigating the Analytical Robustness of the Social and Behavioural Sciences.” Nature 652 (8108): 135–42. https://doi.org/10.1038/s41586-025-09844-9.
Teney, Céline. 2016. “Does the EU Economic Crisis Undermine Subjective Europeanization? Assessing the Dynamics of Citizens’ EU Framing Between 2004 and 2013.” European Sociological Review 32 (5): 619–33. https://doi.org/10.1093/esr/jcw008.