Applied replication for data skills

Chris Moreh

Open Research Conference · Newcastle University · 16 June 2026

Workshop 1 · Open Research Conference · 10:15–12:45 · Room 1.06

1. What survives when science repeats itself? – three Nature papers
2. Whose question is it anyway? – estimands and DAGs
3. You are analyst no. 6 – your own reanalysis, published before lunch

You leave with: your own public repo · a live web page · a row in the class multiverse

Work in pairs – one prepared laptop per pair

Welcome – good to have you here. This morning has three movements. First I’ll show you what happened when Nature asked, three times over, whether social science holds up when you repeat it. Then we sharpen the question itself – estimands and causal graphs. And then you become analyst number six on a real published claim, and you’ll publish your own reanalysis before lunch. You leave with your own public repository, a live web page, and a dot on the class’s chart. One practical thing before we start: find a partner – ideally one prepared laptop per pair.

PART 1 – The three Rs

Talk I · 20 minutes

Part one, then. Three words that sound alike and are not – and the three Nature papers that put numbers on each of them. [18 minutes for this part – about 80 seconds a slide.]

Replication crises

Another replication crisis – terminological confusion?

The term reproducibility has been used as a synonym for a number of different characteristics – reproducibility, robustness, replicability, repeatability, credibility, and trustworthiness.

Successfully rerun the same code on the same data? Re-analyse the same data a different way? Collect fresh data and analyse it in the same way? Using the same data and same analysis but asking a different question? Asking the same question but answering it with fresh data and a different analysis?

Part of the problem is the word itself. For years replication covered everything: rerunning someone’s code, re-analysing their data a different way, collecting new data altogether. Those are three different operations, answering three different questions – but because we called them all replication, the crisis debates had people talking past each other for a decade. So before any numbers, we need vocabulary.

The data \(\times\) analysis matrix

Alipourfard et al. (2021); Nosek et al. (2025)

Here is the whole framework on one slide, and if you take a single thing home today, make it this grid. The question stays the same throughout; what varies is the data and the analysis. Same data, same analysis – reproducibility: if I rerun your recipe, do I get your number? Same data but a different, justifiable analysis – robustness: does your finding survive my defensible choices? New data, same question – replicability. The fourth cell, new data and new analysis, is really generalisation, and we set it aside today. Sit with this for a moment – everything that follows this morning hangs off it.

Related concepts

Repeatability is used when referring to reproducibility, robustness, and replicability as a constellation of concepts intended to assess whether answers to scientific questions are unchanged when different steps of the research process are repeated.

Credibility and trustworthiness refer to the confidence one can have in research findings. Because scientific knowledge claims are tentative, they are not synonymous with truth.

For example, credibility is advanced by addressing potential competing interests in the conducting and reporting of research, by taking existing knowledge into account, by using methods that have been validated, by controlling for biases in the research context, by pursuing precise and reliable evidence, and by calibrating the strength of the claims to the uncertainty of the evidence.

Nosek et al. (2025)

Three SCORE articles

SAME data

DIFFERENT data

SAME methods

DIFFERENT methods

Sánchez-Tójar et al. (2026)

Same four-cell map you have just learned – but now each box carries the actual disciplinary breakdown from its study. Three of the four corners have a SCORE paper behind them: reproduction top-left, replication top-right, reanalysis bottom-left. The fourth, generalisation, we set aside today. Notice the pattern jumps out by discipline rather than by concept: economics and political science sit at the top of reproduction, sociology lags; replication hovers around half almost everywhere; and reanalysis holds the original conclusion three times in four. Same data on the wall, three very different questions.

What is SCORE?

• DARPA-funded programme to gauge the credibility of social-science claims
• 62 journals, papers from 2009–2018, stratified sample
• Each paper’s central claim traced from abstract to the supporting statistic
• Thousands of researchers; three repetition studies built on the same frame

Where do these numbers come from? All three papers sit on one programme, SCORE – funded by DARPA, oddly enough, who wanted to know whether you could predict which findings hold up. The backbone is a stratified sample from sixty-two journals, papers published between 2009 and 2018, with each paper’s central claim traced from the abstract down to the exact statistic that supports it. The three studies then took that same frame and each asked one of our three questions. So when the numbers land in a moment, they are about the same body of literature, asked three different things.

Reproducibility – Miske et al.

• 600 papers sampled (62 journals, 2009–2018)
• Ask the authors for data and code, then try to rerun the reported result
• How often can we even get to the starting line?

(Miske et al. 2026).

First, reproducibility – the gentlest test of the three. The design in one breath: take six hundred papers, ask the authors for their data and code, and try to recompute the central reported result. Same data, same analysis: do you get the same number? Before we even ask how often that works, the first finding is about how rarely you can even start.

Most papers never reach the starting line

• Of the 143 assessable: 53.6% reproduced precisely, 73.5% approximately
• Counting the unassessable, only about 18% reproduce

Read it left to right. Six hundred papers sampled. Data were available for twenty-four percent of them – for nearly three quarters there was neither data nor code. That leaves 143 papers you can actually assess. Of those, just over half reproduce precisely, and about three quarters at least approximately. Which sounds respectable – until you notice what happens if you count the papers nobody could check as failures: the overall rate falls to roughly eighteen percent. The barrier is not arithmetic. It is access.

Why some fields do better

• Journals requiring data sharing: 87.5–100% availability vs 16.0% without
• Political science 54.1% data availability vs education 2.9%
• With data and code: 90.9% approx-reproducible vs 38.1% from rebuilt source

The most substantial barrier … was the unavailability of author data.

Reproducibility success does not mean that the finding is correct.

(Miske et al. 2026).

And access follows policy. Where journals mandate data sharing, availability sits between about ninety and one hundred percent; without a mandate, sixteen. Political science and economics built the habit early and are far ahead of the rest. The first quote is the paper’s own conclusion – the biggest barrier was simply that the data were not there. The second quote deserves a beat of silence: reproducibility success does not mean that the finding is correct. Reproducing a number tells you the computation was faithful. It does not tell you the claim about the world is true.

Robustness – Aczel et al. (Multi100)

• 100 studies, each re-analysed by ≥5 independent analysts
• Same data; each analyst free to choose any justifiable analysis
• A peer panel judges whether each pipeline is appropriate
• This is the paper I was an analyst in

(Aczel et al. 2026).

The middle paper – the one I was in. One hundred studies, and for each one at least five analysts working independently: no teams, original authors barred, everyone handed the same data and the same claim. You were free to analyse it any way you could justify, and a peer panel later judged whether your pipeline was appropriate. Two things worth knowing about the design. The hundred studies had already passed a reproducibility check – so robustness was being measured on the reproducible subset, the best-case material. And we were not blinded: every analyst saw the full original article, published estimates and all, so if anything we were anchored towards the original answer.

Same data, five analysts, different answers

• 34% of reanalysis effect sizes within ±0.05 d of the original (57% within ±0.20)
• 74% reached the same conclusion; 24% null/inconclusive; 2% opposite
• Effect sizes shrank: original mean d 0.73 → reanalysis 0.49
• Analyst-to-analyst variability typically exceeds sampling error

(Aczel et al. 2026).

So what do free, justifiable choices do to the same data? They move the answer – a lot. Only about a third of the reanalyses landed within five hundredths of a Cohen’s d of the original, and in just five studies out of ninety-five did all the analysts land that close. Conclusions held up better than magnitudes: three quarters of reanalyses agreed with the original conclusion. But the effect sizes shrank, from a mean d of point seven three down to point four nine. And the finding I keep coming back to: the spread across analysts is typically larger than the statistical uncertainty inside any single analysis. The analyst, in other words, is a bigger source of noise than the sample. One more detail to hold for later: observational studies – which is what we work on today – came out less robust than experiments.

Replicability – Tyner et al.

• 274 claims from 164 papers, tested on new data
• Median statistical power 99.6%; preregistered; original authors consulted
• The strictest test: does the effect reappear in a fresh sample?

(Tyner et al. 2026).

The third paper is the strictest test: not the same data analysed differently, but genuinely new data. Two hundred and seventy-four claims, each replicated to a preregistered, peer-reviewed protocol, often with the original authors consulted, and with a median statistical power of ninety-nine point six percent. That last number matters: these studies were built so that a real effect would be very hard to miss. So whatever fails here is not failing for lack of trying.

Even success shrinks

• 55.1% of claims replicated by significance; effect sizes roughly halve
• Thirteen success criteria give anywhere from 29% to 75%

(Tyner et al. 2026).

And this is what comes back. The median effect size halves: point two five in the originals, point one zero in the replications. By the significance criterion, fifty-five percent of claims replicate. But notice that did it replicate? is not even one question – the team scored the same studies against thirteen reasonable success criteria and got answers ranging from twenty-nine to seventy-five percent. Two sentences from the paper are worth carrying around. Strictly speaking, there is no such thing as exact replication. And: The problem to solve is not unreplicability per se, it is overconfidence.

The take-away

Reproducible ≠ robust ≠ replicable.

Across SCORE the three are essentially uncorrelated – knowing one barely predicts the others.

So there is no single number for trust. You have to ask which repetition you mean.

Now the bit that genuinely surprised me. You would hope these three properties travel together – that a reproducible finding is more likely to be robust, and a robust one more likely to replicate. Across SCORE, they do not. Measured on the same body of papers, the three indicators are essentially uncorrelated – knowing one tells you almost nothing about the others. So there is no single trust score for a finding. Is it reliable? is an incomplete question until you say which kind of repetition you mean. Hold that thought, because we are about to watch all three play out on one single paper.

Quick check: credibility in the wild

You submit your paper and replication package. It goes out to reviewers.

• Reviewer 1 wants to ensure that your results are reproducible, so she sources your R script to run the entire analysis. You’re in luck – she hasn’t yet updated her dplyr, so the code runs.
• Reviewer 2 is very eager. He’s unconvinced by your modelling choices, so he fits your data to another function to check if your findings are truly robust. You’re in luck again – he recovers your point estimate to a reasonable precision, although the confidence intervals have widened.
• Reviewer 3 takes a special interest because he’s just working on a similar paper on data they published recently. Do your results replicate when your methods are applied to that state-of-the-art dataset?

Quick check: credibility in the wild

You submit your paper and replication package. It goes out to reviewers.

Reviewer 2 just doesn’t want to let go…

• He tried a few alternative analyses of your data, including a fancy new regression method that he just released as an R package. His method gives results that are generally aligned to yours, but for the method to work, it would require a larger dataset; would your results really generalise to this new dataset published by Reviewer 3 that he just read about, if applying his new fancy method to it?

PART 2 – Our case

Talk II · 10 minutes

Part two – one paper. The one I worked on, and the one you are about to.

On the OSF

https://osf.io/6zqct

The Teney (2016) study

• 16 Eurobarometer waves, 2004–2013; ~ 390,000 respondents in 27 countries

The results show, first, that poor economic performances increase negative and decrease positive dimensions of EU framing.

• What does the EU mean to you personally?
  • COSMOPOLITAN: positive non-materialist form of EU framing (Peace; Democracy; Freedom to travel, study and work anywhere in the EU; Cultural diversity; Stronger say in the world)
  • UTILITARIAN: positive materialist form of EU framing (economic prosperity and social protection)
  • COMMUNITARIAN: negative non-materialist form of EU framing (Loss of our cultural identity; More crime; Not enough control at external borders; Unemployment
  • LIBERTARIAN: negative materialist form of EU framing (Bureaucracy; Waste of money)

(Teney 2016).

The paper is Céline Teney’s, from 2016, in the European Sociological Review. Sixteen Eurobarometer waves, 2004 to 2013 – the crisis years – about three hundred and ninety thousand respondents across twenty-seven member states. The outcome comes from a nice survey item: what does the EU mean to you personally? Thirteen response options, which she organises into four framing scales; the cosmopolitan one – peace, democracy, freedom to move, cultural diversity – is the one we will target. The predictors are national unemployment and GDP growth, in three-level multilevel models. We will work on a country-year aggregate of the same data – two hundred and seventy rows – which behaves the same for our purposes and runs in milliseconds.

The claim card, as analysts received it

… poor economic performances … decrease positive dimensions of EU framing

Teney (2016: 619)

• Anchor result – unemployment → cosmopolitan framing, Table 3 Model 1:
• b = −0.00340 · t = −4.03 · p < .001

This is, word for word, the claim every Multi100 analyst was handed: poor economic performances decrease positive dimensions of EU framing. And this is the anchor number behind it, from Table 3, Model 1 – the unemployment coefficient on cosmopolitan framing: minus point zero zero three four, t of minus four. Keep one wrinkle in mind for later. In the paper’s own Model 2, with more controls in, that same coefficient drops by more than half and loses significance. The robustness question is already alive inside the original paper.

Reproducibility

• Data: openly available but not freely redistributable

The required data to replicate my analyses can be downloaded on the Gesis repository website. I am not allowed to forward the Eurobarometer data. Enclosed you´ll find the code to combine the different Eurobarometer data waves and the code for the analysis of the ESR piece

• Code: push-button replication failed

The reproduction attempt failed because the authors used file names that did not correspond with the names of the files available for download. Additionally 16 datasets are listed on the website as being used but the code references 18 files

• Source-data reproduction (rebuild from raw) – not reproduced: could not extract nor re-calculate the original effect size

So what happened when SCORE tried to reproduce this paper? Two attempts, two different failures. And let me be fair to the author here: there was no public archive with the paper, but she did supply her two Stata do-files when asked – what she could not share was the data, because GESIS terms do not allow it. Even with her code in hand, the push-button reproduction failed, and the report’s wording is instructive – the file names in the code did not correspond to the files you could actually download; sixteen datasets listed, eighteen referenced. The second attempt rebuilt the data from source, and could not recover the published effect size at all. So this claim entered the robustness study already carrying two failed reproductions, of two different kinds.

Robustness

• All five negative – same direction · partial r from −0.45 to −0.01 · one not significant

And then five of us re-analysed it, independently. Here is what came back. All five found a negative effect – on direction, unanimity. On size, look at the spread: from essentially zero – that one is not significant – up to a partial correlation of minus point four five. Two orders of magnitude apart, on the same data, for the same claim. The orange dot is me: R and Quarto, minus point two four. We did not even use the same software – two of the five worked in Stata, three in R, which quietly makes today’s case for reproducible pipelines. Five competent people, one fixed question: analysis is a garden of forking paths.

Replication

• Analysis One: specifically serves as the focal test for evaluation of the claim replicated for the purposes of SCORE.

  • 352,114 person-year observations in the analytic sample.
  • b = 0.002, p = 0.046, se = 0.0008, t = 2.004

• Analysis Two: still studying and no full-time education categories in the variable age when finished full-timeeducation were NOT converted to missing values.

  • 379,396 person-year observations in the analytic sample
  • b = 0.002, p = 0.017, se = 0.0007, t = 2.394

• A positive coefficient for unemployment on cosmopolitan framing, and significant at p=value results of 0.05 or less, after controlling for selected predictors, and wave and country-level effects

• In this regard, this replication of the claim using Analysis Two was not successful according to the SCORE criteria

osf.io/zt5y2/files/7efsw

One paper, all three Rs

osf.io/h7432

Put the whole dossier together – every artifact here is public, the link is on the slide. Reproduction: failed, twice, in two different ways. Robustness: mixed – five analysts, one sign, sizes spanning two orders of magnitude. And replication: a preregistered study took the same question to new Eurobarometer data, three hundred and fifty-two thousand observations – and the coefficient flipped sign. Positive, just significant, the wrong way round. So: one paper, three Rs, three different verdicts. That is the uncorrelated-indicators result from part one, embodied in a single case. Asking is this paper reliable? gets you nowhere; asking reliable in which sense? gets you all of this. And now it is your turn – you are the sixth analyst.

You are Analyst no. 6

• Aim 1: Reproduce the original reanalyst C6HJR’s constrained result (Multi100 Task 2)

• Task: Focus on the ‘Cosmopolitan’ dimension for EU framing and on unemployment rate as the independent variable; disregard contextual variables

• Procedure: two-way fixed-effects model (mcosmopolitan ~ unemp_c | cntry + year)

• Result: t ≈ −3.80 (df = 233, N = 270, partial r ≈ −0.242)

You are Analyst no. 6

• Aim 2: Assess the robustness of the original reanalyst C6HJR’s constrained result

• Task: Choose ONE deviation from the specification menu, evaluate the conceptual validity of your claim, clarify your estimand, commit to your chosen model before you run it, and submit your result

• Procedure: ???

• Result: ???

Two tracks

Track A – browser lab (zero install, webR in the page):

codemoreh.github.io/applied-replication/exercise/browser-lab

Track B – template repo (full Positron + git pipeline):

github.com/CodeMoreh/replication-lab

Use this template → clone in Positron

Here is how the next hour works. Two ways in, same destination. Track A is the browser lab: open that first address and R boots inside the page itself – nothing to install, works on any laptop, including the one you borrowed this morning. Track B is the full pipeline: take the template repository, click Use this template, and clone your copy in Positron. In your pairs, make sure at least one of you has a working track – and if you are on Track B, keep the browser lab open in a tab as your spare. One more thing to note: everything you produce today ends up as a single click on a link that drops your result onto a live chart on the Multiverse page. That chart is our results board – we will watch it fill up and read it together at the end.

PART 3 – Estimands & DAGs

Concepts · 8-minute mini-lecture, then guided work

Right – everyone has something running. Eight minutes of ideas, and then you put them to work. The argument of this part, in one sentence: a lot of what we just called analysis variability is really question variability – and if you fix the question precisely, the multiverse narrows.

The red-card study: 29 teams, one dataset

(Silberzahn et al. 2018). Display sketch from published median (1.31) + range (0.89–2.93).

This is the study that started the whole many-analysts conversation. Twenty-nine teams, identical data, one question on paper: are dark-skinned footballers more likely to be red-carded than light-skinned ones? The odds ratios came back ranging from point eight nine – that is, below one – to nearly three. Median one point three one. About two thirds of the teams found a significant effect; a third did not. Same data, same question, twenty-nine answers. One caveat about the figure: the dots are a sketch drawn from the published median and range, not the teams’ actual estimates – it is the spread that carries the message.

They were not answering the same question

Auspurg & Brüderl re-read the 29 analyses and found four different estimands:

• descriptive association · discrimination net of mediators · variance-maximising · exploratory
• The dispersion was largely a question-multiverse, not an analysis-multiverse

(Auspurg and Brüderl 2021)

Why so scattered? Auspurg and Brüderl went back through all twenty-nine pipelines, and their diagnosis is the hinge of this morning: the teams were not answering one question. They found four. Some estimated the plain descriptive association. Some went after discrimination proper – the effect net of everything that legitimately explains red cards. Some built models tuned to explain variance. And some were frankly exploring. Of course the answers differed – they were answers to different questions. That reframes the crisis quite a bit: much of what looks like analytical instability is unstated disagreement about what is being estimated.

What is your estimand?

• The theoretical estimand is stated outside any model – before R opens

(Lundberg et al. 2021).

Which brings us to the most useful question in quantitative research: what is your estimand? Lundberg and colleagues split the answer into three steps. First, the theoretical estimand: a unit-specific quantity and a target population, stated in plain words, before any model exists. Second, the empirical estimand: what in the observable data corresponds to that quantity – and making that link forces your identification assumptions into the open, which is exactly where the causal graph lives. Only third comes estimation, where R finally opens. When they audited a full year of papers in a top sociology journal, the unit quantity was clearly stated in five out of thirty-two. Their demand is the one to remember: every quantitative study must be able to answer the question – what is your estimand?

DAG crash course: three roles

X = exposure, Y = outcome. Green = the causal effect you want; coloured = the path you must reason about.

All the graph theory you need today, in three small pictures. In every panel the green arrow is the same thing – the effect you are actually after, X on Y. What changes is the coloured structure around it. A confounder is a common cause of your exposure and your outcome – that red fork is a back-door path, and if you leave it open you are biased, so you adjust to close it. A mediator sits on the causal path between them – the amber path is real, it is part of the total effect, so if the total effect is what you want you do not adjust for it, or you remove part of the very thing you are measuring. And a collider is caused by both sides – those edges are harmless until you condition on the collider, which opens a spurious path that was not there before. So the rulebook is short: adjust confounders, leave mediators alone for total effects, and never condition on a collider. That is the entire toolkit – the next four slides just give it a bit more depth before you use it.

One number, many stories

The same slope fits four different stories:

• X really causes Y – the slope is the effect

• a common cause inflates it – confounding

• we are seeing a selected sample – collider / selection

• Y causes X (reverse causation), or it is chance

The data look identical in all four. Only an assumption about how the data arose tells them apart – that is what a DAG writes down.

(D’Agostino McGowan et al. 2024; Keele et al. 2020). Scatter simulated for illustration.

Here is a strong correlation – r of about nought point six. Before you reach for a model, sit with an uncomfortable fact: this single number is consistent with at least four different stories. Maybe X really does cause Y, and the slope is the effect. Maybe a common cause is inflating it. Maybe we are quietly looking at a selected sample, and the association is an artefact of who got in. Maybe Y causes X, or it is simply chance. The data cannot choose between these – the scatter looks identical in every case. Only a claim about how the data were generated can. That claim is what a DAG writes down, and it is why we draw one before we fit anything.

Reading a DAG: open and closed paths

Association flows along every open path – not only the causal arrow. A fork or chain is open until you adjust the variable on it; a collider is the reverse – shut until you adjust it.

• Back-door criterion: close every path that enters X from behind, keep the causal X → Y path open
• That adjustment set – and only that set – earns a causal reading

(Hernán and Robins 2020; Huntington-Klein 2021).

A DAG is read through its paths. Association does not only travel down the causal arrow – it flows along every open path between X and Y. So the job is to work out which paths are open and shut the ones you do not want. Two rules do all the work. A path through a fork or a chain is open, and you close it by adjusting the variable in the middle – that is the red back-door fork on the left. A path through a collider is the opposite: shut by default, and you open it by adjusting. The back-door criterion just packages this: close every path that sneaks into X from behind, leave the causal X-to-Y path alone. The set that does that – and only that set – is what you are allowed to read as causal.

Which effect do you even want?

• Total = direct (green) + indirect (amber)
• Adjust for the mediator M → you estimate the direct effect only – a different estimand
• Control for everything silently answers a question you never asked
• Table 2 fallacy: the other coefficients in your model are not all causal effects

(Keele et al. 2020; D’Agostino McGowan et al. 2024).

Now the question the estimand discussion was circling. Suppose M sits between your exposure and outcome. The total effect is the green arrow plus the amber route through M. The direct effect is just the green arrow. These are different numbers and different questions – and which one you get depends entirely on whether you put M in the model. Drop a mediator in as just another control and you have silently switched to the direct effect without telling anyone, including yourself. This is why control for everything is bad advice. And it has a famous corollary – the Table 2 fallacy: when you print a regression table, usually only one coefficient has a clean causal reading. The control variables’ coefficients are not little causal effects of their own.

Spurious by design: selection & feedback

• Selection / collider bias – conditioning, or merely sampling, on a common effect manufactures an association that was never there (much of the early COVID risk-factor literature)
• Reverse causation & feedback – in a panel, today’s outcome can move tomorrow’s exposure; two-way fixed effects absorb stable country traits and common-year shocks, not time-varying feedback (U = unemployment, F = framing)

(Griffith et al. 2020; Cole et al. 2010; Leszczensky and Wolbring 2022).

Two last ways a relationship can be spurious, both of which live in our case. The first is selection, or collider bias. If you condition on – or merely sample on – something that both your variables cause, you manufacture a correlation out of nothing. A lot of the early COVID risk-factor findings were exactly this: conclusions drawn from people who had been tested, which is a selected group. The second is reverse causation and feedback. Our data are a panel, and in a panel today’s outcome can feed tomorrow’s exposure. Two-way fixed effects – which is the model we are about to draw – soak up stable differences between countries and shocks common to a year. They do not, on their own, fix a time-varying confounder or a feedback loop. Keep that in mind when we build the graph for our own paper.

A&B’s resolution: fix the estimand first

• Fix the question → derive the model space → run all 486 justifiable models → SD 0.45 → 0.06

(Auspurg and Brüderl 2021). Display sketches from published medians + SDs (illustrative spread).

And here is the optimistic half of the red-card story. Auspurg and Brüderl fixed one estimand – discrimination, properly defined – used a causal graph to derive every model specification consistent with it, four hundred and eighty-six of them, and ran the lot. Look what happens to the spread: standard deviation point four five in the crowd, point zero six with the question fixed. Same caveat as before – these strips are sketches from the published summary figures. The recipe we take from them is the workshop’s recipe: a precise question, assumptions drawn out explicitly, then a whole multiverse rather than one path, then sensitivity analysis. Their conclusion, roughly: it is research without a clear question that has the credibility problem.

Revenons à nos moutons

The expected change in [unit quantity] among [target population] for a 1-percentage-point difference in unemployment, holding [assumptions] fixed.

• Unit quantity – a person’s cosmopolitan framing, or a country-year mean?
• Target population – which countries, which years?
• You write this sentence in your report before you fit anything

Now apply it to our paper. This is the first thing you write in Task C – the sentence on the slide, with the brackets filled in. The genuinely open choice is the unit. Is your estimand about a person’s framing, the way Teney’s multilevel models are? Or about a country-year average, the way our panel is? Neither is wrong – they are different questions, with different confounding stories, and the coefficient means something different in each. Write the sentence before you fit anything. It will discipline every choice you make afterwards.

The Teney DAG, half built

For online version: dagitty.net/dags.html Paste into Model code: Drag nodes, add edges, watch the adjustment sets update

dag {
unemployment [exposure]
framing [outcome]
country -> unemployment ; country -> framing
year -> unemployment ; year -> framing
unemployment -> framing
}

• Drawn: the path + two confounder pairs (country, year → two-way FE)
• Floating, unwired: growth (confounder?) · bailout (mediator?) · politicisation (mediator?)
• You add the edges you believe, then run adjustmentSets()

And here is the graph you will finish yourselves. The solid part is what we are committing to: unemployment to framing is the path of interest, and country and year are the two confounder pairs – stable national traits on one side, EU-wide shocks in a given year on the other. Ask dagitty for the adjustment set on this graph and it answers: country and year. Which is precisely what a two-way fixed-effects model absorbs – the graph and the baseline model are the same claim, written in two languages. The three pale nodes are yours to wire up, and they are genuine decisions. Is GDP growth a time-varying confounder that fixed effects cannot reach – a slump that raises unemployment and sours sentiment in the same year? Is the bailout a mediator, in which case you leave it alone if you want the total effect? Multi100 told us to disregard politicisation – was that defensible? If you want to play with this live, paste the model code from the slide into dagitty.net and drag things around; the adjustment sets update as you edit. Whatever you decide here becomes your model in the next task – that is the point.

PART 4 – The multiverse

Debrief · 12 minutes

Welcome back. We will get to what the room found in a moment – but first, a story. It happened to all of you about an hour ago, and you may not have noticed.

You just lived a reproducibility story

You fetched rep_data.csv from the fork and fitted the constrained model.

You got t = −3.853.

The result recorded for analyst C6HJR in Multi100 is t = −3.804.

Same model. Same data, supposedly. Two different numbers. Why?

Earlier this morning, every one of you pulled my submitted dataset from the fork and fitted exactly the constrained model – the same one I submitted to Multi100. You got t equals minus three point eight five three. [advance] The number on record for me, in the Multi100 dataset, is minus three point eight zero four. [advance] Same model. Supposedly the same data. Two different numbers. [pause] Why? Take a guess – this is reproducibility, same data and same analysis, coming up short by a hair. And the answer is more interesting than embarrassing.

The discovery – and the correction

• The prep script read EB 70.1 (ZA4819) twice, never EB 72.4 (ZA4994) – the 2009 rows are relabelled 2008 data
• The analyst spotted this during the project and fixed it – the recorded −3.804 came from the corrected pipeline:

APRIL 2025 – Mistake in dataset here; the dataset associated with eb724 should be ZA4994_v3-0-0.dta – analyst’s own annotation

• The official component kept only the earlier folder; the correction is published in the fork: osf.io/6zqct → moreh_rep_vAPR2025/

Here is why. My original data-preparation script read the autumn 2008 Eurobarometer file twice, and never read the 2009 file – so what is labelled 2009 in that dataset is actually 2008 again. I caught it during the project; the annotation on the slide is from the script itself, dated April 2025 – mistake in dataset here. The corrected pipeline is what produced the recorded minus three point eight zero four. What slipped was the archive: the official Multi100 folder kept only the earlier, uncorrected version. The correction now lives, documented where it happened, in the fork you all pulled from this morning. Corrections like this are normal science – the question is only whether the trail stays visible.

This is reproducibility working

The workshop panel, rebuilt from the raw GESIS files, recovers t = −3.804 exactly.

Every step was checkable because the materials were open – the as-submitted data, the corrected script, the rebuild.

Reproducibility working, not failing. Keep the whole trail public.

And the ending is the good part. The workshop dataset you used for the menu – rebuilt independently, from the raw files – recovers minus three point eight zero four exactly. So we know precisely which dataset produced the recorded value, and we know the rest of the pipeline matches to machine precision. Notice what made all of that possible: every step was checkable only because the materials were open. You fetched the submitted data and got one number; you could read the annotation, see the correction, rebuild, and get the other. None of this is a gotcha – not about Teney, not about me. It is what open materials are for. The discipline is keeping the whole trail public. So – with that in mind – where did your own results land?

The class multiverse

• 1,680 specifications · 68.7% claim-consistent direction · 63.9% significant · partial r from −0.59 to +0.70

Pre-computed over the specification menu (data/spec_grid.csv). Class results stream onto the Multiverse page, live.

This is the space your results live in. I took the whole specification menu – every outcome, every exposure form, with and without growth, every estimator, every sample window, weighted and not – and ran all of it. One thousand six hundred and eighty models, each one a dot, ranked by partial correlation. Blue means significant, grey means not. About two thirds point the way the claim says; about two thirds are significant; the red dot is the baseline we all reproduced. Notice you can buy almost any answer you like in here – if you are willing to pick the path after seeing the results. Right – where did the room land? Call them out: did anyone flip the sign? Lose significance? Which choice moved your estimate most?

Where your result lands

Your orange dot lands here → live on the Multiverse page as the room submits

report_result() gives you a one-click submission link; results stream onto the Multiverse page.

Same curve, with the five Multi100 analysts laid across it as dashed lines – me in orange. What I like about this picture: the five published reanalyses are themselves just draws from this same space. Nobody was wrong; we were standing in different places in the garden. Right now this chart shows only the starting position – the five analysts on the empty menu. As you submit, your own results appear as orange dots, live, on the Multiverse page. report_result gives you a one-click link that already has your numbers in it, including the partial correlation computed exactly the way Multi100 computed it. That little conversion is what lets a Stata pooled model and an R fixed-effects model sit on the same axis – yours included.

What we learned

the common single-path analyses in social and behavioural research should not be simply assumed to be robust to alternative analyses (Aczel et al. 2026)

A blueprint: precise question → causal reasoning (DAG) → multiverse → sensitivity analysis (Auspurg and Brüderl 2021)

• Same-conclusion robustness and same-number robustness are different bars
• Justifiability is the criterion – choose with reasons, show the spread

Two sentences to take away. From the Multi100 paper: single-path analyses should not simply be assumed robust. And they add something worth hearing about preregistration – it protects you against choosing opportunistically, but it gives your preselected path no special status among the alternatives. That is why we ran a curve today, not a model. From Auspurg and Brüderl, the constructive recipe we followed: precise question, explicit assumptions, multiverse, sensitivity. And keep the two robustness standards apart in your own writing: same conclusion – seventy-four percent in Multi100 – and same number – thirty-four. When you claim robustness, say which one you mean.

A single analytical path is a choice

• There is no privileged single analytical path – only justifiable ones, explored openly.
• You made one justifiable choice today – transparently, with a reason, inside a multiverse.
• This is a skill.
• And that was the workshop.

codemoreh.github.io/applied-replication

Fork anything. Everything is open.

If the morning had one sentence, it is this one. There is no privileged single analytical path – only justifiable ones, explored openly. And notice that you have already done the right thing once: you made one defensible choice, you wrote down your reason before running it, and you placed your answer inside the space of alternatives rather than pretending it was the only one. That is the transferable habit. Everything else today was tooling in its service.

Take home & resources

• Yours now: a public reproducible repo (Track B) or a saved browser script (Track A); a result on the class curve
• Read: the three Nature 2026 papers (Miske et al. 2026; Aczel et al. 2026; Tyner et al. 2026) · the glossary (Nosek et al. 2025) · Auspurg and Brüderl (2021) · Lundberg et al. (2021) · Teney (2016)
• Explore: the Multi100 data · OSF preregistration templates · fixest & dagitty docs
• At Newcastle: the RSE team · Library open-research pages · data.ncl.ac.uk · ReproducibiliTea

codemoreh.github.io/applied-replication

Fork anything. Everything is open.

What you take with you: a public repository, or your browser script if you worked in Track A; a published page; and a dot on that curve. The website on screen has everything else – the readings, the data, the template, and the full SCORE dossier on the paper if you want to dig further. I will not read the list out, but the Newcastle pointers are worth a look while you are here: the RSE team run excellent training, the library’s open-research people are genuinely helpful, data.ncl is where your own datasets can live – and if this morning’s conversation is one you want to keep having, ReproducibiliTea meets monthly.

References

Aczel, Balazs, Barnabas Szaszi, Harry T. Clelland, et al. 2026. “Investigating the Analytical Robustness of the Social and Behavioural Sciences.” Nature 652 (8108): 135–42. https://doi.org/10.1038/s41586-025-09844-9.

Alipourfard, Nazanin, Beatrix Arendt, Daniel M Benjamin, et al. 2021. Systematizing Confidence in Open Research and Evidence (SCORE). 46mnb_v1. SocArXiv. https://doi.org/10.31235/osf.io/46mnb.

Auspurg, Katrin, and Josef Brüderl. 2021. “Has the Credibility of the Social Sciences Been Credibly Destroyed? Reanalyzing the ‘Many Analysts, One Data Set’ Project.” Socius 7 (January): 23780231211024421. https://doi.org/10.1177/23780231211024421.

Cole, Stephen R., Robert W. Platt, Enrique F. Schisterman, et al. 2010. “Illustrating Bias Due to Conditioning on a Collider.” International Journal of Epidemiology 39 (2): 417–20. https://doi.org/10.1093/ije/dyp334.

D’Agostino McGowan, Lucy, Travis Gerke, and Malcolm Barrett. 2024. “Causal Inference Is Not Just a Statistics Problem.” Journal of Statistics and Data Science Education 32 (2): 150–55. https://doi.org/10.1080/26939169.2023.2276446.

Griffith, Gareth J., Tim T. Morris, Matthew J. Tudball, et al. 2020. “Collider Bias Undermines Our Understanding of COVID-19 Disease Risk and Severity.” Nature Communications 11 (1): 5749. https://doi.org/10.1038/s41467-020-19478-2.

Hernán, Miguel A., and James M. Robins. 2020. Causal Inference: What If. Chapman & Hall/CRC.

Huntington-Klein, Nick. 2021. The Effect: An Introduction to Research Design and Causality. Chapman & Hall/CRC. https://theeffectbook.net/.

Keele, Luke, Randolph T. Stevenson, and Felix Elwert. 2020. “The Causal Interpretation of Estimated Associations in Regression Models.” Political Science Research and Methods 8 (1): 1–13. https://doi.org/10.1017/psrm.2019.31.

Leszczensky, Lars, and Tobias Wolbring. 2022. “How to Deal with Reverse Causality Using Panel Data? Recommendations for Researchers Based on a Simulation Study.” Sociological Methods & Research 51 (2): 837–65. https://doi.org/10.1177/0049124119882473.

Lundberg, Ian, Rebecca Johnson, and Brandon M. Stewart. 2021. “What Is Your Estimand? Defining the Target Quantity Connects Statistical Evidence to Theory.” American Sociological Review 86 (3): 532–65. https://doi.org/10.1177/00031224211004187.

Miske, Olivia, Anna Lou Abatayo, Mason Daley, et al. 2026. “Investigating the Reproducibility of the Social and Behavioural Sciences.” Nature 652 (8108): 126–34. https://doi.org/10.1038/s41586-026-10203-5.

Nosek, Brian A, Timothy M Errington, Noah Haber, Theresa Stankov, and Andrew H Tyner. 2025. A Brief Glossary of Terms about Repeatability: Replicability, Robustness, and Reproducibility. mqfp4_v1. MetaArXiv. https://doi.org/10.31222/osf.io/mqfp4_v1.

Sánchez-Tójar, Alfredo, Jelte M. Wicherts, and Robb Willer. 2026. “Huge Meta-Research Project Puts Claims in Social-Science Papers to the Test.” Nature 652 (8108): 39–41. https://doi.org/10.1038/d41586-026-00805-4.

Silberzahn, R., E. L. Uhlmann, D. P. Martin, et al. 2018. “Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results.” Advances in Methods and Practices in Psychological Science 1 (3): 337–56. https://doi.org/10.1177/2515245917747646.

Teney, Céline. 2016. “Does the EU Economic Crisis Undermine Subjective Europeanization? Assessing the Dynamics of Citizens’ EU Framing Between 2004 and 2013.” European Sociological Review 32 (5): 619–33. https://doi.org/10.1093/esr/jcw008.

Tyner, Andrew H., Anna Lou Abatayo, Mason Daley, et al. 2026. “Investigating the Replicability of the Social and Behavioural Sciences.” Nature 652 (8108): 143–50. https://doi.org/10.1038/s41586-025-10078-y.

Thank you – and your feedback

Feedback poll
(QR / short URL)

Not added for this small workshop