Resources
Readings, links, and tools
Four words that are not synonyms
The four terms below follow the definitions in Nosek et al. (2025). They are not interchangeable, and the day depends on the difference: a finding can survive every robustness check and still be one that nobody can recompute from the original data, or reproduce cleanly and still fail on new data. The ‘in this workshop’ column says where each concept appears in the session.
| Term | Definition | In this workshop |
|---|---|---|
| Reproducibility | Same question · same data · same analysis → same result | Tested by the SCORE programme (Alipourfard et al. 2021); the result reported by the original author (OA) was not reproducible (Miske et al. 2026) |
| Robustness | Same question · same data · justifiable alternative analysis → stable result | The Multi100 design (Aczel et al. 2026); the five-analyst table; the specification work in Part 3 |
| Replicability | Same question · independent new data → same result | Tyner et al. (Tyner et al. 2026); discussed in Part 1 |
| Repeatability | Umbrella term covering all three above | Used in the Nosek et al. glossary as the overarching category |
Core readings
The three Nature 2026 papers (SCORE)
Reproducibility: Miske et al. (2026) – Miske et al. (2026). Investigating the reproducibility of the social and behavioural sciences. Nature, 652, 126–134. https://doi.org/10.1038/s41586-026-10203-5
Robustness / Multi100: Aczel et al. (2026) – Aczel et al. (2026). Investigating the analytical robustness of the social and behavioural sciences. Nature, 652, 135–142. https://doi.org/10.1038/s41586-025-09844-9
Replicability: Tyner et al. (2026) – Tyner et al. (2026). Investigating the replicability of the social and behavioural sciences. Nature, 652, 143–150. https://doi.org/10.1038/s41586-025-10078-y
Vocabulary
Nosek et al. (2025) – Nosek et al. (2025). A brief glossary of terms about repeatability: replicability, robustness, and reproducibility. MetaArXiv. 4 pages; read this one before the day. https://doi.org/10.31222/osf.io/mqfp4_v1
Nosek and Errington (2020) – Nosek & Errington (2020). What is replication? PLOS Biology, 18(3), e3000691. https://doi.org/10.1371/journal.pbio.3000691
Estimands and analytical choices
Auspurg and Brüderl (2021) – Auspurg & Brüderl (2021). Has the credibility of the social sciences been credibly destroyed? Socius, 7. https://doi.org/10.1177/23780231211024421
Lundberg et al. (2021) – Lundberg, Johnson & Stewart (2021). What is your estimand? American Sociological Review, 86(3), 532–565. https://doi.org/10.1177/00031224211004187
Silberzahn et al. (2018) – Silberzahn et al. (2018). Many analysts, one data set: making transparent how variations in analytic choices affect results. Advances in Methods and Practices in Psychological Science, 1(3), 337–356. https://doi.org/10.1177/2515245917747646
Preregistration
These three are the pre-session reading list for the preregistration segment of the day, and all are open access. None is required, because everything is taught on the day, but they let you arrive with the ideas already in hand.
Nosek, Ebersole, DeHaven & Mellor (2018). The preregistration revolution. PNAS, 115(11), 2600–2606. https://doi.org/10.1073/pnas.1708274114 – a short primer on the idea on which preregistration depends: predicting a result versus explaining one after the fact, and the case, like ours, where the data already exist.
The AsPredicted eight-question form – the short template you complete in the session, listed on OSF as the “Preregistration Template from AsPredicted.org”. Live form: https://aspredicted.org; OSF listing: https://help.osf.io/article/158-create-a-preregistration.
van den Akker, Weston, Campbell, et al. (2021). Preregistration of secondary data analysis: A template and tutorial. Meta-Psychology, 5, 2625. https://doi.org/10.15626/MP.2020.2625 – the fuller template built for exactly our situation, reanalysing data that already exist; the tool that a full version of today’s exercise would use.
Reading the multiverse
- Reading a specification curve: what it can and cannot say – this workshop’s own reading for the final part of the day: why a large set of results is not a large set of independent tests, and what a single test across the whole curve can and cannot establish.
The case study paper
- Teney (2016) – Teney (2016). Does the EU economic crisis undermine subjective Europeanization? European Sociological Review, 32(5), 619–633. https://doi.org/10.1093/esr/jcw008
The SCORE programme
- Alipourfard et al. (2021) – Alipourfard et al. (2021). Systematizing Confidence in Open Research and Evidence (SCORE). SocArXiv. https://doi.org/10.31235/osf.io/46mnb
Companion curriculum
Nine modules carry most of the teaching in this course, from a first line of R through to a reanalysis you can version and archive. The workshop at the centre is compressed by design, so it works the single hands-on case and leaves the skills that surround it to the modules; each of them stands on its own and can be taken without the others. The curriculum overview sets out the three strands that the modules group into, and a suggested reading order.
- Getting started with R – for participants arriving with no R at all; best read before the day.
- Quarto and reproducible documents – the medium in which every page of this site is written: what a
.qmdfile is and what rendering does to it, the YAML header and the chunk options that matter, figures, tables and cross-references, citations drawn from a bibliography, and publishing the finished document, all of it worked through on files you can open in this project rather than on a toy example. - The statistical methods behind the workshop – every model fitted on the case’s own data with real outputs read line by line, from a first
lm()through fixed and random effects to multilevel, weighted, Bayesian and ordered-beta models, ending with meta-analysis and the common effect-size axis. - Draw the arrows you believe – causal graphs written as R formulas with
dagify(): confounders, mediators and colliders fitted rather than only described, the EU-frames graph whose adjustment set is what two-way fixed effects absorb, and what fixed effects cannot fix. - The multiverse: origins, uses, and critics – where specification-curve thinking came from, the three purposes a multiverse can serve, and the 2025–26 critical debate read against this workshop’s own grid.
- Simulation as a method – a simulated person-level twin with a known recipe, built to carry the original analysis: parameter recovery, composition against context, recovering individual effects, weights that matter, the degrees-of-freedom story run live, a null multiverse, a lineup, and design precision by simulation.
- One claim, two levels – the published specification fitted at the individual level on a validated simulated twin, with the real-data values beside it: individual circumstance against national context, and what declaring a level of analysis does to the evidence.
- Reproducibility repositories – research-data archives, the OSF nodes behind this case, and preregistration templates in more depth.
- Git and GitHub for reproducible research – version control and publishing your own report, the hands-on practice kept out of the core day so it can be learned on its own terms.
Data and project links
| Resource | URL |
|---|---|
| Multi100 – OSF project | https://osf.io/q5h2c/ |
| Multi100 – GitHub repo (effect sizes) | https://github.com/marton-balazs-kovacs/multi100 |
| OSF fork – data & maintained analysis (workshop fetch target) | https://osf.io/6zqct/ |
| Official Multi100 archival record for analyst C6HJR | https://osf.io/8rtwe/ |
| Full SCORE dossier on the EU-frames case | https://osf.io/h7432/ |
| Multi100 navigation sheet (claims list) | Available from the Multi100 OSF project above |
One paper, all three Rs
The target article (Teney 2016) was put through all three SCORE activities: reproduction, robustness reanalysis, and replication. Few papers are, which is why this one case can carry the whole day. The parent node for the full dossier is https://osf.io/h7432/, and the table below summarises each component.
| Component | What | Outcome |
|---|---|---|
whp4b |
Original materials collected by COS | View-only |
cf8en |
Author-provided code – two Stata do-files (eb_all.do, eb_ml_ESR.do) supplied on request; data cannot be shared under GESIS terms (statement file gxk9r) |
Code available; data restricted |
8fwrt |
Power analysis for the replication | – |
ezhcs |
Push-button reproduction (with author code) | Failed – file names in the code did not match the files available for download; 18 files referenced, 16 listed on the website |
4rjbf |
Source-data reproduction | “Not reproduced” – original effect size could not be extracted or re-calculated |
gr8u6 |
Multi100 robustness – 5 independent reanalysts | All 5 coefficients negative (claim-consistent direction); partial r from −0.006 to −0.452 |
zt5y2 |
Secondary-data replication (Eurobarometer + Eurostat, different waves; preregistered; N = 352,114 person-year obs.) | Not replicated – sign flip: positive coefficient (b = 0.002, t = 2.004, p = 0.046) in both Analysis One and Analysis Two |
6zqct |
Analyst’s maintained fork (workshop fetch target) – carries the working materials and today’s data | https://osf.io/6zqct/ |
8rtwe |
Official Multi100 archival record for analyst C6HJR | https://osf.io/8rtwe/ |
Three Rs, three different verdicts on one paper: reproduction failed (twice, differently); robustness held in sign but spanned two orders of magnitude in effect size; replication returned a sign flip. The glossary states that the three indicators are near-uncorrelated. A single case built entirely from public artefacts makes that concrete.
If you are at Newcastle University
The workshop is open to anyone, so support of the kind below exists wherever you are: the library at your own institution and its research-software or research-data team are the places to ask. As one worked example, participants based at Newcastle University can draw on the following.
| Resource | URL |
|---|---|
| RSE training (Research Software Engineering) | https://rse.ncldata.dev/events |
| NCL Library – open research pages | https://www.ncl.ac.uk/library/academic-support/research/open-research/ |
| Research Data Newcastle | https://data.ncl.ac.uk |
| UKRN Primers (open research primers) | https://www.ukrn.org/primers/ |
| ReproducibiliTea Newcastle | Search ‘ReproducibiliTea Newcastle’ – a journal club for open and reproducible science |
Tool documentation
| Tool | URL |
|---|---|
| Positron (IDE) | https://positron.posit.co |
| Quarto | https://quarto.org |
| quarto-live (webR extension) | https://r-wasm.github.io/quarto-live/ |
| osfr package | https://docs.ropensci.org/osfr/ |
| fixest package | https://lrberge.github.io/fixest/ |
| ggdag package | https://r-causal.github.io/ggdag/ |