Resources
Readings, links, and tools
Glossary
The four terms below follow the definitions in Nosek et al. (2025). The ‘in this workshop’ column says where each concept appears in the session.
| Term | Definition | In this workshop |
|---|---|---|
| Reproducibility | Same question · same data · same analysis → same result | Tested by the SCORE programme (Alipourfard et al. 2021); Teney’s result was not reproducible (Miske et al. 2026) |
| Robustness | Same question · same data · justifiable alternative analysis → stable result | Multi100’s design (Aczel et al. 2026); the five-analyst table; Task D2 |
| Replicability | Same question · independent new data → same result | Tyner et al. (Tyner et al. 2026); discussed in Talk I |
| Repeatability | Umbrella term covering all three above | Used in the Nosek et al. glossary as the overarching category |
Core readings
The three Nature 2026 papers (SCORE)
Reproducibility: Miske et al. (2026) – Miske et al. (2026). Investigating the reproducibility of the social and behavioural sciences. Nature, 652, 126–134. https://doi.org/10.1038/s41586-026-10203-5
Robustness / Multi100: Aczel et al. (2026) – Aczel et al. (2026). Investigating the analytical robustness of the social and behavioural sciences. Nature, 652, 135–142. https://doi.org/10.1038/s41586-025-09844-9
Replicability: Tyner et al. (2026) – Tyner et al. (2026). Investigating the replicability of the social and behavioural sciences. Nature, 652, 143–150. https://doi.org/10.1038/s41586-025-10078-y
Vocabulary
Nosek et al. (2025) – Nosek et al. (2025). A brief glossary of terms about repeatability: replicability, robustness, and reproducibility. MetaArXiv. 4 pages; essential pre-reading. https://doi.org/10.31222/osf.io/mqfp4_v1
Nosek and Errington (2020) – Nosek & Errington (2020). What is replication? PLOS Biology, 18(3), e3000691. https://doi.org/10.1371/journal.pbio.3000691
Estimands and analytical choices
Auspurg and Brüderl (2021) – Auspurg & Brüderl (2021). Has the credibility of the social sciences been credibly destroyed? Socius, 7. https://doi.org/10.1177/23780231211024421
Lundberg et al. (2021) – Lundberg, Johnson & Stewart (2021). What is your estimand? American Sociological Review, 86(3), 532–565. https://doi.org/10.1177/00031224211004187
Silberzahn et al. (2018) – Silberzahn et al. (2018). Many analysts, one data set: making transparent how variations in analytic choices affect results. Advances in Methods and Practices in Psychological Science, 1(3), 337–356. https://doi.org/10.1177/2515245917747646
The case study paper
- Teney (2016) – Teney (2016). Does the EU economic crisis undermine subjective Europeanization? European Sociological Review, 32(5), 619–633. https://doi.org/10.1093/esr/jcw008
The SCORE programme
- Alipourfard et al. (2021) – Alipourfard et al. (2021). Systematizing Confidence in Open Research and Evidence (SCORE). SocArXiv. https://doi.org/10.31235/osf.io/46mnb
Data and project links
| Resource | URL |
|---|---|
| Multi100 – OSF project | https://osf.io/q5h2c/ |
| Multi100 – GitHub repo (effect sizes) | https://github.com/marton-balazs-kovacs/multi100 |
| OSF fork – data & corrected analysis (workshop fetch target) | https://osf.io/6zqct/ |
| Official Multi100 archival record for analyst C6HJR | https://osf.io/8rtwe/ |
| Full SCORE dossier on Teney (2016) | https://osf.io/h7432/ |
| Multi100 navigation sheet (claims list) | Available from the Multi100 OSF project above |
The Teney (2016) SCORE dossier – one paper, all three Rs
The Teney (2016) paper was subjected to all three SCORE activities: reproduction, robustness reanalysis, and replication. The parent node for the full dossier is https://osf.io/h7432/. The table below summarises each component.
| Component | What | Outcome |
|---|---|---|
whp4b |
Original materials collected by COS | View-only |
cf8en |
Author-provided code – two Stata do-files (eb_all.do, eb_ml_ESR.do) supplied on request; data cannot be shared under GESIS terms (statement file gxk9r) |
Code available; data restricted |
8fwrt |
Power analysis for the replication | – |
ezhcs |
Push-button reproduction (with author code) | Failed – file names in the code did not match the files available for download; 18 files referenced, 16 listed on the website |
4rjbf |
Source-data reproduction | ‘Not reproduced’ – original effect size could not be extracted or re-calculated |
gr8u6 |
Multi100 robustness – 5 independent reanalysts | All 5 coefficients negative (claim-consistent direction); partial r from −0.006 to −0.452 |
zt5y2 |
Secondary-data replication (Eurobarometer + Eurostat, different waves; preregistered; N = 352,114 person-year obs.) | Not replicated – sign flip: positive coefficient (b = 0.002, t = 2.004, p = 0.046) in both Analysis One and Analysis Two |
6zqct |
Analyst’s maintained fork (workshop fetch target) – adds corrected April 2025 analysis | https://osf.io/6zqct/ |
8rtwe |
Official Multi100 archival record for analyst C6HJR | https://osf.io/8rtwe/ |
Three Rs, three different verdicts on one paper: reproduction failed (twice, differently); robustness held in sign but spanned two orders of magnitude in effect size; replication returned a sign flip. This is the glossary’s point that the three indicators are near-uncorrelated – made concrete by a single case built entirely from public artefacts.
Newcastle University resources
| Resource | URL |
|---|---|
| RSE training (Research Software Engineering) | https://rse.ncldata.dev/events |
| NCL Library – open research pages | https://www.ncl.ac.uk/library/academic-support/research/open-research/ |
| Research Data Newcastle | https://data.ncl.ac.uk |
| UKRN Primers (open research primers) | https://www.ukrn.org/primers/ |
| ReproducibiliTea Newcastle | Search ‘ReproducibiliTea Newcastle’ – a journal club for open and reproducible science |
Tool documentation
| Tool | URL |
|---|---|
| Positron (IDE) | https://positron.posit.co |
| Quarto | https://quarto.org |
| quarto-live (webR extension) | https://r-wasm.github.io/quarto-live/ |
| osfr package | https://docs.ropensci.org/osfr/ |
| fixest package | https://lrberge.github.io/fixest/ |
| ggdag package | https://r-causal.github.io/ggdag/ |