Resources
Readings, links, and tools
Glossary
The four terms below follow the definitions in Nosek et al. (2025). They are not interchangeable, and the day depends on the difference: a finding can survive every robustness check and still be one that nobody can recompute from the original data, or reproduce cleanly and still fail on new data. The ‘In this course’ column notes where each concept appears in the session.
| Term | Definition | In this course |
|---|---|---|
| Reproducibility | Same question · same data · same analysis → same result | Tested by the SCORE programme (Alipourfard et al. 2021); the result reported by the original author (OA) was not reproducible (Miske et al. 2026) |
| Robustness | Same question · same data · justifiable alternative analysis → stable result | The Multi100 design (Aczel et al. 2026); the five-analyst table; the specification work in Part 3 |
| Replicability | Same question · independent new data → same result | Tyner et al. (Tyner et al. 2026); discussed in Part 1 |
| Repeatability | Umbrella term covering all three above | Used in the Nosek et al. glossary as the overarching category |
Core readings
The three Nature 2026 papers (SCORE)
Reproducibility: Miske et al. (2026) – Miske et al. (2026). Investigating the reproducibility of the social and behavioural sciences. Nature, 652, 126–134. https://doi.org/10.1038/s41586-026-10203-5
Robustness / Multi100: Aczel et al. (2026) – Aczel et al. (2026). Investigating the analytical robustness of the social and behavioural sciences. Nature, 652, 135–142. https://doi.org/10.1038/s41586-025-09844-9
Replicability: Tyner et al. (2026) – Tyner et al. (2026). Investigating the replicability of the social and behavioural sciences. Nature, 652, 143–150. https://doi.org/10.1038/s41586-025-10078-y
Vocabulary
Nosek et al. (2025) – Nosek et al. (2025). A brief glossary of terms about repeatability: replicability, robustness, and reproducibility. MetaArXiv. 4 pages; read this one before the day. https://doi.org/10.31222/osf.io/mqfp4_v1
Nosek and Errington (2020) – Nosek & Errington (2020). What is replication? PLOS Biology, 18(3), e3000691. https://doi.org/10.1371/journal.pbio.3000691
Estimands and analytical choices
Auspurg and Brüderl (2021) – Auspurg & Brüderl (2021). Has the credibility of the social sciences been credibly destroyed? Socius, 7. https://doi.org/10.1177/23780231211024421
Lundberg et al. (2021) – Lundberg, Johnson & Stewart (2021). What is your estimand? American Sociological Review, 86(3), 532–565. https://doi.org/10.1177/00031224211004187
Silberzahn et al. (2018) – Silberzahn et al. (2018). Many analysts, one data set: making transparent how variations in analytic choices affect results. Advances in Methods and Practices in Psychological Science, 1(3), 337–356. https://doi.org/10.1177/2515245917747646
Preregistration
These three are the pre-session reading list for the preregistration segment of the day, and all are open access. None is required, because everything is taught on the day, but they let you arrive with the ideas already in hand.
Nosek, Ebersole, DeHaven & Mellor (2018). The preregistration revolution. PNAS, 115(11), 2600–2606. https://doi.org/10.1073/pnas.1708274114 – a short primer on the idea on which preregistration depends: predicting a result versus explaining one after the fact, and the case, like ours, where the data already exist.
The AsPredicted eight-question form – the short template you complete in the session, listed on OSF as the “Preregistration Template from AsPredicted.org”. Live form: https://aspredicted.org; OSF listing: https://help.osf.io/article/158-create-a-preregistration.
van den Akker, Weston, Campbell, et al. (2021). Preregistration of secondary data analysis: A template and tutorial. Meta-Psychology, 5, 2625. https://doi.org/10.15626/MP.2020.2625 – the fuller template, built for our situation of reanalysing data that already exist, and the tool that a full version of the exercise would use.
Reading the multiverse
- Reading a specification curve – the course reading for the final part of the day: why a large set of results is not a large set of independent tests, and what a single test across the whole curve can and cannot establish.
The case study paper
- Teney (2016) – Teney (2016). Does the EU economic crisis undermine subjective Europeanization? European Sociological Review, 32(5), 619–633. https://doi.org/10.1093/esr/jcw008
The SCORE programme
- Alipourfard et al. (2021) – Alipourfard et al. (2021). Systematizing Confidence in Open Research and Evidence (SCORE). SocArXiv. https://doi.org/10.31235/osf.io/46mnb
Companion curriculum
Sixteen modules hold most of the teaching in this course, from a first line of R through to a reanalysis you can version and archive. The workshop at the centre is compressed, so it works through the single hands-on case and leaves the skills around it to the modules, each of which stands on its own and can be taken without the others. The curriculum overview sets out the three strands into which the modules are grouped, and a suggested reading order.
- Getting started with R – for participants arriving with no R at all; best read before the day.
- Positron, RStudio and VS Code – what an IDE does and what the course needs from one, with Positron pane by pane and RStudio and VS Code set beside it; best read before the day, once R is installed.
- Quarto and reproducible documents – the medium in which every page of this site is written: what a
.qmdfile is and what rendering does to it, the YAML header and the chunk options that matter, figures, tables and cross-references, citations drawn from a bibliography, and publishing the finished document, all of it worked through on files that you can open in this project. - The items behind the scales – the EU-meaning battery taken apart: how the four framing scales and the two composites are built from its thirteen items, how often each item is named and how the items hang together, and how the published scales compare with the structure that the answers hold.
- Binary outcomes – one item of the battery as the outcome on the real respondents, fitted as a logit and as a probit side by side, read through odds ratios and average marginal effects, with errors clustered by country-year and a random intercept for the country-year.
- Weights – design weights inside a country-year cell, precision weights across cells and population weights across countries, taken apart on the course panel, and why only the third changes the estimand.
- Missing answers – what a respondent who mentions no frame at all does to a share built from mentions, the rules that the specification menu offers for such respondents, and multiple imputation set against those rules.
- The statistical methods behind the course – every model fitted on the data from the EU-frames case, with real outputs read line by line, from a first
lm()through fixed and random effects to multilevel, Bayesian and ordered-beta models, ending with meta-analysis and the common effect-size axis. - Causal graphs – causal graphs written as R formulas with
dagify(): confounders, mediators and colliders fitted on data as well as described, the EU-frames graph whose adjustment set is what two-way fixed effects absorb, and what fixed effects cannot fix. - The multiverse and its critics – where specification-curve thinking came from, the three purposes that a multiverse can serve, and the 2025–26 critical debate read against the course grid.
- Simulation as a method – a simulated person-level twin, generated from known parameters so that the original analysis can be fitted on it: parameter recovery, composition against context, recovering individual effects, weights that matter, the degrees-of-freedom argument run live, a null multiverse, a lineup, and design precision by simulation.
- The individual level – the published specification fitted at the individual level on a validated simulated twin, with the real-data values beside it: individual circumstance against national context, and what declaring a level of analysis does to the evidence.
- Social science data sources – a map of the survey programmes, archives, household panels, official statistics, international indicators and political datasets, with the access rule for each, and then the sources used by the course, step by step, including a screen-by-screen walk through the GESIS download.
- Building the analysis file – the two workspace scripts that turn your own sixteen GESIS downloads into the respondent file and then into the panel, reproducing the committed panel to machine precision.
- Reproducibility repositories – research-data archives, the OSF nodes behind this case, and preregistration templates in more depth.
- Git and GitHub for reproducible research – version control and publishing your own report, the hands-on practice that is kept out of the core day so that it can be learned separately.
Data and project links
| Resource | URL |
|---|---|
| Multi100 – OSF project | https://osf.io/q5h2c/ |
| Multi100 – GitHub repo (effect sizes) | https://github.com/marton-balazs-kovacs/multi100 |
| OSF fork – data & maintained analysis (course fetch target) | https://osf.io/6zqct/ |
| Official Multi100 archival record for analyst C6HJR | https://osf.io/8rtwe/ |
| Full SCORE dossier on the EU-frames case | https://osf.io/h7432/ |
| Multi100 navigation sheet (claims list) | Available from the Multi100 OSF project above |
The SCORE dossier on the case
The target article (Teney 2016) was put through all three SCORE activities: reproduction, robustness reanalysis, and replication. Few papers have been, which is why this one case can serve for the whole day. The parent node for the full dossier is https://osf.io/h7432/, and the table below summarises each component.
| Component | What | Outcome |
|---|---|---|
whp4b |
Original materials collected by COS | View-only |
cf8en |
Author-provided code – two Stata do-files (eb_all.do, eb_ml_ESR.do) supplied on request; data cannot be shared under GESIS terms (statement file gxk9r) |
Code available; data restricted |
8fwrt |
Power analysis for the replication | – |
ezhcs |
Push-button reproduction (with author code) | Failed – file names in the code did not match the files available for download; 18 files referenced, 16 listed on the website |
4rjbf |
Source-data reproduction | “Not reproduced” – original effect size could not be extracted or re-calculated |
gr8u6 |
Multi100 robustness – 5 independent reanalysts | All 5 coefficients negative (claim-consistent direction); partial r from −0.006 to −0.452 |
zt5y2 |
Secondary-data replication (Eurobarometer + Eurostat, different waves; preregistered; N = 352,114 person-year obs.) | Not replicated – sign flip: positive coefficient (b = 0.002, t = 2.004, p = 0.046) in both Analysis One and Analysis Two |
6zqct |
Analyst’s maintained fork (course fetch target) – holds the working materials and the data used in the course | https://osf.io/6zqct/ |
8rtwe |
Official Multi100 archival record for analyst C6HJR | https://osf.io/8rtwe/ |
The three activities reached three different verdicts on the same paper. Reproduction failed twice, in different ways. The robustness reanalysis held in sign but spanned two orders of magnitude in effect size, and the replication returned a sign flip. The glossary states that the three indicators are near-uncorrelated, and this case, documented entirely in public artefacts, shows what that means for a single paper.
If you are at Newcastle University
The course is open to anyone, and support of the kind listed below exists wherever you are: the library at your own institution and its research-software or research-data team are the places to ask. As an example, participants based at Newcastle University can draw on the following.
| Resource | URL |
|---|---|
| RSE training (Research Software Engineering) | https://rse.ncldata.dev/events |
| NCL Library – open research pages | https://www.ncl.ac.uk/library/academic-support/research/open-research/ |
| Research Data Newcastle | https://data.ncl.ac.uk |
| UKRN Primers (open research primers) | https://www.ukrn.org/primers/ |
| ReproducibiliTea Newcastle | Search ‘ReproducibiliTea Newcastle’ – a journal club for open and reproducible science |
Tool documentation
| Tool | URL |
|---|---|
| Positron (IDE) | https://positron.posit.co |
| Quarto | https://quarto.org |
| quarto-live (webR extension) | https://r-wasm.github.io/quarto-live/ |
| osfr package | https://docs.ropensci.org/osfr/ |
| fixest package | https://lrberge.github.io/fixest/ |
| ggdag package | https://r-causal.github.io/ggdag/ |