Reproducibility repositories
From a licensed archive to a plan you cannot quietly rewrite
This is the repositories module of the companion curriculum, and it is optional. It works through the repositories that this workshop’s own data actually passed through: which archive holds the raw microdata behind the EU-frames case, why that microdata never enters the workshop’s materials, which OSF nodes the workshop fetches from and why, and how a preregistration – yours, or the one that the Multi100 team filed – gets written down before an analysis runs. The closing section turns to preregistration templates, and it feeds straight into Part 4 of the workshop day, where you complete one yourself.
The module assumes no more than the workshop itself does. You do not need to have taken the git and GitHub module first, though the two are natural companions. This module is mostly about other people’s materials, and about a study plan you commit to in writing. The git and GitHub module is about the tool, git, that lets you keep a record of how your own materials changed over time.
1. The licence decides what travels
The EU-frames case rests on sixteen waves of Eurobarometer survey data, and the German distributor of Eurobarometer is GESIS – Leibniz Institute for the Social Sciences, which describes itself as “the largest German infrastructure institute for the social sciences”. It runs Germany’s central social-science data archive, holding more than seven thousand studies, certified to the CoreTrustSeal standard and organised around the FAIR principles (findable, accessible, interoperable, reusable). This is the institutionally managed kind of repository: an archive run by a funded body that curates a named collection under its own terms, rather than a general-purpose deposit box that accepts anything from anyone.
Those terms are the reason that this workshop’s own repository looks the way it does. The GESIS licence for the Eurobarometer scientific-use files does not permit redistributing the raw microdata, so the dataset that this workshop actually ships – EUframes_cy.csv – is not the microdata itself but a set of derived country-year aggregates, built once from the raw files and then released under their own, more permissive terms (CC BY 4.0 for the derived data; the codebook states this explicitly). The preparation scripts that built them are kept in the template repository as an exhibit: code you can read to see exactly how the aggregates were constructed, not a pipeline you can point at a folder of raw files, because no raw files are there to point at. Every project built on licensed survey data lives under the same discipline: commit only what the licence allows. A workshop that redistributed the raw microdata, even to a closed group of participants, would be breaching the terms under which GESIS makes the data available in the first place.
If you want the raw data yourself – to check the aggregation, to try a person-level specification that the country-year files cannot support, or simply because a licensed archive is worth learning to use – the route is a free registration with GESIS at search.gesis.org, followed by the request process that GESIS runs for the specific Eurobarometer waves that the codebook names. This is the tiered-access pattern common to institutional archives: viewing a catalogue entry is open to everyone, but obtaining the file itself means registering under your own name and agreeing to the conditions set by the archive. Registering is a normal step in a working researcher’s life. It is also the reason that the original author (OA) who built this claim, the Multi100 team who reanalysed it, and this workshop’s own rebuild could all work from the same sixteen waves without any of them ever redistributing a byte of GESIS data to another.
Open the data codebook and read its final section, “Licence”. Write one sentence distinguishing what is licensed under CC BY 4.0 from what remains subject to GESIS terms, and one sentence on what a ‘derived aggregate’ is doing in that distinction – why summarising sixteen waves down to one row per country-year changes what you are allowed to share, even though the underlying facts are the same.
2. Three nodes hold the trail
The Open Science Framework (OSF), run by the non-profit Center for Open Science, is the general-purpose repository behind almost everything else that this workshop touches. Its help pages describe an OSF project as “a flexible, collaborative workspace where researchers can plan, manage, and share any part of their research lifecycle”, and a component as an “optional sub-project nested within a parent project … used to break a large project into smaller parts”. Projects and components form a parent–child tree, and each node can carry files, a wiki, contributors and its own privacy setting. Three nodes in that tree matter for this workshop specifically.
osf.io/8rtwe – the official Multi100 archival record. This is the component that the Multi100 project itself preserved for analyst C6HJR: the materials exactly as originally submitted. It is the archival record in the strict sense – frozen, citable, unchanged since deposit. That immutability is what an archive is for, and it is also its limit, because an archive cannot follow a project onwards.
osf.io/6zqct – the analyst’s maintained fork, and the workshop’s own fetch target. OSF describes this node itself as “a public fork” of 8rtwe, maintained by the facilitator for educational purposes. Alongside the submitted materials it carries a later revision of the analysis script and its exported dataset, plus revised Task 1 and Task 2 documents, and EUframes_cy.csv in its main folder. This is the node that every fetch in this workshop actually targets. Archives preserve a moment and forks maintain a living version, so a project that expects to keep moving usually needs both, with each pointing at the other.
osf.io/h7432 – the full SCORE dossier, one paper put through all three Rs. This is the parent node under which the whole reproducibility apparatus around the EU-frames case sits, and its component table records what each check found.
| Component | What | Outcome |
|---|---|---|
whp4b |
Original materials collected by the Center for Open Science | View-only |
cf8en |
Author-provided code – two Stata do-files, supplied on request; OA’s own statement records that the data cannot be forwarded and must come from GESIS | Code available, data restricted |
8fwrt |
Power analysis for the replication | – |
ezhcs |
Push-button reproduction, run with OA’s own code | Failed. “The authors used file names that did not correspond with the names of the files available for download. Additionally 16 datasets are listed on the website as being used but the code references 18 files.” |
4rjbf |
Source-data reproduction | ‘Not reproduced’ – the effect size “could not be extract[ed] nor re-calculate[d]” from the source data |
gr8u6 |
Multi100 robustness – five independent reanalysts, including analyst C6HJR whose materials this workshop reanalyses | All five coefficients negative (claim-consistent direction); partial r ranging from −0.006 to −0.452, a span of two orders of magnitude |
zt5y2 |
Secondary-data replication – fresh Eurobarometer and Eurostat waves, preregistered, N = 352,114 person-year observations | Not replicated – sign flip. “We found a positive coefficient for unemployment on cosmopolitan framing”: b = +0.002, p = .046 |
That table shows what a well-kept repository record lets a reader establish without emailing anyone. The push-button check failed for a mundane, checkable reason – file names that did not match what was actually available for download – not because the underlying claim was wrong. The source-data check failed differently: the code ran, but the reported effect size could not be recovered from it. The robustness check, the one that this workshop stands inside, held the direction of the claim across five independent analysts while the effect size itself moved by a factor of roughly 75. And the replication, run on genuinely new data under a preregistered plan, flipped sign entirely. One paper, three different verdicts on three different questions – reproduction, robustness, replicability – each one legible only because the materials behind it sit in a public, node-by-node OSF tree rather than in a folder on someone’s laptop.
Open osf.io/h7432 and find the zt5y2 component from the table above. Locate the sentence reporting the sign-flipped coefficient in the materials of that component, and note the exact figures: the coefficient, its standard error, and the significance level. This is the same scavenger task that the in-session OSF segment sets in Part 4 – if you are working through this module before the workshop, you will already know where to look when it comes up.
3. Archives preserve, forks maintain, registrations freeze
Two pieces of OSF vocabulary are needed for the rest of this module. A registration is “a frozen version of your project that can never be edited or deleted, but you can issue a withdrawal of it later”, time-stamped and given its own permanent identifier. This is the mechanism behind preregistration: it captures a study plan before the data are seen (or, for a reanalysis of existing data, before the specific analytical choices are made), so the plan cannot later be quietly rewritten to fit the result. The Multi100 reanalysts themselves worked from this kind of frozen, dated protocol, which is one reason that their five independent results can be compared on equal footing.
A fork, discussed in the git and GitHub module in its git and GitHub sense, has a direct OSF equivalent: your own editable copy of a public node, kept separate from the original so you can change it without touching the original record. osf.io/6zqct is such a fork, and “a public fork” is the term that OSF itself uses for the manoeuvre.
Both pieces of vocabulary are on display in the materials of this case. The registration you can read today is the Multi100 protocol. It froze the scoring rules – what counts as ‘the same conclusion’, how close an effect size must be to count as ‘close’ – before a single reanalyst’s result came back, which is why the agreement rates you meet in Part 1 mean something. The fork you have already used is osf.io/6zqct, the maintained copy of the archival record, where the later script and its annotation live without touching the frozen original at 8rtwe. Registrations freeze plans rather than materials, which is what allows the date of a plan to be compared with the date of a result.
4. A reanalysis needs its own template
The earlier sections of this module have been about materials that other people deposited, and about the plan that the Multi100 team registered in advance. This closing section turns to writing your own, because Part 4 of the workshop asks you to do just that – complete a preregistration template block in your own report, before you run your chosen specification.
An OSF registration, as defined above, is the general mechanism: a frozen, time-stamped, permanent copy of a study plan. What varies between registrations is the template used to structure that plan – the specific set of questions that a researcher answers before the data are touched. OSF hosts several, built for different kinds of study, and two are worth knowing by name.
The AsPredicted-style short template. Originally built by the team behind AsPredicted.org and available as one of the registration forms built into OSF, this is the simplest entry point into preregistration and the one recommended for the in-session exercise. It asks a short, fixed sequence of questions – what the hypotheses are, what the key variables and analysis are, how the sample will be determined, and what would count as excluding data – each answered in a sentence or two rather than a full protocol document. Its appeal for a workshop setting is its brevity. A participant with no prior preregistration experience can complete it in the time that Part 4 sets aside, and what it demands – state the analysis before running it, in a form specific enough that it cannot be quietly reinterpreted afterwards – is what a longer template demands too, only asked in fewer words.
The secondary-data preregistration template (van den Akker and colleagues). A short-form template written for a fresh, not-yet-collected study assumes something that is not true of this workshop’s exercise: that the researcher has not yet seen the data. A reanalysis of an existing, already-public dataset like the Eurobarometer country-year panel breaks that assumption from the outset. You can see summary statistics, published tables, even other researchers’ results, before you commit to a specification of your own. Van den Akker and colleagues’ template is built around that difference. Where a fresh-data template asks what you will collect and how, a secondary-data template asks what you have already seen and what you have not. It wants to know which parts of the dataset or its published results you have already been exposed to, whether your hypothesis was formed before or after that exposure, and what remains genuinely undetermined about your specific analytical choices at the point of writing the plan. For a Multi100-style robustness reanalysis – exactly the kind of exercise that this workshop runs – this is the more accurate match for what preregistration can actually promise: not that you approached the data blind, which would rarely be true of a reanalysis, but that your specific analytical choices were fixed and dated before you ran them.
How the workshop’s own template block maps onto these. The template block you complete in Part 4 is modelled on the structure of the AsPredicted-style short form, a small number of short and specific questions, but adapted for a specification-menu reanalysis rather than a fresh study. Instead of ‘what is your hypothesis and how will you test it’, the block asks you to state your chosen specification – one option from each of the eight menu axes – and to justify that choice against the estimand and DAG you built earlier in the day. This is deliberately the lighter of the two templates, chosen because the Part 4 time budget favours a form that participants can complete and render inside a defined slot rather than one that demands a fuller protocol. The secondary-data template is signposted as pre-session and take-home reading rather than completed live, for anyone who wants to see how the same discipline scales up to a plan detailed enough to submit as a real OSF registration.
Read the question list of the AsPredicted template at aspredicted.org and, separately, skim the structure of the secondary-data template. For a single specification you might choose from the specification menu – say, outcome = mpos, predictor form = raw unemp, two-way fixed effects, all years, no weights – write one sentence answering the central question of the secondary-data template in your own words: what, if anything, had you already seen of the results on this claim before settling on that specification? A truthful answer to that question, written down before anything runs, is what separates a preregistration that constrains you from one that only decorates a result you had already half-decided on.
Sources and attribution
The definitional material in this module – what an OSF project, component and registration are, what a fork means on OSF – draws on the same primary sources as the repositories module of the trust-multilab companion curriculum: the OSF help pages (https://help.osf.io), quoted directly and linked in place wherever they appear above. The GESIS self-description in section 1 is likewise a direct quotation from the website of the institute (https://www.gesis.org/en/institute). None of this definitional material is workshop-specific; it describes the platforms themselves, so the same quotations hold regardless of which project supplies the data trail that is being followed.
Everything else – the GESIS licensing constraint on this workshop’s own data, the three-node OSF trail (8rtwe, 6zqct, h7432) and its component table, and the mapping from the workshop’s own template block onto the AsPredicted-style and secondary-data templates – is this workshop’s own worked example, drawn from the data codebook, the OSF nodes themselves, and the facilitator’s own project notes. Direct quotations from the component reports in the OSF dossier (ezhcs, 4rjbf, zt5y2) are reproduced exactly as they appear in the materials of those components.
For the version-control half of taking a reanalysis further – putting your own report, and eventually your own registration, under a history you can read and correct – see the companion module Git and GitHub for reproducible research.