The one-day track
For a facilitated cohort · four parts of ninety minutes, eight hours from the first slide to the last
This is the format the course was built for, and the one that the slide deck follows: four parts of ninety minutes, with breaks of thirty, sixty and thirty minutes between them. It needs a facilitator, because several of its beats are things that a room does together, such as everyone typing the t-statistic they have just reproduced and holding it until the count. Across the day you work as analyst no. 6 on a Multi100 reanalysis of a claim made by the original author (OA) of a paper about how the European Union is framed across its member states. You reproduce the value on record for one of the five published analysts, write the estimand behind the claim in words, finish a causal graph for it, work through a menu of defensible analyses, and then declare one of them in a shortened preregistration template before you run it. What you leave with is a report of your own, two dots on the class chart, one explored and one declared, and the percentile that places your result among 840 alternatives.
Before the day
The day runs in either of two routes, and you prepare only for the one you pick. Route 1 is R and the Positron editor with a ready-made workspace; Route 2 is the browser lab, which needs nothing installed. The two reach identical numbers, and switching between them on the day costs nothing.
| Work through | What you do | Timing |
|---|---|---|
| Setup | For Route 1, install R and Positron, download the workspace zip and run R/check_setup.R until it prints OK. For Route 2, open the browser lab once and confirm that it loads on the machine you will use. |
30 to 60 minutes, once |
| Exercise overview | Read the claim you will be working on and the table of what the five Multi100 analysts found when they reanalysed it. | 10 minutes |
| Getting started with R | Optional. Worth the time if you have never written R: objects, data frames, reading a CSV, and the pipe. | An evening |
| Reading a specification curve | Optional pre-reading for Part 4, on what a large set of results together can and cannot establish. | 20 minutes |
By the start of the day you have one of the two routes working on the machine you will use, and you know what the claim is and how far apart the five Multi100 analysts landed on it.
Part 1 – Why any of this, and with what tools
Part 1 asks what a claim in social science is for, why people describe the field as being in crisis, and what a computational toolchain has to do with either question. The hands-on work sits at the end, where it runs into the first break, so that a machine that will not cooperate costs nobody any content. A two-minute stretch sits between the five-analyst forest and the tools segment.
| Work through | What you do | Timing |
|---|---|---|
| Slides, Part 1 | Follow the talk: what a claim is for and why the worry is structural rather than a matter of fraud; the twenty-nine teams of the red-card study and the estimates they returned; why ‘replication’ names three different operations; the SCORE programme and its three studies; and Multi100 from the position of an analyst inside it, ending on the five-analyst forest. | 62 minutes |
| Slides, the tools segment | Take in the toolchain: what open source buys you, plain text as the format that survives, the render pipeline, packages, version control, where data and code are archived, licences, and what the tools cannot do. Note anything you might switch to. | 18 minutes |
| Browser lab, Task 1, or the Positron workspace from Setup | Get to the point where a line of R runs and the panel of 270 country-years is in front of you. The three R’s warm-up on the same page fills the wait while the engine downloads. | 8 minutes, and the 30-minute break after it if you need more |
By the end of the part you have R running on your chosen route with the panel of 270 country-years loaded, and you can distinguish the three operations that go under the single word ‘replication’.
Part 2 – The question underneath
Part 2 is the conceptual middle of the day. It reads the paper in full, then argues that analysts who look as though they disagree are often answering different questions, and it gives you the vocabulary for saying which question you are answering. A two-minute stretch sits between the estimands run and the graphs.
| Work through | What you do | Timing |
|---|---|---|
| Slides, Part 2 | The paper in full: the four framing scales, what OA actually fitted, the robustness question already inside the paper, the two failed reproductions and the replication that came back with the opposite sign, and the five questions that the Multi100 analysts asked in the free phase. | 20 minutes |
| Slides, the estimands run | Two studies set beside our five: the red-card estimates return in one line, four different estimands are pulled out of one football question, and the spread collapses once the estimand is fixed. Then Lundberg’s three steps from a theoretical quantity to an empirical one. | 25 minutes |
| Slides, the causal-graph crash course | Confounders, mediators and colliders; open and closed paths; total against direct effects; selection and feedback. Then the return to our own claim, where everyone writes a one-sentence estimand, holds it, and sends it together. | 25 minutes |
Browser lab, Task 2, or R/dag_starter.R in the workspace, with Draw the arrows you believe as the slower companion |
The optional R tutorial runs first if the room needs it, pitched at whoever is least confident with the code. Then look at the panel and find the countries that move sharply around 2009; write the estimand in words; take the # off the arrows you are willing to defend, re-run adjustmentSets(), and write two sentences on what two-way fixed effects adjust for and what they cannot fix. |
18 minutes |
By the end of the part you have the estimand behind the claim written in one sentence, a graph carrying only the arrows you are willing to defend, and the adjustment set that follows from it.
Part 3 – What you can actually do
Part 3 is where the reproduction happens and where the menu of alternatives opens. Two hands-on tasks, with the talk that connects them in between.
| Work through | What you do | Timing |
|---|---|---|
| Slides, Part 3 | The three Multi100 instructions read verbatim, what each of them assumes, the columns of the panel, and the aim of the reproduction you are about to run. | 15 minutes |
| Browser lab, Task 3, or your working script in the workspace | Fit mcosmo ~ unemp_c with country and year fixed effects and land t = −3.804 (df = 233, N = 270), the value on record for analyst C6HJR. Everyone types their t and holds it until the count, so that the identical values arrive together. |
22 minutes |
| Specification menu and Slides | First, why everyone landed the same t while the five published analysts, each writing their own code, did not. Then the eight axes you choose from, one option each: outcome, outcome family, predictor, predictor form, copredictor, estimator, sample and weights. Last, what the constrained task left open, including the unit of analysis, and the claim-alignment convention that puts every result on one scale. | 20 minutes |
| Slides, the methodological tutorial | Optional depth: what each axis actually does to an estimate. The outcome, claim alignment and estimator slides are the core of it. | 13 minutes |
| Browser lab, Task 4, then Multiverse | Call try_spec() three or four times, changing one argument at a time, print them side by side with my_experiments(), then pick the one you would defend and send it with submit_exploration(). Your dot lands on the chart as an open circle. |
20 minutes |
By the end of the part you have reproduced t = −3.804 yourself, tried three or four alternatives one argument at a time, and put an exploratory dot on the class chart.
Part 4 – Reading the multiverse, choosing one path
Part 4 opens on what the room has just built and closes on the single analysis you would defend if you had to pick one. The reading on the specification curve is its companion.
| Work through | What you do | Timing |
|---|---|---|
| Multiverse and Slides | Read the class’s exploratory dots against the full curve and the five published analysts, and find your own among them. | 10 minutes |
| Slides | Bet on which analytical choice moves estimates most before the decomposition is shown, then see it: the outcome fork carries 49.7% of the variation across specifications. Then where the five analysts sit on the aligned grid, and where the wrong-sign results concentrate. | 16 minutes |
| Browser lab, the placement printed beside your result | Read off your own percentile, which is the share of the 840 specifications more negative than the estimate you submitted, and send it with everyone else’s. Anyone without a submission uses the reproduced baseline instead. | 7 minutes |
| Slides, the multiverse tutorial and what the curve leaves out | Optional depth on where the idea came from and what a multiverse is for, followed by the four things that a curve leaves out, which is not part of the optional run. | 10 minutes |
| Reading a specification curve and Slides | Why hundreds of estimates on the same data are not hundreds of independent tests, what a curve-level permutation test would have to do, and how to read one if it were run. Then the discussion beat, argued in chat or at the table. | 12 minutes |
| Slides, with Reproducibility repositories as the follow-up | The three critical positions on multiverse analysis, the argument that a fully specified question has one correct analysis, what preregistration settles and what it leaves open, and the OSF nodes behind the case. | 10 minutes |
| Browser lab, Task 5, or the report skeleton in the workspace | Complete the preregistration block before you run anything, translate the declaration into model syntax yourself, run it, and submit with report_result(). Your dot joins the chart as a filled circle beside the open exploratory ones. |
20 minutes |
| Multiverse | Wrap up with the chart on screen: two sentences worth keeping, and the reminder that the path you declared was one choice among the alternatives you had just seen. | 5 minutes |
By the end of the part you have your own percentile in the 840-specification grid, one specification declared in writing before it was run, and a report of your own carrying the result that came back.
Adapting this track
The first thing to compress is optional depth. The deck carries three runs of extra slides – an R tutorial in Part 2, a methodological tutorial in Part 3 and a multiverse tutorial in Part 4 – and they are a live switch rather than fixed content, so a room that does not want them loses nothing else when they go. Adding ?depth=0 to the address of the deck opens it without them. Take time from a tutorial before you take it from a task, because the tasks are what the following part reads.
Seven things stay in whatever else goes: the five-analyst forest with its software column, the estimand exercise in Part 2, the half-built graph and its adjustment set, the reproduction and the count that follows it, the Task 4 submission, the preregistration moment in Part 4, and the reveal of which fork carries the variation. Each of them is either a beat that a later part reads from, or the one point in the day at which participants do the thing themselves rather than watch it being done.
The nine modules of the companion curriculum hold the material that a single day can only point at, running from a first session with R through causal graphs and the statistical methods behind the case to version control. Each module stands on its own, so a cohort can be sent one of them before the day or after it without needing any of the rest.
If a single day is the wrong shape for your group:
- The three-day track keeps this day exactly as it is and gives it a tools day before and a methods day after.
- The semester track spreads the same material across a term, with the workshop day sitting inside it as the practical core.
- The self-study track drops the clock and the cohort, for anyone working alone.