The multiverse and its critics
Origins, purposes and the current debate
This is the multiverse module of the companion curriculum, and it is optional. The course teaches you to run a multiverse analysis: fix one specification in a preregistration, run it, and read your result against 2,520 pre-computed alternatives. This module is about where the idea came from, what it is claimed to do, and the arguments that are now being made against it. It is a companion to the inference reading, which covers one narrow question, namely what a curve-level test can and cannot establish. The question here is broader. The method you practised on the workshop day is under live debate in 2025–26, the objections raised against it are strong ones, and the aim of this module is that you can take a position in that debate. Throughout, the course materials serve as the worked example, sometimes as an answer to a criticism and sometimes as a case to which the criticism applies.
1. Origins
The founding statement is a 2016 paper by Sara Steegen, Francis Tuerlinckx, Andrew Gelman and Wolf Vanpaemel (Steegen et al. 2016). Their starting point is the construction of the data, before any model is fitted: turning raw data into an analysable dataset involves choices – which respondents to exclude, how to code a variable, where to cut a scale – and several options are usually reasonable. Instead of privileging one arbitrarily assembled dataset, they propose constructing every dataset that a reasonable set of choices could have produced and running the analysis on all of them, so that the reader sees how much the conclusion owes to decisions nobody would defend as the uniquely correct ones. The idea connects directly to Gelman and Loken’s ‘garden of forking paths’ (Gelman and Loken 2014). A researcher who makes each choice once, on the fly, has walked one path through a branching garden, and the single p-value at the end contains no record of the branches. The closing sentence of the 2016 paper sets the ambition plainly: “it should become standard practice to go beyond a single data set analysis and to acknowledge the multiverse of statistical results”, followed immediately by a concession that the critics would later press hard on – “performing a multiverse analysis will often be difficult, and to a large extent subjective” (Steegen et al. 2016, 710).
Two things about the original paper matter for everything that follows. First, its multiverse was customised for critique: the alternative datasets were choices that the original authors of the target study “might themselves have considered”, drawn from their own prior publications. It was therefore a post-mortem on the flexibility of one team, not a general survey of the ways in which the question could be answered, and the distinction matters again in section 2. Second, Steegen and colleagues already foresaw the extension that the course grid puts into practice: alongside the data multiverse they sketch a model multiverse, in which “One specific analysis thus corresponds to a single sample from a model multiverse” (Steegen et al. 2016, 710). They also place the whole idea in an older lineage running through perturbation and sensitivity analysis in economics and Bayesian statistics, and what the 2016 paper added to that lineage was the demand that the whole distribution of results be shown.
The modelling side of the idea was developed separately as specification curve analysis (Simonsohn et al. 2020), which enumerates the defensible ways of specifying the test, estimates all of them, plots the estimates in order and, in their proposal, tests the curve as a whole against a null distribution built by permutation, that is, by reshuffling the data so that any true effect is removed. The Multiverse page uses this form of display. A distinct strand kept the data and the question fixed and varied the analysts instead: 29 teams answering one question about football referees with odds ratios spanning 0.89 to 2.93 (Silberzahn et al. 2018); 73 teams testing one hypothesis about immigration and social policy, with more than 95% of the variance in their results left unexplained even after coding every identifiable decision in the workflow of every team (Breznau et al. 2022); and the Multi100 project, which put at least five independent reanalysts on one claim from each of 100 published studies (Aczel et al. 2026). The EU-frames case you worked on is one of those hundred, and the five analysts whose results you saw in Part 1 are its reanalysts. A further strand proposes Bayesian hierarchical modelling as the way to aggregate such team-science data, extended into a multiverse over inclusion criteria and priors (Hoogeveen et al. 2020). The course design combines these strands: the specification grid is a model multiverse, the five Multi100 analysts and your own class chart are a many-analysts layer on top of it, and the two are meant to be read against each other.
2. Three purposes
Rohrer, Hullman and Gelman ask the organising question of the current debate, which is what a multiverse is actually for. Their answer distinguishes three purposes – “as a tool for reflection and critique, as a persuasive tool, as a serious inferential tool” (Rohrer et al. 2026, 1) – and attaches a characteristic failure to each: “it fails as a persuasive tool when researchers disagree about which variations should be included in the analysis”, and “it fails as a serious inferential tool when the included analyses do not target a coherent estimand” (Rohrer et al. 2026, 1). Their conclusion is mild. The multiverse “does remain a valuable tool”, but they “urge against taking it too seriously” (Rohrer et al. 2026, 1).
The taxonomy matters most for how a multiverse is built, because the three purposes impose incompatible building rules on the universe of specifications. A multiverse built for critique may legitimately mix analyses that answer slightly different questions, because the aim is to document that researchers plausibly disagree about the question – “it may be perfectly sensible to include analyses with different estimands as long as it is plausible that researchers may try out different estimands” (Rohrer et al. 2026, 6). The same set then cannot be read as inference about one quantity, because it no longer targets one quantity. Hullman states the condition in a companion blog post: “what kinds of equivalence are needed will also depend on your goal in using a multiverse” (Hullman 2026). On this argument, a multiverse can be judged correctly built only relative to the purpose that it is meant to serve.
Measured against that standard, the artefacts of the workshop day map unevenly onto the three purposes. The fourteen-axis specification menu and the fork-importance chart serve reflection in Rohrer and colleagues’ sense, since they exist to make visible how much of the outcome depends on decisions that look arbitrary from outside. The live class chart looks like a persuasive device, but section 6 argues that it serves a different purpose. And the curve-level test designed in the inference reading belongs to the inference purpose, which is where the requirement that every analysis target the same estimand presses hardest on the course grid. One universe of specifications, built outwards from the grid of 840, serves all three roles across the day, and by the incompatibility argument this sets the roles in tension with one another. Naming that tension in front of participants is part of what the day teaches. If you can say which purpose each artefact of the workshop day serves and which building rule that purpose imposes, you can read the rest of this module, and the papers it cites, as a participant in the argument.
Take the three purposes and assign each artefact of the workshop day to one of them: the specification menu, the preregistration block of your report, the 2,520-point curve, the fork-importance bars, the class chart, the permutation test designed in the inference reading. For each assignment, write one sentence on what the universe of specifications would have to look like for that purpose to be legitimately served. Note where the same 840 analyses are being used for two different purposes.
3. The pruning critique
The most direct empirical disagreement in the recent literature is between Ganslmeier and Vlandas’s model-uncertainty study and Katrin Auspurg’s reply to it. Ganslmeier and Vlandas cross five specification dimensions at once – control set, fixed-effect structure, standard-error type, sample, and outcome operationalisation – across four political-science literatures, producing roughly 3.6 billion estimates, and find that “the sign and statistical significance of estimates is more contingent on the underlying sample and the operationalization of indicators than on the inclusion of the control set” (Ganslmeier and Vlandas 2025, 6). They are alert to the obvious objection, stating the dilemma themselves: “Running all possible models risks introducing errors in the empirical approach, but selecting a subset of robustness checks risks reintroducing the ‘researcher’s degree of freedom’” (Ganslmeier and Vlandas 2025, 2).
The title of Auspurg’s reply promises that robustness is better assessed with a few thoughtful models than with billions of regressions, but the paper itself argues for a narrower precondition, that “Multiverse analyses are only informative when all included models represent equally plausible strategies for estimating the same estimand” (Auspurg 2025, 1). Her charge is that the set is heterogeneous, in statistical quality and in what each model estimates, so that its distribution pools numbers that cannot meaningfully be compared with one another. She names four kinds of nonequivalence: statistically inferior models (“Many models do not include fixed effects or cluster-robust SE, which are standard in panel data analyses” (Auspurg 2025, 1)); unstable country subsets; causal misspecification from randomly combined covariates; and measurement nonequivalence, where “Outcomes differ in scope … and are sometimes negatively correlated. This clearly indicates that they do not reflect the same construct” (Auspurg 2025, 1). Restricting Ganslmeier and Vlandas’s universe to models she classes as justified (1,152 of them, against 91,008 unjustified) turns a 60/40 split among significant coefficients into 97% pointing the same way. She also rules out filtering by model fit, noting that “Ironically, the authors’ own fit criterion (AIC) consistently favors unjustified models” (Auspurg 2025, 1). The remedy therefore comes before estimation, in what she calls “thorough, theory-based model selection” (Auspurg 2025, 2).
The constructive version of the same demand is in Young and Cumberworth’s book – the current how-to manual for the method, and worth reading whole if this module interests you (Young and Cumberworth 2025). Their rule is estimand consistency: “When an estimand is defined a priori by a set of controls, the multiverse model set must always include those controls” (Young and Cumberworth 2025, 56), and “any ‘necessary control’ requires a clear causal diagram or definitional justification supported by prior research” (Young and Cumberworth 2025, 57). The last clause describes the order of the workshop day, where the DAG in Part 2 comes before the preregistration of Task 5 in Part 4. Their book is about robustness, and they are explicit that robustness does not identify the true effect: “there is nothing in the distribution of estimates per se that tells us which is the true effect” (Young and Cumberworth 2025, 119). The analyst still owes the reader a defended preferred estimate.
Some of Auspurg’s charge applies to the course grid, and the way in which the grid was built answers some of it. The grid is curated: it contains 840 distinct analyses, the forms that provably change nothing (centred and standardised predictors) are removed so that they do not inflate the count, and every axis comes with a note on where it came from. Deduplication, however, removes redundancy, whereas Auspurg’s objection is about heterogeneity, and a perfectly deduplicated universe can still mix models that she would prune.
Her first category names the pooled specifications, which are the cells of the course menu fitted without country and year adjustment, and her fourth describes the outcome axis, where six framing scales of different scope, some negatively correlated with the others, all stand in for ‘EU framing’. The claim-alignment convention answers the polarity half of that, reverse-coding the negative framings so that support for the claim always points the same way. But reverse-coding does not make a communitarian-framing scale the same construct as a cosmopolitanism scale. A further complication bears on her objection, and the simulation module lets you examine it directly.
The six scales are shares of one instrument, constrained to sum to one, so whatever raises one must lower the others, and the coefficients estimated on the panel itself show how unevenly that plays out. Rising unemployment moves cosmopolitan and utilitarian framing one way and communitarian framing the other, roughly three times as far, while libertarian framing barely changes; with growth as the predictor the pattern is different again. An analyst choosing mutil and one choosing mcomm are therefore estimating quite different quantities from a single instrument, before any question of construct validity arises.
This compositional arithmetic does not rescue the outcome fork from her objection, and it overlaps only in part with the point that she makes. Part of the spread across the outcome axis is the arithmetic of a compositional measure and part is the nonequivalence that she names, and separating the two is work that the curve alone will not do for you. The provenance notes on the menu, meanwhile, are disclosures, and disclosure does not supply the justification that she demands. The course grid was built to lie between full enumeration and theory-based pruning, curated enough to escape the ‘billions of regressions’ objection and heterogeneous enough that her categories of nonequivalence leave you real work to do on it, which the next two sections take up.
Pick one axis of the specification menu and write Auspurg’s case for pruning it. Which of her four nonequivalence categories applies, and what would the distribution across the grid look like if the axis were fixed at its best-justified option? Then write the case for keeping it, in Rohrer, Hullman and Gelman’s terms: which purpose does varying this axis serve? The estimator axis and the outcome axis are the two where both cases are strong.
4. The one-correct-analysis position
Lakens, Rasti and Tunç push the criticism one level deeper than Auspurg. Their target is the inference that many people draw from analytic variability, namely that since reasonable analyses disagree, there is no single correct analysis, and researchers should report distributions instead of settling on one analysis. Their reply depends on the distinction between two kinds of uncertainty, which they insist must not be merged. Aleatoric uncertainty is random variation, the kind that statistics quantifies with standard errors. Epistemic uncertainty is uncertainty about which auxiliary assumptions are true – which measure is valid, which units belong in the sample, which adjustment set closes the back-doors. On their view, epistemic uncertainty is reduced by doing research, and widening intervals only absorbs it. Once the auxiliary assumptions are fixed, “there is a fact of the matter about which analysis correctly implements that test” (Lakens et al. 2026, 7), which is the claim that gives the paper its title. Robustness across specifications of unequal justification counts for nothing, because robustness “is a consequence of valid tests, and not a goal in itself” (Lakens et al. 2026, 15). One of their warnings applies directly to teaching exercises: “Variability across specifications based on uninformed hunches, especially when less plausible alternatives are included, provides little actionable insight” (Lakens et al. 2026, 10).
It is tempting to answer them with Task 5 of the course, where you fixed a single specification in your preregistration before running it, since that sounds like their prescription. The two devices, however, solve different problems. Preregistration solves a timing problem: it rules out choosing the specification once you have seen where the numbers fall, and that answers a genuine worry. Rohrer and colleagues note that “in the absence of preregistration, researchers may as well determine what is equivalent based on whether or not it supports their result” (Rohrer et al. 2026, 5). The argument of Lakens and colleagues is about warrant, which does not depend on timing, since an arbitrary choice remains arbitrary when it is preregistered. It demands the groundwork that makes a choice non-arbitrary – validated measures, established adjustment sets, empirical evidence about the auxiliary assumptions – and the course moves in that direction only through the estimand and DAG work of Part 2, which introduces that groundwork without completing it. A participant choosing one cell from a fourteen-axis menu after ninety minutes of instruction is, on their reading, still choosing on an uninformed hunch. That limits what the exercise demonstrates without invalidating it.
Two further caveats belong here, both cutting against the device that the course itself uses. Preregistering a reanalysis of data you have already seen is itself a compromised form of it, since “[w]hen the data already exist, authors can explore the data prior to registering their analysis – making it unclear what if anything was truly preregistered” (Young and Cumberworth 2025, 149). That is why the course points you to the secondary-data template in the repositories module, which is designed for data that already exist. The two devices can also be combined: “one could preregister a multiverse analysis to combine the strengths of both approaches” (Heyman and Vanpaemel 2022, 9). Preregistering the whole universe of specifications, and only then running it, is the strongest available reply to the charge that multiverse analysis launders analytic flexibility.
5. The fork chart
The fork-importance chart of the course attributes 49.7% of the specification variance in the grid, as estimated, to the outcome axis, and under 2% to the estimator, predictor-form, co-predictor and weight axes together. On the claim-aligned scale the facilitator’s extended decomposition of the same grid inverts the ranking: the estimator fork rises to roughly 58%, the outcome fork falls to roughly 11–12%. Numbers like these invite a particular over-claim when they are put to the critics.
The over-claim would be: ‘our dispersion is mostly estimator dispersion, not estimand dispersion, so Auspurg’s diagnosis does not apply to us’. The critics would reply that the estimator fork is an estimand fork in their sense. A pooled specification with clustered standard errors estimates a blend of two associations – countries with higher unemployment framing the EU differently, and countries framing differently as their own unemployment moves over time – while a two-way fixed-effects specification isolates the second alone. Those are different quantities, about which substantive theory can reasonably differ. The statistical methods module works through this decomposition arithmetically. Auspurg’s first nonequivalence category names this axis, and on her reading a large estimator share is evidence for pruning. The claim-aligned recomputation is open to the same objection, since alignment is a display transform that removes differences in polarity but leaves differences in scope, and the caveat that the course attaches to the 49.7% figure therefore cuts both ways.
The decomposition supports a more modest reading, in which it locates where the argument has to happen. That use is endorsed by every party to the debate. Lakens and colleagues “support the use of multimodel analyses to hypothesize about ‘which analytical decisions are most consequential’” (Lakens et al. 2026, 6); Ganslmeier and Vlandas offer their method as something that “can help inform which combinations of choices are more or less warranted” (Ganslmeier and Vlandas 2025, 6); Steegen and colleagues promised pointers to the most consequential choices from the start. Read this way, the fork chart identifies which axes of the grid need a theory. A large outcome share means that you have to decide what ‘EU framing’ means, and a large estimator share on the aligned scale means that you have to decide whether your question concerns change within countries over time or differences between countries.
6. Frequency and probability
One practical rule from the critical literature concerns how to read a specification curve, and it takes the form of a prohibition on treating frequency as probability. The share of specifications supporting a claim – 68.7% in the grid of 840 – is a description of a curated set of analyses, not the probability that the claim is true. Reading it as a vote invites what Rohrer and colleagues call an illusion. Their prescription is that “variation should be treated possibilistically”, meaning that any result occurring anywhere in a justified universe is a live possibility, and that reading has its own condition: “This possibilistic interpretation is only sensible if every universe included is indeed deemed justified” (Rohrer et al. 2026, 6). The Milliways project makes the same point operationally, showing how standard displays emphasise how often an outcome occurs and thereby invite “misleading, probabilistic conclusions” (Sarma et al. 2024). Lakens and colleagues state the bluntest version: “Scientists should not compute the percentage of agreement across different data-analytic strategies, because we do not resolve scientific disagreement by majority vote” (Lakens et al. 2026, 16).
When the course quotes 68.7%, the number describes the curve that you are looking at, and the inference reading exists to police the boundary between that description and any claim about evidence. For the permutation test designed in that reading, this means that a probabilistic device, the permutation p-value, is applied to an object that the current literature says should be read possibilistically. The reading explains what the test licenses under its sharp null, the hypothesis that the effect is zero in every unit, and this module adds a warning about everything that the test does not license.
Rohrer and colleagues also observe that a multiverse persuades only under a precondition: “This can only work if the results are fairly unambiguous across specifications” (Rohrer et al. 2026, 3). At 68.7% claim-direction and 63.9% significant, the EU-frames grid fails that precondition, so the curve cannot persuade anyone that the claim is settled. A reader shown it should come away knowing which decisions determine whether the claim holds. If you ever build a multiverse to defend a result of your own, this is the precondition to check before you lean on it.
On the workshop day it is easy to read the live class chart, on which every submitted specification appears as a dot on one curve, as the persuasive set piece of the day. The critical literature describes it better as a many-analysts demonstration, the classroom analogue of the 29 teams and the 73 teams, with the extra property that every declared specification, drawn as a filled circle, was fixed in the Task 5 preregistration before it was run, while the specifications that were tried with try_spec() and submitted with submit_exploration() in Task 4 are drawn as open circles. Heyman and Vanpaemel, who run multiverse projects with students, describe the design as “comparable to the many-analysts-one-dataset approach used by Silberzahn et al. (2018)” (Heyman and Vanpaemel 2022, 7), and note that when several analysts each make their own analytical choices, “there is a multiverse of multiverse analyses” (Heyman and Vanpaemel 2022, 7). Disagreement among the analysts in a class is no verdict on robustness in either direction, but it demonstrates that the dispersion documented by the many-analysts literature is real, reproducible and, once the menu and the preregistrations are on the table, explainable. The chart therefore serves reflection, practised as a group, and this is also the answer to anyone who asks why the class chart is allowed to straddle zero.
Open the Multiverse page and find the filled circle for the specification that you declared in Task 5 (or the baseline, if you did not submit one). First read its neighbourhood possibilistically: what is the full range of aligned estimates among specifications that differ from yours on exactly one axis? Then catch yourself the moment you start counting – ‘most of my neighbours agree with me’ – and write one sentence on what that count would and would not establish, given how this universe was built.
7. The multiverse in the classroom
Every side of this debate agrees that the multiverse is useful in the classroom. Rohrer, Hullman and Gelman, in the middle of urging everyone not to take it too seriously, concede without reservation that “the multiverse has been a valuable training tool” (Rohrer et al. 2026, 8). Heyman and Vanpaemel build their whole teaching design on it, and the construction of this course follows their two rules of classroom practice. The first puts the motivation for each path before the number of paths, since “the goal should not be to merely devise as many paths as possible” and “[t]he key is that the alternatives are properly motivated” (Heyman and Vanpaemel 2022, 4). That is why the course offers a menu of justified axes.
The second rule is pastoral: “one should be cautious that students do not completely lose faith in (psychological) science” (Heyman and Vanpaemel 2022, 9). A day spent watching one published claim scatter across 2,520 estimates can leave participants cynical about research, and the day is structured to work against that. The materials of this case that are public on the OSF form a record that anyone can check, and preregistration, practised in Task 5, makes the record of your own future analyses equally checkable. Taught with that caution in mind, the multiverse shows that the decisions behind an analysis can be identified and defended, and that they belong to the analyst who made them.
Hullman gives a further reason why this literacy will matter more in future. Multiverse construction is becoming cheap, because “lots of forms of robustness checking that used to be difficult are now trivially easy” (Hullman 2026) once a generative-AI agent can build the universe from a paper. When anyone can produce a specification curve on demand, the scarce skill lies in judging what belongs in the universe, naming the purpose that it serves, and refusing the readings that its construction cannot support.
Sources and further reading
The 2025–26 debate read here: Ganslmeier and Vlandas (2025) and the reply by Auspurg (2025); Rohrer et al. (2026) with Hullman’s blog companion (Hullman 2026); and Lakens et al. (2026). The origins: Steegen et al. (2016), Gelman and Loken (2014), Simonsohn et al. (2020); a systematic framework for judging which analytic decisions are equivalent is in Del Giudice and Gangestad (2021). The many-analysts strand: Silberzahn et al. (2018), Breznau et al. (2022), Aczel et al. (2026), and, for the estimand diagnosis that Part 2 of the course builds on, Auspurg and Brüderl (2021). The book-length how-to is Young and Cumberworth (2025); the tooling is Sarma et al. (2023) and Sarma et al. (2024); the classroom design is Heyman and Vanpaemel (2022). Quotations from the two PsyArXiv preprints are cited by the pagination of the preprints themselves.