Reading a specification curve
Pre-session reading for the inference module
The multiverse in this course is built on the aggregated grid of 840 country-year specifications that forms the backdrop of the class chart (data/spec_grid.csv), together with the family universe of 1,680, which adds a beta-regression twin to each of those specifications and is described on the specification menu. The same menu logic extends to the individual respondents, where a person-level universe of 3,840 specifications, and a joint universe of 5,520 uniting the two levels, can be enumerated from the menu before any of it is fitted, and neither has been computed. This reading explains the inferential logic that governs a design like that: why a multiverse is reported descriptively, why a single formal test can nonetheless be attached to it, why such a test belongs on the aggregated universe alone, and how its result should be read. The reasoning is elementary but seldom set out in full, and the vocabulary of ‘multiverse’ and ‘specification curve’ is young enough that the two terms are routinely conflated.
Description and inference
A multiverse analysis, in the sense given to the term by Steegen and colleagues, is a descriptive exercise: identify every defensible way of turning raw data into a test of the claim, run all of them, and display the resulting distribution instead of a single hand-picked estimate (Steegen et al. 2016). Nothing in that founding formulation involves a joint hypothesis test. The founding paper reports grids of estimates and p-values and asks the reader to see how conclusions shift across the grid; its contribution is transparency about analytical contingency. The inferential layer arrived later and separately, under the name ‘specification curve analysis’, when Simonsohn, Simmons and Nelson supplemented the descriptive curve with a formal question – could a curve like this have arisen from data in which the effect does not exist? – and a resampling method for answering it (Simonsohn et al. 2020). The two layers can be used separately, and the largest robustness exercise yet conducted in the social sciences chose to use only the descriptive one. Multi100 reported the dispersion of its 504 reanalyses in explicitly exploratory terms, with no inferential statistics anywhere in the paper (Aczel et al. 2026).
This course follows that precedent for everything you have seen: the grids, the fork decomposition and the placement of the five analysts are descriptive throughout. A single permutation-based test can nonetheless supplement them – fixed in its design before it runs, and confined to the aggregated universe alone. The remainder of this reading explains each half of that sentence.
Estimates and tests
The descriptive curve invites vote counting. With 63.9% of specifications significant, against the 5% that chance alone would produce, the count can look as though it settles the question. It cannot, because the specifications are not independent of one another. All 840 results come from the same 270 country-years, which are averaged from the same 416,698 respondents and analysed 840 times in slightly different ways. If the panel happens, by sampling accident alone, to contain a negative-looking alignment between unemployment and cosmopolitan framing, then most specifications will detect that same accident together. The points on the curve rise and fall together, because every one of them is fitted to the same data. If unemployment and framing were in fact unrelated (a true null hypothesis), the share of significant specifications would therefore have no known distribution. It could be 10% or 70%, and no formula can say which, because the correlations among hundreds of overlapping specifications cannot be derived mathematically (Simonsohn et al. 2020).
Every individual cell of the multiverse still has its own conventional inference – an estimate, a standard error, a confidence interval – and each of those is as interpretable as it would be in a stand-alone paper. What is missing is a warranted statement about the curve as a whole. The gap matters because analytical flexibility is the mechanism by which false positives are produced. Simmons, Nelson and Simonsohn’s demonstration that undisclosed flexibility “allows presenting anything as significant” is the same arithmetic read in the opposite direction (Simmons et al. 2011). A multiverse inverts the disclosure problem, putting every fork on display rather than hiding it in a garden of unreported paths (Gelman and Loken 2014), but display alone does not turn a set of correlated results into a body of evidence with known error rates.
The permutation test
The solution that Simonsohn and colleagues adapted is among the oldest ideas in statistics: if the sampling distribution of a quantity cannot be derived, generate it by rearranging the data so that the null hypothesis is true by construction (Fisher 1935; Ernst 2004). Here the null hypothesis is that national unemployment and EU framing are unrelated, and the rearrangement that makes it true is a within-country permutation of the macroeconomic series. Each country keeps its own ten-year unemployment history and its own outcome data, but the assignment of unemployment values to years is shuffled – jointly with GDP growth, so that the two macro series keep their mutual correlation – and the alignment between economic conditions and framing responses is thereby destroyed. On each such rearrangement the aggregated universe of 840 Gaussian specifications is refitted, and the curve is summarised by the median partial correlation, the share of specifications significant in the claim-consistent direction, and an aggregate z following Stouffer’s rule, which converts the p-value of each specification into a z score, sums those scores and divides by the square root of their number. Five hundred rearrangements give five hundred versions of what this multiverse looks like when nothing is going on. The observed curve is set beside them, and its rank in that set is the p-value.
The construction deals with the dependence problem set out in the previous section. Every null draw refits the same 840 specifications on the same panel, so every null curve has the same correlation structure that made the observed curve impossible to test by analytical means, and the dependence is reproduced inside the reference distribution instead of being assumed away. For this reason curve-level permutation has become the standard inferential test for large specification exercises, including well-known applied examples (Orben and Przybylski 2019; Rohrer et al. 2017).
The test also has limits. The permutation destroys temporal alignment, and with it any year-to-year persistence in the predictor, so the null hypothesis that it builds in is ‘no association at any alignment’, which a sceptic could note is subtly stronger than ‘no causal effect’. Two series that trend together can lead the test to reject such a null without any causal connection between them, which is why the fixed-effects cells of the universe – which remove shocks common to every country in a given year – matter for interpretation. It is also why the result of the test is a statement about alignment beyond chance and never about causation, so the estimand questions of Part 2 still apply however small the p-value.
The level of the identifying variation
The design confines this test to the aggregated universe, and the main reason is substantive. The predictor in this study varies only at the country-year level: all 416,698 respondents observed in a given country and year share one value of the unemployment rate. All the information in the data about the alignment of unemployment with EU framing therefore comes from 270 pairings, one for each country-year cell, in every specification and at every level of analysis. Refitting the 3,840 person-level specifications under each of 500 permutations would test the same 270-cell alignment – the person-level rows are, from the point of view of the null hypothesis, an elaborate disaggregation of the same evidence – at several hundred times the computational cost, and with the added numerical problems of maximum-likelihood mixed models fitted 1.9 million times. A person-level universe would be worth building as a description. It would show how the size of standardised effects, and their conversion to a common metric, depend on the unit of analysis, a point that recurs throughout the day. It would have essentially nothing further to contribute to the question ‘is the alignment distinguishable from chance?’, because it contains no additional identifying variation with which to answer it.
This is also the reply to an objection that a careful reviewer should raise: that the original study modelled individuals, so a test confined to country-year aggregates excludes the model family that the original study itself used. The objection conflates the two layers. Model families are fully represented in the descriptive universes, where the random-effects specifications have cells of their own and their attenuated estimates can be read on their own terms. The permutation layer asks a single question about a single alignment, at the level at which that alignment is observed.
Reading the five hundred draws
The design fixes five hundred permutations in advance, and the consequences for interpretation are mechanical. A permutation p-value is computed with the observed statistic included in its own reference distribution – Phipson and Smyth’s correction, which guarantees that an estimated p can never be exactly zero (Phipson and Smyth 2010) – so with B = 500 the smallest attainable two-sided value is 2/(B + 1), roughly 0.004. Monte Carlo error, the random error that comes from using a finite number of permutations, follows the binomial rule: an estimated p near 0.05 has a standard error of about √(0.05 × 0.95/500) ≈ 0.01. Both facts bear on how a result should be read. A reported p of 0.03 or 0.08 should be treated as ‘near the conventional threshold, with one point of Monte Carlo noise either way’. The choice of B trades that noise against computation in a way that has been formalised for the bootstrap and applies here unchanged (Davidson and MacKinnon 2000). Had finer resolution been needed, a larger B would be fixed in advance. For a supplementary test whose role is calibration rather than discovery, this resolution is sufficient, and its cost, hours of computation against days for a person-level equivalent, is proportionate.
Without the test
The conclusions of the day are built so that they would lose nothing if the test were never run. The decomposition showing that the outcome fork accounts for half the specification variance while estimators, predictor forms, copredictors and weights together account for under 2%; the placement of the five Multi100 analysts at their respective percentiles of the curve; the reading of the recorded dispersion as a multiverse of different questions rather than of different analyses of one question (Auspurg and Brüderl 2021; Lundberg et al. 2021) – all of these are claims about the composition of analytical variation, and they are established by listing the specifications and decomposing their variance, with no null distribution involved. The distinction between same-conclusion and same-number robustness is likewise descriptive vocabulary (Aczel et al. 2026). The permutation test answers one narrow supplementary question: whether a curve this claim-consistent could plausibly arise when nothing is going on. Fixing that test in advance, at the level where its identifying variation is found, with a resolution stated before anything runs, and in a form that is cheap to reproduce, adds a calibration that the descriptive layers cannot supply, while the conclusions of the day rest on the layers that need no simulation to be understood (Steegen et al. 2016; Del Giudice and Gangestad 2021).