Enter the multiverse

A visual overview of 2,520 justifiable data processing and statistical analysis pipelines

The specification curve below contains results from 2,520 justifiable analyses of our dataset to test the central claim that “poor economic performances … decrease positive dimensions of EU framing” (Teney 2016, p. 619).

The negative framings (mcomm, mlib, mneg) are reverse-coded (1 − y) before plotting, to align the direction of their meaning to those of the positive framings (mcosmo, mutil, mpos). This is the same convention expected in the preregistration exercise (claim_align). Because equally justifiable analyses can rely on different outcome scales and predictor forms, raw coefficients are not comparable across specifications. The chart therefore plots partial correlations \(r\) (calculated as \(r=t/\sqrt{t^2+df}\)) to make the results comparable. Thus, a negative partial \(r\) value supports the claim, while a positive one goes against it. The horizontal line at zero is the boundary between support for the claim and evidence against it.

On the claim-aligned scale every supportive specification sits below zero: 69.8% of the family universe of 1,680 (68.7% of the gaussian half, 71.0% of the beta half), with 63.5% significant at the 5% level overall. The two family curves all but coincide, and in the cosmopolitan universe (outcome mcosmo) the wrong-sign pattern is identical in each family. The same 33 pooled and 8 two-way specifications run against the claim whichever likelihood is used. The beta family sits on the menu beyond the constraints set by Multi100. The specification menu describes both the extra axis and the claim-alignment convention, and the module on statistical methods explains the beta model itself. Across the 840 growth-predictor specifications the direction agrees with the claim just as often (70.5%), but only 46.9% are significant – the two halves of OA’s two-indicator construct are not equally strong evidence, exactly as analyst KQXUE’s free-phase growth null hinted.

Interpreting the curve

Each point is one specification. The points are ranked by their claim-aligned partial r, so the curve runs from the strongest support for the claim on the left to the strongest evidence against it on the right. The 2,520 specifications are a subset of the specification menu, all fitted at its standard measurement settings; the estimand choices page draws the full menu one question at a time.

Blue points use the linear (gaussian) model and green points its beta-regression twin. Triangles have unemployment as the predictor and squares GDP growth, whose results are sign-flipped as well, so that a point below zero supports the claim whatever its predictor. The blue triangles are the 840 specifications of the Multi100 menu, and the beta twins and the growth predictor are course additions to it. Each family is ranked separately, and faded points are not significant at 5%. Name tags mark the five Multi100 analysts, with C6HJR’s gold tag on the baseline that you reproduce in Task 3, and a red tag marks the published coefficient of the original author (OA). Hover over any mark for its details.

Section 9 of Getting started with R builds a curve like this one from the data. Section 6 of the multiverse module explains why the share of points below zero describes this set of analyses and is not the probability that the claim is true, and the inference reading sets out what a test on the whole curve can add.

Class results

The live class chart below is where your own submitted results are shown on the specification curve. If you have submitted your results via the form, pressing Run Code below will run the refresh_multiverse() function that updates the plot with the latest recorded submission data.

Submissions from the class are displayed as black circles, numbered in order of arrival. An open circle shows an explored specification, one that you tried with try_spec() in Task 4 and submitted after seeing its result. A filled circle shows a declared specification, the one that you fixed in your preregistration in Task 5 before running it. Each submission also carries the tag that submit_exploration() and report_result() print, such as amber-otter-42, and your two circles share it. Hovering over a circle shows its tag and the specification behind it. Typing your tag into the my_tag line of the cell below and pressing Run Code draws your circles in orange, joined in the order in which you submitted them.