Specification Menu
The fourteen decisions, by level
The specification menu defines the space of justifiable analyses you choose from. Every option on it is defensible, but you have to write the defence yourself, and the menu is arranged so that you know what kind of defence each choice needs.
The menu mirrors the structure of Gelman’s ‘garden of forking paths’: at each decision point there is more than one reasonable choice, and choosing without reasons is what turns a multiverse into noise. The course runs the Multi100 sequence in reverse. Its analysts first chose freely and were set a constrained task months later; you start from the standardised specification and reopen the decisions one axis of the menu at a time.
The fourteen decisions are of three kinds. Some change what you are estimating, so that two people who choose differently are answering different questions and their results are not rival estimates of one quantity. Some change how a named thing is measured, leaving the question alone. Some change neither, only how the fixed quantities are fitted and how sure the answer is. A decomposition of the variance across the pre-computed universe, the combinations of options that were fitted in advance, shows how much each kind matters. Taken one at a time, the estimand decisions account for 30.8% of the variance in the estimate between them, the measurement decisions 0.9%, and the estimation decisions 0.6%, with 67.7% in the interactions between axes – the estimator decides how much every other decision matters, so on a fully crossed grid that is where much of the variance has to be. When each interaction is attributed to the level of the decisions that produce it, 85.0% of the variance lies between questions, 8.0% is added by measurement within a question, and 7.0% by the estimation. Choose level by level, and write your reasons at the level to which each choice belongs.
The estimand statement
Write the statement that your analysis will complete:
Among units in population, the association between poor economic performance, measured as X, and EU framing, measured as Y, comparing what with what.
Move one decision and reread the statement. If a bold slot other than the two “measured as” slots changes, you have made an estimand decision. If only a “measured as” slot changes, you have made a measurement decision. If nothing in the statement changes, you have made an estimation decision. The completed statement is your estimand (Lundberg et al. 2021), and it is what a reader has to know before your coefficient means anything.
The test sorts decisions relative to the statement that you wrote, and a statement written differently would sort them differently. Write the statement finely enough to name the published series behind the unemployment rate and every decision on the menu rewrites it; write it as loosely as “the economy and EU framing” and none does. The statement is written at the resolution of the claim, which names the constructs and the comparison and not the figures that stand for them. One decision lies on the boundary between Level 1 and Level 2: whether a respondent’s answers enter as a share of their own mentions or as a count of the items in the set. Both are built from the same thirteen items, which is a reason to call the choice a matter of measurement. But a share and a count remain different quantities however well the items were measured, one the weight of a framing among the frames that a person holds and the other its volume, and two people who differ on it would not accept each other’s number as an estimate of the same thing. The menu applies that criterion and places the scoring rule at Level 1, and the closing section reports what the other placement would change.
Level 1 – Estimands
| Decision | Options (code) |
What it changes | Share of variance | Crosses p = .05 |
|---|---|---|---|---|
| Frame set | cosmopolitan mcosmo · utilitarian mutil · communitarian mcomm · libertarian mlib · positive composite mpos · negative composite mneg |
EU framing: which meanings count as the outcome. The claim speaks of dimensions in the plural; Task 2 fixed the cosmopolitan scale and the claim does not. How the thirteen items group into the four scales is examined on The items behind the scales | 2.6% | 70% |
| Scoring rule | share of a respondent’s own mentions share · count of the items in the set that were mentioned count · count per item in the set rate |
EU framing: what “more of it” means. The original author (OA) and three analysts used the share, which is compositional, so mentioning anything else lowers it; two analysts used a count, which tracks how much a person says about the EU at all. The rate is the count divided by the number of items in the set: the same fit as the count for the linear family, and a proportion that the beta family can take | 0.1% | 28% |
| Predictor | unemployment unemp · GDP growth growth |
poor economic performance. Task 2 fixed unemployment; three of the five analysts chose a GDP measure in place of unemployment in Task 1 | 0.4% | 0% |
| Adjustment set | none none · the other macro measure other |
comparing what with what: holding growth fixed asks a different question from not doing so. Whether it is a confounder or a second predictor is a question for the causal graph | 0.1% | 8% |
| Sample | all 27 states 2004–2013 all · 2004–2008 y2004_2008 · 2009–2013 y2009_2013 · excluding bailout recipients no_bailout · excluding Greece and Spain no_GR_ES |
population. Conventionally run as a robustness check; it changes which countries and years the claim is about | 5.0% | 69% |
| Whose average | none none · respondents interviewed n_cell · effective sample size n_eff · adult population pop |
units. Unweighted, every country counts once; by population, every European counts once; by sample size or effective size, the weight is a precision argument that leaves the estimand alone. See Weights | 0.2% | 23% |
| Estimator | two-way fixed effects fe_twoway · country fixed effects fe_country · random effects re · pooled OLS pooled_cl |
comparing what with what: a country with itself over time, or countries with one another | 22.5% | 71% |
Level 2 – Measurement
The question stays fixed, and these decisions change how a named construct becomes a column of numbers.
| Decision | Options (code) |
What it changes | Share of variance | Crosses p = .05 |
|---|---|---|---|---|
| No-opinion handling | score 0 zero · drop drop · drop “don’t know” and blank, keep “other” at 0 split · multiply impute mi_mice, mi_amelia |
What a respondent who mentioned nothing means. Four of the five Multi100 reanalyses scored 0; OA’s code dropped them, and so did the code of KQXUE. See Missing answers | 0.0% | 6% |
| Item set | all 13 items all · without the “unemployment” item un |
OA’s own sensitivity set: an outcome item that names unemployment shares content with an unemployment predictor | 0.8% | 13% |
| Cell weighting | as interviewed unweighted · Eurobarometer W1 design weight design |
Whether the cell mean estimates the population of the country or the sample achieved there | 0.0% | 2% |
| Published series | unemployment: World Bank wb_nat · Eurostat 15–74 es_1574_act · Eurostat 20–64 as share of population es_2064_pop · labour force survey 25–64 es_2564_lfs · tipsun20 es_1574_tips ; growth: World Bank wb_growth · Eurostat es_growth |
Which figure stands for the construct. The five analysts merged five different ones. Two of the series lack some country-years, so choosing either changes the sample as well as the values | 0.1% | 17% |
Level 3 – Estimation
With the question and the measures fixed, these decisions change the statistical model fitted to them and how sure the answer is said to be. Two of the three are about the fit and can move the estimate; the third is about the inference and cannot.
| Decision | Options (code) |
What it changes | Share of variance | Crosses p = .05 |
|---|---|---|---|---|
| Predictor form | raw raw · logged log (growth: raw only) |
The fit, as a proportional or an additive effect. Centring and standardising are not separate analyses, because shifting or rescaling the predictor cannot move t, r or p | 0.0% | 6% |
| Outcome family | linear gaussian · logit-link beta beta |
The fit, and whether the model respects the bounds of the scale. Available for the share and the per-item rate at the standard measurement settings; a count is not a proportion and has no beta twin | see statistical methods | – |
| Standard errors | conventional conventional · clustered by country clustered |
The inference only. The estimate is untouched and its standard error changes. One analyst reported both on the same fit, t = −6.7 against −3.4. Its share of the variance is nil by construction, so the crossing column is the one to read | 0.6% | 20% |
Admissible combinations
The fourteen decisions cannot all be combined freely. Some options exist only given an earlier choice, and the lab tells you when you have combined two that do not go together:
- Outcome family ← scoring rule. The beta family needs a proportion strictly inside 0 and 1: the share and the per-item rate qualify, the count does not, and at the respondent level none of them would, because a quarter of respondents score exactly zero and a quarter exactly one. The beta twins are fitted at the standard measurement settings only (all 13 items, no-opinion respondents scored zero, cells as interviewed, the World Bank series).
- Standard errors ← estimator. Clustering by country is close to mandatory under pooled OLS on a country panel, a judgement about serial correlation (errors that persist from year to year within a country) under fixed effects, and a guard on the variance components under random effects. Every pairing is on the menu; the justification differs.
- Whose average ← estimator. Random effects is never fitted with regression weights, because frequency weights do not combine consistently with the estimation of its variance components.
- Predictor form ← predictor. GDP growth takes negative values and enters raw only.
- Imputed no-opinion options ← the standard settings. The two imputation engines were run for the unemployment arm at the standard measurement and estimation settings only.
The pre-computed universe
The universe is computed in tiers. The linear family with both standard-error conventions is fitted in full, every combination of every axis apart from the two imputation engines. With the other tiers added, the 1,560 files hold 461,760 rows that were fitted or imputed, or left empty where a series has no data. Multiple imputation is fitted for every estimand cell that has unemployment as the predictor and the answers scored as shares, at the standard measurement settings, with ten imputations pooled by Rubin’s rules, under two engines. The beta family is fitted for every estimand cell at the standard measurement settings too, under the share and under the per-item rate. The two expensive axes are nested rather than crossed because the beta twins reproduce the linear results cell for cell in the part of the space where both were fitted, so nothing measurable is lost by not multiplying them through everything else. A further 224,640 rows are written out rather than fitted: the per-item rate under the linear family, which is the count rescaled by the number of items in the set.
The grid is stored as one file per combination of the six Level 1 decisions other than the scoring rule, which selects rows within the file; with the scoring rule the menu asks 3,120 distinct questions, counting the per-item rate and the count that it rescales as one question. Once you have settled the Level 1 decisions you have chosen a question, and its file contains every way of measuring and estimating within it: a few hundred rows that you can read as a table.
Two of the series cover fewer country-years than the others. The tipsun20 table has no United Kingdom row and nothing before 2009 for any country but France and Sweden, and the Eurostat growth series has no United Kingdom row, so the cells that cannot be fitted under them appear in the grid as empty rows. A series with no data for half the panel changes the sample as well as the measure, and the missing answers module draws on this example.
Claim alignment
One convention applies to every submission from the class. The claim predicts a negative unemployment coefficient for the positive framings (mcosmo, mutil, mpos) and a positive one for the negative framings (mcomm, mlib, mneg). Plotted raw, the class chart would straddle zero by construction and read as noise. The convention is claim-alignment: a negative framing is reverse-coded before fitting,
d$mcomm_ca <- 1 - d$mcomm # claim-aligned communitarian framingso that ‘supports the claim’ reads as a negative sign for every outcome, and the class chart is drawn on that scale. Reverse-coding a bounded scale flips the sign of the coefficient and leaves its magnitude, standard error, |t| and p-value untouched (exactly so for the beta family too, since logit(1 − μ) = −logit(μ)), so nothing about the evidence changes, only the direction convention it is stated in. It is nonetheless a real analytic decision, so it is set by a switch in your preregistration, claim_align, which is TRUE by convention and is applied for you. If your outcome is a positive framing, there is nothing to reverse-code.
One further flip applies when your claim variable is GDP growth. The claim then predicts a positive coefficient on the positive framings, so claim-aligned reporting multiplies your b, t and r by −1 (the lab applies this for you). After both conventions, ‘supports the claim’ reads as a negative sign for every specification on the menu, whatever it measures.
Worked example
Choice: frame set mpos, scoring rule share, predictor unemp, adjustment none, sample all, whose average none, estimator fe_twoway; no-opinion drop, item set all, cell weighting unweighted, series wb_nat; form raw, family gaussian, standard errors conventional.
Justification (two sentences per level): The “positive dimensions” wording in the claim maps more directly onto the composite mpos scale than onto mcosmo alone, scored as a share of each respondent’s own mentions because the claim is about how people frame the EU rather than how much they say about it, and the comparison that the claim makes is a country with itself as its economy moves, which is what two-way fixed effects identify; nothing in the claim restricts the countries or the years, and the claim is about countries, so every cell counts once. Respondents who mentioned nothing are dropped, following OA’s handling rather than the zero used in four of the five reanalyses, because a person with no framing of the EU is not a person with zero positive framing; the World Bank series is retained because it is the one on which the panel was built. The raw percentage is kept because a proportional effect has no warrant in the claim, and conventional standard errors are reported on the judgement that, once the country effects have removed the stable differences between countries, the errors that remain do not persist from year to year within a country.
A justification of this kind makes a specification defensible, because it connects each choice to the wording of the claim or to a causal argument, at the level to which the choice belongs.
Preregistration
Fix your specification in a preregistration before you run it. That is what stops you picking the path that happens to give the answer you want. You preregister it by completing the preregistration block in your report before you add the model-fitting code, and you run the model only after that. It is a real template – the OSF “Preregistration Template from AsPredicted.org”, shortened to five fields – so the exercise mirrors what a registered secondary-data analysis actually does.
- In the workspace: the report skeleton (
index.qmd) contains the preregistration block in Task 5. Fill in your fourteen choices and their justification, and render the report before you add and run the model code. The rendered block, which carries the date of the render, is your preregistration. - On the browser lab page: there is no render step, so complete the commented preregistration block and copy it into your notes. That copy is your preregistration.
An optional mirror form may be used to log preregistrations from the class, and the link is shared in the session. It is not required. The completed block in your report is your official preregistration.
The source of the numbers
Every specification on the menu is already in the pre-computed universe, so the browser lab never fits a menu specification live. Your Level 1 choices name an estimand cell: six of them name its file and the scoring rule selects within it. run_spec() fetches that file, finds the row that your remaining choices select, and prints its b, se, t (or z for the beta family), df and claim-aligned partial r, and report_result() puts the same numbers into a pre-filled link to the class-results form. The values arrive claim-aligned: for a negative framing the coefficient, statistic and r flip sign, and for a GDP-growth predictor they flip as well. If a machine will not run the cells at all, the facilitator can open the same file under data/grid/ and read the row by hand.
Reporting a result
After running your model, submit your result to the class-results form. You do not fill it in by hand and you do not transcribe anything. report_result() takes no arguments: it reads your preregistered choices, builds the specification label itself with all fourteen decisions in it, and prints a one-click link to the form with your result already filled in, on the claim-aligned scale. Open it and press Send, and your result appears on the live Multiverse chart.
The helper standardises every submission to a partial correlation, \(r = t / \sqrt{t^2 + df}\) – the same conversion that Multi100 used – so that models on different outcome scales and estimators can all be placed on one comparable axis.
In the workspace the helper is defined in R/spec_tools.R, which the report and the working script both source; on the Browser lab page it is already loaded:
source("R/spec_tools.R")
report_result()This prints your submission link, ready to click.
Justifying each decision
The statement first
Before defending any single choice, complete the estimand statement above. The first slot is countries or Europeans, and your weighting decides it. The second is which member states and which years, and your sample decides it. The two “measured as” slots are your predictor and outcome constructs, and every Level 2 decision concerns how one of them is measured. The outcome construct is itself two decisions, which meanings count and what “more” of them means, and both rewrite the statement. The last slot is what your estimator compares. A Level 1 decision changes a bold slot other than the two “measured as” slots, and that is what separates it from the decisions on the other two levels.
Level 1 – Estimands
Frame set. The four EU-frames dimensions are built from thirteen answer options, and mpos and mneg are composites of the positive and negative pairs. All are in EUframes_cy.csv, and the codebook gives the exact definitions. Under the share rule the four scales are shares of one instrument and are constrained to sum to one, so whatever raises one lowers the others, and choosing between them is a choice of what the claim is about. The data recover the two negative scales item for item but not the split between the two positive ones, and they support the composites best of all. The evidence is set out on The items behind the scales. The negative framings need the claim-alignment step described above.
Scoring rule. A share of a person’s own mentions and a count of the items in the set give two different meanings to being “more cosmopolitan”. The share is compositional, so mentioning anything else lowers it. The count is absolute: across the country-year cells it tracks how many items respondents mention at all (a correlation of 0.85 with the mean number of items mentioned, against -0.09 for the share), and the two constructions of the cosmopolitan outcome correlate at only 0.42 across the cells. The per-item rate is the count divided by the number of items in the set, so that a two-item and a five-item scale are on one footing; for the linear family it changes nothing but the units of the coefficient, and it is the version of the count that the beta family can take. Say which of the three the claim is about.
Estimator choice. This axis accounts for 22.5% of the variance in the whole universe, because it decides which comparison you are making. Two-way fixed effects controls for time-constant country traits and year-specific EU-wide shocks, so it compares a country with itself. Country fixed effects removes country traits but leaves year shocks in the error. Random effects blends the within-country and between-country comparisons, and assumes the country-level heterogeneity is uncorrelated with the covariates. Pooled OLS ignores the panel structure and compares countries with one another. On this claim the pooled and the within-country specifications give answers of opposite sign, and no amount of robustness checking reconciles them, because they are answers to different questions.
Adjustment set. From the causal graph: is the other macro measure a confounder (a common cause of the predictor and of EU framing), a second predictor of interest, or a mediator on the path from the predictor to framing? Your answer determines whether including it is defensible for your estimand. See the causal graphs module.
Sample. Dropping Greece and Spain removes the two countries with unemployment rates far above the rest during the crisis. If the effect is driven mainly by them, the coefficient will shrink. This is conventionally called a sensitivity check, but it changes which countries the claim is about and so changes the estimand statement, which makes it an estimand decision whatever it is called. The same goes for restricting to the crisis years or the pre-crisis years.
Whose average. Unweighted, every country-year is one observation and the coefficient describes the average country; Malta counts as much as Germany. Weighted by the number of respondents or by the effective sample size (the number of respondents, discounted for the unevenness of the design weights), the weight is a precision argument and the estimand does not move. Weighted by adult population, the coefficient describes the average European, which is a different quantity. The three arguments are separated, with the numbers, in Weights.
Level 2 – Measurement
No-opinion handling. This interlocks with the scoring rule, because under the share rule a respondent who mentioned nothing has no share at all and something has to be decided about 26,325 people, while under a count their count is a genuine zero. The zero, drop and split rules and the two imputation engines are set out in Missing answers. In brief, the rule changes who is in the population as well as how they are scored, so it reaches further than its place on this level suggests, but in these data the choice hardly changes the results, which is why it stays at Level 2.
Item set. OA’s own code builds a second version of every scale without the item “unemployment”, because an outcome that counts “unemployment” among the meanings of the EU shares content with a predictor that is the unemployment rate. Under that set, a respondent whose only mention was “unemployment” has nothing left and joins the no-opinion group.
Cell weighting and published series. Both decisions barely move the estimate. The design weight changes the cell means by up to a few hundredths and the distribution by nothing measurable. Which unemployment series is merged changes the scale of the coefficient and hardly anything else, because the four full-coverage series correlate above 0.995 once country and year means are removed. The fifth series changes the sample instead, because it has no rows for half the panel.
Level 3 – Estimation
Predictor form. Log-transforming unemployment makes sense if you believe the effect is proportional rather than additive. Centring or standardising cannot change the t-statistic, because shifting a variable leaves its coefficient and standard error unchanged, and rescaling it changes both by the same factor. The justification here is technical rather than substantive, as it is for every decision on this level.
Outcome family. Beta regression is a defensible fork because the share-built and rate-built outcomes are bounded 0–1 proportions and the country-year cell means lie strictly inside the interval, so a logit-link beta model respects those bounds where a linear model does not. A count-built outcome is a number of items, not a proportion, and has no beta twin: under the beta family, choose the rate if you want the construct that the count measures. Beta fits report a z statistic rather than a t; report_result() handles both, and the partial-correlation conversion is unchanged. The theory is in the statistical methods module.
Standard errors. Clustering by country changes the standard error and nothing about the estimate, which makes it the one decision on the menu that is about inference rather than the answer. It is on the menu because one of the five analysts computed both on the same fit and the two disagreed by a factor of two, and because it shows that an estimation decision can decide whether a result is reportable while leaving the estimate unchanged. Its share of the variance is nil, and moving it alone crosses the threshold in 20% of cells.
A decision can account for almost none of the variance in the distribution and still decide whether your result clears p = 0.05, because significance is a property of one cell and variance is a property of the whole space. In the pre-computed universe the predictor form accounts for 0.0% of the variance and crosses the threshold in 6% of cells. Pin the question on all seven Level 1 decisions and leave the other seven free, and 1,600 of the 3,120 questions still include both a significant and a non-significant result among their 240 variants; count the scoring rule as a measurement decision instead, so that it too is free, and 1,121 of 1,560 cells do among 480 variants. The difference between the two proportions shows how much depends on where the scoring rule is placed. Pin only the outcome and the estimator, the two decisions that every methods section states, and 24 of 24 blocks straddle the threshold, which is why you fix your specification in the preregistration before you run it.