The case at the respondent level, and the second claim of the paper
Most pages on this site work at the country-year level: 270 rows, one per country and year, which is the level of the course exercise. The study on which the exercise is based worked at a different level. Its central result comes from a model fitted to individual survey respondents, and some patterns in data on individuals cannot be detected in country-year averages.
This page fits that model. It is optional, it is not part of the workshop day, and it exists so that the individual-level version of the argument can be run and inspected. It runs on two datasets, and section 2 says which is which: the real respondents, for anyone who has built the respondent file from their own GESIS downloads, and a synthetic twin of them, which ships with the course and on which every code chunk on this page runs. The closing section applies the same data to the second claim of the paper, about the education gap, which is by its nature a claim about individuals inside countries. The original author is referred to throughout as OA, following the convention of the rest of the site.
1. The level of analysis
The claim is about countries: when a national economy performs badly, do the people in it frame the EU less positively? That is a statement about country-years, and the course panel is built to test it directly.
But the same words license a second reading. Perhaps unemployed people frame the EU differently, so a country with more of them shows a lower average without anything having changed for anyone else. The first reading is contextual, in that living through high unemployment changes how everyone frames the EU. The second is compositional, in that the mix of people changes while each person frames the EU as before.
A country-year panel cannot separate them, because it contains no people. Individual data can, and this page does it.
2. The two datasets
The Eurobarometer microdata behind the case are GESIS-licensed and cannot be redistributed, so this page cannot ship the real respondents. They can still be obtained: the sixteen survey files can be downloaded with a free GESIS account, and the build script in the workspace turns them into one respondent file, data_private/eb_respondents.rds, whose codebook is published even though the file is not. The data sources module explains the download, and the data management module the build. Every number on this page labelled ‘real’ was computed on that file by a script in the course repository, and a reader who holds the file can rerun the script. The code chunks shown on the page run on the twin, so that the page renders for everyone, and swapping the twin for the respondent file in the setup chunk is a one-line change.
What the course ships is a synthetic twin: EUframes_person_full.csv, 41,660 simulated respondents across the real 270 country-years, with every variable that OA’s published regressions use. No real respondent appears in it. The simulation module explains how it is built. In short, a respondent’s characteristics come from a correlation structure estimated on the real file, and their answers from a model of the survey instrument conditional on those characteristics.
What licenses a number from the twin is the validation, not the care taken in building it: the same model was fitted to the real respondents and to the twin, and the two sets of coefficients were compared.
Table 1: Every coefficient of OA’s published model, fitted on 416,698 real respondents and on the twin. The last column is the difference over the two-sample standard error.
valid |>summarise(`coefficients compared`=n(),`within 1 standard error`=sum(abs(z) <1),`within 2 standard errors`=sum(abs(z) <2),`largest discrepancy`=round(max(abs(z)), 2))
# A tibble: 1 × 4
`coefficients compared` `within 1 standard error` `within 2 standard errors`
<int> <int> <int>
1 64 46 60
# ℹ 1 more variable: `largest discrepancy` <dbl>
60 of 64 agree within two standard errors, where roughly three would be expected to fall outside by chance alone. The comparison is stored in _model-outputs/twin_validation.csv and can be read line by line.
Two cautions apply to that result. The yardstick above is the two-sample standard error, which asks whether a difference exceeds the sampling noise of the twin itself – the question that a reader quoting one of its numbers has. Judged instead against the precision that 416,698 respondents afford, only 39 agree, because the twin holds a tenth as many people and cannot stand in at full precision. And the model validated is OA’s specification with its cross-level interactions removed. The twin reproduces that model, but it is not validated for the interacted form and should not be quoted for it.
3. The published model
fit <-function(y) { f <-reformulate(c("female", "mage", "edu3", "ses", "urban","mgrowth", "munemp", "(1 | cntry)", "(1 | cy)"),response = y)lmer(f, data = twin, REML =FALSE)}m_cosmo <-fit("cosmo")parameters::model_parameters(m_cosmo, effects ="fixed",ci_method ="satterthwaite")
Uncertainty intervals (equal-tailed) and p-values (two-tailed) computed
using a Wald t-distribution with Satterthwaite approximation.
Among the macro terms, a one-point rise in a country’s unemployment rate goes with a fall of about 0.0054 in an individual’s cosmopolitan framing score, net of who they are.
The term sesunemployed is the respondent’s own employment status. Being unemployed rather than a manual worker moves the score by -0.028 – roughly 5 times as much per unit. Individual circumstance matters far more to an individual than the national rate does.
The comparison is easy to misread, and the next section works through the arithmetic.
4. Individual effects and aggregate associations
The two coefficients above apply to different units, so they are not rival explanations of one thing. The individual effect applies to whoever is unemployed, and the contextual effect applies to everyone. The question is how much of the country-level association the individual effect could account for.
That depends on a quantity that neither coefficient contains: how much a country’s unemployment rate changes who is in the sample. Measured on the real respondents, a one-point rise in the published rate adds 0.0074 to the share of respondents who are themselves unemployed, which is well under one percentage point.
b_ind <-fixef(m_cosmo)[["sesunemployed"]]b_ctx <-fixef(m_cosmo)[["munemp"]]grad_w <- grad$gradient[2]tibble(quantity =c("individual effect of being unemployed","extra unemployed share per point of the national rate","compositional contribution to the country-level slope","the country-level slope itself","compositional share of it"),value =c(b_ind, grad_w, b_ind * grad_w, b_ctx, b_ind * grad_w / b_ctx)) |>mutate(value =signif(value, 3))
# A tibble: 5 × 2
quantity value
<chr> <dbl>
1 individual effect of being unemployed -0.0278
2 extra unemployed share per point of the national rate 0.00743
3 compositional contribution to the country-level slope -0.000207
4 the country-level slope itself -0.00544
5 compositional share of it 0.038
So the compositional path accounts for roughly 4% of the country-level association, because an effect several times the size, spread over a group that barely grows, moves the average hardly at all.
That figure is computed from the coefficients of the twin itself, and the unemployed contrast is where the twin performs worst: it comes back at -0.0278 against -0.0350 on the real respondents, the largest single discrepancy in the validation table. Using the real value the compositional share is 4% rather than 4%. Both are small, so the conclusion does not depend on which you use, even though the twin is visibly imperfect on this coefficient.
This calculation should be asked for whenever an aggregate association is explained by ‘it must be the people who changed’. It is rarely done, and when it is done the compositional explanation usually does not hold up. It needs an individual-level effect and a prevalence gradient, meaning the amount by which the share of people in the group rises as the aggregate predictor rises. The second is usually assumed, although it can be measured.
5. Two designs
The course panel and OA’s own model are both defensible readings of the same claim, and they do not agree.
Table 2: The unemployment coefficient on cosmopolitan framing under two designs. The aggregate model is C6HJR’s published Multi100 specification; the individual model is OA’s published one, fitted on the twin.
agg <-lm(mcosmo ~ unemp + growth +factor(year) +factor(cntry), data = panel)tibble(design =c("country-year panel, two-way fixed effects","individual, country and country-year random intercepts"),n =c(nrow(panel), nrow(twin)),estimate =c(coef(agg)[["unemp"]], fixef(m_cosmo)[["munemp"]]),removes_common_year_shocks =c("yes", "no")) |>mutate(estimate =signif(estimate, 3))
# A tibble: 2 × 4
design n estimate removes_common_year_…¹
<chr> <int> <dbl> <chr>
1 country-year panel, two-way fixed effec… 270 -0.00356 yes
2 individual, country and country-year ra… 41660 -0.00544 no
# ℹ abbreviated name: ¹removes_common_year_shocks
The individual-level estimate is about half again as large, and the reason is in the last column. OA’s design includes country and country-wave random intercepts but no year effects. Nothing in it removes a shock that hit every country at once, and 2009 brought such a shock: unemployment rose everywhere and EU framing moved everywhere, for reasons that include but are not limited to unemployment. The two-way fixed-effects design removes that shock and OA’s design does not, so part of the shock ends up in her estimate.
Neither estimate is wrong, because the two designs address different questions, which is the central argument of the course applied at the individual level. The difference between them is larger than the difference made by most of the choices on the specification menu.
The disagreement between the designs is starker on the libertarian dimension:
On the panel, unemployment has almost no association with libertarian framing, while at individual level the association is clear. When an effect appears under one defensible design and not under another, that is a strong argument for fixing the design in a preregistration before any model is fitted.
6. Levels and evidence
The multilevel structure matters here for the standard error, and for what the model counts as information.
Table 3: The unemployment coefficient under three random-effects structures, fitted on the twin. The real-data values are read from a committed table produced by the same models on the 416,698 licensed respondents.
ml <-list("country only"= cosmo ~ unemp + (1| cntry),"country and country-year"= cosmo ~ unemp + (1| cntry) + (1| cy),"with a random slope"= cosmo ~ unemp + (1+ unemp | cntry) + (1| cy))keys <-c("two_level", "three_level", "slopes")imap(ml, \(f, nm) { m <-lmer(f, data = twin) p <-coef(summary(m))["unemp", ] k <- keys[match(nm, names(ml))]tibble(structure = nm,twin_estimate = p[["Estimate"]], twin_t = p[["t value"]],twin_df = p[["df"]],real_estimate =rv(k, "unemp"), real_df =rv(k, "unemp", "df"))}) |>list_rbind() |>mutate(across(where(is.numeric), \(x) signif(x, 4)))
# A tibble: 3 × 6
structure twin_estimate twin_t twin_df real_estimate real_df
<chr> <dbl> <dbl> <dbl> <dbl> <dbl>
1 country only -0.00534 -8.99 28910 -0.00479 391200
2 country and country-year -0.00564 -5.24 251. -0.00524 253.
3 with a random slope -0.00601 -2.95 15.4 -0.00571 18.9
In the first row, declaring only the country level lets the model treat every respondent as fresh information about unemployment, and the degrees of freedom run into the tens of thousands on the twin and into the hundreds of thousands on the real file. But the predictor takes only 270 distinct values in the whole dataset. A model that reports six-figure degrees of freedom for a variable with 270 distinct values is counting people as though they were countries.
Once the country-year level is declared, the estimate barely moves, but the degrees of freedom drop to the order of the number of cells. That is the correct count, and the t falls accordingly.
The variance components below lead to the same conclusion.
Table 4: Variance components, on the twin and on the real respondents. Standard deviations, from the country and country-year model.
# A tibble: 3 × 3
component twin real
<chr> <dbl> <dbl>
1 between countries 0.0646 0.0613
2 between country-years 0.0446 0.0385
3 between people 0.356 0.368
Variation between people accounts for far more than either of the other components, around 95% of the total. The share reflects the shape of the outcome at this level. A respondent who mentions two things and finds one of them cosmopolitan scores exactly one half. The score is bounded, takes only a few values, and its scatter within any cell is enormous compared with the differences between cells. That is also why so many respondents add so little precision to the estimate for a country-level predictor.
The country-year component in the twin is about a sixth larger than the real one, because the cells of the twin hold a tenth as many respondents, so their means are noisier, and some of that sampling noise is attributed to the cell level rather than to people. The country and residual components agree with the real ones to within a few per cent.
Whenever the predictor is measured at a higher level, a person-level sample size suggests a precision that the design does not have. Two of the five Multi100 analysts fitted person-level models, and this is a large part of why their strengths of evidence differ so much from those of the other three.
7. The limits of the dataset
The twin will support the specification validated above and things close to it: the four framing dimensions, the published covariate set, country and country-year random structures, and the composition-against-context arithmetic. Coefficients from those may be quoted with the validation table beside them.
It will not support three things. Anything at the survey-wave level, because the twin is built on 270 country-years and its finest cluster is the country-year, while the real file has sixteen waves across ten years. The interacted form of the published model, which was excluded from the validation because a synthesis built to reproduce a covariance structure does not reproduce a cross-level interaction unless one is put into it. And analyses of individual framing items, since the twin reproduces the average salience of each item but not its country-by-country variation. That limit does not affect any model using the four dimension scores, which covers every model on this page.
Its weight is not a survey weight.w1 reverses an over-sampling of younger respondents that the generator introduces so that weighting has something to correct. It is a teaching device with a known answer and should not be read as the Eurobarometer design weight.
TipTry it
Fit the model of section 3 again with ses removed, and watch what happens to munemp. It should barely move, and the reason is section 4: the compositional path is a few per cent of the total, so deleting the variable through which it runs changes little.
Then do the same with mgrowth removed, and with the country-year random intercept removed. One of those three matters far more than the others; find out which, on data whose answer you can check.
8. Claim 2 – the education gap
The second hypothesis of the paper is about people inside countries, and it cannot be tested on the 270-row panel at all. The abstract states the finding in one sentence, which serves as the claim card for a second replication exercise in the style of the first:
“…high unemployment rates affect the EU framing of high-educated citizens to a larger extent…” (Teney 2016, p. 619), or, in other words from the same abstract, “high unemployment rates are associated with a smaller gap between high- and low-educated citizens in the way they frame the EU”.
Unemployment × high education → cosmopolitan EU framing: b = −0.0031, t = −6.93, p < .001 (Table 3, Model 1), beside an education gap of b = 0.0771 (t = 41.79) and an unemployment slope of b = −0.0034 (t = −4.03) at the lowest education band.
The estimand statement, in the form that the specification menu uses: among respondents in the 27 member states surveyed between 2004 and 2013, the difference between high- and low-educated respondents in the association between the national unemployment rate, measured as the World Bank series, and cosmopolitan EU framing, built as the share of a respondent’s mentions, comparing each country with itself over time. The quantity is an interaction, the amount by which the association with unemployment differs between education bands, and that is what makes the claim individual-level. Education is a property of respondents, unemployment a property of country-years, and the question is whether people who differ on the first respond differently to the second.
Three ways to estimate it. The published model is a three-level random-intercept model with the interaction terms and the full covariate set, fitted on the respondents. The same interaction can be fitted with country and year fixed effects instead of random intercepts. Those effects remove the average level of framing in each country and the shocks common to all countries in a year. They do not absorb the national unemployment rate, which moves within each country from one year to the next, so both its slope and its difference between bands remain estimable. Because the three bands of a country share one country effect, the difference between bands draws on differences between countries in the size of the education gap as well as on movement within each country. And the interaction can be fitted on cells rather than respondents. The education-band panel holds one row per country, year and band, and a fixed-effects regression on its 810 rows, weighted by the respondents behind each mean, addresses the same question with the same data at the aggregate level. The first two were run on the real respondents. The third runs here, on public data, in a few lines:
Table 5: The education gap and its interaction with unemployment on the cosmopolitan scale, under three designs and two rules for the respondents who named nothing. The published model is fitted on the real respondents with OA’s covariates and random intercepts for countries and country-waves; the fixed-effects model on the same respondents with errors clustered by country-year; the cell model on the education-band panel, which is built under the zero rule. Coefficient, with t in brackets. The first row is OA’s Table 3, Model 1.
design
rule
unemployment
education: high
mid x unemployment
high x unemployment
OA, Table 3 Model 1
no-opinion dropped
−0.0034 (−4.03)
0.0771 (41.79)
−0.0014 (−3.23)
−0.0031 (−6.93)
fixed effects, education-band cells
no-opinion scored zero
−0.0036 (−3.31)
0.1637 (18.01)
0.0016 (1.47)
0.0001 (0.10)
fixed effects, respondents
no-opinion dropped
−0.0030 (−3.45)
0.0943 (27.56)
0.0003 (0.48)
−0.0014 (−1.75)
fixed effects, respondents
no-opinion scored zero
−0.0033 (−4.25)
0.1174 (35.61)
0.0016 (2.04)
−0.0001 (−0.07)
published model, respondents
no-opinion dropped
−0.0044 (−5.74)
0.0933 (50.01)
−0.0011 (−2.77)
−0.0029 (−6.82)
published model, respondents
no-opinion scored zero
−0.0051 (−6.60)
0.1162 (62.81)
0.0004 (0.95)
−0.0014 (−3.23)
The published design under the published rule reproduces OA’s interaction almost exactly, −0.0029 against her −0.0031, with a t of −6.82 against her −6.93. The other terms of the row are further apart, the education gap 0.0933 against her 0.0771 and the unemployment slope −0.0044 against −0.0034, and the change from her Eurostat unemployment series to the World Bank series used in the course does not account for the gap: on the education-band panel, swapping one unemployment series for another moves the education gap by less than 0.0001. The refit also keeps 382,335 respondents, where the note to her Table 3 reports 335,566. That row is the reproduction, and it is the one to quote. The next row changes one thing, the rule for the respondents who named no item at all: OA’s code dropped them, because dividing by a mentions total of zero left them missing, and the course scores them zero on every scale, as the Multi100 reanalyses did. The interaction halves. The respondents who name nothing are disproportionately low-educated, and scoring them zero pulls down the cosmopolitan mean of the low band in every cell in a way that does not move with unemployment, so the gap between the bands and its movement both change. A decision that the missing-answers module classifies as a measurement choice therefore moves the coefficient on which the claim rests.
The fixed-effects rows change the design instead. The fixed-effects design estimates a slope for unemployment as well as its difference between bands, because the national rate moves within each country from year to year and the country and year effects do not absorb it. Under the published rule the slope is −0.0030 (t = −3.45), against −0.0044 (t = −5.74) in the published design. The interaction under the published rule halves again with fixed effects, from −0.0029 to −0.0014, and loses conventional significance, and it falls to zero under the zero rule. The cell row, fitted on 810 aggregates that anyone can download, agrees with the respondent-level fixed-effects row under the same rule on the unemployment slope and on both interactions, which confirms that for the interaction the aggregate route addresses the same question as the respondent route. The two rows differ on the education gap, 0.1637 in the cells against 0.1174 among the respondents, and the difference comes mainly from the covariates: the respondent model holds sex, age, occupation and size of community constant, while the cell model holds none of them. Taken as a whole, the table shows the claim holding under the design and the rule that OA used, and weakening step by step as the fixed-effects design replaces the published one and as the respondents who named nothing are kept rather than dropped. The unemployment slope repeats the finding of section 5: OA’s design includes no year effects, and unemployment and framing moved everywhere at once in 2009. The interaction does not follow that reasoning, because year effects that the three bands share cannot remove a shock in which the bands moved differently. The fixed-effects rows also drop the random intercepts for country-waves, and on the 810 cells that is the change which moves the interaction: a random-intercept model with intercepts for countries and country-years gives about −0.0010 with or without year effects, and about zero once the country-year intercepts are removed.
The results reported in the paper include two qualifications that a replication should keep in view. The interaction holds for the cosmopolitan, communitarian and libertarian scales and not for the utilitarian one, where “unemployment rate affects to the same extent the level of utilitarian framing of low-, middle-, and high-educated citizens” (p. 628). And the second sentence of the hypothesis, that the gap would move more on the materialist scales, is contradicted by the table in the paper itself: “in contrast to my hypothesis, the interactions between education and unemployment rate are no larger for the materialist dimensions of EU framing” (p. 628). The measurement module explains why that comparison presupposes a grouping that the data only half support. The other three outcomes are in the committed table for anyone who wants to check the first qualification against the real respondents:
Table 6: The high × unemployment interaction on all four scales: OA’s Table 3, Model 1, and the published model refitted on the real respondents with the no-opinion respondents dropped, as her code dropped them. Coefficient, with t in brackets.
outcome
published
refitted
cosmo
−0.0031 (−6.93)
−0.0029 (−6.82)
util
−0.0004 (−1.73)
−0.0002 (−0.82)
comm
0.0020 (5.63)
0.0016 (4.81)
lib
0.0015 (4.84)
0.0015 (4.99)
A replication exercise of your own. The pieces for a second exercise in the style of the first are all on this page: a claim card, an estimand statement, a public dataset on which the aggregate design runs in the browser lab or the workspace, the respondent file for anyone who built it, and OA’s published values to reproduce. The menu is smaller than the one for the first claim, and each of its axes is a decision that you can defend or not. The axes are the frame set, the scoring rule, the rule for the respondents who named nothing, the education coding (the published three bands, or the four bands that keep students separate), the estimator (random intercepts, fixed effects, or cells), the sample, the weight, and whether growth enters beside unemployment. Fix one specification in a preregistration, as in Task 5, then run it and set the interaction you get against the rows of the table above. The table already shows that two of those decisions, the design and the no-opinion rule, decide whether the claim is found.
TipTry it
Refit m_edu above with edu3 * munemp replaced by edu3 * mgrowth alone, and read the growth interactions against the claim in the paper that “GDP growth impacts mainly on the EU framing of low-educated citizens” (p. 619). Then refit the cosmopolitan model with the utilitarian scale as the outcome and read the interaction that the paper found absent.
Sources
The specification is OA’s published Table 3 Model 1 with its cross-level interactions dropped, taken from the author-provided Stata code in the SCORE dossier (Teney 2016). The twin is generated by companion/_model-outputs/simulate_person_joint.R from the public country-year panel and committed calibration files. The calibration was estimated once, locally, from a person-level rebuild of the sixteen GESIS waves that is never committed and never redistributed – the data sources module explains that regime. The validation table comes from validate_twin.R, the real-data multilevel reference from fit_person_multilevel.R, and the prevalence gradient from fit_person_anchor.R, all in the same folder. The models in the code on this page are fitted when it is built, on the twin, the panel and the panel by education band, all of which ship with the site. The real-data values beside them are read from the saved outputs of those scripts and of fit_claim2.R, which fits the second claim on the respondents.
References
Teney, Céline. 2016. “Does the EU Economic Crisis Undermine Subjective Europeanization? Assessing the Dynamics of Citizens’ EU Framing Between 2004 and 2013.”European Sociological Review 32 (5): 619–33. https://doi.org/10.1093/esr/jcw008.