Weights

Design weights, precision weights and population weights

The specification menu has a regression weight axis with four settings and a cell weight axis with two, and between them the two axes cover three different decisions. The three are made at two different levels of the analysis and are justified by different arguments, and only one of them changes what the analysis is estimating. When all three go under the one word ‘weighting’, a change of question can be reported as though it were a technical robustness check.

This module takes the three decisions apart, using the course panel throughout. It is best read alongside the causal graphs module and the section of the specification menu on what each decision commits you to.

The three decisions

Table 1: The three decisions that are grouped under the word ‘weight’ in this analysis.

	Level	The argument for it	Does it change the estimand?
Design weighting	inside a country-year cell	The achieved sample is not a miniature of the country, so weight the respondents before averaging them	No. It makes the cell mean a better estimate of the same quantity
Precision weighting	across cells	A cell mean built on more interviews is measured more precisely, so let it count for more	No. It is an efficiency argument about estimating the same slope
Population weighting	across cells	The claim is about Europeans, not about countries, so a country should count in proportion to how many people live in it	Yes. The slope now describes a different average

The first is a measurement decision, because it changes how well the number in the cell estimates the thing it names. The second is an estimation decision, which changes the efficiency of the fit and leaves the target alone. The third is an estimand decision: it changes whose average the coefficient describes, and two analysts who resolve it differently are answering different questions.

Design weights within the cell

Every Eurobarometer wave is released with design weights, because the people who complete an interview never quite match the population. Some groups are easier to reach than others, some refuse more often, and the achieved sample tilts accordingly. The W1 national weight reverses that tilt so that the weighted sample matches the known distribution of the national population by sex, age, region and size of locality.

Applying it changes the country-year means by a small but not negligible amount. Across the 270 cells, the largest change to the cosmopolitan scale is 0.0433 on a 0–1 scale whose mean is about 0.51, and the weighted and unweighted cell means correlate at 0.9939.

Choosing among Eurobarometer weights

A Eurobarometer file contains several weight variables, and they are not interchangeable. The national weight (W1) makes the sample from each country representative of that country. The European weights (W5 onwards, “WEIGHT EU6” to “WEIGHT EU27” and their relatives) additionally scale each country by its population, so that pooling all respondents together gives an estimate for the EU population as a whole. An extrapolation factor (WEX) grosses the sample up to population counts.

Choosing the wrong one changes the question being answered, and no warning appears in the output. Using a European weight and then averaging countries double-counts population size; using a national weight and then pooling respondents across countries gives you an average over respondents, which is an artefact of how many interviews each country happened to get. Whenever a survey provides more than one weight, read the codebook before choosing, because the choice among them is, in effect, the choice between a country-level and a person-level estimand.

Effective sample size

Weighting improves representativeness at a cost in precision. When respondents have unequal weights, the weighted mean is less precise than an unweighted mean of the same number of interviews, because a few heavily weighted cases contribute a disproportionate share of the estimate. Kish’s effective sample size is the standard way of saying how much precision is left:

\[n_\text{eff} = \frac{\left(\sum_i w_i\right)^2}{\sum_i w_i^2}\]

For a cell where every weight is equal, \(n_\text{eff} = n\). As the weights spread out, it falls. In this panel the median cell has \(n = 1540\) interviews and \(n_\text{eff} = 1234\), a design effect (the ratio of \(n\) to \(n_\text{eff}\)) of 1.09; the worst cell loses 41.3% of its sample to the weighting.

The practical consequence is that if you are going to weight cells by their precision, \(n_\text{eff}\) is the right variable and \(n\) is an approximation to it.

Regression weights across cells

Once you have 270 country-year means, you have to decide how much each one counts in the regression. There are four defensible answers, and each makes a different claim.

Table 2

Setting	What it weights by	The claim it makes	Kind
`none`	nothing; every cell counts once	The claim is about countries. Malta and Germany are two observations	estimand: the average country
`n_cell`	respondents interviewed	Cells measured on more interviews are more reliable evidence	precision
`n_eff`	effective sample size	The same argument, using the precision that the weighting actually leaves	precision
`pop`	adult population of the country that year	The claim is about Europeans. A German counts as much as a Maltese	estimand: the average European

: The four regression weights available in spec_grid_weights.csv. {#tbl-candidates}

Sample size and population size

It is tempting to treat the respondent count (n_cy in the panel, n_cell on the menu) as a size variable, on the reasoning that bigger countries get bigger samples. They do not, or not by much. Eurobarometer fields roughly a thousand interviews in each member state whatever its size, with smaller allocations for the smallest states and a larger one for Germany. So across this panel:

the respondent count runs from 500 to 3614, a ratio of 7.2 to 1;
the adult population runs from 329,409 to 70,761,158, a ratio of 215 to 1;
and the two correlate at only 0.35 across the 270 cells.

Weighting by n_cell therefore does not make the analysis representative of the European population. It gives better-measured cells more say in the fit, which is a different and much weaker claim. If you want the average European, you have to weight by population and state that you have done so.

The published specification under each weight

The table below shows C6HJR’s published Multi100 specification (cosmopolitan framing, two-way fixed effects, all years, raw unemployment, no co-predictor) fitted under all eight combinations of the two ways of building the cell means and the four regression weights.

Table 3

Cell mean	Regression weight	b	t	p	r
unweighted	none	−0.00347	−3.80	0.000182	−0.242
unweighted	n_cell	−0.00263	−3.04	0.00264	−0.195
unweighted	n_eff	−0.00269	−3.11	0.00214	−0.199
unweighted	pop	−0.003	−4.74	3.8e−06	−0.296
design-weighted	none	−0.00379	−4.12	5.32e−05	−0.260
design-weighted	n_cell	−0.00292	−3.33	0.00101	−0.213
design-weighted	n_eff	−0.00297	−3.40	0.000804	−0.217
design-weighted	pop	−0.00331	−5.24	3.54e−07	−0.325

: C6HJR’s published Multi100 coefficient under every weighting choice. The first row is the published Multi100 result. {#tbl-c6hjr}

Every row supports the claim and every row is significant, so no weighting choice flips the sign or removes the result. The size of the association does change, with the partial correlation running from −0.325 to −0.195, a spread of about two-thirds of the smaller value, across eight versions of a specification that any methods section would describe in the same sentence.

The two kinds of weight pull in opposite directions. Precision weighting by the respondent count weakens the result, from t = −3.80 to t = −3.04. Population weighting strengthens it, to t = −4.74. An analyst who reasons “I should weight, because weighting is more careful” without saying which argument they are making could end up with either.

Kish’s correction changes almost nothing, with n_eff and n_cell giving t = −3.11 and −3.04. \(n_\text{eff}\) is the correct precision weight, but the design effects in this panel are small enough that choosing it over n_cell does not change the conclusion.

The denominator

When asked what the EU means to them, 26,325 respondents, 6.3% of the sample, mentioned nothing at all. How they are scored is a decision that no weight can change. The code of the original paper dropped those respondents, as the missing answers module explains, and so did one of the five Multi100 reanalyses. The other four, among them the one that produced the recorded t of −3.804, kept them at zero, and the course panel follows them, scoring those respondents 0 on all four framing scales, so a person with no opinion enters the data as a person with no cosmopolitan framing.

That would be a defensible convention if the no-opinion rate were the same everywhere, but it differs widely between countries.

Table 4

Country	No opinion
ES	16.0%
GB	14.9%
PT	12.7%
IT	8.5%
LU	2.5%
AT	2.3%
SE	2.0%
CY	1.9%

: The four highest and four lowest no-opinion rates. {#tbl-noop}

An eight-fold difference in the rate of a response that is coded as a substantive zero is a measurement problem that no weight will fix, because the weights correct the composition of the respondents and leave the scoring of their answers untouched. If you think the no-opinion respondents should be dropped rather than zeroed, that is an argument about what the outcome variable is, and it belongs with the measurement decisions at Level 2 of the specification menu.

Weights in estimation

Every respondent in a Eurobarometer file comes with a survey weight. This section explains what those weights are, how estimation uses them, and why they fit awkwardly into multilevel models, where the choice of software can change the results. It sets out the mechanics behind the design-weight decision discussed above, and every fit shown here was run on the respondent file that is produced by the build script in the workspace.

Weights exist because samples are not miniature populations. Some people are harder to reach, some decline, some designs deliberately over-sample small groups; the result is that a 60-year-old rural woman may stand for more people than an urban student who was easier to find. A design weight \(w_i\) is, at its cleanest, the inverse of person \(i\)’s probability of ending up in the sample, \(w_i = 1 / \pi_i\). Post-stratification then adjusts the weights so that the weighted sample matches known population totals, such as the number of people in each age group. Estimation uses the weights by letting each observation count \(w_i\) times. The weighted mean is the simplest case,

\[ \hat{\bar{Y}} = \frac{\sum_i w_i\, y_i}{\sum_i w_i} \]

and weighted least squares generalises it, choosing coefficients to minimise \(\sum_i w_i \left( y_i - \beta_0 - \beta_1 x_i \right)^2\) so that under-represented kinds of people pull the line with their population share rather than their sample share.

The weighting scheme of the Eurobarometer itself, documented in the weighting overview of the GESIS Eurobarometer Data Service (GESIS Eurobarometer Data Service, n.d.), is a complete worked example of that logic, and the data file for every wave labels its weights plainly. The basic one is W1, “WEIGHT RESULT FROM TARGET”, which is the variable w1 in the rebuilt person file. It is a pure post-stratification weight (redressment, in the vocabulary of the Eurobarometer). An iterative procedure adjusts the weights to one margin after another until each national sample matches the known distribution of its population by sex, age, region (NUTS 2) and size of locality, while keeping the real number of cases in each country. W1 makes each national sample representative of its own country, and it does not rescale countries against one another.

Of the other weights in the file, W3 (“WEIGHT GERMANY”) and W4 (“WEIGHT UNITED KINGDOM”) combine the separately drawn East/West German and Great Britain/Northern Ireland samples into united national totals, the same splits that are merged by the DE and GB recodes in the course codebook. The long W5–W30 series (“WEIGHT EU6” through “WEIGHT EU27”, euro zone in and out) add the population-size scaling needed when similarly sized national samples must stand for populations as different as Malta’s and Germany’s, and WEX (“WEIGHT EXTRA POPULATION 15+”) extrapolates results to the actual population aged fifteen and over. The same routine applies to any survey: find the weighting documentation, identify the weight that matches the population that your question is about, and report in the methods which one you used. For a question about the within-country association between framing and unemployment, W1 is the natural choice, and it is the natural weight to offer in any person-level universe of specifications built from this menu.

In R, design-based estimation is the job of the survey package (Lumley 2004). You declare the design once – weights, and where the data supply them, clusters and strata – and then every estimator in the package takes it into account:

library(survey)

des <- svydesign(ids = ~1, weights = ~w1, data = person)   # weights only; the file has no primary sampling unit (PSU) ids
m_svy <- svyglm(cosmo ~ unemp, design = des)
summary(m_svy)


Call:
svyglm(formula = cosmo ~ unemp, design = des)

Survey design:
svydesign(ids = ~1, weights = ~w1, data = person)

Coefficients:
             Estimate Std. Error t value Pr(>|t|)    
(Intercept) 0.5055600  0.0014713  343.61   <2e-16 ***
unemp       0.0001262  0.0001538    0.82    0.412    
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

(Dispersion parameter for gaussian family taken to be 0.1398685)

Number of Fisher Scoring iterations: 2

The W1-weighted pooled slope is +0.0001 with t = 0.8, which is no association at all, and its sign is the opposite of the one that the claim predicts. The weights have corrected the representation of the sample, but they cannot correct a mis-specified model, and this is the pooled estimator of section 3 of the statistics module again, blending within-country and between-country variation across 27 countries. The output also shows that svyglm() reports design-robust (linearised) standard errors by default. There is no classical standard-error option to forget, which is one reason that survey statisticians like the package.

Combining weights with a multilevel model is harder, and the first difficulty is conceptual. A multilevel model needs to know the selection probability at each level – the probability that the country is included, then the probability that the respondent is included given the country – but surveys typically publish one all-in weight per person, and there is no unique way to split it. The standard answer is pseudo-maximum-likelihood, which weights the contribution of each level to the likelihood (Rabe-Hesketh and Skrondal 2006), and it leaves a further choice open: level-1 weights can be scaled in more than one defensible way, the choice can move variance estimates noticeably, and the recommendation in the literature is to report sensitivity across scalings rather than trust any one (Carle 2009).

Because implementations differ in those choices, the same weighted multilevel model can return different numbers in different software, and knowing which convention you are running is part of reporting the model. Three practical points follow. First, the weights argument of lmer() is not a survey-weights argument. It treats weights as precision weights, a different quantity entirely, and if you pass it w1 it returns answers that are inconsistent with the sample design, without any warning. Second, the mixed command in Stata accepts probability weights ([pw = ...]) and implements pseudo-maximum-likelihood with its own scaling options (pwscale()); many of the weighted multilevel results in the applied literature were produced under its conventions. Third, the R package that follows the Stata conventions is WeMix (Bailey et al. 2023), which takes one weight column per level and fits by pseudo-maximum-likelihood. In our own comparative survey work it has reproduced to three decimals the fixed-effect estimates (in the multilevel sense, the coefficients that do not vary across groups) of weighted mixed models in Stata, which makes it the tool of choice when a published Stata result needs re-implementing in R.

Two of the five Multi100 analysts worked in Stata, and their public code shows these choices being made in practice. Analyst PRL47, the person-level multilevel analyst of section 6 of the statistics module, builds the level-2 cell identifier and fits three-level random intercepts with mixed. These lines are from their Multi100 upload:

egen cquarter=concat(country quarter)    // the country-by-quarter cell id
mixed cosmo || country: || cquarter:     // three-level random intercepts

The weight is missing from that second line. PRL47’s script carefully assembles a survey weight for every wave (gen weight=w3a4a), then uses it only in descriptive checks (sum cosmo cosmo2 [w=weight]). It never enters the model, because including it there would mean committing to [pw = ...] and to a defensible pwscale() setting, which is the pseudo-maximum-likelihood decision described above. Building the weight is easy; using it inside a multilevel model requires a decision that has to be defended, and PRL47’s code stops short of making it. A bare mixed, with no estimation options, estimates by maximum likelihood with model-based standard errors, whereas lmer() in R defaults to REML (restricted maximum likelihood), so a line-for-line port disagrees with the original for no substantive reason unless the default on one side is overridden. The correspondence for Stata users is as follows:

* Stata – PRL47's estimator (maximum likelihood, the Stata default)
mixed cosmo || country: || cquarter:
* what a design-weighted version would have needed
mixed cosmo [pw = weight] || country: || cquarter:, pwscale(size)

# the R equivalent – lmer() defaults to REML, so request maximum likelihood to match Stata
lmer(cosmo ~ 1 + (1 | country) + (1 | cquarter), data = d, REML = FALSE)

# the design-weighted equivalent – WeMix, pseudo-maximum-likelihood, one weight per level
mix(cosmo ~ 1 + (1 | country), data = d, weights = c("weight", "w_country"))

On the data of this case, supplying a constant weight of 1 at the country level reproduces the common ‘weights at level 1 only’ scenario:

library(WeMix)

person <- person |>
  mutate(w_lvl2 = 1)   # unit weight at the country level – weighting at level 1 only
m_wemix <- mix(cosmo ~ unemp + (1 | cntry), data = person,
               weights = c("w1", "w_lvl2"))
summary(m_wemix)

Call:
mix(formula = cosmo ~ unemp + (1 | cntry), data = person, weights = c("w1", 
    "w_lvl2"))

Variance terms:
 Level    Group        Name Variance Std. Error Std.Dev.
     2    cntry (Intercept) 0.003741   0.001394  0.06116
     1 Residual             0.136155   0.004246  0.36899
Groups:
 Level Group n size mean wgt sum wgt
     2 cntry     27        1      27
     1   Obs 416677        1  416698

Fixed Effects:
             Estimate Std. Error t value
(Intercept)  0.555062   0.015043  36.898
unemp       -0.004915   0.001488  -3.302

lnl= -175909.61 
Intraclass Correlation= 0.02674

Compare this with the unweighted two-level fit of section 6 of the statistics module, and read the comparison carefully, because the two fits differ in more than the weights. The weighted slope, −0.0049, is almost the same as the unweighted −0.0048, so the W1 weights barely move the estimate, and the association between framing and unemployment is not an artefact of who is over- or under-represented within national samples. The standard errors, though, are 0.0015 here against 0.0002 there, and they are not comparable at a glance. Most of that gap comes from the variance estimator, since WeMix reports design-linearised (robust) standard errors while lmer() reports model-based ones, so two settings changed between the outputs and only one of them is the weighting. The rule from section 3 of the statistics module applies here too: before attributing a difference to the weights, make sure that every other setting, the variance estimator included, is the same in both fits.

The weighting axes of the course are simpler than any of this. The specification menu offers three regression weights – the number of respondents behind each country-year average (n_cell), the effective sample size that remains once the design weights have been applied to those respondents (n_eff), and the adult population of the country (pop) – and, separately, a choice of whether the cell means themselves are built with or without the Eurobarometer design weights (cell_weight). The first two regression weights are precision arguments about aggregated cells: a mean built from 3,000 people is measured more precisely than one built from 300. The population weight is a different kind of object, because it changes whose average the estimate describes. The design weights belong to the respondent-level data, and the cell_weight axis applies them there, when the cell means are built, producing a second version of the panel. The earlier sections of this page discuss each of these in turn. The second Stata analyst in this case, KQXUE, works with the aggregated cells, and in their code two lines settle the whole weighting question by default:

collapse cosmo util, by(cntry year)    // unweighted country-year means
xtreg cosmo unemployment, fe           // = the within estimator of section 4 of the statistics module

The collapse command averages every respondent equally, so the design weights are gone before any model is fitted, and each cell then counts equally in xtreg whether 300 people or 3,000 stand behind it. That is the choice which the n_cell option on the menu switches on and off. (The Stata–R correspondence continues here too: xtreg, fe after xtset cntry_num year is the within estimator in plm.) Across the grid, the weighting choices on the menu contribute less than one per cent of the specification variance.

What to take away

Say which of the three decisions you are making, and give the argument that goes with it. The word ‘weighted’ on its own does not specify a model.
Design weighting and precision weighting leave the estimand alone. Population weighting changes it, and a paper that switches to population weights in a robustness section has changed its question without saying so.
When a survey provides several weights, the choice among them usually encodes the country-versus-person decision. Read the codebook first.
Look at what your weight variable actually varies over before trusting it as a proxy for anything. Here the respondent count spans a factor of 7.2 and the population it is standing in for spans a factor of 215.

Exercise

Pick any specification from the menu and look it up in data/spec_grid_weights.csv under all four regression weights. Write one sentence per weight stating what the coefficient is the average of.
Two of those four sentences describe the same quantity. Which two, and why?
The no_bailout sample setting and the pop weight both change the population that the claim is about. One of them is conventionally reported as a robustness check and the other as a modelling choice. Should either be?

References

Bailey, Paul, Blue Webb, Claire Kelley, Trang Nguyen, and Huade Huo. 2023. WeMix: Weighted Mixed-Effects Models Using Multilevel Pseudo Maximum Likelihood Estimation. https://american-institutes-for-research.github.io/WeMix/.

Carle, Adam C. 2009. “Fitting Multilevel Models in Complex Survey Data with Design Weights: Recommendations.” BMC Medical Research Methodology 9 (1): 49. https://doi.org/10.1186/1471-2288-9-49.

GESIS Eurobarometer Data Service. n.d. Weighting Overview (Standard and Special Eurobarometer). Accessed July 17, 2026. https://www.gesis.org/en/eurobarometer-data-service/data-and-documentation/standard-special-eb/weighting-overview.

Lumley, Thomas. 2004. “Analysis of Complex Survey Samples.” Journal of Statistical Software 9 (8): 1–19. https://doi.org/10.18637/jss.v009.i08.

Rabe-Hesketh, Sophia, and Anders Skrondal. 2006. “Multilevel Modelling of Complex Survey Data.” Journal of the Royal Statistical Society: Series A (Statistics in Society) 169 (4): 805–27. https://doi.org/10.1111/j.1467-985X.2006.00426.x.