Building the analysis file
Foundation to intermediate – from sixteen survey files to the respondent file, the scales and the panel
This module of the companion curriculum is about the work between a download and a model: reading survey files with their labels, finding the same question in sixteen files that name it sixteen ways, harmonising the respondent characteristics, scoring a battery of items into scales, averaging respondents into country-years, and merging a macro series in by country and year. Every page of the course that fits a model starts from a file that this work produced, and this page is built around the two scripts in the workspace that do that work: prep/01_build_respondents.R, which you run once, and prep/02_scales_and_panel.R, which you read and run step by step. The first rebuilds the respondent file used by the course from your own GESIS downloads and checks the result against the course build. The second rebuilds the country-year panel that is distributed with the course, to machine precision, and then builds the panel by education band on which the individual-level extension is based.
The module assumes the R module, and the data sources module explains the downloads themselves step by step. If you have not downloaded the sixteen files, everything here can still be read, because the numbers that the scripts print are quoted on the page; nothing can be run.
The GESIS terms of use apply to everything that the first script writes. The respondent file stays in data_private/, which is kept out of version control by the .gitignore file in the workspace. It is never uploaded, shared, or opened in a session that an AI assistant can read, because the terms prohibit processing the data with AI systems. It is deleted when the project ends. The aggregates distributed with the course are subject to none of these restrictions, and the second script shows where the line between the two falls.
1. What the first script does
prep/01_build_respondents.R is written to be run as a whole: set the folder that holds the sixteen files at the top, then source the script. It does five things, and it prints a report at the end.
It checks the files. The script contains an inventory of the sixteen studies: the file name under which GESIS serves each study, the study number, the wave, the year to which the course assigns the wave, and the MD5 checksum of the file from which the course panel was built. A missing file stops the script with the study number to fetch. The two studies that GESIS serves zipped, ZA4506 and ZA4744, are unpacked beside their archives. The checksum of every file is compared with the inventory, and the report says, file by file, whether yours is byte for byte identical to the one that the course used. The comparison makes the rest of the course reproducible on your own machine. If the sixteen checksums match, every number on this site can be rebuilt from your folder.
It finds the question in each file. The EU-meaning battery occupies sixteen adjacent columns in every file, fourteen items followed by “other” and “don’t know”, but the position of those columns differs. In the twelve older files the columns are numbered (v63 to v78 in EB 61.0, v105 onwards in EB 62.0) and in the four newest they are named after the question (qa12_1 to qa12_16 in EB 76.3). The inventory records the start of the block or the question stem for each wave, and the script reads the sixteen columns by that rule and checks that every value is 0 or 1. The respondent characteristics are found differently, by their variable labels rather than their positions, because the occupation item in particular moves from wave to wave. If a label pattern matches two variables, the script stops instead of picking one.
It harmonises Germany and the United Kingdom, which the survey samples as two regions each, into one country each, and keeps the twenty-seven member states of the period.
It codes the respondent characteristics as the published models code them. Sex as a female indicator. Education as the age at which full-time schooling finished, with the two codings that the menu offers: the published three bands, which replace “still studying” with the respondent’s current age and cut at 16 and 20, and a four-band version that keeps the students as a band of their own. Occupation collapsed into the eight groups of the published models, with the unemployed as their own group. Urbanisation in three categories. Left–right placement on its ten-point scale, missing in the four waves that did not ask it.
It compares the result with the course files. The number of respondents in every country-year is checked against the committed panel, and the counts of respondents who named none of the thirteen items, split into those who said “don’t know”, those who gave another answer and those who left the question blank, are checked against the committed cell file. When the checksums match, all 270 cells match on both counts.
On a folder holding the sixteen files that the course used, the printed report looks like this:
study eb year rows_eu27 identical_to_course_file
ZA4056 61.0 2004 16216 TRUE
ZA4229 62.0 2004 26807 TRUE
...
ZA5689 79.3 2013 26605 TRUE
check value
files found 16
files identical to the course build 16
country-years in the committed panel 270
country-years rebuilt 270
cells with an identical respondent count 270
cells with identical no-opinion counts (none, DK, other, blank) 270
respondents: 416698 rows, 31 columns, written to data_private/eb_respondents.rds
Every check passed: your file is the one the course was built from.Run the script and read the report. Then open data_private/build_report.csv and find the wave with the fewest respondents. It is the spring 2004 wave, fielded before the ten accession states joined; every later wave has about twenty-six and a half thousand respondents in the twenty-seven states.
2. What the second script teaches
prep/02_scales_and_panel.R is written to be read from the top and run a section at a time. It makes the measurement decisions of the course explicit one at a time, in the order in which the pipeline needs them.
2.1 Which items form which scale
The published paper assigns the thirteen scored items to four scales, cosmopolitan, utilitarian, communitarian and libertarian, and the course adds the positive and the negative composite. The script states the assignment as four vectors of item names, since at this point a scale is no more than a list. The euro item is kept in the respondent file and never scored, as in the paper. The measurement module tests whether the data agree with the assignment.
2.2 Three scoring rules
A respondent’s answers to a set of items can become one number in more than one way, and the course menu offers three: the share, which is the published rule and divides the mentions of items in the set by the respondent’s own total; the count, which is the number of items in the set that were mentioned; and the per-item rate, which divides the count by the number of items in the set. The script computes all three for every scale, side by side, from the same item columns. The share is compositional, meaning that the four scales sum to one for anyone who mentioned anything. The count measures how many items were mentioned, and its four scales have no such constraint. That is why the specification menu treats the rule as a decision about what the outcome is rather than about how it is measured.
2.3 The respondent who named nothing
About six per cent of respondents mentioned none of the thirteen items, and a share has no denominator for them. The published rule scores them zero on every scale, and the script does the same. The missing-answers module sets out the alternatives, which are dropping them, splitting them across the scales, or imputing them. The first script keeps the “don’t know”, “other” and blank codes apart because the three groups are different people, and treating them alike is itself a decision.
2.4 From respondents to country-years
The course panel averages every respondent in a country-year with the same weight, which the course calls ‘as interviewed’. The design weight that GESIS supplies with each file, which corrects a national sample to its population, is the other choice, and the script computes both means so that the two can be compared in the weights module. It also records the count of respondents in each cell, which is used by a precision weight at the next stage.
2.5 The macro series
The unemployment rate and the growth series join the panel by country and year. The script merges them from the file that contains every series offered by the menu, which is distributed with the course so that the merge works offline, and shows the two lines that fetch the same World Bank indicators live with the WDI package. A live fetch can differ in the last decimals because the Bank revises past years, so the course records which release of each series it used. The bailout indicator is not merged from any file. It is coded by rule from the record of assistance programmes, and the codebook states the rule.
2.6 Does it reproduce the panel?
The rebuilt panel is compared column by column with the panel distributed with the course. On the file from which the course was built, the six as-interviewed scale means, the respondent counts and the two macro series agree to machine precision, and so do the design-weighted and the count-built means in the cell file:
column max_abs_difference
mcosmo 5.551115e-17
mutil 2.775558e-17
mcomm 5.551115e-17
mlib 2.775558e-17
mpos 5.551115e-17
mneg 5.551115e-17
n_cy 0.000000e+00
unemp 0.000000e+00
growth 0.000000e+00A difference of five parts in a hundred million billion is the rounding of floating-point arithmetic, not a difference in the data.
A reproducible data pipeline, in the sense used in this course, is one in which anyone with the same sixteen files and this script gets the same 270 rows, and can alter any decision above and find out what changes in the panel.
2.7 The panel by education band
The last section builds a second panel, one row per country, year and education band, with the same six scales and the number of respondents behind each mean. It is the aggregate used in the individual-level extension of the course to ask whether unemployment narrows the gap between the high- and the low-educated. Unlike the respondent file, it can be committed and shared, because a cell mean over at least 43 people, and in most cells several hundred, reveals nothing about any of them.
Change the scoring rule in section 2.2 from the share to the count for the cosmopolitan scale, re-run sections 2.4 to 2.6, and read the column of differences. The as-interviewed cosmopolitan mean now differs from the course panel in every cell, and the count-built column of the cell file agrees instead. Then change the no-opinion rule so that respondents who named nothing are dropped rather than scored zero, and read the respondent counts. Each is a one-line change in the script and a decision on the specification menu.
3. Beyond this case
The same five steps (read with labels, find the question, harmonise, score and aggregate) recur in most survey work, and the reading section of the R module has the functions for the file formats and the fetching subsection the packages that reach the archives directly. Two habits from these scripts apply to any project. Keep an inventory of the files you read, with their versions and checksums, in the script itself, so that the data on which a result rests are stated where the result is computed. And keep the licensed and the shareable in different folders from the first day, with the ignore rule written before the first file is downloaded.