html`<p class="estimand">Among <b>${labels.units[sample]}</b>, the association between <b>${labels.predictor[predictor]}</b>
<span class="measured">(measured as ${labels.measured.measure[measure]})</span> and <b>${labels.outcome[outcome]}</b>
<span class="measured">(built ${labels.measured.item_set[item_set]}, ${labels.measured.noop[noop]}, ${labels.measured.cell_weight[cell_weight]})</span>,
comparing <b>${labels.comparison[estimator]}</b>${adjustment}, with <b>${labels.weighting[weights]}</b>.</p>
<p class="small" style="margin-top:0.6rem">The words in blue state the question, and the words in green say how its variables are measured. Change a Level 2 choice and only the green words change, while a Level 3 choice leaves the statement as it is.</p>`problem !== null
? html`<p class="small" style="color:#96131b">${problem}</p>`
: html`<span class="chip ${verdict[0]}">${verdict[1]}</span>
<div class="readout">
<div><div class="k">Coefficient</div><div class="v">${d3.format(".5f")(hit.estimate)}</div></div>
<div><div class="k">${statLabel} on ${Math.round(hit.df)} df</div><div class="v">${fmt(hit.statistic)}</div></div>
<div><div class="k">p</div><div class="v">${hit.p < 0.001 ? "< 0.001" : fmt(hit.p)}</div></div>
<div><div class="k">aligned r</div><div class="v">${fmt(hit.r_ca)}</div></div>
</div>
<p class="small">Without changing the question, the answer can be moved from <b>${fmt(d3.min(shard, d => d.r_ca))}</b> to <b>${fmt(d3.max(shard, d => d.r_ca))}</b> by the Level 2 and Level 3 choices alone (median ${fmt(d3.median(shard, d => d.r_ca))}; ${Math.round(100 * d3.mean(shard, d => d.sig ? 1 : 0))}% of the ${shard.length} answers significant). Among the ${questions.length.toLocaleString()} questions that the menu can ask, ${Math.round(100 * placement)}% have a median answer more claim-supporting than the answer from the chosen specification.</p>`colourBy_input = Inputs.select(
new Map([
["Significance only", "none"],
["No-opinion respondents", "noop"],
["Item set", "item_set"],
["Cell means", "cell_weight"],
["Published series", "measure"],
["Predictor form", "pred_form"],
["Outcome family", "family"],
["Standard errors", "se"]
]), {label: "Colour the dots by", value: "none"})Plot.plot({
height: 440,
width: 1100,
marginLeft: 52,
marginBottom: 40,
style: {fontSize: "12px"},
x: {label: `${curve.length} ways of answering this question, ranked from most to least claim-supporting →`, ticks: 5},
y: {label: "↑ Claim-aligned partial r (below zero supports the claim)", domain: [-0.75, 0.45], grid: true},
color: colourBy === "none" ? {legend: false} : {legend: true, label: axisLabel[colourBy], range: palette},
marks: [
Plot.ruleY([0], {stroke: "#5b6b78", strokeWidth: 1}),
cell === undefined ? null : Plot.ruleY([cell.median_rca], {stroke: "#0e3b6a", strokeDasharray: "3,3", strokeOpacity: 0.6}),
Plot.dot(curve, {x: "rank", y: "r_ca", r: 3.6, stroke: "none", fillOpacity: 0.55,
fill: colourBy === "none" ? (d => d.sig ? "#0e3b6a" : "#b7c2cc") : colourBy,
title: d => `${d.construction}, ${d.noop}, items: ${d.item_set}, cells: ${d.cell_weight}, ${d.measure}\n${d.pred_form}, ${d.family}, se: ${d.se}\nr_ca = ${d.r_ca.toFixed(3)}${d.sig ? " (p < .05)" : ""}`}),
Plot.dot(curve.filter(d => d.isCurrent), {x: "rank", y: "r_ca", r: 8, fill: "none", stroke: "#c77c00", strokeWidth: 2.5}),
Plot.text(curve.filter(d => d.isCurrent), {x: "rank", y: "r_ca", text: () => "chosen", dy: -16, fill: "#c77c00", fontWeight: 600})
]
})Each dot is one way of answering the question that is fixed by the Level 1 choices: one combination of the Level 2 and Level 3 options, read from the pre-computed grid. Dark blue dots are significant at the five per cent level, grey ones are not. The orange ring is the chosen specification, and the dashed line is the median answer to the question. Hover over a dot for its options. The menu above the chart colours the dots by one Level 2 or Level 3 choice, to show where each of its options falls along the curve.
Plot.plot({
height: 24 * rows.length + 70,
width: 1100,
marginLeft: 200,
marginBottom: 40,
style: {fontSize: "12px"},
x: {label: "Median claim-aligned partial r of the question (below zero supports the claim) →", domain: [-0.75, 0.45], grid: true},
y: {label: null, domain: rows},
marks: [
band ? Plot.rectX([{x1: band[0], x2: band[1]}], {x1: "x1", x2: "x2", fill: "#c77c00", fillOpacity: 0.12}) : null,
Plot.ruleX([0], {stroke: "#5b6b78"}),
Plot.ruleY(firstOfAxis, {stroke: "#d6dde5", strokeWidth: 1, dy: -12}),
Plot.dot(neighbours.filter(d => d.exists && !d.isCurrent), {x: "median", y: "row", r: 5.5, fill: d => clusterColour[d.cluster], fillOpacity: 0.85, stroke: "none",
title: d => `${d.row}\nmedian r_ca ${d.median.toFixed(3)}, range ${d.lo.toFixed(3)} to ${d.hi.toFixed(3)}, ${Math.round(100 * d.share_sig)}% significant`}),
Plot.dot(neighbours.filter(d => d.exists && d.isCurrent), {x: "median", y: "row", r: 7, fill: "#c77c00", stroke: "none",
title: d => `${d.row} (chosen)\nmedian r_ca ${d.median.toFixed(3)}`}),
Plot.text(neighbours.filter(d => !d.exists), {x: -0.74, y: "row", text: () => "not in the universe", textAnchor: "start", fill: "#9aa8b3", fontSize: 11})
]
})One row for every option of every Level 1 choice, grouped by choice in the order of the sidebar. The orange dots are the seven options chosen now. Each other dot is what the median answer to the question becomes if that one choice is switched to that option and the other six are left as they are: blue for the four choices that name the variables, teal for the two that say whom the question is about, bronze for the estimator. The shaded band covers the middle 80% of the answers to the current question, across every combination of the Level 2 and Level 3 options. A dot outside the band means that switching that one choice moves the answer further than all the measurement and estimation decisions together can. Hover over a dot for the range of answers to that question and the share of them that is significant.
Plot.plot({
height: 340,
width: 1100,
marginBottom: 40,
style: {fontSize: "12px"},
x: {label: "Median claim-aligned partial r of each question that the menu can ask →", domain: [-0.75, 0.45]},
y: {label: "↑ Questions", grid: true},
marks: [
Plot.rectY(questions, Plot.binX({y: "count"}, {x: "median_rca", fill: "#b9c9dc", thresholds: 40})),
Plot.ruleX([0], {stroke: "#5b6b78"}),
cell === undefined ? null : Plot.ruleX([cell.median_rca], {stroke: "#c77c00", strokeWidth: 2.5}),
Plot.frame({stroke: "#ddd"})
]
})Every question that can be set with the seven Level 1 choices, each summarised by the median of its answers, with the chosen question marked in orange. The per-item rate, which rescales the count, is not counted as a question of its own. Below zero supports the claim on every scale, because negative framings are reverse-coded and the sign convention for growth is flipped.
The statement is how the course tests whether two analyses ask the same question. Change a Level 1 choice and the statement changes, because a different question is now being asked of the data; two analysts who set it differently were not disagreeing about method. Change a Level 2 choice and only the green words change, while a Level 3 choice leaves the statement as it is. The Level 1 choices are listed in three groups. The frame set, the scoring rule, the predictor and the adjustment set name the variables. The sample and the weight say whom the question is about, and the estimator says how the country-years are compared.
The answers to this question ranks every answer to the question that is fixed by the Level 1 choices, one for each combination of the Level 2 and Level 3 options in the pre-computed grid, from most to least claim-supporting. Dark blue dots are significant at the five per cent level, grey ones are not, and the orange ring is the chosen specification. Colour the dots by one choice to find out where each of its options falls.
One Level 1 choice at a time shows what each alternative option of a Level 1 choice would do to the answer, holding the other six as they are. Each dot is the median answer to the question that the switch produces, the orange dots mark the chosen question, and the shaded band behind them covers the middle 80% of the answers to the current question. A dot outside the band means that switching that one choice moves the answer further than every Level 2 and Level 3 choice together.
All the questions places the current question among every question that the menu can ask, each summarised by its median answer. The per-item rate, which rescales the count, is not counted as a question of its own.
The specification menu gives, for every choice, the share of the variation in results that is attributed to it and how often changing that choice alone takes a result across p = .05. The weighting and missing-answers pages examine two of the Level 2 choices in detail.