The sample you did not get

In 1936, a magazine polled its readers about the coming US presidential election, collected around 2.4 million responses, and predicted the wrong winner by a wide margin. A pollster working with a few thousand respondents got it right. The magazine's sample was roughly a thousand times larger.

The reason is the single most useful idea in applied statistics, and it is not taught anything like as often as sample size is: a larger sample reduces variance, and does nothing whatsoever about bias.

Where the survey went wrong

The magazine drew its list from telephone directories, club memberships and automobile registrations. In 1936 those things correlated with wealth, and wealth correlated with the vote. The sample was not a small version of the electorate. It was a large version of a particular slice of it.

Adding respondents from the same lists makes the estimate more precise. It does not move it toward the truth, because every additional respondent is drawn from the same skewed pool. Precision improves, accuracy does not, and the result is a confidently stated wrong answer.

That is worse than a noisy right one, because the confidence is real. The standard error genuinely shrinks. Every internal check on the arithmetic passes.

The arithmetic of it

Take a population that is 60 percent in favour of something. Two groups respond at different rates.

Group X is 50 percent of the population and responds 30 percent of the time. Group Y is the other 50 percent and responds 10 percent of the time. Within X, 40 percent are in favour. Within Y, 80 percent are.

Check that against the stated population figure: half of 40 percent plus half of 80 percent is 20 plus 40, which is 60 percent. Correct.

Now take 10,000 people, 5,000 in each group, and see who answers.

• From X: 5,000 times 0.30 is 1,500 respondents, of whom 40 percent are in favour, giving 1,500 times 0.40, which is 600.
• From Y: 5,000 times 0.10 is 500 respondents, of whom 80 percent are in favour, giving 500 times 0.80, which is 400.

Total respondents: 1,500 plus 500, which is 2,000. Total in favour: 600 plus 400, which is 1,000.

The estimate is 1,000 divided by 2,000, which is 50 percent. The truth is 60 percent. The survey is off by 10 points.

Now scale to a million people, keeping every rate identical.

• From X: 500,000 times 0.30 is 150,000 respondents, 40 percent in favour, giving 60,000.
• From Y: 500,000 times 0.10 is 50,000 respondents, 80 percent in favour, giving 40,000.

Total: 200,000 respondents, 100,000 in favour. The estimate is 100,000 divided by 200,000, which is 50 percent.

Identical. A hundred times the data, the same error, and a confidence interval a tenth as wide wrapped around it. The bias is a property of the response rates, and no quantity of additional respondents touches it.

Group X, responding three times as often, supplies 75 percent of the responses while being 50 percent of the population. That over-representation is the entire error, and it is visible only if you know the true group sizes.

Survivorship, the same error wearing a different hat

During the Second World War, analysts examined returning aircraft to decide where to add armour, and found the most damage on the wings and fuselage. The proposal was to armour those areas. Abraham Wald's observation was that the sample consisted of aircraft that came back, and the armour belonged where the returning planes showed no damage, because hits there were the ones that prevented a plane from returning at all.

The pattern recurs constantly in ordinary analysis.

Analysing "customers who have been with us two years" to find what drives retention is a study of survivors. The customers who left are gone from the table, and they are the ones the question is about. Studying successful startups to find the traits that cause success omits the failed startups that had the same traits. Evaluating a model on the requests it answered omits the ones that timed out, which are disproportionately the hard ones.

The tell is always the same: the filter that produced the dataset is related to the outcome being studied. Ask what had to happen for a row to exist, and if the answer involves the thing you are measuring, the sample is selected on the outcome.

Missing data, in three flavours

Whether missing values can be repaired depends on why they are missing, and there are three cases with genuinely different consequences.

Missing completely at random. A sensor dropped readings because of a network fault unrelated to what it was measuring. Losing these rows costs precision and introduces no bias. This is the benign case and it is the rarest.

Missing at random, given what you observed. Older respondents skip the income question more often, but within each age band the skippers earn about the same as the answerers. Age is recorded, so the gap can be modelled and the missingness corrected for. The name is misleading: the data is not missing randomly, it is missing in a pattern the observed columns can account for.

Missing not at random. High earners skip the income question regardless of age or anything else recorded. The reason a value is missing depends on the value itself. Nothing in the observed data can fix this, because the information needed to correct it is precisely the information that is absent.

The third case is common and it is the one that gets treated as if it were the first. Dropping incomplete rows, or filling them with the column mean, silently assumes the missingness is benign. When high earners are the ones skipping, mean imputation pulls their incomes toward the average of people who answered, and the estimate of average income comes out too low with a tidy confidence interval around it.

The distinction cannot be made from the data. It is a claim about the mechanism that produced the gaps, and like the causal claims in Correlation, causation and the third variable, it has to be argued and written down.

Why more data can make it worse

There is a genuinely counter-intuitive consequence, and Meng's analysis of the 2016 election polling makes it precise: as a biased sample grows, the confidence interval narrows around the wrong answer, so the reported uncertainty becomes an increasingly poor guide to the actual error.

A small biased sample is wrong and admits to being uncertain. A large biased sample is wrong and claims precision. The second is more dangerous, because the usual quality signal, a tight interval, now points the wrong way.

This is worth holding onto when working with large operational datasets, which are almost never random samples of anything. Server logs cover users who reached the server. Support tickets cover people who bothered to complain. Clickstream data covers sessions that did not crash. Each is enormous and each is selected, and their size buys precision on a population that is not the one anybody wants to reason about.

The training data case

For anyone building models rather than running surveys, the same error arrives in a specific and costly form: the data a model trains on is the data some earlier process produced, and that process was rarely neutral.

A lending model trained on repayment history sees only applicants who were approved. Whether the rejected applicants would have repaid is unrecorded, because the decision that created the dataset also determined which outcomes exist. Train on that and the model learns the previous policy's blind spots as though they were facts about borrowers. The gap cannot be closed by adding more historical rows, since every one of them passed through the same filter.

A support triage model trained on tickets that were resolved omits the ones that were abandoned. A recommendation model trained on clicks sees only items the previous recommender chose to display, so items it never showed have no data and stay unshown, which is a feedback loop rather than a static bias.

The practical response is the same one used in medicine and advertising: hold out a small randomised slice. Approve a small random fraction of applicants who would have been rejected, show a small random fraction of unranked items, and accept the cost as the price of an unbiased sample to calibrate against. It is deliberately buying the data the operational process will never generate on its own.

What to do

Say what the population is, in a sentence, before analysing. Not "our data" but the actual group the conclusion is meant to describe. Most selection problems become visible at this step, which is why it is worth doing in writing.

Ask what had to happen for a row to exist. Every filter between the world and the table is a potential selection effect. List them.

Compare the sample to a known benchmark. If the population's group proportions are known from another source, check them against the sample's. The 75 percent against 50 percent gap in the worked example above is exactly what this check surfaces.

Reweight when you can. If group X is 75 percent of responses and 50 percent of the population, weight its responses by 50 divided by 75, which is 0.667, and weight Y by 50 divided by 25, which is 2.0. Redo the earlier sum with those weights: X contributes 600 times 0.667, which is 400, and Y contributes 400 times 2.0, which is 800, from a weighted total of 1,000 times 0.667 plus 1,000 times 2.0. Working it through returns 60 percent. Reweighting fixes the bias exactly when the variable causing the imbalance is known and recorded, and not otherwise.

Report the response rate and who is missing. A result from a 4 percent response rate is a different object from one at 70 percent, and a reader cannot judge it without being told.

The version worth carrying

Sample size answers "how precise is this estimate", and sampling method answers "what is it an estimate of". Only the second question can be answered wrongly in a way that more data will not fix.

The next article turns to what happens when the thing being measured changes over time, and why a model validated on a random split of historical data can fail immediately in production.

References

1. The ASA Statement on p-Values: Context, Process, and Purpose. Ronald L. Wasserstein and Nicole A. Lazar, American Statistical Association, 2016.
2. Statistical paradises and paradoxes in big data: Law of large populations, big data paradox, and the 2016 US presidential election. Xiao-Li Meng, The Annals of Applied Statistics, 2018.
3. Causal Inference in Statistics, A Primer. Judea Pearl, Madelyn Glymour and Nicholas P. Jewell, Wiley, 2016.