Correlation, causation and the third variable

"Correlation is not causation" is the most widely known statement in statistics and one of the least useful, because it tells you what not to conclude and nothing about what to do instead. It is repeated at the point where the real work starts.

The useful version is a question. When two things move together, there are only a few structures that can produce that, and they call for different responses. Knowing which one you are looking at is the job.

The worked example

A company sells a subscription product. Someone notices that customers who contact support are far more likely to renew, and proposes encouraging support contact to raise renewals.

Here are 1,000 customers.

Contacted support: 400 customers, of whom 340 renewed.
Did not contact support: 600 customers, of whom 330 renewed.

Renewal rate among those who contacted support: 340 divided by 400, which is 0.85. Renewal rate among those who did not: 330 divided by 600, which is 0.55.

A gap of 30 percentage points. The correlation is real and it is large. No arithmetic error will make it go away, and if the only two columns you have are "contacted support" and "renewed", there is nothing further to compute.

Now add a column that existed all along: whether the customer had actually deployed the product, as opposed to signing up and never switching it on.

Customers who had deployed (500 of them):

Contacted support: 350, of whom 315 renewed. Rate: 315 divided by 350, which is 0.90.
Did not contact support: 150, of whom 132 renewed. Rate: 132 divided by 150, which is 0.88.

Customers who had not deployed (500 of them):

Contacted support: 50, of whom 25 renewed. Rate: 25 divided by 50, which is 0.50.
Did not contact support: 450, of whom 198 renewed. Rate: 198 divided by 450, which is 0.44.

Check the totals. Contacted support is 350 plus 50, which is 400, and renewals are 315 plus 25, which is 340. Did not contact is 150 plus 450, which is 600, and renewals are 132 plus 198, which is 330. The same thousand customers, the same 670 renewals.

The 30 point gap has become a 2 point gap in one group and a 6 point gap in the other. Nothing was removed. One column was added.

What happened

Deployment drives both. A customer who has deployed the product has a reason to contact support, because they are using something, and a reason to renew, because it is doing something for them. Support contact was mostly a marker of being a real user.

Splitting by deployment holds that fixed. Inside the deployed group everyone has already cleared that bar, so comparing across support contact compares like with like. The 30 point gap collapses to somewhere between 2 and 6 points, which is a plausible size for a real effect of good support, and a completely different business case from the one the headline number implied.

Acting on the raw number would mean pushing customers toward the support queue. The split says the money belongs in getting customers deployed, and that a fair test of support quality is worth running separately.

The general shape: a confounder is a variable that causes both of the things you are comparing. It manufactures correlation between them with no causal arrow running between the two at all.

When the direction flips completely

The example above weakened an effect. A confounder can also reverse one, which is worth seeing because it destroys any intuition that the raw number is at least directionally safe.

Two treatments for kidney stones, in the shape of the classic case.

Treatment A: 81 of 87 small stones succeed, and 192 of 263 large stones succeed. Treatment B: 234 of 270 small stones succeed, and 55 of 80 large stones succeed.

Small stones: A is 81 divided by 87, which is 0.931. B is 234 divided by 270, which is 0.867. A wins. Large stones: A is 192 divided by 263, which is 0.730. B is 55 divided by 80, which is 0.688. A wins.

A is better on small stones and better on large stones. Now pool them.

A: 81 plus 192 is 273 successes, out of 87 plus 263, which is 350 patients. That is 273 divided by 350, or 0.780. B: 234 plus 55 is 289 successes, out of 270 plus 80, which is 350 patients. That is 289 divided by 350, or 0.826.

B wins overall, having lost both halves.

The mechanism is in the group sizes. Treatment A was given mostly to large stones, 263 of its 350 cases, and treatment B mostly to small stones, 270 of its 350. Large stones are harder for both treatments. The pooled figure for A is dragged down by the hard cases it took on, and B's is lifted by the easy ones. This is Simpson's paradox, and the arithmetic above is the whole of it.

Note that the pooled number is not miscalculated. It correctly answers a question nobody wanted to ask: what happened on average given who was assigned to what. The question of interest is what would happen to a given patient under each treatment, and the pooled number does not answer it.

Four structures behind a correlation

When two variables move together, one of these is happening.

One causes the other. The thing you hoped for. It supports intervention: change the cause and the effect follows.

A confounder causes both. Deployment above. Intervening on the correlated variable does little, because you are moving a marker rather than a cause.

Selection. The correlation is created by how the data got collected. If a dataset holds only customers who stayed at least a year, every conclusion is conditional on surviving that year, and the population you care about is the one you cannot see.

Chance. With enough variables, some pairs correlate for no reason. Test twenty independent hypotheses at the conventional five percent threshold and you should expect about one apparently significant result from nothing at all: 20 times 0.05 is 1.

The four are not distinguishable from the correlation itself. They are distinguished by knowledge of how the data was generated, which is why the analysis cannot be done from the table alone.

Which columns to control for

The natural instinct is to control for everything available. That is wrong, and the way it is wrong is specific.

Controlling for a confounder, a variable that causes both sides, removes bias. Controlling for a variable that sits on the path between cause and effect removes the effect you were trying to measure. If a training programme improves skills and skills improve performance, then controlling for skills makes the programme look useless, because the mechanism has been held fixed.

There is a third case that surprises people. Controlling for a variable that both sides cause creates a spurious association where none existed. If two independent traits each raise the chance of admission to a programme, then among admitted people the two traits will appear negatively related, because an admitted person who is low on one must have been high on the other to get in. The correlation is manufactured by conditioning on the shared consequence.

So the rule is not "control for more". It is that the decision requires a claim about which variables cause which, before the data is touched. That claim is an assumption, it should be written down where a reader can disagree with it, and it cannot be extracted from the correlation matrix.

What settles it

A randomised experiment. Assigning the treatment at random makes it independent of every confounder, measured or not, which is the property no amount of statistical adjustment can buy. Adjustment can only handle variables you thought of and recorded; randomisation handles the ones you did not.

Randomise support contact and the deployment problem disappears, because deployed and undeployed customers land on both sides in similar proportions. The comparison is then like with like by construction rather than by argument.

When randomisation is impossible, for cost or ethics or because the event already happened, the honest position is that the estimate rests on an assumption about what was measured. State the assumption. Do not present the number as though it did not need one.

When you cannot randomise

Plenty of questions cannot be settled by experiment. You cannot randomly assign customers to churn, or firms to a recession. The honest options are narrower than the literature sometimes suggests, and they share one feature: each replaces randomisation with an explicit, checkable assumption.

Find a natural experiment. Sometimes the world randomises for you. A policy that applies to customers who signed up after a cutoff date, but not before, splits a population in a way unrelated to the customers themselves. Comparing just either side of the boundary approximates a randomised comparison, and the assumption is that nothing else changes sharply at that date.

Compare the same unit over time, against a control that did not change. If a feature launches in one market and not another, the difference in the two markets' trends is an estimate of its effect. The assumption is that both markets would have moved in parallel without the launch, and the check is whether they did so before it.

Adjust, and state what you assumed. Splitting by deployment status above is the simplest version of this. It works when the confounder is known, measured and complete, and every one of those three words is doing work.

None of these is as good as randomisation, because each rests on a claim that cannot be verified from inside the dataset. That is not a reason to avoid them. It is a reason to write the claim down next to the number, so that a reader who doubts it knows exactly which assumption to argue with.

The version worth carrying

Not "correlation is not causation", which stops the conversation. Better: a correlation is a fact about the data, and a causal claim is a fact about the world, and getting from one to the other requires an argument that the data cannot supply on its own.

That argument is usually a randomised experiment, and when it is not, it is a written statement of which variables cause which and why the ones that matter were measured.

The next article takes the fourth structure above, chance, and works through what a p-value actually promises, which is considerably less than it is usually asked to deliver.

References

Causal Inference in Statistics: A Primer. Judea Pearl, Madelyn Glymour and Nicholas P. Jewell, Wiley, 2016.
The Book of Why, The New Science of Cause and Effect. Judea Pearl and Dana Mackenzie, Basic Books, 2018.
Simpson's Paradox. Stanford Encyclopedia of Philosophy, 2021.