What a p-value actually promises

An experiment comes back with p equal to 0.03 and gets described as "97 percent likely to be real". That reading is wrong, and it is wrong in a direction that matters: the actual probability that the finding is false can be several times higher than the p-value suggests. The gap is not a subtlety. It is often the difference between shipping a change and not.

The definition, stated carefully

A p-value answers one question: if there were no real effect, how often would data at least this extreme turn up by chance?

Read the conditional. It assumes there is no effect and asks about the data. It does not assume the data and tell you about the effect, which is the thing anybody actually wants to know.

Written as probabilities, a p-value is the probability of data this extreme given no effect. The question of interest is the probability of no effect given this data. Those two are not interchangeable, and the difference between them is not a technicality.

An example that makes the asymmetry obvious. The probability that someone is holding an umbrella given that it is raining is high. The probability that it is raining given that someone is holding an umbrella is lower, and how much lower depends entirely on how often it rains where you are. Same two facts, different conditional, wildly different numbers. Reversing a conditional probability requires knowing how common the underlying thing is, and a p-value contains no such information.

Working the number that matters

Take a team running A/B tests on a product. Historically about one in ten of the changes they try has a real effect. That base rate is the missing ingredient.

Run 1,000 tests, at the conventional threshold of 0.05, with 80 percent power, which is the standard target and means a test has an 80 percent chance of detecting a real effect when one exists.

Of the 1,000 tests, 100 involve a real effect, which is the one in ten base rate.

Detected: 80 percent of 100, which is 80 true positives.
Missed: the remaining 20.

The other 900 involve no real effect, and the threshold decides what fraction of those slip through.

False positives: 5 percent of 900, which is 0.05 times 900, or 45.
Correctly negative: 855.

Now count the tests that came back significant: 80 plus 45, which is 125.

Of those 125 wins, 45 are not real. That is 45 divided by 125, which is 0.36.

Thirty-six percent of the significant results are false, in a setup using the standard 0.05 threshold and standard power. Not five percent. The 0.05 is the false positive rate among tests where nothing is happening, which is a different denominator from the one the reader cares about.

What changes the answer

The base rate does most of the work, which is why it deserves to be stated out loud before an experiment rather than after.

Suppose the team gets better at picking ideas and half of what they try is real.

500 real, 80 percent detected, giving 400 true positives.
500 null, 5 percent false, giving 25 false positives.
Significant results: 425. False share: 25 divided by 425, which is 0.059.

Six percent, against thirty-six. Same threshold, same power, same statistics. The only change is the quality of the hypotheses going in.

Now the other direction. A team mining a large dataset for whatever turns up, where perhaps one idea in a hundred is real.

10 real, 80 percent detected, giving 8 true positives.
990 null, 5 percent false, giving 49.5 false positives.
Significant results: 57.5. False share: 49.5 divided by 57.5, which is 0.861.

Eighty-six percent of the significant findings are noise. The arithmetic is unremarkable and the conclusion is not: an exploratory search over many weak hypotheses produces mostly false positives even when every individual test is run correctly.

This is the core of Ioannidis's argument about published findings, and it requires no misconduct anywhere. Every test in these three scenarios was performed properly.

The base rate is also where the previous article connects. A confounded comparison, of the kind worked through in Correlation, causation and the third variable, does not produce a large p-value. It produces a small one, because the correlation it rests on is genuinely there in the data. Significance testing checks whether a pattern could be chance. It has nothing at all to say about whether the pattern means what you think it means, and a confounded result will sail through it.

Power, and why underpowered studies mislead twice

Power is the probability of detecting a real effect when one exists. Dropping it does more damage than the obvious.

Repeat the first scenario, one in ten real, at 20 percent power, which is common in small studies.

100 real, 20 percent detected, giving 20 true positives.
900 null, 5 percent false, giving 45 false positives.
Significant: 65. False share: 45 divided by 65, which is 0.69.

Sixty-nine percent false, up from thirty-six, purely from a weaker test.

There is a second effect that is less obvious and worse. In an underpowered study, an effect can only clear the significance threshold if the noise happened to push it upward. So the effects that do get published from underpowered studies are systematically overestimated, sometimes by large factors. The finding is not merely more likely to be wrong; when it is right, the size is inflated. This is why a well-powered replication so often produces a smaller effect than the original, and why "it replicated but smaller" is the expected outcome rather than a puzzle.

Twenty tests, one finding

The counting error that produces most spurious results deserves its own arithmetic, because "we ran a few analyses" hides it well.

A single test at the 0.05 threshold has a 5 percent chance of a false positive when nothing is going on, so it has a 95 percent chance of correctly finding nothing. Run two independent tests and the chance that both come back clean is 0.95 times 0.95, which is 0.9025. So the chance of at least one false positive is 1 minus 0.9025, which is 0.0975, nearly ten percent.

Run twenty. The chance all twenty stay clean is 0.95 raised to the power of 20. Working it up: 0.95 squared is 0.9025, to the fourth is 0.9025 squared, which is 0.8145, to the eighth is 0.8145 squared, which is 0.6634, to the sixteenth is 0.6634 squared, which is 0.4401. Multiply by the fourth power for the last four tests: 0.4401 times 0.8145, which is 0.3585.

So the chance of at least one false positive across twenty tests is 1 minus 0.3585, which is 0.6415. Roughly a two in three chance of finding something significant when there is nothing there at all.

This is why testing several metrics, several segments, or several variants and reporting the one that worked is not a minor reporting choice. Four variants against five metrics is twenty tests, and at that point a significant result is the expected outcome of a null experiment rather than evidence against it.

The standard correction is to divide the threshold by the number of tests. Twenty tests at 0.05 divided by 20 gives a threshold of 0.0025 per test, which restores the overall false positive rate to about 5 percent. It is conservative and there are better methods, but the crude version is far better than not counting at all. The number that has to be corrected for is the number of tests run, not the number reported, which is why the count has to be recorded as the work happens.

What p equals 0.03 does and does not tell you

It says: if nothing were going on, data this extreme would appear about 3 percent of the time.

It does not say the hypothesis is 97 percent likely to be true. It does not say the effect is large or that it matters. With a large enough sample, an effect too small to be worth acting on will produce a tiny p-value, because the p-value combines effect size and sample size and reports neither.

It also says nothing about whether the experiment was well designed. A p-value computed on a biased sample is a precise statement about a biased sample. The arithmetic cannot see the bias.

What to report instead

Report the effect size with a confidence interval, and lead with it.

"Conversion rose by 1.2 percentage points, 95 percent interval from 0.3 to 2.1" is a more useful sentence than "the result was significant at p equal to 0.03". It gives the size, the direction and the uncertainty, and it lets a reader decide whether the smallest plausible value in that range is worth the engineering effort.

Two habits that follow from the arithmetic above:

State the base rate before running the test. If the honest answer is that most ideas in this area do not work, then a single significant result is weak evidence, and everybody should know that before the number arrives rather than after.

Count every test you ran. Twenty tests at a 0.05 threshold produce about one significant result from pure chance, since 20 times 0.05 is 1. Reporting only the one that worked converts a null result into a finding. This is the same counting error as the exploratory scenario above, and correcting for the number of comparisons is the standard fix.

The version worth carrying

A p-value is a statement about data under an assumption, not a statement about the world. Converting it into a probability that a finding is real requires a base rate, and once you supply a realistic one, the conventional threshold turns out to be far weaker evidence than its reputation suggests.

The practical consequence is not to abandon significance testing. It is to stop treating a single p-value below 0.05 as a decision, and to report the size of the effect and its uncertainty as the primary result, with the p-value as a footnote.

The next article turns to the sample itself, and to the kinds of missing data that no amount of statistical care can repair after the fact.

References

The ASA Statement on p-Values: Context, Process, and Purpose. Ronald L. Wasserstein and Nicole A. Lazar, American Statistical Association, 2016.
Why Most Published Research Findings Are False. John P. A. Ioannidis, PLoS Medicine, 2005.
Simpson's Paradox. Stanford Encyclopedia of Philosophy, 2021.