Scaling laws and the Chinchilla correction

Most articles about research explain a result that turned out to be right. This one is more useful, because it explains a result that a lot of people read slightly wrong, spent an enormous amount of money acting on, and then corrected in public with an experiment.

The question underneath is narrow and entirely practical. You have a fixed amount of compute. You can spend it on a bigger model, or on more training tokens for a smaller model, and you cannot have both. Which buys more?

Two knobs and one budget

A useful approximation, and the one the compute-optimal work is built on, is that the cost of training a transformer is roughly proportional to the number of parameters multiplied by the number of training tokens. Call those N and D. Doubling either one roughly doubles the bill.

That single fact is what makes this a real trade rather than a preference. If the budget is fixed, N times D is fixed. Pushing N up means pulling D down by the same factor.

Kaplan and colleagues, in 2020, established that model performance improves smoothly and predictably as you scale, and that the improvement follows power laws rather than hitting a wall. That result held up and it is genuinely useful: it says that loss keeps falling as you add parameters, data and compute, so a plan to spend ten times more compute has a forecastable payoff instead of an unknown one.

What circulated afterwards was a rule of thumb about how to split a budget, and the version that spread put most of the extra budget into parameters. Models grew enormously while the token counts they were trained on grew much more slowly. GPT-3 arrived with 175 billion parameters trained on the order of 300 billion tokens, and larger models followed on comparable amounts of data.

The experiment that settled it

In 2022, Hoffmann and colleagues did the thing that resolves this kind of disagreement. Rather than reinterpreting existing curves, they trained "over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens", varying the split deliberately, and measured which split won at each budget.

Their conclusion is one sentence: "for every doubling of model size the number of training tokens should also be doubled".

The two quantities should grow together. Not parameters first, with data trailing. Together, in step.

To show that this is not merely a curve-fitting preference, they built a model to the recipe. Chinchilla has 70 billion parameters and was trained on four times more data than Gopher, DeepMind's own earlier 280 billion parameter model. On a comparable compute budget, the smaller model won. Chinchilla outperforms Gopher, GPT-3 at 175 billion, Jurassic-1 at 178 billion, and Megatron-Turing NLG at 530 billion, and it "reaches a state-of-the-art average accuracy of 67.5% on the MMLU benchmark, greater than a 7% improvement over Gopher".

A model with one quarter of Gopher's parameters beat it, and beat a model more than seven times its size.

The arithmetic, checked

The claim that these two runs cost about the same is the load-bearing part, so it is worth doing rather than accepting.

Take N times D for each, in units of parameters times tokens.

• Gopher: 280 billion times 300 billion. That is 2.8 times 10 to the 11, multiplied by 3 times 10 to the 11. Multiply the front numbers, 2.8 times 3, which is 8.4. Add the exponents, 11 plus 11, which is 22. So 8.4 times 10 to the 22.
• Chinchilla: 70 billion times 1.4 trillion. That is 7 times 10 to the 10, multiplied by 1.4 times 10 to the 12. Multiply the front numbers, 7 times 1.4, which is 9.8. Add the exponents, 10 plus 12, which is 22. So 9.8 times 10 to the 22.

Two budgets within about 17 percent of each other, and the smaller model wins clearly rather than narrowly. That gap is not what is producing the result.

Now check Chinchilla against the rule it was built from. Twenty tokens per parameter, at 70 billion parameters, is 70 billion times 20, which is 1.4 trillion. That is exactly the token count used, which is the point: Chinchilla is the rule made into a model.

Run the same check on GPT-3. Twenty tokens per parameter at 175 billion parameters calls for 175 times 20, which is 3,500 billion, or 3.5 trillion tokens. It was trained on roughly 300 billion. By this rule it received under a tenth of the data its size called for, and the honest way to say what that means is that a large part of what was paid for in parameters was never trained into them.

Using the rule forwards

The comparisons above start from two models that already exist. In practice the question arrives the other way round: a budget is agreed first, and the shape has to be chosen. The rule answers that directly.

Suppose the budget allows a parameter-token product of 10 to the 22, which is roughly an eighth of the Gopher run above. Write D as 20 times N, since that is the ratio being aimed at, and substitute:

• N times D is N times 20N, which is 20 N squared.
• Set 20 N squared equal to 10 to the 22, so N squared is 5 times 10 to the 20.
• The square root of 5 times 10 to the 20 is about 2.2 times 10 to the 10.

That gives roughly 22 billion parameters, and 20 times that is roughly 440 billion tokens. Two numbers, from one budget and one ratio, and the arithmetic is simple enough to redo on a napkin when somebody proposes a model shape in a meeting. Anything far off that line is a decision to be undertrained or oversupplied with data, which is fine when made deliberately and expensive when made by default.

Why balance wins

The reason is worth getting right, because there is a tempting explanation that is wrong.

The wrong one says the larger model must have passed over its data several times and learned less each time from a staler dataset. That is not what happened. These runs are close to a single pass over their token budgets, and repetition is not the mechanism.

The real mechanism is that the loss has two separate ways to be bad, and a fixed budget has to pay down both. One term shrinks as parameters increase, and it represents the model lacking the capacity to express what the data contains. The other shrinks as tokens increase, and it represents the model not having seen enough text to pin down the parameters it already has. Push all the budget into parameters and the second term dominates: an enormous model with too few tokens has capacity it has no evidence to fill. Push it all into tokens and the first term dominates: a tiny model reads a library and cannot represent what it read.

The minimum total loss sits where neither term is left large, and what Hoffmann and colleagues measured is where that point actually falls. It falls near the middle, which is why the recommendation is that N and D double together.

An analogy that survives contact with the mathematics: parameters are the size of the notebook and tokens are the time spent studying. A vast notebook filled in for one afternoon is worse than a modest notebook filled in over a year, at the same total cost. Neither an empty notebook nor an unread library is where you want to be.

What this changed, and what it did not

The practical consequence is that a great many models trained before 2022 are undertrained for their size, GPT-3 among them, and that the same compute spent in the Chinchilla shape would have produced a stronger model. That is a statement about allocation, not about whether those models were interesting. The finding in GPT-3 and learning from the prompt, that a large model learns a task from examples in its context with no weight update, is not affected by how its budget was split.

Two limits on the rule are worth carrying.

It is an empirical summary, not a law. It describes where the optimum sat for the model family, data and budgets that were tested. Twenty tokens per parameter is a starting point for a plan, not a constant of nature, and later practice has often pushed well past it deliberately.

It also optimises for the wrong thing about half the time. Compute-optimal means cheapest to reach a given loss during training. It says nothing about what a model costs to run afterwards, and a model that will serve millions of requests is worth making smaller than the compute-optimal point and training for longer than the rule suggests, because inference cost scales with parameters and is paid forever. Training happens once. The rule answers the training question only, which is a distinction Training a network without wasting a month treats from the practical side.

The part worth copying

A widely held belief about how to spend hundreds of millions of dollars of compute turned out to be a misreading, and it was corrected by somebody training 400 small models and publishing the curve.

That is a cheaper experiment than it sounds. The models are small, the sweep is systematic, and the answer transfers to budgets far larger than any single run in the study. The alternative, which the field ran for two years, was to infer the answer from results that had not been designed to test it.

When a decision rests on an assumption nobody has measured directly, the measurement is usually less expensive than the decision. The next article turns to a different kind of correction, one where the training signal stops being text at all and starts being human judgement.

References

1. Scaling Laws for Neural Language Models. Jared Kaplan and colleagues, arXiv, 2020.
2. Training Compute-Optimal Large Language Models. Jordan Hoffmann and colleagues, arXiv, 2022.
3. Language Models are Few-Shot Learners. Tom B. Brown and colleagues, arXiv, 2020.