Chain of thought and what it does not prove

A language model predicts one token at a time, and each token costs computation. If you ask it a math question directly, it gives you an answer in one or two tokens. If you ask it to show its working, it writes intermediate steps, which takes more tokens, and more tokens mean more arithmetic operations have access to the answer. Wei and colleagues show what happens when you ask the largest models to think step by step.

A concrete example

Take a word problem: "Sarah had 15 apples. She gave 7 to her friend. How many does she have now?"

The direct approach is to prompt the model for the answer alone. The model sees the question and outputs "8", using one token for the final answer. It has done the arithmetic in the hidden computation that produced that token.

The intermediate-steps approach asks the model to show its working. The model might write:

"Sarah started with 15 apples. She gave 7 to her friend. That means she has 15 minus 7, which equals 8. So Sarah has 8 apples left."

That is on the order of thirty tokens rather than one. Each of them is produced by its own forward pass through the whole network, and each one is written back into the context that the next pass reads. The difference is not the complexity of the arithmetic. A language model does not compute 15 minus 7 by hand. The difference is the depth of computation available before the final answer.

A language model applies a fixed amount of computation per token: one forward pass through the network. Writing intermediate steps does not teach the model arithmetic. It gives the model more forward passes in which to arrive at an answer, and more forward passes mean better opportunity to resolve the problem.

Scale matters for the effect

On GSM8K, a set of grade school maths word problems, prompting "a 540B-parameter language model" with "eight chain of thought exemplars", meaning eight worked examples showing the desired format, "achieves state of the art accuracy on the GSM8K benchmark". The comparison worth noting is what it beat: a finetuned GPT-3 paired with a verifier, a separate model trained to check candidate answers. Eight examples in a prompt outscored a system that had been trained on the task and given a checker.

The effect is reported as substantial only at large scale. Smaller models in the same families gain little or nothing from being asked for intermediate steps, and on some tasks they do worse, because a small model asked to produce a chain of reasoning produces a chain of plausible sentences with an incorrect answer at the end of it. The gain shows up at the top of the size range tested.

No settled account explains why the gain arrives where it does. The usual story is that a larger model has the capacity to represent the intermediate steps well enough for them to be worth writing down, and that a smaller one spends its capacity elsewhere. That is a plausible description rather than a demonstrated mechanism, and it is worth holding loosely.

Claims about reasoning emerging only at large scale have been contested in the five years since publication, and the current picture is more subtle than an emergence threshold. Some capabilities do appear suddenly at larger scales, while others grow gradually. The straightforward reading of the Wei paper is that asking for reasoning steps helps at the scale of the largest models being tested, and the practical implication is that chain-of-thought prompting is a technique for large models specifically, not a universal improvement.

The critical honesty: outputs are not traces

Here is what the written steps are not. They are not a record of the computation that produced the answer. They are outputs of the model, generated by the same process that generates any token. A model can write correct reasoning steps and arrive at the wrong answer. It can write nonsensical reasoning and produce the correct answer. The text you see is what the model chose to emit, not a transcript of what went on inside.

The mechanism at work is nothing more than token budgeting. By writing intermediate steps, the model uses more tokens. Using more tokens means more opportunities for the network to update its internal state as it processes the problem. More depth of computation tends to solve harder problems better. But that depth is arithmetic computed in the hidden layers. The visible text is an output, not evidence of that arithmetic.

Why does this matter? Because it is tempting to read the written reasoning as an explanation of the answer. If the model writes "15 minus 7 is 8", you might think the model has explained its reasoning, and you might trust the answer more. But the text and the answer are two independent outputs from the same model. A model that has learned to write plausible-sounding reasoning steps can write them whether or not they reflect what actually happened during inference.

This is the same issue that arose with attention weights in the article on Attention is a lookup you can learn. Attention weights are evidence about the model, not an explanation of it. Reading them as an explanation is one of the more common mistakes made with these systems. Chain-of-thought reasoning has the same structure: the model outputs reasoning steps, and those steps are evidence about what the model is capable of emitting, not a transparent window into its computation.

What the paper actually showed

The contribution is concrete: asking for intermediate steps improves answer accuracy on mathematical reasoning benchmarks, at least for the largest models tested. That is useful. It does not mean the model is thinking. It does not mean the reasoning is real. It means that more tokens, and therefore more computation, help the model solve harder problems. The mechanism is not mysterious. It is the same mechanism that runs through all of transformer inference: deeper processing means better results for harder tasks.

The paper does not show that models were reasoning before and are now reasoning more. It shows that the 540-billion-parameter model gets better answers when you ask for steps. That is a practical finding about prompting strategy. It is valuable for building systems. It is not evidence that the steps the model writes are how the model thinks, and treating them as such is a common misreading.

The follow-up work has been mixed. Some results replicate cleanly. Others depend heavily on the specific task, the model size, and the quality of the examples provided. The picture of reasoning, emergence, and scaling has turned out to be more complicated than a simple scaling threshold, and the field has moved toward more modest claims about what intermediate-step prompting does and does not accomplish.

Why intermediate reasoning helps anyway

The key mental model remains true even without the mystique. A language model generates one token at a time, using one forward pass per token. Each forward pass is an opportunity for the network to update what it is computing toward. A problem that requires reasoning benefits from more forward passes. You do not need magic or emergence. You just need to understand that depth of processing matters for task difficulty, which has been true since the earliest work on deep learning and remains true here.

Consider a harder example to see why depth matters. Suppose you ask the model to solve a multi-step problem: "Alice has twice as many books as Bob. Bob has 3 more books than Carol. Carol has 5 books. How many books does Alice have?" A direct prompt gives the model one or two tokens to reach the answer. A chain-of-thought prompt lets the model write: "Carol has 5 books. Bob has 3 more than Carol, so Bob has 5 plus 3, which is 8 books. Alice has twice as many as Bob, so Alice has 2 times 8, which is 16 books." That is roughly 30 tokens. The model has gone through 30 forward passes instead of 2, and those 28 extra passes give it much better opportunity to hold intermediate values and combine them correctly.

When you ask the model to show work, you are not asking it to access some hidden reasoning faculty. You are asking it to emit more tokens before the answer, which buys it more depth. The model then outputs steps that, if it has learned the task well, tend to match what human reasoning looks like. The reasoning is not where the answer comes from. It is a byproduct of having written it down as part of the process of reaching the answer.

That distinction matters when Chain of thought builds on GPT-3, because GPT-3 was already good at mimicking reasoning in its outputs. The innovation was showing that this mimicry improves accuracy when you also ask for the final answer. That is a practical contribution. It does not license us to treat the mimicry as explanation.

It also matters for anyone building systems that rely on model reasoning, because the visible steps can mislead you about what the model is actually capable of. A model can be confident and wrong. It can be hesitant and right. The steps it writes are one form of confidence signal, but they are not truth. This is where work like Hallucination is a design problem becomes essential: the system itself has to verify or validate the answer, not trust that good-looking reasoning steps mean a good answer.

The most useful mental model is the simplest one: you are not unlocking hidden reasoning. You are changing the token budget and therefore the compute depth available to the model before it commits to an answer. Everything that follows flows from that fact. When you read the model's reasoning steps, you are not reading a window into its mind. You are reading text that the model chose to emit because it was trained to emit reasonable-sounding text when asked to show its work. The remarkable finding is that this text, whether or not it reflects true reasoning, correlates with correct answers at scale. That is the finding to build on, without misinterpreting it as something more profound.

References

Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. Jason Wei and colleagues, arXiv, 2022.
Emergent Abilities of Large Language Models. Jason Wei and colleagues, arXiv, 2022.
Are Emergent Abilities of Large Language Models a Mirage?. Rylan Schaeffer, Brando Miranda and Sanmi Koyejo, arXiv, 2023.
On the Opportunities and Risks of Foundation Models. Rishi Bommasani and colleagues, arXiv, 2021.