Language models predict the next token

A support engineer pastes a customer email into a chat interface and asks for the order number. The model answers ORD-4471-K. The format is right, the prefix matches every order the company has ever issued, and the check digit convention is correct. The number is not in the email. It is not in the database either.

Nobody trained the model to lie. The behaviour follows from what the objective asks for, and once the objective is clear the failure stops being surprising and starts being predictable.

The objective, exactly

A language model is a function from a sequence of tokens to a probability distribution over the next token. That is the whole contract. Given The invoice was paid on, it produces a number for every token in its vocabulary, and those numbers sum to one.

Take a toy vocabulary of five tokens and suppose the network's final layer emits these scores, called logits, for the position after The cat sat on the:

• mat 4.2
• floor 3.1
• roof 1.8
• table 1.5
• idea 0.4

Logits are not probabilities. They are turned into probabilities by exponentiating each one and dividing by the total, the same softmax used for multi-class output in Logistic regression and decision boundaries. Exponentiating gives 66.7, 22.2, 6.05, 4.48 and 1.49, which total 100.9. So the probabilities are 0.661, 0.220, 0.060, 0.044 and 0.015.

Two things follow from that arithmetic and both matter later. Every token gets a non-zero probability, including idea, because the exponential of any finite number is positive. And the gap between the top two is a ratio of exponentials, so a logit difference of 1.1 becomes a probability ratio of almost exactly 3 to 1.

Training minimises the negative log of the probability the model assigned to the token that actually came next, averaged over an enormous number of positions. Assign 0.661 to the correct token and the loss at that position is 0.414. Assign 0.015 and it is 4.20. The gradient descends that loss through the same chain rule worked through in Backpropagation worked by hand, and the architecture doing the computing is the stack described in The transformer block piece by piece.

Notice what the loss never mentions. It does not mention truth, facts, sources, or the customer's database. It rewards assigning high probability to the token that followed in the training text. ORD-4471-K is exactly what a well-fitted model should produce after "the order number is", because in the text it learned from, that position is usually followed by something with precisely that shape.

The token is not the word

The unit is not English. Sennrich, Haddow and Birch introduced subword segmentation to solve a translation problem: rare and unknown words could not be handled by a fixed vocabulary, so they proposed encoding them as sequences of subword units, evaluating a segmentation based on the byte pair encoding compression algorithm. Their measured gain was 1.1 and 1.3 BLEU over a back-off dictionary baseline on the WMT 15 English-German and English-Russian tasks.

The consequence for a modern model is structural. A vocabulary of perhaps 100,000 entries covers all text because anything unfamiliar decomposes into pieces. Common words are single tokens. Rare ones are several. A long number is usually several. An unusual surname may be four.

This is the source of a whole family of confusing behaviours. Ask a model to reverse the letters of a word and it struggles, because it never saw letters. Ask it to add two ten-digit numbers and the digits were grouped into chunks that do not align with place value. Count tokens rather than words when estimating cost, because the ratio is not one and it is worse for code, for non-English text, and for anything with unusual formatting.

Why this beat counting

Before neural models, the standard approach counted. Take the previous two words, look up what followed them in a large corpus, and use those frequencies. It works until the two words never co-occurred, which for any realistic vocabulary is most of the time.

Bengio, Ducharme, Vincent and Jauvin named the problem and the fix in 2003. The difficulty, they wrote, is intrinsic because of the curse of dimensionality: a word sequence in the test set is likely to differ from every sequence seen in training. Their proposal was to learn a distributed representation for words, so that each training sentence informs the model about an exponential number of semantically neighbouring sentences. Their neural approach improved significantly on state-of-the-art n-gram models and could take advantage of longer contexts.

That is the hinge the entire field turns on. A counting model treats cat and dog as unrelated symbols. A model with learned representations places them near each other, so a sentence about one adjusts predictions about the other. What "near" means is the subject of Embeddings are coordinates for meaning.

Generation is a second decision

Predicting a distribution and producing text are different steps, and confusing them causes a lot of misplaced blame.

Given the distribution, something has to choose. Always taking the highest-probability token is greedy decoding. Sampling in proportion to the probabilities is the other extreme. Temperature sits between them: divide every logit by a constant before the softmax. With the five logits above and a temperature of 0.5, every logit doubles, the top gap of 1.1 becomes 2.2, and mat rises from 0.661 to about 0.890. With a temperature of 2, the gap halves and mat falls to about 0.437.

Holtzman, Buys, Du, Forbes and Choi documented what happens when likelihood is pushed too hard. Their counter-intuitive observation was that although likelihood works well as a training objective, using it as a decoding objective leads to text that is bland and strangely repetitive. They found that decoding strategies alone can dramatically affect quality even from the same model, and proposed nucleus sampling: draw from the dynamic nucleus of the distribution, which allows diversity while truncating the less reliable tail.

Two practical consequences. A model that repeats itself may not need retraining; it may need a different decoder. And a model that produces different answers to the same question is not necessarily unstable, because temperature above zero makes variation the design, not the defect.

What the objective does imply

Scale improves it, predictably. Kaplan and colleagues studied empirical scaling laws for cross-entropy loss and found that loss scales as a power law with model size, dataset size, and training compute, with trends spanning more than seven orders of magnitude. They also found that architectural details such as width or depth have minimal effects within a wide range, and that larger models are significantly more sample-efficient, so compute-efficient training means training very large models on relatively modest data and stopping well before convergence.

Read that last clause slowly. It says the standard early-stopping advice from Train, test, and the lie of a single score has a different shape here: you stop not because the model has begun to overfit but because the compute is better spent on a bigger model.

Scale also produced a capability nobody wrote an objective for. Brown and colleagues showed that scaling greatly improves task-agnostic few-shot performance, with a model of 175 billion parameters applied without any gradient updates or fine-tuning, sometimes reaching competitiveness with prior fine-tuned approaches. Examples placed in the prompt condition the distribution. Nothing about the weights changes.

What it does not imply

The objective is silent on truth, and silence is not an oversight that a larger model repairs.

A model has no representation of a claim's source. Fluent text and accurate text look identical to a next-token loss, because both are simply sequences that occurred. When the training text contained a fact many times, the fact is easy to predict and usually comes out right. When it appeared twice, or never, the model still produces the shape of an answer, because a shape is exactly what it learned.

The model also has no access to your data unless you put it in the prompt. That is the whole premise of Retrieval-augmented generation, honestly, and it is why the support engineer's problem was never fixable by a better model.

Confidence deserves separate care. The probability the model assigns to a token is a claim about text continuation, not about the world. A sentence can be assigned 0.98 and be false, and the argument in What probability buys you applies here with more force than anywhere else in this series, because the output is prose rather than a number and there is no scale beside it to check.

Practical rules that follow

• Ask what the prompt contains, not what the model knows. If a fact is not in the context and not overwhelmingly common in text, treat the answer as generated rather than retrieved.
• Measure tokens, not words, for cost and context limits. Test with your own worst-case input: code, tables, and non-English text all inflate the ratio.
• Separate model problems from decoder problems. Repetition, blandness, and run-to-run variation are usually decoder settings.
• Do not read fluency as a signal of correctness. The training objective made fluency free.
• When an answer must be checkable, design the system so the answer arrives with the text it came from. That constraint shapes everything in the rest of this series.

The next article takes the piece this one deferred: what those learned representations actually are, why arithmetic on them appears to work, and where the geometry stops meaning what people say it means.

References

1. A Neural Probabilistic Language Model. Yoshua Bengio, Rejean Ducharme, Pascal Vincent and Christian Jauvin, Journal of Machine Learning Research volume 3, 2003.
2. Neural Machine Translation of Rare Words with Subword Units. Rico Sennrich, Barry Haddow and Alexandra Birch, arXiv, 2015.
3. The Curious Case of Neural Text Degeneration. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes and Yejin Choi, arXiv, 2019.
4. Scaling Laws for Neural Language Models. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu and Dario Amodei, arXiv, 2020.
5. Language Models are Few-Shot Learners. Tom B. Brown and colleagues, arXiv, 2020.