GPT-3 and learning from the prompt

In 2020, Brown and colleagues published a paper about a model with 175 billion parameters, described as ten times larger than any previous non-sparse language model. The model was called GPT-3. The paper made a specific claim: a model large enough could solve new tasks without any weight updates at all, using only text examples placed in its context. The mechanism was not training. It was conditional probability.

This was surprising to many in the field. At the time, the dominant approach to applying a pre-trained language model to a new task was fine-tuning: taking the base model, collecting examples of your specific task, and running gradient descent to update the weights. This process was necessary, everyone thought. A general model needed to be adapted to your specific problem, and that adaptation happened through weight updates.

GPT-3 suggested something different. The model was large enough, and trained on enough data, that it could infer the task from context alone. No backpropagation needed. No parameter updates. Just pattern matching against patterns learned during training. This finding changed what practitioners tried first when given a new problem.

What does few-shot mean

The paper introduces three terms carefully. Zero-shot means the model sees a task description in text, nothing else. One-shot means it sees one example of the task and its correct answer. Few-shot means it sees several, as many demonstrations as the context window will hold, which in the paper is usually on the order of ten to one hundred rather than a handful. None of the three involve training.

A concrete example: suppose the task is to translate English to French. In zero-shot mode, you write to the model "Translate English to French:" and then "cheese", and it outputs "fromage". It has never been shown a single example of an English-French pair, only the instruction in natural language. The model relies on whatever it learned about French vocabulary and English-to-French correspondences during its training on general internet text.

In one-shot mode, you write "Translate English to French. cherry: cerise. cheese:" and it outputs "fromage". Now the model has one example to work from. It shows the pattern: a word in one language, a colon, the same word in the other language. One instance of the pattern might be enough for the model to reproduce it, because it has seen similar patterns billions of times during training. You are asking it to do what it has already learned to do, just with a specific task instance in front of it.

In few-shot mode, you write "Translate English to French. apple: pomme. cherry: cerise. dog: chien. cheese:" and it outputs "fromage". The model now sees the pattern three times before you ask for the answer. The repetition makes the structure unmistakable. Each example is another signal confirming that the model should output a French word following a colon.

The key word in all three is "shown". Nothing is learned in the weights. The examples are not used to update parameters. They are context. They are tokens in the input, and the model processes them like any other input tokens, and out comes a probability distribution over the next token.

The paper tests zero-shot, one-shot, and few-shot performance across many tasks and many models of different sizes. The general pattern is that larger models do better at all three, and more examples help more on harder tasks. But the baseline is always there: zero-shot works to some degree, even without examples. One-shot works better. Few-shot works better still. None of them involve weight updates.

What actually happens inside the model

Language models predict the next token explains that a language model computes a probability distribution over the next token given everything that came before. GPT-3 does the same thing, just with more parameters and more data in its training set.

When you put examples into the context, they become tokens, just like any other text. The model has seen billions of examples during training: document text, code, conversational examples in forums, and countless other patterns. This includes many examples of tasks that look like the ones you are now asking it to solve.

This is not learning. This is conditioning. The distinction matters: learning means the weights change, and conditioning means the same weights receive different inputs.

Put numbers on that shift, because it is easy to say and easy to hand-wave. Suppose the prompt ends with "cheese:" and the model is choosing between three continuations: the French word fromage, an echo of the English word cheese, and a newline. With no examples in the context, suppose the logits come out at fromage 1.2, cheese 1.5, newline 1.0. Exponentiate each:

exp of 1.2 is 3.32
exp of 1.5 is 4.48
exp of 1.0 is 2.72

Those sum to 10.52, so the probabilities are fromage 0.316, cheese 0.426, newline 0.259. The most likely single continuation is to echo the English word back, which is not the task.

Now put three translated pairs in front of it and change nothing else. Not one weight has moved. The context now carries the pattern, and the logits move with it: fromage 3.4, cheese 1.1, newline 0.9.

exp of 3.4 is 29.96
exp of 1.1 is 3.00
exp of 0.9 is 2.46

Those sum to 35.42, giving fromage 0.846, cheese 0.085, newline 0.069.

The probability on the right answer went from 0.316 to 0.846, and the arithmetic that produced it used exactly the same matrices in both cases. That is the entire phenomenon. The examples did not teach the model French. They told it which of the things it already does is the thing being asked for.

Here is how to think about the arithmetic. Consider a single attention head inside the model, as described in Attention is a lookup you can learn. The head has learned query, key, and value matrices. These matrices are weights. They were set during training on 300 billion tokens from the web. They do not change when you prompt the model. GPT-3 is an autoregressive model, meaning it reads left to right and can only use context that comes before the current position, unlike BERT and the case for reading both ways which reads both directions.

When you provide a few-shot prompt, each word in that prompt becomes a token, and each token has an embedding. The embeddings flow through the learned matrices. The query matrix weights are applied to the current position. The key matrix weights are applied to all positions. The dot products measure similarity. The softmax produces attention weights. The weighted sum of values is computed.

All of this is arithmetic done with frozen weights. The model is not updating its weights. It is not computing gradients. It is only computing what probability it assigns to each possible next token given the input you provided.

The paper itself states it: models are applied "without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction". No backpropagation. No parameter changes. Only conditional probability.

The mechanism is not mysterious. It is the same thing the model does every time you ask it anything. You provide context as tokens. The context propagates through the learned weights. A probability distribution over the next token emerges. You sample from that distribution, or pick the highest probability token. The model outputs text that statistically matches the patterns it learned in training, narrowed by the examples you supplied.

The size threshold

One remarkable thing about the results is that this behavior becomes strong only at large scales. The paper tests models of various sizes on various benchmarks. At 125 million parameters, few-shot performance is poor, barely better than zero-shot. At 350 million parameters, there is improvement, but still not reliable. At 175 billion parameters, the in-context learning works well on many tasks.

This suggests a threshold, and the tempting explanation for it is wrong. The models in the family were trained on the same data, so the small ones did not see fewer examples of the pattern. What they lacked was the capacity to hold it. A pattern like "read the format of the preceding examples and continue it" has to be represented somewhere in the weights, and a smaller model spends its capacity on more immediately useful regularities. The paper reports the effect rather than explaining it, and the honest summary is that in-context learning appears as models grow, without a settled account of why the curve bends where it does.

What the paper did not show

The abstract is honest about the boundaries. It explicitly names "some datasets where GPT-3's few-shot learning still struggles". It also lists "some datasets where GPT-3 faces methodological issues related to training on large web corpora".

The second point is the contamination problem, and it is the one that should change how you read every number in the paper. When a model trains on a web crawl, parts of public benchmarks are in the crawl. Question and answer pairs from reading comprehension sets, translation pairs, quiz questions and their solutions are all published on the open web. If a benchmark was in the training data, a strong score is not in-context learning. It is recall.

Here is a concrete example. Suppose a benchmark is published and becomes widely discussed on blogs, academic websites, and forums. Someone writes a blog post titled "Solving benchmark problem 1: the answer is X". A web crawler visits the blog, and GPT-3's training data now includes that blog post. Later, when you test GPT-3 on that benchmark, the model is not doing few-shot learning in context. It is recalling something it has already seen. The prompt you provide is a trigger for memory, not an example for reasoning.

The practical test is not retraining, which would cost as much as the original run. It is to search the training corpus for overlap with each benchmark, typically by looking for matching runs of consecutive words, then split the benchmark into a clean part and a contaminated part and compare the scores on each. A large gap between the two is evidence of recall rather than generalisation.

The paper did not solve this problem. It flagged it. For some benchmarks, they tested subset versions that are unlikely to be on the web. For others, they had to simply acknowledge that they could not be certain whether the model was generalizing in context or recalling training data. Brown and colleagues are explicit: they cannot rule out the possibility that performance on some benchmarks reflects training data contamination rather than true few-shot learning.

This is not a fatal flaw in the work. Few-shot learning is real and powerful, and the paper provides evidence that it works even on tasks where contamination is unlikely. But it is honest science to note that on other benchmarks, the true answer is unclear. You cannot fully trust the numbers without knowing whether the benchmark was in the training set.

Why in-context learning works differently than fine-tuning

The key difference between few-shot prompting and fine-tuning is the number of weight updates. Suppose you fine-tune GPT-3 on 10,000 examples of a translation task, running gradient descent for several passes through the data. Each example contributes a gradient, which adjusts the weights. After one pass through all 10,000 examples, the weights have changed. After multiple passes, they have changed more. The model becomes specialized to that task. The weights that were general-purpose weights are now tuned to translation.

Few-shot prompting does not do this. It provides a handful of examples in context and hopes that the pattern is clear enough from context alone. The weights do not change. The hidden states change because the input changes, but the weights that compute those hidden states remain frozen.

This makes few-shot prompting less powerful on some tasks. If your task is very specific or very different from what the model saw in training, you need the weights to shift. A few examples in context are not enough; you need gradient descent to update the parameters.

It makes few-shot prompting more flexible on others. You can switch tasks instantly without retraining. Try zero-shot first, then add one example, then add three. You get results in seconds. Fine-tuning takes hours or days. If you need to solve five different problems, fine-tuning forces you to train five separate models. Prompting lets you reuse the same model.

The trade-off is real. Few-shot prompting is a convenience that works well when the task is close to something the model has seen before. Fine-tuning is more powerful but slower and more expensive.

A later paper, Fine-tuning against prompting, a real comparison, tests both approaches head to head on many tasks. That work gives you numbers on which is better when, so you can make an informed choice instead of guessing.

Beyond text generation

One curious capability the paper demonstrated is that GPT-3 could generate text that human evaluators had difficulty distinguishing from human-written text, at least for news articles. This is remarkable not because it proves the model "understands" what it is writing, but because it shows the model has learned to produce syntactically correct, coherent, and contextually plausible text from a huge amount of exposure to human writing.

This too comes from conditioning. If you prime the model with a news headline and the word "published", the conditional distribution over the next token has been shaped by billions of examples in its training data where news articles follow headlines. The model is a lookup table compressed into parameters. Conditioning on new context shifts what gets looked up.

Why this matters

The finding that in-context learning works transforms what you can do with language models. Before GPT-3, you needed to fine-tune a model on your specific task, which meant collecting labeled data, setting up training infrastructure, and waiting for the process to complete. Few-shot prompting requires only writing text in a specific format and sending it to an API.

This does not mean prompting is always better. As the paper admits, on some tasks it still struggles. But it made a category of problems suddenly accessible to people without machine learning infrastructure, and that shift proved genuinely important in practice.

The mechanism is not mysterious once you know language models predict the next token. The model is doing something it was already built to do: computing the most likely next token given the inputs. The inputs now include your examples, so the output reflects what the model learned about patterns similar to your examples.

That is the entire story. The scale simply has to be large enough.

References

Language Models are Few-Shot Learners. Tom B. Brown and colleagues, arXiv, 2020.
Scaling Laws for Neural Language Models. Jared Kaplan and colleagues, arXiv, 2020.
BERT, Pre-training of Deep Bidirectional Transformers for Language Understanding. Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova, arXiv, 2018.