Fine-tuning against prompting, a real comparison

A team has 400 support tickets labelled into eleven categories by the people who handle them. The current classifier is a prompt: a page of instructions, the eleven category definitions, six worked examples, and the ticket. It runs to about 900 tokens before the ticket is added, and it is right roughly four times in five.

Someone proposes fine-tuning. Someone else says 400 examples is not enough data. A third person says the prompt is already too long and every request pays for it. The meeting ends without a decision, which is the correct outcome, because nobody in the room has yet said what either option would change.

Two different objects

Prompting changes the input. The weights are fixed, and the instructions and examples condition the distribution over the next token at generation time. Brown and colleagues established how far that goes: scaling up language models greatly improves task-agnostic few-shot performance, and their 175 billion parameter model was applied without any gradient updates or fine-tuning, sometimes reaching competitiveness with prior state-of-the-art fine-tuning approaches.

Fine-tuning changes the weights. You continue training on your examples, so the behaviour is baked in and the prompt can shrink to almost nothing.

Everything else follows from that difference, and most of it is not about accuracy.

The arithmetic nobody does before the meeting

Take the 900-token instruction block. Suppose the service handles 20,000 tickets a month and the average ticket is 300 tokens.

With the prompt, each request sends 1,200 input tokens, of which 900 are the same every time. Across a month that is 18 million tokens of instructions, against 6 million tokens of actual ticket. Three quarters of what you pay for is the same block of text repeated 20,000 times.

With a fine-tuned model, the instruction block goes away. A short system line plus the ticket is perhaps 340 tokens, so the monthly volume drops from 24 million to 6.8 million. That is a 72 percent cut in input tokens, and a proportional cut in latency for the part of the response time driven by processing the input, which matters for the reasons set out in Putting a model behind an interface.

Against that, fine-tuning costs a training run, a per-model hosting charge on most platforms, and an evaluation set you have to build anyway. The break-even is a volume, not an opinion, and it is usually reached earlier than people expect. Prompt caching changes these numbers substantially where it is available, so measure your own provider's behaviour rather than assuming either result.

Do the arithmetic before the meeting and the discussion becomes short.

What fine-tuning is genuinely good at

Format and behaviour. If you need output that always parses, always uses your eleven category names and nothing else, and never adds a friendly preamble, this is the tool. A few hundred examples teach a format far more reliably than any amount of instruction, because you are showing rather than describing.

Style and register. The tone of your organisation, the vocabulary of your industry, the conventions of your document type. These are almost impossible to specify in prose and trivial to demonstrate.

Shrinking the model you need. This is the result worth internalising. Ouyang and colleagues found that outputs from their 1.3 billion parameter InstructGPT model were preferred by human labellers to outputs from the 175 billion parameter GPT-3, despite having 100 times fewer parameters, with improvements in truthfulness and reductions in toxic output generation and minimal performance regressions on public natural language processing datasets.

A hundredfold reduction in parameters, and people preferred the smaller model. Not because it knew more, but because it had been trained to do the thing being asked. On a narrow task with real examples, a small fine-tuned model very often beats a large general one, and it costs a fraction as much to serve.

What fine-tuning is bad at

Adding facts. This is the expensive misunderstanding. Fine-tuning on a thousand documents does not install those documents in a way you can look up or cite. It adjusts a distribution over text. The model becomes fluent in the register of your documents while remaining unable to tell you which one said what, and it will produce confident answers in that register about things none of them said. The material in Retrieval-augmented generation, honestly exists because this does not work.

Anything that changes weekly. Prices, policies, staff, stock. A weight update is a deployment. If the fact has a shelf life shorter than your release cycle, it belongs in the context, not the weights.

Small, messy label sets. Four hundred examples labelled inconsistently by five people will teach the model the inconsistency faithfully. The argument in The data comes first is not softened by the model being large; it applies exactly as written. Measure agreement between your labellers before you spend anything on training.

Keeping everything else intact. Training narrows. A model tuned hard on eleven ticket categories gets better at those eleven and quietly worse at the adjacent things it used to handle, which for this team meant summarising a thread and drafting a reply. Nobody notices, because the evaluation set was built from the classification task and measures only that. Ouyang and colleagues describe checking for exactly this, reporting minimal performance regressions on public datasets alongside their gains, and that pairing is the practice to copy. Keep a held-out set for the capabilities you are not training, run it before and after, and treat a drop there as a cost of the fine-tune rather than a surprise.

The cost objection is much weaker than it was

The standard reason to avoid fine-tuning used to be that it meant a full copy of the model per task, and a training run nobody could afford.

Hu and colleagues removed most of that. Full fine-tuning retrains all parameters, they observe, and deploying independent instances of a fine-tuned 175 billion parameter model is prohibitively expensive. Their method freezes the pre-trained weights and injects trainable rank decomposition matrices into each layer of the transformer architecture, the architecture assembled in The transformer block piece by piece. Compared to fine-tuning that model with Adam, they report reducing the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times, performing on par with or better than fine-tuning on several model families, with higher training throughput and, unlike adapters, no additional inference latency.

Dettmers, Pagnoni, Holtzman and Zettlemoyer pushed the memory cost further. Their approach reduces memory usage enough to fine-tune a 65 billion parameter model on a single 48GB GPU while preserving full 16-bit fine-tuning task performance, and their resulting model family reached 99.3 percent of the performance level of ChatGPT on their evaluation while requiring only 24 hours of fine-tuning on a single GPU.

Read those two results together and the meeting changes shape. Fine-tuning is no longer a capital decision. It is closer to a build step, which shifts the real question from cost to maintenance.

The maintenance question, which decides most cases

A prompt is text. Anyone on the team can read it, change one line, and see the effect in a minute. It lives in version control next to the code, and a bad change is reverted in seconds.

A fine-tuned model is an artefact produced by a pipeline. To change its behaviour you need the training data, the script, the environment, and someone who remembers how it was run. Six months later the person who trained it has changed teams, and the question "why does it classify escalations as billing" has no answer that anyone can read.

That asymmetry decides more real projects than accuracy does. It is not an argument against fine-tuning; it is an argument for treating a fine-tune as a system with a repository, a pinned dataset version, a rebuild script that works from scratch, and a written record of what it was trained to do. The habits in Reading a model you did not build are the same habits, applied to the model you will not remember building.

A decision rule you can use

Work down this list and stop at the first line that matches.

1. The task needs facts from your documents, and answers must be checkable against a source. Retrieval, not fine-tuning. Fine-tuning will not put the documents anywhere you can cite from.
2. The task is unclear, the requirements are still moving, or nobody has 200 clean examples. Prompt. You cannot fine-tune towards a target you have not defined, and building the examples defines it.
3. The output format matters more than the content, or the tone must match your organisation. Fine-tune. This is the strongest case there is.
4. The prompt works but volume makes it expensive, or a smaller model would do if it knew the task. Fine-tune, and expect the cost arithmetic above to be the whole justification.
5. None of these. Prompt, and keep the examples you collect. They become the training set if you ever need one.

The two are not rivals. The common production shape is a prompt that reached 80 percent, an evaluation set built to prove it, retrieval added for the facts, and a fine-tune on the accumulated examples once the requirements stopped moving and the volume justified it. Each step is cheap because the previous one produced what it needed.

Every one of those steps assumes you can tell whether the system got better, and that assumption has been unexamined for four articles. The next one takes it apart.

References

1. Language Models are Few-Shot Learners. Tom B. Brown and colleagues, arXiv, 2020.
2. LoRA: Low-Rank Adaptation of Large Language Models. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang and Weizhu Chen, arXiv, 2021.
3. Training language models to follow instructions with human feedback. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike and Ryan Lowe, arXiv, 2022.
4. QLoRA: Efficient Finetuning of Quantized LLMs. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman and Luke Zettlemoyer, arXiv, 2023.