LoRA and fine-tuning without the bill

Fine-tuning a large language model usually means updating all of its weights. A 175-billion parameter model like GPT-3 requires 175 billion floats in memory, plus gradients, plus optimizer state. The memory footprint balloons to terabytes. The GPU time is weeks. The cost is prohibitive.

LoRA, Low-Rank Adaptation, solves this by observing that not all weights need to change equally. When you fine-tune a model, the weight updates are structured. They are not random noise across all 175 billion parameters. Most weights barely move. The change has structure, and structure is compressible.

Hu and colleagues show a way to learn only a low-rank approximation to the weight update, not the update itself. The result is that LoRA can reduce the number of trainable parameters by 10,000 times relative to full fine-tuning of GPT-3 175B. The GPU memory requirement is reduced by 3 times. Unlike adapters, which add extra layers to the model, LoRA has no additional inference latency, because the low-rank update can be merged into the weights after training.

The mechanism, with arithmetic

Start with a single weight matrix W of size 1000 by 1000. This matrix holds 1,000,000 parameters. During full fine-tuning, you would compute gradients for all 1,000,000 values and update them. You would need to store those gradients in memory, and you would need the gradient of the gradient if using a second-order optimizer, which is expensive. The memory cost balloons: 175 billion parameters becomes 700 gigabytes of GPU memory, more than any single machine has.

Instead, decompose the weight update into two small matrices:

• A matrix B of size 1000 by 8
• A matrix A of size 8 by 1000

The product B times A is still a 1000 by 1000 matrix, the same shape as the original weight matrix. But B and A together hold only 8,000 plus 8,000 parameters, which is 16,000 total. The fraction is 16,000 divided by 1,000,000, which is 0.016, or 1.6 percent.

That is the compression on one matrix. Scaling it to a whole model is not a matter of multiplying 1.6 percent through, and the paper's own headline number shows why: Hu and colleagues report reducing the trainable parameters by 10,000 times against full fine-tuning of GPT-3 175B, which puts the trained part in the tens of millions rather than at 1.6 percent of 175 billion.

Two things account for the gap, and both are worth understanding. The first is that the saving gets better as matrices get larger. For a square matrix of side d at rank r, the pair holds 2 times d times r weights against d squared, so the fraction is 2r divided by d. At d of 1000 and r of 8 that is 16 divided by 1000, the 1.6 percent above. At a width of 12,288, which is the order of the largest GPT-3 layers, the same rank 8 gives 16 divided by 12,288, or about 0.13 percent. The bigger the model, the better the trade. The second is that LoRA is not applied to every weight. The paper adapts selected matrices, in practice the attention projections, and leaves the rest alone.

Why does this work? Because of rank. The number 8 is the rank. Rank is a property of matrices: a rank-8 matrix can be expressed as the sum of eight outer products. Concretely, if you decompose B times A, you get:

B times A equals the sum of column i of B times row i of A, for i from 1 to 8.

Each outer product is 1000 by 1000. You are summing eight of them. A full-rank 1000 by 1000 matrix would be the sum of 1000 outer products, so you are discarding 992 of them. The model learns the eight most important directions in which the weights should change.

Think of an outer product as a rank-1 matrix: a single direction of change in a high-dimensional space. The weight matrix W is 1000-dimensional. A rank-1 update moves in one direction. A rank-8 update can move in eight independent directions. Most of the structure in a weight update lives in a low-dimensional subspace. The early dimensions capture the most important changes, and the later dimensions capture smaller adjustments that may not generalize well to new tasks anyway. LoRA selects the eight most important directions and discards the rest.

Why is this enough in practice? Empirically, weight updates during fine-tuning are low-rank. The model does not need to adjust every dimension equally. Some weights move a lot, others move a little, and most of the change can be captured by eight directions. The authors tested this on RoBERTa, DeBERTa, GPT-2 and GPT-3, and found that low-rank updates were sufficient across all of them.

This is an empirical fact about neural networks, not a theoretical proof. Rank is a hyperparameter you choose. A higher rank captures more of the full update but costs more to train. Hu and colleagues typically use ranks from 8 to 64.

Scaling and training

The decomposition needs a scaling factor. During training, the update is initialized as B at zero (no change initially) and A at random. So B times A starts at zero. To prevent the learning rate from compensating with very large values, Hu and colleagues scale the update by alpha over r, where r is the rank and alpha is a hyperparameter. The update is then:

new weight equals W plus alpha divided by r times B times A.

The scaling does not change the rank constraint, it just sets the scale of the initial change. With this initialization, the model can train stably.

Inference: the key advantage

Here is why LoRA has no inference latency. After training, you compute B times A exactly once. This is a 1000 by 1000 matrix, the same shape as the original weight matrix W. You add it to W in-place: W becomes W plus B times A. You now have a single updated weight matrix. During inference, there is no mention of B or A. They are not stored. The model is unchanged in structure and runs at the same speed.

This is different from adapters, which add new layers to the model. An adapter is a bottleneck module that every forward pass must compute. It adds latency. With LoRA, the low-rank update is baked into the weights after training. Inference is unaffected.

What LoRA does not do

LoRA adapts some weight matrices and leaves others frozen. Hu and colleagues concentrate on the attention projections, and their reported experiments adapt the query and value projections in particular, rather than the feed-forward blocks that hold most of a transformer's weights.

That choice is a real one, and the trade runs the opposite way to the intuition. Adapting more matrices costs more trainable parameters, not fewer. The interesting question is how to spend a fixed budget: the same number of trainable weights can go into a high rank on a few matrices or a low rank spread across many. The paper's finding is that spreading across more matrices at lower rank tends to do better than concentrating at high rank on fewer, which is a useful default and not a law.

LoRA also does not put knowledge into a model that was not there. A rank 8 update to a handful of matrices is a small nudge to an existing set of capabilities, not a mechanism for absorbing a new body of facts. If a system needs information the base model never saw, the answer is retrieval rather than a low-rank adapter, for the reasons set out in Retrieval-augmented generation, honestly.

The useful way to hold the distinction: fine-tuning of any kind, LoRA included, is good at changing form, style, format and which of a model's existing behaviours it reaches for. It is a poor and expensive way to teach facts.

The low-rank constraint also means LoRA can only change weights along eight directions. A full fine-tuning can change them along 1000. If the optimal adaptation of the weights requires a change in a direction orthogonal to all eight of LoRA's directions, LoRA cannot do it. Hu and colleagues show empirically that this rarely happens on their tested tasks, but it is a real limitation. For very different domains or very specific use cases, a higher rank or full fine-tuning may be necessary.

The initialisation is worth one more sentence, because it is what makes the method safe to start. B begins at zero and A at random, so B times A is exactly zero on the first step and the adapted model is bit for bit identical to the base model before any training happens. Nothing is learned from scratch: everything the model knows is still in the frozen weights, and the update starts as a no-op and grows from there.

Performance and practice

The method is evaluated across four model families: RoBERTa and DeBERTa on language understanding, GPT-2 on natural language generation, and GPT-3 at 175 billion parameters. The consistent finding is that LoRA performs on par with or better than full fine-tuning despite training a small fraction of the parameters. The result holding across encoder-only and decoder-only models, and across understanding and generation tasks, is what makes it a general technique rather than a trick that suits one architecture.

What changes is not the size of the model but the size of the thing you train and the thing you keep. The base weights still have to be loaded for the model to run at all, so LoRA does not make a large model fit on a small card. What it removes is the optimiser state and the gradients for every frozen weight, which is where Hu and colleagues report the GPU memory requirement falling by 3 times.

The storage argument is the one that compounds. Full fine-tuning for ten tasks means ten complete copies of the base model. LoRA means one base model and ten small sets of B and A. That is what makes a per-task or per-customer adapter practical to ship and to swap at run time, and it is a large part of why the technique spread as fast as it did.

Because the updates are additive, two adapters can be summed, and the sum of two rank 8 updates has rank at most 16. Whether that produces a model good at both tasks is a separate and much less certain question: the two updates were each trained without knowledge of the other, and adding them can as easily interfere as compose. Treat it as something to try and measure, not as a property you get for free.

Understanding LoRA requires understanding rank, understanding that weight updates have structure, and understanding why that structure can be approximated. These ideas extend beyond language models. Any large neural network can be fine-tuned more cheaply by learning a low-rank update to its weights. Dimensionality reduction without the hand-waving discusses rank and dimensionality more formally. Fine-tuning against prompting, a real comparison discusses when fine-tuning is worth doing at all. RLHF and how InstructGPT was taught to follow describes one critical use of fine-tuning: steering model behavior toward human preferences, which LoRA can do at scale.

References

1. LoRA, Low-Rank Adaptation of Large Language Models. Edward J. Hu and colleagues, arXiv, 2021.
2. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Colin Raffel and colleagues, arXiv, 2019.
3. The State of Sparsity in Deep Neural Networks. Trevor Gale, Erich Elsen and Sara Hooker, arXiv, 2019.