BERT and the case for reading both ways

In October 2018, Devlin, Chang, Lee and Toutanova released BERT, a model that inverted the direction of sequence processing. Where GPT-1 (released months earlier) read left to right, predicting each token from all previous tokens, BERT read both ways simultaneously, letting each token attend to the entire sentence. The models were built from the same Transformer blocks, but the training objective changed everything.

The immediate results were dramatic. BERT pushed the GLUE benchmark, a suite of language understanding tasks, to 80.5%, a 7.7 percentage point absolute improvement over the prior state of the art. On the MultiNLI task, it reached 86.7%, a 4.6 point improvement. On SQuAD v1.1, the reading comprehension benchmark, it achieved 93.2 F1, a 1.5 point improvement. On SQuAD v2.0, which includes unanswerable questions, it scored 83.1 F1, a 5.1 point improvement.

But the deeper contribution was different. BERT showed that pretraining on bidirectional context produces stronger representations for understanding than left-to-right prediction. The trade, however, is severe: bidirectionality makes generation impossible. A model that has seen the entire context cannot predict what comes next in a natural way. That constraint has defined the encoder-only branch ever since.

The pretraining objective, the core innovation

Attention is all you need, what the paper actually says describes the Transformer encoder-decoder architecture. BERT uses only the encoder, letting each token attend to the entire input in both directions, and then adds a masked language modeling objective on top.

Take the sentence "the cat sat on the mat", six tokens long. Next token prediction, the objective used by GPT and described in Language models predict the next token, shows the model "the cat" and asks for the third token, then shows it "the cat sat" and asks for the fourth. Every position is predicted from its left context only.

BERT masks positions instead. Roughly 15 percent of token positions are chosen at random, and the model predicts what was there using every other position, on both sides. Note that the selection is over positions in the sequence, not over entries in the vocabulary: it is a choice of which slots to blank, not which words are eligible.

There is a detail here that matters more than it first appears. A chosen position is not always replaced by the mask symbol. In 80 percent of cases it becomes [MASK], in 10 percent it is replaced by a random token, and in 10 percent it is left exactly as it was, with the model still asked to predict it. The reason is that [MASK] never appears when the model is later fine-tuned on a real task, so a model that only ever saw [MASK] at the masked slot would learn a feature that vanishes the moment it is used. The random and unchanged cases force it to build a usable representation at every position rather than only at the marked ones.

The arithmetic, on a four word vocabulary

Take the sentence with position three masked: "the cat [MASK] on the mat". To keep the numbers checkable, pretend the vocabulary holds only four words: sat, mat, cat, the.

A model produces a score, called a logit, for each word, and a softmax turns those scores into probabilities. Reading both directions, the model can see "the cat" on the left and "on the mat" on the right, which between them pin the answer down to a past tense verb. Suppose the logits come out as sat 3.2, mat 0.9, cat 0.4 and the 0.1.

Exponentiate each one:

• exp of 3.2 is 24.53
• exp of 0.9 is 2.46
• exp of 0.4 is 1.49
• exp of 0.1 is 1.11

Those sum to 29.59. Divide each by the total to get the probabilities:

• sat: 24.53 divided by 29.59, which is 0.829
• mat: 2.46 divided by 29.59, which is 0.083
• cat: 1.49 divided by 29.59, which is 0.050
• the: 1.11 divided by 29.59, which is 0.038

They sum to 1.000, and the correct word holds 83 percent of the mass.

Now take the same position away from the bidirectional model and give it only the left context, "the cat", which is what a left-to-right model has when predicting position three. The right side is gone, so "on the mat" cannot help. Plausible continuations after "the cat" are far more varied, and the logits flatten: say sat 1.6, the 1.3, mat 1.1 and cat 0.6.

• exp of 1.6 is 4.95
• exp of 1.3 is 3.67
• exp of 1.1 is 3.00
• exp of 0.6 is 1.82

Those sum to 13.44, giving sat 0.368, the 0.273, mat 0.223 and cat 0.135.

The right answer still ranks first, and it is nowhere near as certain: 0.368 against 0.829.

Put that in the units training actually uses. The loss on a single token is the negative natural logarithm of the probability given to the correct word. For the bidirectional case that is minus the log of 0.829, which is 0.188. For the left-to-right case it is minus the log of 0.368, which is 1.000. The same model architecture, the same sentence, the same target word, and more than five times the loss, purely because half the evidence was withheld.

That gap is the whole argument for bidirectional pretraining. Predicting a word from both sides is an easier problem, so the representation the model needs in order to solve it can be richer and more specific than one built under a harder, thinner signal.

The gap is also the trap. The easier problem is not the problem generation poses. A model that has learned to fill gaps using both sides has never once been asked to continue a sentence from the left alone, which is the only thing a generator ever does.

The architecture and parameters

BERT-base has 12 layers, a hidden size of 768, and 12 attention heads, for a total of 110 million parameters. BERT-large, the model used for the benchmark results above, has 24 layers, a hidden size of 1,024, and 16 attention heads, for a total of 340 million parameters. Both use the same Transformer encoder architecture as Attention is all you need, what the paper actually says describes.

The pretraining procedure masks roughly 15 percent of token positions, chosen at random across the sequence. For each one the model predicts the missing token from a softmax over the whole vocabulary, which for the English models runs to a little over thirty thousand entries rather than the four used above. The loss for each masked token is the cross-entropy between the predicted distribution and the true token.

BERT also includes a second pretraining objective called next sentence prediction (NSP). Given two sentences, the model predicts whether they are consecutive or randomly sampled from different places in the corpus. This objective was intended to help the model learn sentence-level relationships. Later work questioned whether NSP contributes meaningfully to downstream performance, and most subsequent models dropped it.

Why bidirectionality and generation are incompatible

Generation requires producing tokens one at a time, with no knowledge of future tokens. This is called autoregressive decoding: the next token is generated given all previous tokens, and then that token becomes part of the context for the next step. Attaching a language model head to a Transformer encoder and doing autoregressive decoding is mathematically possible, but it produces incoherent output, because the encoder was trained to see the full sentence, not to make good predictions from partial ones.

The model learns to rely on later context to resolve ambiguity. If the sentence is "The bank flooded", the token "bank" learns to look forward to "flooded" to disambiguate the sense. The attention weights let "bank" take most of its signal from "flooded" rather than from "the" or itself. When you try to generate and can only look backward, "bank" has no forward signal and must choose between financial and aquatic based only on "the", which provides almost no clue. The model makes an arbitrary choice.

This is not a defect that training harder can fix. The constraint is fundamental to the pretraining objective. A model trained on masked language modeling has never been asked to predict tokens given only left context. Its representations are built around bidirectional attention. Adding a left-to-right mask at test time does not change the learned representations.

Fixing this requires retraining the model with a left-to-right mask, which is what GPT does. That model can generate coherently but gives up the bidirectional context that made BERT effective at understanding.

This is not a shortcoming of BERT. It is the core tradeoff that defines the encoder-only branch. Encoders are best for tasks where you have the full input and need to extract understanding: classification, sentiment analysis, sequence labeling, named entity recognition, extractive question answering, and semantic similarity. Decoders are best for tasks where you generate output one token at a time: language generation, machine translation, summarization, and dialogue.

The field solved this by building models optimized for each use case. BERT-style encoders remain the best choice for understanding. GPT-style decoders remain the best choice for generation. The occasional attempt to build a single model that does both usually produces something that is good at neither. The split reflects a real constraint: you must choose what the model learns to optimize for during pretraining, and that choice locks in the downstream capabilities.

What BERT showed about scale and pretraining

BERT's success had two components. First, the masked language modeling objective is more effective for learning useful representations than next-token prediction on the same corpus. Second, pretraining on a large unlabeled corpus and then fine-tuning on downstream tasks is more effective than training only on the labeled data for each task.

The scale of pretraining matters. BERT is trained on BooksCorpus (800 million words) and English Wikipedia (2.5 billion words). That is roughly 3.3 billion words total. The fine-tuned models that achieve the benchmark results are then trained on much smaller task-specific datasets, sometimes with only a few thousand labeled examples. The disparity in scale is enormous: the pretraining dataset is thousands of times larger than any downstream task.

This pattern, established by BERT and refined by subsequent work, showed that pretraining and fine-tuning is a practical recipe for building understanding systems. The encoder learns patterns from a large unlabelled corpus, and a small task head added on top adapts those patterns to the specific problem. There is no decoder anywhere in BERT, which is exactly why it cannot generate. The pretraining phase builds a foundation of general language understanding, and the fine-tuning phase specializes that foundation to a specific problem.

BERT did not invent fine-tuning or transfer learning, but it showed that fine-tuning a large bidirectional Transformer on moderately-sized downstream tasks produces results that far exceeded prior art. The magnitude of the improvement was striking: a 7.7 percentage point gain on GLUE was substantial. That combination, now standard practice, was not obvious before BERT was released. The field had been focused on training task-specific models from random initialization, and BERT proved that pretraining was worth the effort and computational cost.

The encoder-only branch

BERT opened a branch of the field focused on building better encoders for understanding. RoBERTa improved the pretraining procedure by adjusting masking patterns and training longer. ELECTRA changed the pretraining objective by using discriminative training instead of generative. Sentence-BERT specialized in producing good sentence embeddings for retrieval by adding a contrastive loss. DistilBERT reduced the model size through distillation. Each made specific improvements to the encoder paradigm.

None of them changed the fundamental constraint: an encoder trained on bidirectional context is not a language model. It cannot generate text in any natural way. That property is a feature, not a bug, for the tasks encoders are built for, but it closes the door on generation. The constraint is not about the implementation but about the learning signal. An encoder trained to recover masked words in both directions has learned something different than an encoder trained to predict the next token.

The field split. The encoder-only branch, built from BERT, optimized for understanding and classification. The decoder-only branch, built from GPT, optimized for generation. Recent work has explored encoder-decoder combinations and hybrid architectures, but the basic split has held: if you need to understand existing text, extract information, or classify it, use an encoder-only model. If you need to generate text, predict continuations, or translate, use a decoder-only model. The split is not arbitrary; it flows from the pretraining objective.

BERT did not predict this divergence. It did not propose that the field should specialize. Devlin and colleagues built an encoder and showed it was effective for understanding. The rest followed from recognizing that constraint as a feature and building separate systems for separate tasks. Nine years later, the split remains the organizing principle of how large language models are built.

References

1. BERT, Pre-training of Deep Bidirectional Transformers for Language Understanding. Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova, arXiv, 2018.
2. Attention Is All You Need. Ashish Vaswani and colleagues, arXiv, 2017.
3. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Colin Raffel and colleagues, arXiv, 2019.