Embeddings are coordinates for meaning

Two support tickets. The first says card declined at checkout, tried three times. The second says payment keeps failing when I try to pay. They share the word try and nothing else that matters. A keyword search ranks them as unrelated. An embedding model scores their similarity at 0.83 and puts them in the same cluster, which is the behaviour you wanted.

Then a third ticket arrives: my refund went through fine. It scores 0.79 against the first one. Same vocabulary of money and transactions, opposite meaning, and the number barely moved.

Both results come from the same mechanism. Understanding why the first works explains why the second does, and it stops the second from being a surprise in production.

What a vector actually is here

An embedding is a list of numbers assigned to a piece of text. Nothing more. A typical model produces 384, 768, or 1,536 of them per input.

Take a deliberately small case: three dimensions, four words, coordinates invented for the arithmetic.

• invoice = (0.8, 0.5, 0.1)
• receipt = (0.7, 0.6, 0.2)
• refund = (0.6, 0.7, 0.1)
• bicycle = (0.1, 0.2, 0.9)

Cosine similarity is the dot product divided by the two lengths. For invoice and receipt: the dot product is 0.56 plus 0.30 plus 0.02, which is 0.88. The lengths are the square roots of 0.90 and 0.89, which are 0.9487 and 0.9434. So the similarity is 0.88 divided by 0.8950, or 0.983.

For invoice and bicycle: the dot product is 0.08 plus 0.10 plus 0.09, which is 0.27. The length of bicycle is the square root of 0.86, or 0.9274. So the similarity is 0.27 divided by 0.8798, or 0.307.

Cosine ignores length entirely and measures only direction. Double every number in invoice and its similarity to everything is unchanged. That is usually what you want for text, because a longer document should not automatically look more similar to everything, and it is why almost every vector database defaults to cosine rather than Euclidean distance.

Note the second number, 0.307. Nothing in the space is at zero. Two texts about entirely different subjects still score a few tenths, and in a real 768-dimensional model that floor is often higher than beginners expect. A raw similarity score is close to meaningless without knowing the distribution of scores for unrelated text in that specific model.

Where the coordinates come from

Nobody assigns them. They fall out of a prediction task, and the task is the one from Language models predict the next token in an earlier form.

Mikolov, Chen, Corrado and Dean proposed two architectures for computing continuous vector representations of words from very large data sets, measured on a word similarity task, and observed large improvements in accuracy at much lower computational cost: less than a day to learn high quality word vectors from a 1.6 billion word data set. The training signal is co-occurrence. A word is asked to predict its neighbours, or to be predicted from them, and words appearing in similar company end up with similar coordinates.

That single sentence explains both of the opening results. card declined and payment failing appear in similar company, so they land close together. So do refund went through and card declined, because refunds and declines are discussed in almost identical surroundings by almost identical people. The space encodes topical company, not agreement. It has no operator for negation, no notion of a claim being the opposite of another, and no reason to have acquired one, because nothing in the objective ever asked.

This is the same lesson as How a dataset becomes features, arrived at from the other direction. Representations are learned to serve a task, so they carry exactly what that task rewarded.

The arithmetic that made the metaphor famous

The demonstration everyone remembers is king minus man plus woman lands near queen. It is a real effect. The claim built on top of it, that the space contains an interpretable gender axis you can reason with, has been examined carefully and it does not hold in the form usually stated.

Nissim, van Noord and van der Goot address exactly this. Analogies, they write, are often used to illustrate the amazing power of word embeddings and also to expose how strongly human biases are encoded in vector spaces trained on natural language, with examples like man is to computer programmer as woman is to homemaker. Recent work, they note, has shown that analogies are in fact not an accurate diagnostic for bias, but this does not mean that they are not used anymore, or that their legacy is fading. Rather than the intrinsic problems with the task, they discuss a series of issues involving implementation as well as subjective choices that might have yielded a distorted picture of bias in word embeddings. Their conclusion is worth quoting in shape: they stand by the truth that human biases are present in word embeddings, and the need to address them, but analogies are not an accurate tool to do so, and the way they have been most often used has exacerbated some possibly non-existing biases and perhaps hidden others.

The engineering lesson generalises past bias. A demonstration that requires you to select which outputs count is a demonstration about the selection as much as about the space. Before treating any vector arithmetic as a capability, ask what the answer set was and what was excluded from it.

Three places the metaphor breaks

Direction is not meaning; it is company. The refund ticket above is the whole argument. Antonyms are among the most reliably similar pairs in these spaces, because opposites are discussed together constantly. If your system needs to distinguish a complaint from a compliment, similarity will not do it, and no amount of dimensionality will fix it.

A word does not have one vector any more. Static embeddings gave each word a fixed point. Contextual models do not, and the geometry is stranger than that upgrade suggests. Ethayarajh measured it directly and found that the contextualized representations of all words are not isotropic in any layer of the contextualizing model: rather than spreading through the space, they occupy a narrow region. Representations of the same word in different contexts still have greater cosine similarity than those of two different words, but this self-similarity is much lower in upper layers, which produce more context-specific representations. His headline measurement is the one to carry: in all layers of ELMo, BERT and GPT-2, on average, less than 5 percent of the variance in a word's contextualized representations can be explained by a static embedding for that word.

Less than 5 percent. Whatever a contextual embedding of bank is, it is overwhelmingly not a fixed property of the word.

Distances are not calibrated across models. A cosine of 0.83 in one model and 0.83 in another are not comparable quantities, and neither is comparable across domains within one model. Swapping the embedding model silently invalidates every threshold in your code. This is a versioning problem before it is a machine learning problem, and it is discussed further in Putting a model behind an interface.

The cost that decides your architecture

There are two ways to ask a neural model whether two texts are similar, and the difference between them is the reason vector databases exist at all.

A cross-encoder reads both texts together and scores the pair. It is more accurate and it cannot be precomputed, because the score does not exist until both texts are in hand. A bi-encoder embeds each text alone, so every document can be embedded once and stored.

Reimers and Gurevych measured the gap in the paper that made the second approach standard. Feeding both sentences into BERT, they write, causes a massive computational overhead: finding the most similar pair in a collection of 10,000 sentences requires about 50 million inference computations, roughly 65 hours. Their modification using siamese and triplet network structures derives sentence embeddings comparable with cosine similarity, and this reduces the effort for finding the most similar pair from 65 hours with BERT or RoBERTa to about 5 seconds with SBERT, while maintaining the accuracy from BERT.

Sixty-five hours to five seconds is the entire design argument for storing vectors. It also explains the two-stage pattern used in serious retrieval systems: a bi-encoder retrieves a few dozen candidates cheaply, then a cross-encoder reranks only those. The expensive model runs 50 times instead of 10,000, and both properties are kept.

What to do with all this

• Calibrate before you threshold. Embed a few hundred pairs you know to be unrelated and look at the distribution. Set the cut-off from that, in that model, on that content.
• Never use similarity where you mean agreement. Sentiment, contradiction, and polarity need a model trained for them, and the reasoning about choosing a metric from consequences in The cost of a wrong answer applies unchanged.
• Version the embedding model like a schema. Changing it means re-embedding the corpus. A mixed index of vectors from two models is silently broken, and nothing will throw.
• Reduce dimensions for a picture, not for a decision. The caveats in Dimensionality reduction without the hand-waving are all still in force when the input is embeddings.
• Rerank if precision matters. The two-stage pattern is a well-understood cost trade, not a sophistication.

Coordinates for meaning is a useful phrase as long as you remember the coordinates were fitted to predict company. The next article puts them to work, and looks honestly at what retrieval built on them can and cannot repair.

References

1. Efficient Estimation of Word Representations in Vector Space. Tomas Mikolov, Kai Chen, Greg Corrado and Jeffrey Dean, arXiv, 2013.
2. How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings. Kawin Ethayarajh, arXiv, 2019.
3. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. Nils Reimers and Iryna Gurevych, arXiv, 2019.
4. Fair Is Better than Sensational: Man Is to Doctor as Woman Is to Doctor. Malvina Nissim, Rik van Noord and Rob van der Goot, Computational Linguistics volume 46 issue 2, 2020.