Embeddings = Lossy Compression

An embedding is a list of numbers. A long list. You hand a model a sentence and it hands you back 1,536 floats (OpenAI’s text-embedding-3-small), or 3,072 (large), or 384 (MiniLM), or whatever the model in question produces. The trouble starts when people talk about these numbers as if they are the meaning. They’re not. They’re a lossy compression of the sentence into a fixed-size box.

This sounds pedantic. It isn’t. Once you accept that embeddings are compression, every weird behaviour you’ve ever seen from a vector search stops being mysterious and starts being predictable. I’ve been kicking this framing around for a while, partly because I wrote about compression codecs more generally a while back, and the parallels are tighter than people give credit for.

The JPEG frame

Think of a JPEG. It’s a lossy compression of an image. It throws away high-frequency detail that you probably won’t notice, and in exchange you get a file that’s a fraction of the raw pixel size. You can decode it back to something that looks like the original, but the original pixels are gone. You can compare two JPEGs and reason usefully about similarity. You cannot recover the bits you discarded.

An embedding does the same thing to a piece of text. It throws away most of what makes the original text the original text – word order beyond a certain locality, rare vocabulary, syntactic structure, the specific lexical choices – and keeps a representation that’s good enough for the model’s training objective. The objective for most embedding models is roughly: texts that mean similar things should land near each other in this vector space. That’s it. Not “preserve all information.” Not “reconstruct the input.” Just “preserve similarity.”

And like a JPEG, it has a quality knob. That knob is the dimensionality. 384 floats is a low-bitrate JPEG: small, fast, more loss. 3,072 floats is a high-bitrate JPEG: bigger file, more fidelity, more compute per comparison. The reason teams default to 1,536 isn’t because someone proved it’s optimal. It’s because that’s what OpenAI picked, and the world rounds to OpenAI.

What this frame explains

The compression framing pays for itself almost immediately, because it explains the failure modes that otherwise look like sorcery.

Why does vector search return documents that are topically close but don’t contain your search term? Because the embedding compressed away the surface form. Two documents about “Postgres performance” sit near each other in the space regardless of whether either contains the literal word “Postgres.” Useful when you’re searching paraphrased questions. Catastrophic when you needed to find a specific product code or error string.

Why does the same query give different results on different embedding models? Because they’re different compression schemes, with different priorities baked into the training objective. text-embedding-3-small compresses differently from all-MiniLM-L6-v2, which compresses differently from a Cohere model. There is no canonical embedding of a sentence, any more than there’s a canonical JPEG quality. Asking “which embedding model is best?” is the wrong question; the right one is “which lossy projection of language do I want to live with?”

Why do longer documents embed worse than short ones? Because compressing 5,000 words into 1,536 floats throws away more than compressing 50 words into the same 1,536 floats. The fixed-size box is the problem. This is also why naive RAG against long PDFs is a mess: the chunking is doing most of the work, not the embedding. I’ll come back to that one separately.

Why does cosine similarity between two embeddings often hover around 0.7, even for unrelated texts? Because the trained model uses only a fraction of the available vector space. Most of the geometry is empty. The interesting differences live in a narrow range, not across the whole [-1, 1] interval that cosine could theoretically produce. The compression schemes don’t spread their outputs uniformly.

Bits and bytes

If we’re taking the compression framing seriously, let’s look at the bits.

1,536 floats at 32-bit precision is 6,144 bytes per embedding. A million documents costs you about 6 GB just in vectors. A hundred million costs you 600 GB. This is before any index structure. People underestimate the storage bill until they hit the wall.

And just like JPEG, you can quantise. Drop from float32 to float16 and you halve the storage with usually-negligible recall loss. Drop to int8 and you quarter it. Binary quantisation (one bit per dimension) cuts it by 32x at the cost of a serious recall hit unless you rerank. Most production vector stores support this; most teams don’t turn it on because they don’t realise it’s there.

-- pgvector example: store as halfvec to halve storage
ALTER TABLE documents
  ADD COLUMN embedding_h halfvec(1536);
UPDATE documents
  SET embedding_h = embedding::halfvec(1536);
CREATE INDEX docs_emb_h_hnsw
  ON documents
  USING hnsw (embedding_h halfvec_cosine_ops);

That single change typically halves storage and shaves query memory pressure, with recall holding within a percentage point or two on the kinds of corpora I’ve seen. It’s the closest thing to a free lunch in the vector world. Try it before you reach for a bigger server.

Matryoshka and the progressive JPEG

The most interesting recent embedding work is Matryoshka representation learning. The idea is that you train one model that produces, say, 3,072 floats, but with the constraint that the first 512 floats are also a usable embedding on their own. So is the first 1,024. So is the first 2,048. You can truncate the vector at inference time and lose only gracefully.

This is exactly a progressive JPEG. One file, multiple usable resolutions, slider at the consumer’s end. OpenAI’s v3 embeddings are Matryoshka. Cohere’s recent models are. Anyone serious is. The implication is that you can store the full 3,072-dimensional embedding and let the query workload pick the precision it needs – long-tail queries with strict latency budgets use a 512-d truncation; precision-sensitive queries use the full vector. Two indexes, one column, no re-embedding.

Evaluating compression on your own data

The practical takeaway: every published embedding benchmark trains your intuition on someone else’s data. MTEB is fine as a sanity check; it’s useless as a procurement input. Your corpus has its own loss profile. The only honest evaluation is on your own data.

The minimum-viable version of this:

1. Take 1,000 documents from your real corpus.
2. Embed them with two or three candidate models.
3. Build a small set of 30–50 queries you know the right answer to.
4. Compute recall@5 and MRR for each model.
5. Then, for the leading model, quantise to float16 and to int8. Re-check. Usually you lose less than you’d guess.

An afternoon’s work that’s worth more than reading a year of blog posts about which embedding model is “best.” The answer is always “best for what, on what data, at what cost.”

The magic dissolves once you see the bits. Which is, on the whole, a useful thing to happen.

Next up – the chunking problem, where most RAG pipelines actually live or die.