Embeddings — From word2vec to Instruction-Tuned Vectors & Production RAG

Embeddings are dense vector representations of text where semantic similarity maps to geometric proximity. The evolution from 2013 to 2024 is not incremental improvement — each generation addressed a fundamental limitation that made the previous generation inadequate for real tasks.

word2vec (Mikolov et al., 2013): 300-dimensional vectors trained via shallow neural networks. Revolutionary because it demonstrated that semantic relationships encode as vector offsets: "king − man + woman ≈ queen." Limitation: every word maps to one vector regardless of context. "bank" in "river bank" and "bank account" get identical representations.

ELMo (Peters et al., 2018): First contextual embeddings — BiLSTM trained on language modeling, producing word vectors that depend on the surrounding sentence. Fixed the polysemy problem. Limitation: extracting a sentence-level embedding by averaging token vectors is noisy, and BiLSTMs are slow to compute.

BERT (Devlin et al., 2018): Transformer-based bidirectional encoder, trained on masked language modeling. Produces rich contextual token embeddings. Limitation: BERT's [CLS] token and mean-pooled embeddings are not optimized for semantic similarity — BERT was fine-tuned on sentence-pair tasks using cross-encoding, not bi-encoding.

Sentence-BERT (Reimers & Gurevych, 2019): Adds a siamese network fine-tuned on NLI and semantic textual similarity (STS) datasets. Enables independent encoding of query and document, making large-scale semantic search practical. For 10,000 sentences, BERT cross-encoder takes 65 hours to compute all pairs; SBERT reduces this to 5 seconds.

Instruction-tuned embeddings (2022+): E5-instruct, text-embedding-3, GTE-Qwen2 support task-specific prompts that shift the representation space for the intended use case (retrieval, classification, clustering). This is the current production standard.

word2vec trains a shallow 2-layer neural network on one of two tasks. CBOW (Continuous Bag of Words) predicts the center word from surrounding context words. Skip-gram predicts surrounding context words given the center word. Skip-gram works better for rare words because it creates more training examples per token.

The training trick that made word2vec practical is negative sampling: for each positive (word, context) pair, sample 15 random negative words and train the model to distinguish them from the true context word. This approximates the full softmax over the entire vocabulary without computing all 100K+ probabilities per step.

Why analogical reasoning works: word2vec encodes syntactic and semantic relationships as consistent vector directions across the entire vocabulary space. The direction "capital city" consistently points from country vectors to their capital vectors. So "Paris − France + Italy ≈ Rome" works because the model has learned a consistent linear subspace for this relationship. This is an emergent property of training on co-occurrence statistics, not an explicit design choice.

Where word2vec breaks down: (1) Polysemy — "jaguar" always maps to one point between its car and animal usages. (2) Out-of-vocabulary words — any word not in training gets [UNK]. (3) Sentence-level tasks — averaging word vectors loses word order information entirely. "Dog bites man" and "Man bites dog" have identical mean word2vec embeddings. These failures motivated contextual embeddings.

A common interview misconception: "just use BERT's [CLS] token as your sentence embedding." This produces mediocre results for semantic similarity tasks and performs worse than averaged GloVe embeddings on STS benchmarks. Why? BERT was pre-trained to predict masked tokens and next sentence prediction — tasks that do not require the [CLS] token to encode the entire meaning of the sentence in a form optimized for cosine similarity. The [CLS] representation is optimized for BERT's specific pre-training objectives, not for downstream similarity.

Sentence-BERT fixes this with a siamese and triplet network structure. Two BERT encoders share weights. During training on NLI data: anchor ("A dog is running"), entailment ("An animal is moving"), contradiction ("Nothing is moving"). The model minimizes distance between anchor and entailment, maximizes distance from contradiction. After fine-tuning on STS-Benchmark, SBERT achieves Spearman correlation 0.869 vs BERT's 0.293 on the same task.

Instruction-tuned embeddings extend this with task-specific prompts. E5-instruct and text-embedding-3 prepend a task description: "Represent this sentence for retrieval: {query}" vs "Represent this passage to answer retrieval questions: {passage}". This asymmetric treatment of query and document significantly improves retrieval where queries are short and documents are long — the classic RAG scenario. On BEIR (heterogeneous retrieval benchmark), instruction-tuned models improve NDCG@10 by 8–15 points over non-instructed variants on out-of-domain datasets.

Matryoshka Representation Learning (MRL, Kusupati et al., 2022) trains a single embedding model such that any prefix of the embedding vector is still a meaningful representation. The first 256 dimensions encode the most salient semantic features; dimensions 257–3072 add progressively finer detail. text-embedding-3-large (OpenAI, 3072 dimensions) uses MRL.

Why this matters for production: embedding storage and ANN search cost scale linearly with embedding dimension. A 10M document index at 3072 dimensions × 4 bytes = 123GB. The same index at 256 dimensions = 10.2GB — 12× smaller, 12× faster in FAISS dot-product search, with roughly 95% of the full-dimension retrieval quality on BEIR benchmarks.

The MRL workflow for cost-latency tradeoff: use truncated embeddings (e.g., 256d) for first-stage ANN recall — retrieving the top-100 candidates quickly at low cost. Re-rank those 100 with full 3072d embeddings (or a cross-encoder) to get the final top-10. This two-stage approach achieves near-full-quality retrieval at a fraction of the storage and compute cost.

The non-obvious insight: MRL does not degrade full-dimension quality. The training loss forces the model to encode the most important information in the first dimensions — this constraint acts as a form of regularization that slightly improves full-dimension performance on some benchmarks compared to a non-MRL model of identical architecture.

Production embedding systems have three infrastructure problems that MTEB scores do not capture: storage cost, query latency, and index update frequency.

FAISS (Facebook AI Similarity Search) is the standard library for ANN search. Key index types: IndexFlatIP — exact search, O(N·d), baseline for recall measurement; IndexIVFFlat — partitions vectors into K clusters (typically K=sqrt(N)), searches only the nearest clusters (nprobe clusters), roughly O(nprobe × N/K × d); IndexIVFPQ — same partitioning but with Product Quantization compression, reduces memory by 4–32× with 2–8% recall loss.

When to use PQ quantization: when your full-precision index exceeds available RAM. For 100M documents at 1024d × 4 bytes = 400GB — this does not fit on a standard server. PQ with 8 bytes/vector compresses to 800MB, enabling 100M vector search on a single machine. The recall trade-off is typically 92–96% vs exact search.

Index update strategy: FAISS indices are static — you cannot add vectors in real-time. Production systems use a two-tier approach: a small real-time index (last 24 hours of new documents) searched exactly with IndexFlatIP, merged into the main IVF-PQ index nightly. Tools like Milvus, Weaviate, and Qdrant manage this operational complexity.

Embedding dimensions and the MRL sweet spot: for most RAG applications, 256–512 dimensions from an MRL model captures 90–95% of full-dimension recall while reducing FAISS memory by 6–12×. The empirical rule: go to 1024d only if you are working with highly technical or specialized domains where fine-grained distinctions matter (medical text, legal documents, code search).