Embeddings & semantic search

Unlike sparse bag-of-words vectors (one dimension per vocabulary word, mostly zeros), dense embeddings pack semantic information into every coordinate. The phrase “refund policy for annual subscriptions” and “how do I cancel my yearly plan and get money back” share few keywords but should produce vectors with high cosine similarity—because the model was trained so that paraphrases and related concepts cluster together.

Meaning in vector space is geometric: direction often matters more than magnitude. “King − man + woman ≈ queen” is the classic word2vec intuition; modern sentence embeddings extend that to full passages. You never interpret individual dimensions by hand—the model learns a space where downstream similarity metrics (cosine, dot product) correlate with human judgment of relatedness.

How text becomes a vector

1. Tokenize — same subword tokenizers as LLMs (BPE, SentencePiece); long docs are truncated to model max length (often 512–8192 tokens).
2. Encode — transformer encoder (or dual-tower) runs forward pass; token hidden states are pooled (mean, CLS, or last-token) into one vector.
3. Normalize — many APIs L2-normalize output so cosine similarity equals dot product (see next section).
4. Store / compare — persist vector with document metadata; at query time embed the question and rank corpus by similarity.

Bi-encoder vs cross-encoder (preview)

A bi-encoder embeds query and document independently, then compares vectors in O(1) per pair after indexing. Fast at scale—this is what semantic search and RAG retrieval use. A cross-encoder concatenates query + document into one forward pass and outputs a relevance score directly—more accurate but O(n) per query over n documents, so it is used as a reranker on top-20 bi-encoder hits, not over the full corpus. Track 2 covers reranking in depth.

Cosine similarity

Measures the angle between two vectors, ignoring magnitude. Range [−1, 1] for general vectors; [0, 1] for non-negative embedding spaces is common in practice.

cos(q, d) = (q · d) / (||q|| × ||d||)

When it matters: default for text embeddings; robust when document length varies (short titles vs long PDFs) because long texts do not automatically get larger vectors unless you skip normalization.

Dot product (inner product)

q · d = Σ qᵢ dᵢ — no division by norms. Faster on GPU/ SIMD; many ANN libraries (FAISS inner-product, Pinecone dotproduct metric) use it when vectors are pre-normalized to unit length.

When it matters: if both vectors are L2-normalized, dot product ranks identically to cosine. OpenAI embedding responses are normalized—store them as-is and use dot product or cosine interchangeably for ranking.

L2 (Euclidean) distance

||q − d||₂ = √(Σ (qᵢ − dᵢ)²) — smaller is closer. For unit vectors, L2 distance monotonically relates to cosine: ||q−d||² = 2(1 − cos(q,d)).

When it matters: some indexes (FAISS L2, pgvector <-> operator) are built for Euclidean distance. Convert mentally: minimizing L2 on normalized vectors equals maximizing cosine.

Normalization

L2 normalization scales a vector to unit length: v' = v / ||v||. Always normalize query and corpus the same way. If you normalize at index time but send a raw query vector, rankings skew. If your provider returns normalized vectors, do not re-normalize twice (harmless but redundant).

Hosted API models

Open-source / self-hosted

Dimension trade-offs

Higher dimensions can capture finer distinctions but cost more storage, memory, and index build time. OpenAI and Nomic support matryoshka embeddings: truncate trailing dimensions with modest quality loss— useful for storing 1536-dim for quality and 512-dim for a fast pre-filter.

Semantic search

Users type natural language; you embed the query and return the nearest document chunks. Production example: Notion’s Q&A embeds workspace pages and retrieves top-K chunks before the LLM synthesizes an answer— keyword search alone misses “PTO rules” vs “how many vacation days do I get.”

Clustering

Group support tickets, log lines, or survey responses by embedding + k-means / HDBSCAN. Production example: Zendesk-style auto-topic discovery—surface emerging outage themes before tags exist.

Classification

Embed labeled examples; classify new text by nearest centroid or a linear probe on frozen embeddings. Cheaper than fine-tuning an LLM for high-volume routing (intent detection, spam, PII sniffing). Production example: route “billing” vs “technical” tickets with 95% accuracy using 50 labels per class.

Anomaly detection

Normal behavior forms a cluster; outliers are far from neighbors in embedding space. Production example: flag unusual admin CLI commands or novel fraud narratives that keyword rules never seen.

Recommendation

“Similar articles” = nearest neighbors of the current page embedding; user history = average of clicked item vectors. Production example: Shopify product recommendations from catalog description embeddings, refreshed nightly.

Deduplication

Near-duplicate chunks (boilerplate, copy-pasted policies) waste index space and pollute RAG. Drop pairs with cosine > 0.95 before indexing. Production example: ingest pipeline for a legal RAG corpus removes 12% redundant paragraphs, improving answer precision.

Manual, NumPy, and scikit-learn (Python)

For unit tests, compare your service’s top-K IDs against a brute-force NumPy baseline on 1,000 vectors. Production search should use an ANN index, not a Python loop over millions of rows.

import numpy as np
from sklearn.metrics.pairwise import cosine_similarity

query = np.array([0.12, -0.04, 0.33, 0.91])  # from embedding API
doc = np.array([0.10, -0.02, 0.30, 0.88])

# Manual (explicit norms)
def cosine(a, b):
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

# NumPy one-liner on normalized vectors
q_norm = query / np.linalg.norm(query)
d_norm = doc / np.linalg.norm(doc)
manual = float(np.dot(q_norm, d_norm))

# sklearn (query vs matrix of docs)
corpus = np.vstack([doc, np.random.randn(4)])
sklearn_scores = cosine_similarity(query.reshape(1, -1), corpus)[0]

print("manual:", manual, "sklearn top:", corpus[np.argmax(sklearn_scores)])

// Plain Java — cosine between two float arrays
public static double cosine(float[] a, float[] b) {
  double dot = 0, na = 0, nb = 0;
  for (int i = 0; i < a.length; i++) {
    dot += a[i] * b[i];
    na += a[i] * a[i];
    nb += b[i] * b[i];
  }
  return dot / (Math.sqrt(na) * Math.sqrt(nb));
}

// Spring AI + pgvector: store/query in PostgreSQL
// CREATE EXTENSION vector;
// CREATE TABLE chunks (id bigserial, content text, embedding vector(1536));
// CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);

@Autowired JdbcTemplate jdbc;

public List<ChunkHit> search(float[] queryEmbedding, int k) {
  String sql = """
    SELECT id, content, 1 - (embedding <=> ?::vector) AS score
    FROM chunks ORDER BY embedding <=> ?::vector LIMIT ?
    """;
  String vec = Arrays.toString(queryEmbedding).replace("[","{").replace("]","}");
  return jdbc.query(sql, (rs, row) -> new ChunkHit(
      rs.getLong("id"), rs.getString("content"), rs.getDouble("score")),
      vec, vec, k);
}

Batch similarity against a corpus

Shape conventions: query (d,), corpus (n, d), scores (n,). Use matrix multiply corpus @ query when both are L2-normalized row vectors.

def top_k(query: np.ndarray, matrix: np.ndarray, k: int = 5):
    """matrix: (n_docs, dim), L2-normalized rows."""
    scores = matrix @ query  # dot = cosine if normalized
    idx = np.argpartition(-scores, k)[:k]
    idx = idx[np.argsort(-scores[idx])]
    return idx, scores[idx]

public record ScoredId(long id, double score) {}

public List<ScoredId> topK(float[] query, List<float[]> docs, List<Long> ids, int k) {
  PriorityQueue<ScoredId> heap = new PriorityQueue<>(Comparator.comparingDouble(ScoredId::score));
  for (int i = 0; i < docs.size(); i++) {
    double s = cosine(query, docs.get(i));
    if (heap.size() < k) heap.offer(new ScoredId(ids.get(i), s));
    else if (s > heap.peek().score()) { heap.poll(); heap.offer(new ScoredId(ids.get(i), s)); }
  }
  return heap.stream().sorted(Comparator.comparingDouble(ScoredId::score).reversed()).toList();
}

Why exact search fails at scale

• Latency — 1M vectors × 1536 dims × float32 ≈ 6 GB; scanning every query violates p99 < 100ms targets.
• Memory bandwidth — brute force is memory-bound; ANN structures reduce comparisons via graphs or clustering.
• Concurrency — 500 QPS × full scans melts CPU; HNSW serves thousands of QPS on modest hardware with tuned ef_search.

HNSW (Hierarchical Navigable Small World)

Multi-layer graph: greedy search on upper layers, refine on lower layers. Default in pgvector, Weaviate, many Milvus configs. Pros: excellent recall/latency balance, no training phase. Cons: high memory (graph edges); slow inserts compared to IVF; rebuild painful on huge static corpora.

IVF (Inverted File Index)

k-means clusters vectors into nlist cells; at query time probe only nprobe nearest centroids. Pros: lower memory, faster bulk build. Cons: needs training on representative sample; recall drops if nprobe too low.

Tuning trade-offs

• ef_search (HNSW) — higher → better recall, slower queries; start at 64–128 for top-10 retrieval.
• nprobe (IVF) — higher → better recall, more cells scanned.
• Always measure recall@K on a labeled eval set—latency means nothing if the right chunk is never in the candidate set.

Cache key design

• hash(model_id + model_version + normalized_text) — SHA-256 of UTF-8 text after NFC normalization and strip.
• Include input type if provider uses asymmetric encoders (Cohere query vs document).
• Store vectors as float16 or binary quantization in Redis only if recall tests pass—float32 JSON is simpler to debug.

Where to cache

TTL strategies

Query cache: TTL 24h–7d; invalidate on model version bump. Corpus: content-hash keyed—when document body unchanged, skip re-embed on nightly job. Negative cache: do not cache failed API responses long; retry transient 429/503 with backoff.

Cost savings math

Suppose 1M queries/month, avg 20 tokens/query, 40% cache hit rate, text-embedding-3-small at $0.02/1M tokens:

• Uncached tokens: 1M × 20 = 20M tokens → ~$0.40/month embed cost
• With 40% hits: 12M billed tokens → ~$0.24/month (40% savings on queries)
• At 10M queries/month and 50-token queries: savings ≈ $5/month API—but latency drops 30–80ms per hit, which matters more at scale

Corpus caching saves more: re-indexing 500K unchanged chunks costs ~$5 per full re-run; content-addressed storage avoids that entirely.

import hashlib, json, redis
from openai import OpenAI

r = redis.Redis.from_url("redis://localhost:6379/0")
client = OpenAI()
MODEL = "text-embedding-3-small"

def cache_key(text: str) -> str:
    digest = hashlib.sha256(f"{MODEL}:{text.strip()}".encode()).hexdigest()
    return f"emb:{MODEL}:{digest}"

def embed(text: str) -> list[float]:
    key = cache_key(text)
    if cached := r.get(key):
        return json.loads(cached)
    resp = client.embeddings.create(model=MODEL, input=text)
    vec = resp.data[0].embedding
    r.setex(key, 86400 * 7, json.dumps(vec))  # 7d TTL
    return vec

@Service
public class EmbeddingCache {
  private final StringRedisTemplate redis;
  private final EmbeddingModel embedder;
  private static final String MODEL = "text-embedding-3-small";

  public float[] embed(String text) {
    String key = "emb:" + MODEL + ":" + sha256(MODEL + ":" + text.strip());
    String cached = redis.opsForValue().get(key);
    if (cached != null) return parseJson(cached);
    float[] vec = embedder.embed(text);
    redis.opsForValue().set(key, toJson(vec), Duration.ofDays(7));
    return vec;
  }
}

CLIP trains on image–caption pairs: matching pairs are pulled together, mismatches pushed apart. At inference, encode an image through the vision tower and text through the text tower; cosine similarity measures cross-modal relevance. OpenAI’s clip-vit-base-patch32 (via API or local) and open models (OpenCLIP, SigLIP) follow the same pattern.

When to use multimodal vs text-only

Text-only embeddings (OpenAI, Cohere, bge) win on pure RAG over documentation—lower cost, longer context, better lexical-semantic blend with BM25. Multimodal wins when the query or asset is inherently visual or when alt-text is missing/untrustworthy.

Indexing pipeline

1. Ingest — webhook, S3, CMS sync; dedupe by content hash.
2. Chunk — split on headings/tokens (400–800 tokens with overlap); attach metadata (source, ACL, updated_at).
3. Embed batch job — queue workers; batch 64–256 texts per API call; rate-limit aware; write to vector store + object storage backup.
4. Index build — HNSW after bulk load (pgvector: create index CONCURRENTLY); verify recall on golden queries.
5. Publish — blue/green index alias swap; never partial-query during rebuild without dual-write.

Query path

Request → authZ filter (tenant_id, ACL) → embed query (cached) → ANN top-K → optional rerank → return to RAG or UI. Log latency breakdown: embed_ms, ann_ms, rerank_ms.

Monitoring drift

• Model version pin — log embedding_model on every vector; alert on mixed versions in one index.
• Recall@K weekly — labeled question → expected chunk IDs; block deploy if drop > 2 pts.
• Score distribution — sudden shift in mean top-1 cosine may indicate bad normalization or wrong model.
• Coverage — % corpus embedded in last 24h; stale chunks after CMS edits.
• Cost — embed API spend vs cache hit rate; anomalous spikes from runaway re-index cron.

Production readiness checklist

• Single embedding model per index; documented migration runbook for model upgrades
• Content-hash skip for unchanged chunks on re-index
• Query + corpus embedding cache with model-scoped keys
• ANN index tuned with measured recall@K, not default params only
• Metadata filters applied before or during ANN (tenant isolation)
• Brute-force eval job on sample set in CI
• Backup vectors outside primary DB (S3 parquet) for disaster rebuild