If you've built a semantic search feature, a RAG pipeline, or a recommendation engine in the last few years, you've almost certainly reached for a vector database. They've gone from a niche research tool to a staple of modern AI infrastructure—yet the core mechanics remain fuzzy for many developers who use them. This post explains what's actually happening under the hood: how text (or images, or code) becomes a vector, how those vectors get indexed for fast retrieval, and how a similarity query finds the right answer in milliseconds across millions of entries.
A vector database stores embeddings—dense numerical representations of content produced by a machine learning model. An embedding model (like OpenAI's text-embedding-3-small or the open-source all-MiniLM-L6-v2) takes an input and maps it to a fixed-length array of floating-point numbers, typically 384 to 3,072 dimensions depending on the model.
The key property is that semantic similarity maps to geometric proximity. Sentences with related meaning end up close together in the high-dimensional space; unrelated sentences end up far apart. "The cat sat on the mat" and "A feline rested on a rug" will have embeddings separated by a small angular distance. "The cat sat on the mat" and "Quarterly revenue grew 12%" will be far apart.
This geometry is what makes embeddings useful. You're not matching keywords—you're comparing meaning. The embedding model has already encoded world-knowledge into the vector space during training, so queries that don't share a single word with a document can still return it as a top match.
One important constraint: the embedding model is fixed at index time. If you embed your documents with model A, you must embed your queries with model A. Mixing models produces meaningless distance comparisons because the coordinate systems are different.
The naive approach to finding the closest vector is a flat (exhaustive) scan: compute the distance from the query vector to every stored vector and return the top-k smallest. For a few thousand vectors this is fine. For 100 million vectors, it's impossibly slow.
Vector databases solve this with Approximate Nearest Neighbor (ANN) indexes. Rather than guaranteeing the exact nearest neighbor, ANN algorithms find results that are very likely to be in the true top-k, in a fraction of the time. In practice, recall@10 above 95% is achievable at latencies that exact search cannot touch.
The most widely deployed ANN algorithm today is HNSW (Hierarchical Navigable Small World). HNSW builds a multi-layer graph during indexing:
A query starts at the top layer, greedily moves toward the query vector, then descends through layers until it reaches the bottom, where it performs a local beam search to collect candidates. This hierarchical structure turns an O(n) scan into something closer to O(log n), with search times that scale gracefully to hundreds of millions of vectors.
Other approaches include IVF (Inverted File Index)—which clusters vectors into Voronoi cells and only searches the closest clusters—and Product Quantization (PQ), which compresses vectors to reduce memory footprint at the cost of some recall. Most production databases combine these: IVF+PQ for high-scale low-memory deployments, HNSW for latency-sensitive applications.
The tradeoff space has three axes: recall (are the true nearest neighbors in the results?), latency (how fast is each query?), and index build time. Tuning parameters like ef_construction (HNSW graph connectivity during build) and ef_search (beam width during query) lets you slide along these axes depending on your application's requirements.
Once the index is built, a query is a vector too. The database finds stored vectors that are "close" to the query vector using a distance metric. The three most common:
Most embedding models for text are trained with cosine similarity in mind, and many vector databases normalize vectors automatically—so "cosine" and "dot product" behave identically. Check your model's documentation before choosing a metric: using L2 on cosine-trained embeddings can degrade recall significantly.
Suppose you're building a support ticket router. You have 50,000 past tickets, each labeled with the team that resolved it. You want new incoming tickets to be automatically routed to the right team.
Setup:
import openai
import chromadb
client = chromadb.Client()
collection = client.create_collection("tickets", metadata={"hnsw:space": "cosine"})
# Embed and store historical tickets
for ticket in historical_tickets:
embedding = openai.embeddings.create(
input=ticket["text"], model="text-embedding-3-small"
).data[0].embedding
collection.add(
ids=[ticket["id"]],
embeddings=[embedding],
metadatas=[{"team": ticket["team"]}]
)
# Route a new ticket
new_ticket = "Login button not working on Safari mobile after the 2.4 release"
query_embedding = openai.embeddings.create(
input=new_ticket, model="text-embedding-3-small"
).data[0].embedding
results = collection.query(query_embeddings=[query_embedding], n_results=5)
# Top results will likely be past Safari/mobile/auth tickets → route to Frontend team
The query vector lands near historical vectors for browser-compatibility bugs and authentication issues. You don't need any keyword overlap with "Safari" or "login"—tickets about "mobile auth regression" or "iOS sign-in failure" will surface too.
Vector databases are a purpose-built layer for one specific operation: finding semantically similar content at scale. They work because embedding models compress semantic meaning into geometric space, ANN indexes make nearest-neighbor search tractable on millions of vectors, and distance metrics quantify closeness in a way that aligns with that geometry.
Understanding these three layers—embeddings, ANN indexing, and distance metrics—lets you make meaningful engineering decisions: which embedding model to choose, which index type fits your latency and recall requirements, and why certain queries return surprising results. The abstraction is elegant, but it's not magic, and knowing the mechanics helps you build more reliable systems on top of it.