How Vector Databases Work

You've heard the pitch: "We built it on top of a vector database." But what does that actually mean? If you've shipped a retrieval-augmented generation (RAG) pipeline or semantic search feature, you've relied on one. If you haven't yet, you will.

Here's what's happening under the hood — without the hand-waving.

The Problem with Traditional Databases

Standard databases are great at exact matches. WHERE name = 'Alice' is fast. WHERE description SOUNDS LIKE 'machine learning for images' is not — at least not without a lot of painful workarounds.

The core issue is that text, images, and audio don't have natural numeric keys. You can't sort "dog" and "puppy" by meaning. That's the problem vector databases solve.

Step 1: Turning Meaning into Numbers (Embeddings)

A vector database stores data as embeddings — arrays of floating-point numbers that represent semantic meaning.

An embedding model (like OpenAI's text-embedding-3-small or Sentence Transformers) reads a piece of text and outputs something like:

"What is photosynthesis?" → [0.23, -0.81, 0.44, 0.09, ..., 0.67]  // 1536 numbers
"How do plants make food?" → [0.25, -0.79, 0.41, 0.11, ..., 0.65]  // very similar
"Best pasta recipes"       → [0.91,  0.12, -0.33, 0.77, ..., -0.44] // very different

Those two questions about photosynthesis land close together in 1536-dimensional space. The pasta query lands far away. The model learned these relationships by training on billions of text examples — it doesn't need a shared keyword to recognize semantic similarity.

This is what makes vector search different from full-text search. You're not matching tokens. You're measuring meaning.

Step 2: Finding Neighbors Fast (Indexing and ANN)

Here's the catch: comparing a query vector against millions of stored vectors is expensive. A brute-force search — calculate cosine similarity against every single record — takes O(n) time. At 10 million documents, that's too slow for real-time use.

Vector databases solve this with Approximate Nearest Neighbor (ANN) algorithms. The tradeoff: you accept a small loss in recall (you might miss 1–2% of the closest matches) in exchange for search times that are milliseconds instead of seconds.

The most widely used approach is HNSW (Hierarchical Navigable Small World graphs). Here's the intuition:

Build a multi-layer graph where each node is a vector.
Higher layers are sparse (few connections, for long jumps). Lower layers are dense (many connections, for precise navigation).
At query time, enter at the top layer and greedily navigate toward the query, descending layers as you zero in.

Think of it like a city map: you start with the highway network (fast, coarse), then zoom into neighborhood streets (slower, precise). You reach your destination without checking every road in the city.

Other common index types include IVF (Inverted File Index, which partitions vectors into clusters) and PQ (Product Quantization, which compresses vectors to reduce memory). Most production vector databases — Pinecone, Weaviate, Qdrant, pgvector — let you tune these parameters based on your recall vs. latency tradeoff.

Step 3: The Similarity Search

With an index built, a query follows three steps:

Embed the query — run it through the same model used to embed your documents.
Search the index — find the k nearest vectors by distance (cosine similarity, dot product, or Euclidean, depending on your setup).
Return results — retrieve the original documents or metadata attached to those vectors.

The distance metric matters. Cosine similarity is common for text (it ignores magnitude, just measures angle). Dot product is faster but assumes your vectors are normalized. Euclidean distance works better for some image embedding spaces.

A Concrete Example: Semantic Search Over a Knowledge Base

Say you're building a support chatbot backed by a docs library. Here's how the pipeline looks in practice:

import openai
from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams, PointStruct

client = QdrantClient(":memory:")

# 1. Create a collection
client.create_collection(
    collection_name="docs",
    vectors_config=VectorParams(size=1536, distance=Distance.COSINE),
)

# 2. Embed and store your documents
docs = [
    {"id": 1, "text": "How to reset your password"},
    {"id": 2, "text": "Billing and subscription management"},
    {"id": 3, "text": "Setting up two-factor authentication"},
]

for doc in docs:
    response = openai.embeddings.create(input=doc["text"], model="text-embedding-3-small")
    vector = response.data[0].embedding
    client.upsert("docs", points=[PointStruct(id=doc["id"], vector=vector, payload={"text": doc["text"]})])

# 3. Query at runtime
query = "I forgot my login credentials"
query_vec = openai.embeddings.create(input=query, model="text-embedding-3-small").data[0].embedding

results = client.search("docs", query_vector=query_vec, limit=2)
for r in results:
    print(r.payload["text"], "— score:", round(r.score, 3))
# Output:
# How to reset your password — score: 0.891
# Setting up two-factor authentication — score: 0.743

The query "I forgot my login credentials" never appears in any document, but the search still surfaces the right result. That's the payoff of embedding-based search.

What This Means for Your Architecture

Vector databases aren't a replacement for Postgres or Elasticsearch — they're a complement. Use them when:

Semantic similarity matters more than keyword matching (RAG, recommendations, semantic dedup).
Your data is unstructured — text chunks, images, audio, code snippets.
Exact lookup alone won't cut it — users phrase the same question twelve different ways.

The practical decision isn't which vector database to use. It's understanding what you're indexing, which embedding model fits your data domain, and what recall/latency tradeoff your use case can tolerate.

Get those three things right, and the rest is implementation detail.