If you have used a semantic search feature, a recommendation engine, or a retrieval-augmented generation (RAG) pipeline, you have already relied on a vector database. Yet many developers treat them as a black box. This post explains the core mechanics—embeddings, approximate nearest neighbor (ANN) indexing, and similarity search—so you can make informed decisions about when and how to use them.
A vector database stores, indexes, and queries high-dimensional numerical vectors. Unlike a relational database, which matches rows by exact column values, a vector database finds rows that are similar to a query—even when no exact match exists. That distinction is what makes vector databases useful for unstructured data: text, images, audio, and any other input you can encode numerically.
The workflow has three stages:
An embedding is a fixed-length array of floating-point numbers that encodes the meaning or features of an input. A text embedding model, for example, maps the sentence "The cat sat on the mat" to a vector like [0.12, -0.85, 0.44, ...] with 768 or 1536 dimensions, depending on the model.
The key property: inputs with similar semantics end up geometrically close in the vector space. "The dog rested on the rug" would produce a vector near the cat sentence, while "quarterly revenue targets" would land far away.
You generate embeddings using a pre-trained model. Popular options include OpenAI's text-embedding-3-small, Sentence Transformers from Hugging Face, or Cohere's Embed API. You call the model once per item at ingest time, store the resulting vector alongside any metadata (document ID, source URL, timestamp), and repeat for every document in your corpus.
Storing millions of vectors is straightforward. The hard part is answering the question "which stored vectors are closest to this query vector?" in milliseconds rather than seconds.
Brute-force exact search—computing the distance from the query to every stored vector—is accurate but slow. For 10 million 1536-dimensional vectors, a linear scan can take several seconds per query. That is unacceptable for interactive use.
Approximate Nearest Neighbor (ANN) algorithms trade a small amount of accuracy for large speed gains. The two most widely deployed approaches are:
HNSW (Hierarchical Navigable Small World): Builds a multi-layer graph where each node connects to nearby neighbors. At query time, the algorithm starts at the top (sparse) layer, greedily navigates toward the query, then descends to denser layers to refine the result. HNSW delivers sub-millisecond queries and high recall, but consumes significant memory because the graph lives in RAM.
IVF (Inverted File Index): Clusters vectors using k-means, then assigns each vector to its nearest cluster centroid. A query first identifies the closest centroids (the nprobe parameter controls how many), then searches only those clusters. IVF is more memory-efficient than HNSW but requires tuning nprobe to balance speed against recall.
Most production vector databases—Pinecone, Weaviate, Qdrant, pgvector—expose one or both of these index types, sometimes with compression (scalar or product quantization) layered on top to reduce memory further.
Once the index is built, a similarity search takes a query vector and returns the k most similar stored vectors. The choice of distance metric determines how "similarity" is measured:
Suppose you are building a customer support bot that retrieves relevant knowledge-base articles.
import openai
import qdrant_client
from qdrant_client.models import Distance, VectorParams, PointStruct
client = qdrant_client.QdrantClient(":memory:")
# Create a collection with HNSW indexing
client.create_collection(
collection_name="kb_articles",
vectors_config=VectorParams(size=1536, distance=Distance.COSINE),
)
# Embed and upsert documents
articles = [
{"id": 1, "text": "How to reset your password"},
{"id": 2, "text": "Billing and subscription management"},
{"id": 3, "text": "Connecting your account to Slack"},
]
for article in articles:
vector = openai.embeddings.create(
input=article["text"], model="text-embedding-3-small"
).data[0].embedding
client.upsert("kb_articles", [PointStruct(id=article["id"], vector=vector)])
# Query at runtime
query_vector = openai.embeddings.create(
input="I forgot my login credentials", model="text-embedding-3-small"
).data[0].embedding
results = client.search("kb_articles", query_vector=query_vector, limit=2)
# Returns the "reset your password" article as the top hit
The entire pipeline—embed, index, query—runs in under 50 lines of code. The key insight is that "forgot my login credentials" and "reset your password" have no words in common, yet their embeddings are geometrically close, so the search surfaces the right article.
Vector databases solve a specific problem: finding semantically similar items in large collections of unstructured data, quickly. The three building blocks are:
When you need exact keyword matching, reach for Elasticsearch or Postgres full-text search. When you need semantic similarity—for RAG pipelines, recommendations, duplicate detection, or image search—a vector database is the right tool.
Understanding these internals helps you choose the right index type, tune recall vs. latency trade-offs, and debug unexpected search results. Most of the "magic" in modern AI applications is just well-engineered vector arithmetic.