Vector Databases Explained

Vector databases power modern AI applications by enabling lightning-fast similarity search across millions of high-dimensional embeddings. Understanding how they work, from ANN algorithms to indexing strategies, is essential for building retrieval-augmented generation, recommendation engines, and semantic search.

The Problem Vector Databases Solve

Traditional databases excel at exact matches. Give me every user where country = 'US' and age > 25. These queries are fast because databases index discrete values with B-trees and hash maps. The entire field of relational database optimization is built around efficiently answering queries on structured, categorical, or numerical data.

But what if your query is "find me images that look like this one" or "find documents that mean something similar to this paragraph"? There is no column to filter on. The concept of "similar" lives in a continuous, high-dimensional space that traditional indexes cannot navigate. You cannot build a B-tree over the concept of semantic meaning.

Vector databases solve this problem. They store and search high-dimensional numerical vectors (arrays of floating-point numbers) and find the ones most similar to a query vector. This capability underpins modern AI applications from semantic search to recommendation engines to retrieval-augmented generation (RAG).

The market has exploded. Purpose-built vector databases like Pinecone, Weaviate, Milvus, and Qdrant have raised hundreds of millions in venture capital. Traditional databases have added vector extensions (pgvector for PostgreSQL, Atlas Vector Search for MongoDB). The technology has moved from academic curiosity to production infrastructure in under three years.

What Are Embeddings?

Before vectors can be searched, data must be converted into vectors. This conversion is called embedding, and it is the foundation of everything vector databases do.

An embedding model takes unstructured data (text, images, audio) and maps it to a fixed-size numerical vector. The key property: similar items produce similar vectors. A sentence about "machine learning algorithms" will have a vector close to a sentence about "AI training methods" but far from a sentence about "chocolate cake recipes."

Text embedding models like OpenAI's text-embedding-3-large, Cohere's embed-v3, or open-source models like BGE, E5, and GTE produce vectors with 768 to 3072 dimensions. Each dimension captures some learned aspect of meaning, though individual dimensions are not human-interpretable. The model learns these dimensions during training on massive text corpora, organizing the vector space so that semantically related content clusters together.

Image embeddings work similarly. Models like CLIP produce vectors where images of dogs cluster together, separate from images of cats, and an image of a golden retriever sits closer to other golden retrievers than to bulldogs. CLIP is particularly interesting because it embeds both images and text into the same vector space, enabling cross-modal search.

Audio embeddings (from models like OpenAI's Whisper encoder or CLAP) map sound to vectors. Music recommendation, voice search, and audio similarity all rely on this capability.

The embedding model determines the quality of your search. A better model produces vectors where "similar" aligns more closely with your definition of similar. Choosing the right embedding model is often more impactful than choosing the right vector database. Benchmark your embedding model on your specific data before committing to a database.

Similarity Metrics: Measuring Closeness

Given a query vector and a collection of stored vectors, similarity search finds the closest matches. "Close" is defined by a distance metric, and the choice of metric matters.

Cosine similarity measures the angle between two vectors, ignoring their magnitude. Two vectors pointing in the same direction have cosine similarity of 1, regardless of their length. This is the most common metric for text embeddings because embedding models typically normalize their output, making magnitude meaningless. The formula is the dot product divided by the product of magnitudes.

Euclidean distance (L2) measures the straight-line distance between two points in the vector space. Unlike cosine similarity, it accounts for magnitude. Useful when the scale of the vector matters, such as when embedding models produce unnormalized vectors or when you want to distinguish between "strongly about topic X" and "weakly about topic X."

Dot product (inner product) combines direction and magnitude. It is equivalent to cosine similarity when vectors are normalized (which many embedding models produce by default). Some systems use dot product for efficiency since it requires fewer operations than cosine similarity.

Manhattan distance (L1) sums the absolute differences across dimensions. Less common but useful in specific domains where individual dimensions represent independent features.

The naive approach to finding nearest neighbors computes the distance between the query vector and every stored vector. This is called brute-force search or exact nearest neighbor search. It guarantees perfect results but scales linearly with dataset size. For a million vectors at 1536 dimensions, each query requires approximately 6 billion floating-point operations. At a billion vectors, this is impractical for real-time applications.

Approximate Nearest Neighbors (ANN)

The breakthrough that makes vector databases practical is accepting "close enough" instead of "exact." Approximate Nearest Neighbor algorithms trade a small amount of accuracy (typically finding 95-99% of the true nearest neighbors) for orders-of-magnitude speed improvements.

HNSW (Hierarchical Navigable Small World)

HNSW is the most widely used ANN algorithm in production systems. It builds a multi-layer graph where each node is a vector and edges connect nearby vectors.

The bottom layer contains every vector, densely connected. Higher layers contain progressively fewer vectors with longer-range connections. Think of it like a transportation network: highways (top layers) for covering large distances quickly, then local roads (bottom layers) for the final approach.

Search starts at the top layer, greedily navigating toward the query vector's neighborhood, then descends to lower layers for finer-grained search. This hierarchical approach achieves logarithmic search complexity: searching a billion vectors might require evaluating only a few thousand.

HNSW parameters that matter: M controls the number of connections per node (typical values: 16-64; higher means better recall but more memory and slower builds). efConstruction controls index build quality (typical: 100-400; higher means slower builds but better graph structure). At query time, efSearch controls the accuracy/speed tradeoff (typical: 50-200; higher means better recall but slower queries).

The primary downside of HNSW is memory consumption. The graph structure adds significant overhead on top of the raw vector storage. A million 1536-dimensional vectors might need 6GB for vectors plus 8-16GB for the graph structure.

IVF (Inverted File Index)

IVF partitions the vector space into clusters using k-means clustering. Each query vector is compared to cluster centroids first, and then only vectors in the nearest clusters are searched exhaustively.

The nlist parameter controls the number of clusters (typical: sqrt(N) where N is vector count). The nprobe parameter controls how many clusters to search at query time. More clusters with fewer probes means faster but less accurate; fewer clusters with more probes means slower but more accurate.

IVF is memory-efficient because it does not store a graph structure, but it typically achieves lower recall than HNSW at the same query speed. It excels at batch queries where throughput matters more than single-query latency.

Product Quantization (PQ)

PQ compresses vectors to reduce memory consumption. It splits each vector into sub-vectors (segments) and quantizes each segment independently using a codebook. A 1536-dimensional float32 vector (6KB) can be compressed to 48-192 bytes, a 30-125x reduction.

The codebook is learned from the data during index construction. Each segment is replaced by the index of its nearest codebook entry. Distance computation works directly on the compressed representation using precomputed lookup tables.

PQ is often combined with IVF (IVF-PQ) to handle billion-scale datasets that would not fit in memory otherwise. The tradeoff is reduced accuracy due to quantization error, but with enough segments and codebook entries, the degradation is manageable.

ScaNN and DiskANN

Google's ScaNN uses anisotropic vector quantization, which accounts for the direction of quantization error relative to the query. Standard quantization treats all directions equally, but errors in the direction of the query matter more than orthogonal errors. This produces better results than standard PQ at the same compression ratio.

Microsoft's DiskANN stores the index on SSD rather than in memory, enabling billion-scale search on commodity hardware. It uses a Vamana graph (similar to HNSW) with a caching strategy that keeps frequently accessed nodes in memory while storing the full graph on disk. Query latency is higher than in-memory indexes (1-5ms vs sub-millisecond) but the cost savings are dramatic.

The Vector Database Landscape

Pinecone

A fully managed, cloud-native vector database. You send vectors through an API; Pinecone handles indexing, replication, scaling, and infrastructure. Strengths: zero operational overhead, consistent performance, metadata filtering, hybrid search combining vectors with keyword matching. Limitations: vendor lock-in, cost at scale (pricing per vector stored and queried), no self-hosted option. Best for: teams that want to ship fast without managing infrastructure.

Weaviate

An open-source vector database with built-in vectorization. You can send raw text or images, and Weaviate calls embedding models automatically through its module system. Supports hybrid search (combining BM25 keyword matching with vector similarity using reciprocal rank fusion). Can be self-hosted or used as Weaviate Cloud Services. Strengths: schema-aware, built-in vectorization, GraphQL API. Limitations: more complex to operate than simpler alternatives, resource-intensive.

Chroma

A lightweight, open-source embedding database designed for AI application development. Runs in-process (embedded mode) or as a server. Popular for prototyping and smaller-scale applications. Strengths: dead simple API, pip-installable, great for experimentation and development. Limitations: limited scalability, fewer production features than purpose-built databases.

Milvus

An open-source vector database built for scale, developed by Zilliz. Supports multiple index types (HNSW, IVF, DiskANN, GPU-accelerated indexes), hybrid search, and multi-tenancy. Used in production at scale by organizations processing billions of vectors. Strengths: massive scale, GPU acceleration, rich index options. Limitations: complex to operate, significant resource requirements for the distributed deployment.

pgvector

A PostgreSQL extension that adds vector similarity search to your existing Postgres database. Store vectors alongside relational data, query them with SQL. Strengths: no new infrastructure, familiar tooling, transactional consistency, join vectors with relational data in a single query. Limitations: not as fast as purpose-built vector databases at large scale (though pgvector 0.7+ with HNSW indexing has narrowed the gap significantly), limited to PostgreSQL.

Qdrant

A Rust-based, open-source vector database with strong filtering capabilities. Supports payload (metadata) filtering during vector search, which is critical for production use cases where you need "find similar items that also match these criteria." Written in Rust for performance and memory safety. Strengths: fast, efficient filtering, good documentation, gRPC API. Limitations: smaller ecosystem than Milvus or Weaviate.

Use Cases in Production

Retrieval-Augmented Generation (RAG)

The most common use case today. Embed your documents, store them in a vector database, and retrieve relevant chunks when a user asks a question. The retrieved chunks are injected into the LLM's context, grounding its response in your actual data rather than its training data.

Critical considerations for RAG: chunk size affects retrieval quality (too large dilutes relevance, too small loses context; 256-512 tokens is a common starting point). Embedding model choice matters more than database choice. Overlap between chunks (50-100 tokens) preserves context at boundaries. Re-ranking retrieved results with a cross-encoder model (like Cohere's reranker or BGE-reranker) often improves final quality by 10-30%. Hybrid search (vector + BM25) catches cases where keyword matching outperforms semantic matching.

Traditional keyword search fails when users express concepts differently from how documents are written. A search for "how to fix a leaky faucet" should match a document titled "Repairing Dripping Taps." Vector search handles this naturally because both phrases map to similar embeddings.

Hybrid search (combining vector similarity with BM25 keyword matching) often outperforms either approach alone. Most production search systems use this combination. The vector component catches semantic similarity while the BM25 component handles exact term matches and rare terms that embedding models may underweight.

Recommendation Engines

Embed items (products, articles, songs) and users into the same vector space. A user's embedding reflects their preferences; item embeddings reflect their characteristics. Finding items close to a user's embedding produces personalized recommendations. This approach handles the cold start problem better than collaborative filtering because it works with item features rather than requiring interaction history.

Anomaly Detection

Normal behavior produces embeddings that cluster together. Anomalous behavior produces outlier embeddings far from any cluster. Vector databases can efficiently identify vectors that are far from any cluster centroid, flagging potential anomalies for review. Applications include fraud detection, network intrusion detection, and quality control in manufacturing.

Models like CLIP embed both images and text into the same vector space. This enables searching an image library with text queries ("sunset over mountains") or finding images similar to a reference image. Fashion, e-commerce, and content moderation all use this capability. Newer multimodal models extend this to video, audio, and other modalities.

When You Need One (and When You Do Not)

You probably need a vector database if you have more than 100,000 vectors, if you need sub-second query latency at scale, if your vectors change frequently (adds, updates, deletes), if you need to combine vector similarity with metadata filtering, or if you are building a production RAG system with multiple data sources.

You probably do not need a dedicated vector database if you have fewer than 50,000 vectors (brute-force search on NumPy arrays works fine and takes milliseconds), if your vectors are static and you only need batch processing, if you are prototyping and can tolerate slower queries, or if your use case is simple enough that pgvector alongside your existing PostgreSQL database would handle it.

pgvector occupies an interesting middle ground. If you already run PostgreSQL and your scale is moderate (under 5-10 million vectors), pgvector lets you add vector search without new infrastructure. The operational simplicity often outweighs the performance gap versus purpose-built solutions. You also get the enormous advantage of joining vector search results with relational data in a single query, something no standalone vector database can match.

Operational Considerations

Memory planning: HNSW indexes live in memory. A million 1536-dimensional float32 vectors consume approximately 6GB for the vectors alone, plus 2-4x for the graph structure. Plan for 15-25GB per million vectors as a conservative estimate. PQ-based indexes reduce this dramatically at the cost of accuracy.

Indexing latency: Building an HNSW index is not instant. A million vectors might take 5-15 minutes; a billion vectors can take hours or days. Some databases support incremental indexing (adding vectors to an existing index); others require full rebuilds. Understand your database's indexing model before committing.

Consistency model: Unlike relational databases, most vector databases offer eventual consistency. A newly inserted vector might not appear in search results immediately. If your application requires strong consistency (e.g., a user uploads a document and immediately searches for it), verify your database's guarantees and plan accordingly.

Filtering performance: Real queries often combine vector similarity with metadata filters ("find similar products under $50 in the electronics category"). Pre-filtering (filter then search) may miss similar vectors outside the filter. Post-filtering (search then filter) may return too few results if the filter is selective. Most modern vector databases use a hybrid approach, but the performance characteristics vary significantly.

Dimensionality: Higher-dimensional vectors capture more information but require more memory and slower search. The "curse of dimensionality" means that in very high dimensions, distance metrics become less discriminating, as all vectors become roughly equidistant. Most practical applications use 256 to 3072 dimensions. If your embedding model produces very high-dimensional vectors, consider dimensionality reduction (PCA, Matryoshka embeddings) as a preprocessing step.

Embedding model versioning: When you upgrade your embedding model, all existing vectors must be re-embedded. The new model produces vectors in a different space that are incompatible with old vectors. Plan for periodic re-embedding and design your pipeline to handle this migration gracefully.

Building a Production Pipeline

A production vector search pipeline involves several components beyond the database itself.

Embedding service: A model server (or API) that converts raw data into vectors. Must handle batching for throughput and caching for frequently embedded queries. Consider running embedding models locally (with ONNX or TensorRT) for latency-sensitive applications.

Chunking strategy: For documents, you must decide how to split text into searchable units. Fixed-size chunking (e.g., 512 tokens) is simple but can split sentences or paragraphs awkwardly. Semantic chunking (splitting at paragraph or topic boundaries) often outperforms fixed-size chunking. Recursive chunking tries large chunks first and subdivides only when needed. Overlap between chunks (10-20%) preserves context at boundaries.

Index management: As your data grows, you need strategies for incremental updates, periodic re-indexing, handling embedding model upgrades (which require re-embedding everything), and managing index versions. Consider blue-green deployment patterns for index updates.

Monitoring: Track recall (are the right results being returned?), latency (p50, p95, p99), throughput (queries per second), and resource utilization (memory, CPU, disk). Set up alerts for latency spikes and recall degradation. Monitor embedding model drift over time.

Caching: Frequently issued queries can be cached at the vector search layer. If your application has a power-law query distribution (some queries are much more common than others), caching the top queries reduces load and latency significantly.

The vector database itself is just one component in a larger system. Getting the retrieval pipeline right requires attention to every stage, from data preparation through embedding to search and re-ranking. The database stores and indexes the vectors, but the quality of those vectors, the intelligence of your chunking strategy, and the sophistication of your re-ranking pipeline determine whether your application succeeds or merely runs.