A practical comparison of RAG databases by scale. Open source and managed options for proof of concept, small volume, larger volume, and enterprise workloads. Plus scaling techniques and how to pick.
The phrase "RAG database" usually means the vector store that holds your document chunks. In practice, a working RAG system needs three things from its database layer:
Pure vector libraries (FAISS, Annoy, ScaNN) do the first one well but are not databases. They have no persistence layer, no concurrent access control, no transactional updates. They belong inside a database, not as one.
A few systems blur the line. PostgreSQL with the pgvector extension is a full relational database that also does vector search. Elasticsearch and OpenSearch are search engines that added vector fields. MongoDB Atlas added vector search to a document database. These hybrid systems often beat pure vector databases on operational simplicity because you already know how to run them.
Every database in this article is either open source, available as a managed service, or both.
Open source databases (pgvector, Chroma, LanceDB, Qdrant, Weaviate, Milvus, Vespa, OpenSearch) give you full control of the data, no per-vector pricing, and no vendor lock-in. The cost is operational: you run the cluster, you patch it, you monitor it, you pay for the compute. Most open source vector databases also offer a managed cloud version run by the same company, which is often the pragmatic choice.
Managed services (Pinecone, Vertex AI Search, Azure AI Search, Bedrock Knowledge Bases, MongoDB Atlas, Redis Cloud) trade flexibility for speed of delivery. You skip the operational layer entirely. The trade-off is data residency considerations, less control over indexing, and pricing that scales with usage.
For an Australian business with sensitive data, an open source vector database self-hosted in an AWS Sydney region (or Azure Australia East, or GCP Australia Southeast 1) is often the right answer. Data stays on infrastructure you control, but you do not have to operate the storage hardware.
Scale: under 100,000 vectors. A few hundred documents at most. One node, single process, no high availability needed. The goal is to prove the RAG idea works on your data, not to ship to production.
What matters at this tier: how fast you can ingest, query, and tear down. You do not want to spend a week setting up infrastructure to test a hypothesis.
What to avoid at this tier: anything that requires a managed cloud account, a sales call, or a multi-node deployment. The PoC tier is about iteration speed, not raw performance.
Scale: 100,000 to 1 million vectors. A few thousand to tens of thousands of documents. Single tenant, modest query volume (under 10 queries per second). This is where most internal knowledge assistants, customer support bots, and small business document search systems land.
What matters: durability, backups, predictable query latency, and ideally hybrid search so exact-term queries (ISO codes, model numbers, policy IDs) still hit.
The deciding factor in this tier is usually existing infrastructure. If you run Postgres, use pgvector. If you run Redis, use Redis vector. If you have nothing, Qdrant Cloud or Pinecone Serverless gives you the fastest path to production.
Scale: 1 million to 50 million vectors. Hundreds of thousands of documents, or millions of small chunks (think product catalogues, support ticket archives, multi-year knowledge bases). Query volume of 10 to 200 queries per second. May span multiple tenants.
What matters: index choice, sharding, query latency under load, hybrid search, and ideally quantization to keep memory costs sensible. Single-node databases start to creak. You want something that can scale horizontally.
At this tier, indexing choice matters. HNSW gives the best latency but uses more memory. IVF saves memory at slightly lower recall. DiskANN (available in Milvus and Vespa) keeps most of the index on disk and is the right answer when memory cost becomes the bottleneck.
Scale: 50 million vectors and up, often into the billions. Multi-tenant by design. Strict requirements around isolation, audit logging, data residency, encryption at rest and in transit, role-based access, and SLAs measured in nines. Query patterns include high concurrency, mixed workloads (search plus filter plus aggregate), and global geographic distribution.
What matters: the maturity of the platform around the database (observability, security, compliance), not just the database itself. The vector search engine is one component in a larger system.
At this tier, the decision is rarely made on benchmarks. It is made on what your security team will approve, what your data residency rules allow, and what your platform team can operate. Pick the option that fits the organisation, not just the workload.
The same database can serve very different scales depending on which scaling features you turn on. Understanding the levers helps you avoid moving databases when load grows.
The biggest single lever. The index determines how the database finds nearest neighbours without comparing the query against every stored vector.
Compress the stored vectors so they take less memory. A 1536-dimensional float32 vector takes 6 KB. The same vector quantized to int8 takes 1.5 KB. Binary quantization takes 192 bytes. The trade-off is recall, but in practice the accuracy drop is small (often under 2%) for huge memory savings.
Most production-grade vector databases support at least scalar quantization. Qdrant, Milvus, and Vespa support binary quantization, which is the most aggressive compression and the most useful at very large scale.
Split the index across multiple nodes. Each shard holds a portion of the vectors. Queries fan out to all shards and merge results. This is the standard horizontal scaling pattern.
Qdrant, Weaviate, Milvus, Pinecone, Elasticsearch, and OpenSearch all support sharding natively. pgvector does not shard automatically; if you outgrow a single Postgres node, you typically move to a purpose-built vector database rather than sharding pgvector manually.
Keep multiple copies of each shard for availability and read throughput. Replication is essential for production deployments and standard in every database listed here at the managed-cloud level. Self-hosted replication adds operational complexity, which is why a managed service starts to look attractive once HA matters.
Combine vector similarity with keyword (BM25) search. Pure vectors miss exact-term queries. Pure keyword search misses semantic matches. Hybrid search beats either alone for almost every business document collection.
Weaviate, Qdrant, Vespa, Elasticsearch, OpenSearch, MongoDB Atlas, and Azure AI Search support hybrid search natively. Pinecone supports it through sparse-dense vectors. pgvector can do it with a CTE that combines a pg_trgm or full-text query with the vector search.
Frequently queried vectors stay in fast storage (RAM, NVMe SSD). Rarely queried vectors move to cheaper storage (S3, Azure Blob, Google Cloud Storage). The database reloads them on demand. This is how vector databases handle billions of vectors without blowing the storage budget.
Pinecone, Milvus, and OpenSearch Serverless implement this natively. Self-hosted setups can approximate it with thoughtful collection design.
Most enterprise RAG systems serve multiple customers, departments, or business units from the same vector database. There are three common patterns:
Searching only documents from the last 30 days, or only documents tagged as "policy", is a metadata filter applied alongside the vector search. The way the database implements this matters at scale.
Pre-filter implementations apply the metadata filter first and then search vectors within the filtered set. Fast when the filter is selective, slow when the filter matches most of the corpus. Post-filter implementations search vectors first and then filter. The opposite trade-off. Qdrant and Weaviate let you tune this. Pinecone and Milvus do it automatically.
If you want a one-paragraph answer for picking a database, here it is.
Start with pgvector if you already run Postgres. Most RAG projects never outgrow it. Move to a purpose-built vector database when you can show a real reason (recall drops, latency exceeds your target, dataset size makes Postgres uncomfortable). Moving too early costs more than moving too late.
No. They are libraries for similarity search, not databases. They have no persistence, no concurrent writes, no transactional updates, no metadata filtering. Most production vector databases use FAISS or a similar library inside, with a database layer built around it. Use them directly only inside a single-process application where you control the lifecycle (rebuilds, restarts, deduplication).
At small scale, no. The fixed cost of a small managed instance is often lower than the engineering time to set up and operate a self-hosted database. Open source becomes cheaper as the dataset grows, because per-vector pricing on managed services scales linearly while a self-hosted cluster scales more like a step function.
Yes. The data format is the same across most databases (high-dimensional float vectors plus metadata). Re-embedding is rarely needed if you stick with the same embedding model. The bigger question is whether your application code is coupled to a specific database. Use a thin retrieval interface so the underlying store is swappable.
For most cases, namespaces or collections inside a shared database give the right balance of cost and isolation. Enforce tenancy at the query layer by always filtering on a tenant_id metadata field. For regulated industries (healthcare, finance, government), separate indexes per tenant or separate databases may be required. Match the isolation strategy to the actual regulatory and contractual requirements rather than overengineering.
For most knowledge bases, no. Vector search plus metadata filtering covers the use cases. Graph databases (Neo4j, ArangoDB) make sense when relationships between entities are core to retrieval (think medical records, supply chain entities, organisational hierarchies). Some teams combine a vector database for semantic retrieval with a graph database for entity-relation lookups. This is sometimes called "GraphRAG" and is still an active area of work.
Annually, or when your scale changes by 10x. Vector database tooling is moving fast. A database that was the right answer 18 months ago may not be today. Keep the retrieval layer behind a clean interface so swapping is not a rewrite.
How vector databases power retrieval, indexing strategies, hybrid search.
The fundamentals of vector storage and similarity search.
How to prepare documents before they hit the vector store.
The full RAG pipeline, retrieval included.
Tell us what you're working on. We'll come back with a practical recommendation and clear next steps.