Vector Databases Explained: The Storage Layer for AI Search
How vector databases store meaning and enable AI-powered search. What they are, how they work, and which options to consider for your RAG system.
What is a vector database?
A vector database is a specialised database designed to store and search vectors: lists of numbers that represent the meaning of text, images, or other data. In the context of AI and RAG systems, it's where you store the embeddings of your document chunks.
Traditional databases search by exact match or keyword. Vector databases search by similarity. This means you can find content that's semantically similar to a query, even if it uses completely different words.
How they work
Here's a simplified version:
- Your text is converted into a vector (a list of perhaps 1,536 numbers) by an embedding model.
- The vector is stored in the database alongside the original text and any metadata.
- When you search, your query is also converted into a vector.
- The database calculates how "close" the query vector is to every stored vector.
- The closest matches are returned. Those are the most semantically similar results.
The "closeness" is measured using mathematical methods like cosine similarity or Euclidean distance. In practice, you don't need to worry about the maths. The database handles it.
Why this matters: A user searching for "annual leave policy" will find a document titled "Employee Holiday Entitlements" because the meanings are similar, even though the words are different.
Why RAG needs them
In a RAG system, the vector database is the retrieval engine. When a user asks a question:
- The question is embedded into a vector
- The vector database finds the most relevant document chunks
- Those chunks are passed to the language model as context
- The model generates an answer grounded in the retrieved content
Without a vector database, you'd need to search by keyword, which misses synonyms, paraphrases, and conceptual matches. Keyword search finds "leave policy." Vector search also finds "time off entitlements," "holiday allowance," and "PTO guide."
Popular options
| Database | Type | Best for |
|---|---|---|
| pgvector | PostgreSQL extension (free) | Teams already using PostgreSQL, smaller datasets, cost-sensitive |
| Pinecone | Managed cloud service | Fast setup, low ops overhead, scaling without management |
| Weaviate | Open-source / managed | Hybrid search (vector + keyword), complex filtering |
| OpenSearch | Open-source / AWS managed | Existing AWS stacks, combined text + vector search |
| Qdrant | Open-source / managed | High performance, advanced filtering, self-hosted |
| ChromaDB | Open-source | Prototyping, local development, small projects |
How to choose
For most Australian business RAG deployments, we recommend:
- If you're already on PostgreSQL: Start with pgvector. It's free, familiar, and handles moderate scale well.
- If you want managed with minimal ops: Pinecone or Weaviate Cloud. More expensive but you don't manage infrastructure.
- If you're on AWS: Amazon OpenSearch with vector search is a natural fit and keeps data in your VPC.
- If you need hybrid search: Weaviate or OpenSearch both handle vector + keyword search natively.
The choice matters less than you think at the start. Most vector databases perform similarly for datasets under a million documents. Start with what's easy and migrate later if needed.
Key takeaways
- Vector databases store numerical representations of text that capture meaning, not just keywords.
- They enable semantic search, finding content that means the same thing even with different words.
- For most business RAG systems, pgvector (free, runs on PostgreSQL) or a managed service like Pinecone work well.
- The vector database is a critical but not the only component. Chunking and retrieval strategy matter just as much.