What is a vector database?
A vector database stores and searches embeddings – numerical representations of text, images, audio, or other data. Instead of looking only for exact keyword matches, it retrieves items whose vectors are close in meaning. In LLM applications, this makes vector databases useful for semantic search, retrieval-augmented generation (RAG), recommendations, and memory.
Hi there! Was this email forwarded to you? Are you new on this website? Sign up here
What's been hot recently? Vector databases! An essential part of the Foundation models/Large Language Models operations cycle, or FM/LLMOps.
The "Vector Database Market" research report forecasts significant growth in the sector from $1.5 billion in 2023 to $4.3 billion by 2028, a CAGR of 23.3% according to a 2023 forecast.
In this Token, we discuss why these databases matter for AI, how they work, and their role in handling complex data; we also explore alternative solutions and provide expert insight on security. Plus you get a curated list of open-sourced vector databases and search libraries. Let's start!
Vector databases have their roots in information retrieval concepts and high-dimensional data indexing techniques developed in the late 20th century. The idea began in the 1960s and 1970s with the vector space model, a method for representing documents not just as plain text but as a series of points in a space with many dimensions, almost like plotting dots on a complex graph. This approach was key for understanding how similar different documents were.
Then, in the 1990s, new techniques like R-trees, KD-trees, and Locality-Sensitive Hashing (LSH) came along. These methods were better at organizing and handling intricate data, paving the way for today's vector databases.
Traditional databases, which most people are familiar with, were great for simple, structured data like numbers and text. However, as the world started dealing with more complex types of data, particularly from fields like machine learning and deep learning, a different kind of database was needed. This is where vector embeddings come into play. These are essentially lists of numbers that represent complex data patterns, like a snapshot of the information a computer learns from data. In the 2010s, vector databases were developed specifically to manage these vector embeddings. They make it easier to store, search, and analyze this advanced data, helping computers to understand and work with it more effectively.
Therefore, Vector databases became an essential part of FMs/LLMs operations or FM/LLMOps. As AI systems move toward agentic architectures, the role of vector databases is expanding — from passive retrieval to active memory and iterative search. See how this shift plays out in agentic vector databases.
Which Vector Databases Are Used with LLMs?
There is no single best option. The sensible choice depends on scale, filters, latency, deployment constraints, and whether your team already runs a database or search platform. These are six widely used options with different strengths:
Tool | What it is | Good fit |
|---|---|---|
A PostgreSQL extension for exact and approximate vector search | Teams that want vectors beside relational data, joins, and existing Postgres operations | |
A fully managed vector database | Production retrieval with minimal infrastructure management | |
An open-source vector database with managed deployment options | Hybrid search, multimodal data, and schema-rich applications | |
An open-source distributed vector database, also available as a managed service | Large collections and distributed vector search | |
An open-source vector search engine with managed deployment options | Filtering-heavy retrieval, hybrid search, and self-hosting | |
An open-source retrieval database designed for AI applications | Prototypes and smaller Python-based applications |
Are Vector Databases Dead?
No. Longer context windows do not remove the need to find fresh, relevant, permissioned information without sending an entire corpus to a model. Dedicated vector databases still make sense when an application needs large-scale similarity search, hybrid retrieval, metadata filtering, multi-tenancy, or careful index operations.
But a separate vector database is not mandatory for every RAG system. PostgreSQL with pgvector, an existing search engine, or even a small in-process index may be enough. Vector retrieval also breaks in predictable ways: the embedding model may not fit the domain, chunks may be poorly designed, the index may be stale, filters may reduce recall, or approximate search may miss useful results. Evaluate the whole retrieval pipeline – including citations, access control, deletion, latency, and answer quality – rather than judging the database in isolation.
Why are vector embeddings crucial for LLMs?
Vector embeddings are numerical representations of data objects (like words or images) in a high-dimensional space, and that’s crucial for foundation models and LLMs. Here is why:
Semantic Information Capture: They encode semantic and syntactic information, enabling models to understand context and meaning.
Generalization: Embeddings help models generalize from training data to novel inputs.
Transfer Learning* Enablement: Facilitate the adaptability of foundation models to specific tasks while retaining broad knowledge.
High-Dimensional Data Handling: They effectively reduce data dimensionality, making it more manageable for neural networks.
Computational Efficiency: Lower the computational complexity by transforming sparse data into denser, lower-dimensional representations.
Transfer learning is applying knowledge gained in one task to enhance learning in a related but different task. Foundation models are enabled by transfer learning and scale.
So vector databases play a supportive role in FMOps/LLMOps by offering a specialized environment and efficiently managing the high-dimensional vector embeddings these models produce. This facilitates quick access needed for FM&LLM processing. Designed to handle complex data, these databases enable effective semantic searches within the embeddings, important for contextual language tasks. Their scalability and performance are essential for handling the large volumes of data generated by LLMs, ensuring smooth data management and contributing to the practical functionality of these AI models.
How a Vector Database Pipeline Works
Imagine a pipeline resembling a sophisticated assembly line in a factory:
Data Ingestion: at the start, raw materials (data) are prepared and refined (might involve cleaning, normalization, or other transformations to make it suitable for analysis by the foundation model).
Vectorization: Preprocessed data is fed into a foundation model/LLM, where it's converted into vectors. These vectors are numerical forms capturing the main features and characteristics of the data.
Indexing: The vectors are then sorted and organized in an elaborate storage system, ready for quick retrieval.
Query Processing: Upon request, the exact needed items (query results) are rapidly located and delivered, possibly undergoing some final touch-ups (post-processing) to ensure they meet the user's needs perfectly.
Post-processing: The final step is to convert these vector results back into a human-readable form or into a format that can be used by other systems. This might involve mapping vectors back to their corresponding data entities (like text snippets, image thumbnails, etc.).
Feedback Loop (Optional): User interactions, such as clicks or likes, can be used to refine the foundation model and vectorization for improved accuracy and relevance.
Vector Database Pipeline. Source: Turing Post
Can I Use Vector Search Libraries/Vector Indices instead?
Some people use vector search libraries, such as Approximate Nearest Neighbor (ANN) search, for tasks like semantic search, where you need to quickly find the most relevant text embeddings from a massive pool, or in recommendation systems where finding items similar to a user's interests is essential; and argue that vector databases are just a wrapper around ANN.
But they are not. A vector database as a whole is a more extensive system designed for the comprehensive management of vector data, offering features like cloud-nativity, multi-tenancy, and scalability — critical for production AI agent workloads. In contrast, libraries like FAISS, ScaNN, and HNSW are lightweight ANN tools for building vector indices, suitable for smaller datasets but limited in scalability.
Vector databases operate at a different abstraction level compared to vector search libraries. Vector databases are complete services, while ANN libraries are components to be integrated into applications. For example, inserting unstructured data into a vector database is straightforward, but it's more complex and less scalable with libraries like FAISS or ScaNN.
Can I use Vector Search Plugins for Traditional Databases?
Sometimes, traditional databases and search systems can be optimized with vector search plugins. However, these plugins are not comprehensive solutions for vector data management. They lack full-stack capabilities for embedding management and vector search, making them less optimized for unstructured data applications.
So the answer is – yes, it's possible to use vector search libraries or indices, or vector search plugins for traditional databases. However, a vector database offers a more integrated solution for optimizing Foundation Model and Large Language Model operations.
Vector Database Security: What to Ask Before You Choose
We asked Patrick Walsh, CEO and co-founder of IronCore Labs, to answer these questions.
“Vector databases are relatively young and immature from a security perspective. They’re a target for copies of a lot of data in better-protected and better-understood systems. When you’re considering which vector database to use, here are three questions to ask:
How is the data protected? If it’s encrypted at rest, that means it’s effectively unencrypted on a running server where anyone with access to it can see the unencrypted data. Ask for application-layer encryption that will protect insiders and attackers from seeing your data, whether it's held by a third party or in your infrastructure.
If hosted by a third party, who has access to your data (both the embeddings and metadata)? Do engineers, for example? Is there a fraud detection workflow that sends some of your data to a group of anonymous reviewers as happens at Microsoft with prompts in their OpenAI models?
What sort of permissions are supported? Can you make it so an API key that can query can’t also manage and delete indices? Can you make it so a user can only see data that they have privileges for?
There are only a few solutions on the market aimed at securing vector databases, and most are aimed at redacting information before it enters a vector database.
Sometimes people assume embeddings are already secure since they’re not immediately human-readable. While embeddings are machine representations of data that aren’t meaningful to us, they are very meaningful to the computer. Embeddings can easily be reversed back to their original inputs – whether they represent sentences, images, or biometrics. Open-source tools are widely available that make it easy to attack vector databases. These tools are based on papers that have had huge success, for example, a paper out of Cornell recovered 92% of exact phrases, including full names, back out of sentence embeddings.” Embedding inversion is only one entry point. Poisoned documents and retrieval-borne prompt injection reach the model through the same pipeline, which we map in our guide to LLM vulnerabilities.
Vector Databases in FMOps: Key Takeaways
Recent developments have thrust vector databases into the spotlight, where they play a vital role in the world of advanced AI, especially Large Language Models (LLMs). This article explores why these databases are crucial for AI, how they handle complex data, and what considerations to keep in mind when choosing one. The important questions from our experts help address security concerns. Given their significance in FMOps, expecting notable growth and sustained investment is reasonable. Keeping an eye on future vector database innovations, such as distributed architectures and indexing methods, is crucial for AI leadership.
Frequently Asked Questions
What is a vector database?
A vector database stores embeddings and retrieves the items whose vectors are most similar to a query vector. This enables meaning-based search across text, images, audio, and other data.
How does a vector database work with an LLM?
The application converts a user query into an embedding, searches the database for similar items, and sends the most relevant results to the LLM as context. This is a common RAG workflow.
Which vector databases are commonly used with LLMs?
Common choices include pgvector, Pinecone, Weaviate, Milvus, Qdrant, and Chroma. The right choice depends on scale, filtering, latency, hosting, and the infrastructure a team already operates.
Do I need a vector database for RAG?
Not always. Small projects may work with an in-process index, while teams already using PostgreSQL or a search engine may add vector search there. A dedicated database becomes more useful as scale, filtering, multi-tenancy, and operational requirements grow.
Are vector databases obsolete because LLM context windows are larger?
No. Large context windows can reduce retrieval for some small, static datasets, but they do not solve freshness, permissions, citations, latency, or the cost of repeatedly sending large corpora to a model.
Useful links
For a comprehensive list of open-source vector databases, embedding libraries, and platforms with vector capabilities, see our curated open-source vector databases and libraries.
Please give us feedback
Thank you for reading, please feel free to share with your friends and colleagues. In the next couple of weeks, we are announcing our referral program 🤍
Previously in the FM/LLM series:
