Token 1.12: What is Vector Database's Role in FMOps?

What's been hot recently? Vector databases! An essential part of the Foundation models/Large Language Models operations cycle, or FM/LLMOps.

The "Vector Database Market" research report forecasts significant growth in the sector from $1.5 billion in 2023 to $4.3 billion by 2028, a CAGR of 23.3%.

In this Token, we discuss why these databases matter for AI, how they work, and their role in handling complex data; we also explore alternative solutions and provide expert insight on security. Plus you get a curated list of open-sourced vector databases and search libraries. Let's start!

Vector databases have their roots in information retrieval concepts and high-dimensional data indexing techniques developed in the late 20th century. The idea began in the 1960s and 1970s with the vector space model, a method for representing documents not just as plain text but as a series of points in a space with many dimensions, almost like plotting dots on a complex graph. This approach was key for understanding how similar different documents were.

Then, in the 1990s, new techniques like R-trees, KD-trees, and Locality-Sensitive Hashing (LSH) came along. These methods were better at organizing and handling intricate data, paving the way for today's vector databases.

Traditional databases, which most people are familiar with, were great for simple, structured data like numbers and text. However, as the world started dealing with more complex types of data, particularly from fields like machine learning and deep learning, a different kind of database was needed. This is where vector embeddings come into play. These are essentially lists of numbers that represent complex data patterns, like a snapshot of the information a computer learns from data. In the 2010s, vector databases were developed specifically to manage these vector embeddings. They make it easier to store, search, and analyze this advanced data, helping computers to understand and work with it more effectively.

Therefore, Vector databases became an essential part of FMs/LLMs operations or FM/LLMOps. As AI systems move toward agentic architectures, the role of vector databases is expanding — from passive retrieval to active memory and iterative search. See how this shift plays out in agentic vector databases.

Vector embeddings are numerical representations of data objects (like words or images) in a high-dimensional space, and that’s crucial for foundation models and LLMs. Here is why:

So vector databases play a supportive role in FMOps/LLMOps by offering a specialized environment and efficiently managing the high-dimensional vector embeddings these models produce. This facilitates quick access needed for FM&LLM processing. Designed to handle complex data, these databases enable effective semantic searches within the embeddings, important for contextual language tasks. Their scalability and performance are essential for handling the large volumes of data generated by LLMs, ensuring smooth data management and contributing to the practical functionality of these AI models.

Imagine a pipeline resembling a sophisticated assembly line in a factory:

Feedback Loop (Optional): User interactions, such as clicks or likes, can be used to refine the foundation model and vectorization for improved accuracy and relevance.

Some people use vector search libraries, such as Approximate Nearest Neighbor (ANN) search, for tasks like semantic search, where you need to quickly find the most relevant text embeddings from a massive pool, or in recommendation systems where finding items similar to a user's interests is essential; and argue that vector databases are just a wrapper around ANN.

But they are not. A vector database as a whole is a more extensive system designed for the comprehensive management of vector data, offering features like cloud-nativity, multi-tenancy, and scalability offering features like cloud-nativity, multi-tenancy, and scalability — critical for production AI agent workloads. In contrast, libraries like FAISS, ScaNN, and HNSW are lightweight ANN tools for building vector indices, suitable for smaller datasets but limited in scalability.

Vector databases operate at a different abstraction level compared to vector search libraries. Vector databases are complete services, while ANN libraries are components to be integrated into applications. For example, inserting unstructured data into a vector database is straightforward, but it's more complex and less scalable with libraries like FAISS or ScaNN.

Sometimes, traditional databases and search systems can be optimized with vector search plugins. However, these plugins are not comprehensive solutions for vector data management. They lack full-stack capabilities for embedding management and vector search, making them less optimized for unstructured data applications.

So the answer is – yes, it's possible to use vector search libraries or indices, or vector search plugins for traditional databases. However, a vector database offers a more integrated solution for optimizing Foundation Model and Large Language Model operations.

We asked Patrick Walsh, CEO and co-founder of IronCore Labs, to answer these questions.

“Vector databases are relatively young and immature from a security perspective. They’re a target for copies of a lot of data in better-protected and better-understood systems. When you’re considering which vector database to use, here are three questions to ask:

There are only a few solutions on the market aimed at securing vector databases, and most are aimed at redacting information before it enters a vector database.

Sometimes people assume embeddings are already secure since they’re not immediately human-readable. While embeddings are machine representations of data that aren’t meaningful to us, they are very meaningful to the computer. Embeddings can easily be reversed back to their original inputs – whether they represent sentences, images, or biometrics. Open-source tools are widely available that make it easy to attack vector databases. These tools are based on papers that have had huge success, for example, a paper out of Cornell recovered 92% of exact phrases, including full names, back out of sentence embeddings.”

Recent developments have thrust vector databases into the spotlight, where they play a vital role in the world of advanced AI, especially Large Language Models (LLMs). This article explores why these databases are crucial for AI, how they handle complex data, and what considerations to keep in mind when choosing one. The important questions from our experts help address security concerns. Given their significance in FMOps, expecting notable growth and sustained investment is reasonable. Keeping an eye on future vector database innovations, such as distributed architectures and indexing methods, is crucial for AI leadership.

Useful links

For a comprehensive list of open-source vector databases, embedding libraries, and platforms with vector capabilities, see our curated open-source vector databases and libraries.

Thank you for reading, please feel free to share with your friends and colleagues. In the next couple of weeks, we are announcing our referral program 🤍