An Introduction to Vector Databases

The rise of large language models and novel implementations such as retrieval augmented generation (RAG) Have been enabled by vector databases. Vector database are a type of database which is optimised for storing vectors. Vector databases have been key to the success of RAG and large language models (LLMs) because the transformer model which underpins LLMs is not able to process textual data. Instead, transformers process text which has been tokenized and stored in vectors. This article provides an introduction to vector databases and their benefits for LLM based applications.

‍

What Are Vector Databases?

To fully understand vector databases, it’s essential to understand what vector data is. Vector data refers to a coordinated representation of an object. A spatial object would be represented by a coordinate location (x,y) whereas non-spatial objects such as text can be tokenized and then mapped to a vector using a vectorisation process.

The Characteristics of Vector Databases

Handling Vector Data

In order to maintain efficient processing and retrieval, vector databases store vector data in a hierarchical manner.

‍

Additionally, vector databases utilise indexing to make accessing of vector data as fast as possible. This is accomplished through the use of indexing structures such as Quad-trees, R-trees, or grids, all of which make attending to input queries easier.

‍

RAG works by vectorising the prompt from the LLM and searching for the closest vector in the vector database in order to retrieve the answer to the prompt.

Geospatial Applications

Beyond generative AI, vector databases can be used for geospatial applications. Namely, vector databases are used in urban planning, environmental management, geographic information systems (GIS), disaster/emergency response, and asset management. Vector databases provide the framework necessary for managing, analysing, and implementing visuospatial data.

‍

What Makes Up Vector Databases

Vector databases consist of several components that make it possible for them to store vector data efficiently, analyse the data, and answer queries accurately.

‍

Storage Engine

The storage engine of vector databases is responsible for storing and retrieving vector data from memory.

‍

The storage engine implements highly specific data structures that are ideal for storing spatial (vector) data. As mentioned above, structures such as indexes, quad-trees, and R-trees are examples of the storage methods employed by storage engines.

‍

Indexing and Querying

Indexing and querying involve improving the performance of retrieval and query processing.

‍

For instance, spatial indexes order vector data using spatial metrics, such as through the nearest neighbour approach. Additionally, this portion of vector databases processes queries in the most efficient order.

‍

Geospatial Functions

Lastly, the geospatial functions of vector databases perform the statistical analysis and processing of vector data. This can include calculations, spatial predictions, topology assessments, and more! Interestingly, geospatial functions allow users to perform a wide range of manipulations to vector data and create outputs that enable model building, among other useful applications.

Applications of Vector Databases

Vector databases are useful in a variety of capacities, such as in geographic information systems, location based services, and LLM powered RAG systems.

‍

Retrieval Augment Generation (RAG)

Large language models can leverage private data in order to perform tasks or respond to queries. The private data can be embedded into a vector database which can then assist the LLM with its tasks and queries. In this case, the prompt is vectorised via the same vector embedding technique and used to query the vector database. The similar vectors are then used to provide context to the LLM when it tries to perform its tasks. RAG is not suitable for applications which require accurate querying because the querying of the vector database is approximate.

‍

Geographic Information Systems (GIS)

Geographic information systems (GIS) display spatial or geographic information and enable users to manage, analyse, and present this information.

‍

GISs capture data from several sources, depending on what is loaded into them. For example, satellite imagery, cities, boundaries, elevation models, and more can be displayed with GIS.

‍

Conclusion

This article has provided an overview of vector databases and their applications across geospatial and AI applications. Whilst vector databases have proven to be a key enabled of LLM based applications such as RAG, they are not suitable for use cases which require accurate results. This is why TextMine leverages knowledge graphs for queries which require factual results. To find out more about our approach and products you can read our article on knowledge graphs and our overview of Vault.

‍

About TextMine

TextMine is an easy-to-use data extraction tool for procurement, operations, and finance teams. TextMine encompasses 3 components: Legislate, Vault and Scribe. We’re on a mission to empower organisations to effortlessly extract data, manage version controls, and ensure consistency access across all departments. With our AI-driven platform, teams can effortlessly locate documents, collaborate seamlessly across departments, making the most of their business data.