Indexing

Weaviate uses two types of indexes to facilitate fast, scalable search and filtering. It uses vector indexes for vector searches and inverted indexes for filtering and keyword searches.

This page introduces the vector indexes and inverted indexes available in Weaviate, so you can arrive at an informed indexing strategy.

Weaviate offers four types of vector indexes, Hierarchical Navigable Small World (HNSW) indexes, flat indexes, dynamic indexes, and HFresh indexes.

• HNSW indexes enable fast, scalable vector searching that works well even with very large data sets.
• Flat indexes are memory-efficient indexes that work best with small data sets.
• Dynamic indexes switch from a flat index to an HNSW index when a collection or tenant reaches a threshold size.
• HFresh indexes use a cluster-based approach with HNSW for centroids, providing memory efficiency by keeping most data on disk.

HNSW are high-performance, in-memory indexes. HNSW indexes scale well, meaning that vector searches remain fast even for very large data sets.

HNSW indexes achieve this by building a multi-layered graph of objects, allowing for fast, approximate nearest neighbor searches.

While HNSW indexes enable fast searches, they use a lot of hot resources, as they load the graph structure and vectors into memory.

Consider using compression to reduce the size of for your HNSW indexes. Weaviate offers several ways to compress your data:

• Rotational Quantization (RQ) (recommended)
  RQ reduces the size of each vector dimension from 32 bits to 8 bits (or 4 bits, or 1 bit) without requiring training. RQ first applies a fast pseudorandom rotation to the vector, then quantizes each dimension. The rotation spreads information evenly across dimensions, so 8-bit RQ reaches up to 98-99% recall with no configuration and no training phase. The 4-bit and 1-bit widths compress further and depend on rescoring for their recall.

• Product Quantization (PQ)
  PQ reduces the size of the vector embedding in two ways. PQ trains on your data to create custom segments. PQ creates segments to reduce the number of dimensions, and segments are stored as 8 bit integers instead of 32 bit floats. Compared to dimensions, there are fewer segments and each segment is much smaller than a single dimension.

  The PQ compression algorithm is configurable. You control the number of segments, segment granularity, and the size of the training set.

• Binary Quantization (BQ)
  BQ reduces the size of each vector dimension to a single bit. This compression algorithm works best for vectors with high dimensionality.

• Scalar Quantization (SQ)
  SQ reduces the size of each vector dimension from 32 bits to 8 bits. SQ trains on your data to create custom buckets for each dimension. This training helps SQ to preserve data characteristics when it maps information from the 32 bit dimensions into 8 bit buckets.

Flat indexes are memory-efficient. They are disk based indexes that perform brute-force vector searches. These searches are fast for small data sets, but the search speed increases linearly as the number of indexed objects grows.

Flat indexes are best suited for cases where the number of objects is low and will not grow significantly.

Binary quantization (BQ) can improve flat indexes' search speeds. BQ improves search time by reducing the amount of data to read, and speeding up time taken to calculate the distance between vectors.

Dynamic indexes offer a flexible approach to indexing. A dynamic index starts as a flat index and converts automatically to an HNSW index when the object count reaches a threshold value.

In multi-tenant configurations where different tenants have different object counts, dynamic indexes are a good index choice. Collections with dynamic indexes have less overhead since tenants can use flat indexes when the HNSW index isn't needed.

The default index conversion threshold is 10,000 objects. You can configure the threshold value when you create the dynamic index.

This table shows how a dynamic index changes as the number of objects in a collection grows. The example configuration is for a dynamic index with the following properties:

• A conversion threshold of 10,000 objects.
• Flat index with BQ configured.
• HNSW index with SQ configured
• The training threshold for SQ is 100,000 objects.

HFresh indexes are cluster-based vector indexes that use HNSW for the centroid index. They partition vectors into posting lists (clusters) and search only the most relevant clusters for a query.

HFresh indexes are well-suited when memory efficiency is a priority, especially with high-dimensional vectors. They use mandatory 1-bit rotational quantization (RQ) for postings and 8-bit RQ for centroids. Only the compressed centroid index is kept in memory. The posting lists live on disk, so memory usage stays low even as the collection grows. The trade-off is lower peak query throughput than HNSW, so HFresh is a good fit when you can tolerate higher query latency in exchange for smaller memory requirements.

For configuration details, see HFresh index parameters.

Building an HNSW index can be resource-intensive and slow down the time it takes to import objects into Weaviate.

Asynchronous vector indexing allows you to import objects into Weaviate without waiting for the HNSW index to be built.

This allows decoupling of data ingestion from vector index creation.

With asynchronous vector indexing, you can import objects into Weaviate without waiting for the HNSW index to finish building. You should be aware that vector searches during this time may be based on an incomplete index, which can affect search results.

Weaviate uses inverted indexes for keyword searches as well as filtering.

There are multiple types of inverted indexes, including:

• indexSearchable: for keyword searches
• indexFilterable: for faster filtering
• indexRangeFilters: for faster filtering on numerical ranges

Aspects of these indexes can be configured at the collection and property level.

The inverted index is configurable on a collection level. The collection level settings determine BM25 parameters, and what metadata is indexed for filtering. For example, you can configure whether timestamps, null state, or property lengths are indexed.

For details, see set inverted index parameters

Property level configuration is more limited. Individual indexes can be turned on or off at the property level, and keyword tokenization options can be set.

indexSearchable determines whether a property is indexed for keyword searches. indexFilterable determines whether a property is indexed to speed up match-based filtering. indexRangeFilters determines whether a property is indexed for numerical range filters.

For more on filters, see Filtering.

For more information, see these documentation pages and blog posts.

To configure indexing, follow the steps on these pages:

• Configure vector indexes
• Configure inverted indexes

For more documentation details, see:

• Vector indexing
• Inverted indexes

Have a question or feedback? Here's how to reach us.