# Indexing Weaviate uses two types of indexes to facilitate fast, scalable search and filtering. It uses vector indexes for vector searches and inverted indexes for filtering and keyword searches. This page introduces the [vector indexes](#vector-indexes) and [inverted indexes](#inverted-indexes) available in Weaviate, so you can arrive at an informed indexing strategy. :::callout{intent="info" title="Indexing and resource usage"} Vector indexes can use _hot_ or _warm_ resources, depending on the index type. Inverted indexes use _warm_ resources. For more on resource types, see [Managing resources](managing-resources.md). ::: :::callout{intent="tip" title="Indexing recommendations"} - For multi-tenant configurations, use a [dynamic index](#dynamic-indexes). - Consider the size of your data set when choosing an index type: - If your object count is low, consider a flat index. - For larger data sets, use an HNSW index. - If you are unsure, use a dynamic index. - If you can trade off some accuracy, consider using [compression](managing-resources-compression.md): - For HNSW indexes, [PQ](../how-to-configure-weaviate/compression-pq-compression.md)/[SQ](../how-to-configure-weaviate/compression-sq-compression.md)/[BQ](../how-to-configure-weaviate/compression-bq-compression.md) can reduce memory requirements. - For flat indexes, [binary quantization (BQ)](../how-to-configure-weaviate/compression-bq-compression.md) can improve search speeds. - Consider your query needs when configuring inverted indexes: - If disk space is not a concern, [index all properties](#property-level-configuration) for searching & filtering. - If metadata filtering is important, index it [at the collection level](#collection-level-configuration). ::: ## Vector indexes Weaviate offers four types of vector indexes, [Hierarchical Navigable Small World (HNSW) indexes](#hnsw-indexes), [flat indexes](#flat-indexes), [dynamic indexes](#dynamic-indexes), and [HFresh indexes](#hfresh-indexes). - HNSW indexes enable fast, scalable vector searching that works well even with very large data sets. - Flat indexes are memory-efficient indexes that work best with small data sets. - Dynamic indexes switch from a flat index to an HNSW index when a [collection](../concepts/data.md#collections) or [tenant](../concepts/data.md#multi-tenancy) reaches a threshold size. - HFresh indexes use a cluster-based approach with HNSW for centroids, providing memory efficiency by keeping most data on disk. #### HNSW indexes [HNSW](../indexing/vector-index.md#hierarchical-navigable-small-world-hnsw-index) are high-performance, in-memory indexes. HNSW indexes scale well, meaning that vector searches remain fast even for very large data sets. HNSW indexes achieve this by building a multi-layered graph of objects, allowing for fast, approximate nearest neighbor searches. While HNSW indexes enable fast searches, they use a lot of [_hot_ resources](managing-resources.md#-hot), as they load the graph structure and vectors into memory. Consider using [compression](managing-resources-compression.md) to reduce the size of for your HNSW indexes. Weaviate offers several ways to compress your data: - **[Rotational Quantization (RQ)](../how-to-configure-weaviate/compression-rq-compression.md)** (_recommended_)\ RQ reduces the size of each vector dimension from 32 bits to 8 bits (or 4 bits, or 1 bit) without requiring training. RQ first applies a fast pseudorandom rotation to the vector, then quantizes each dimension. The rotation spreads information evenly across dimensions, so 8-bit RQ reaches up to 98-99% recall with no configuration and no training phase. The 4-bit and 1-bit widths compress further and depend on [rescoring](../how-to-configure-weaviate/compression-rq-compression.md#4-bit-rq) for their recall. - **[Product Quantization (PQ)](../how-to-configure-weaviate/compression-pq-compression.md)**\ PQ reduces the size of the vector embedding in two ways. PQ trains on your data to create custom segments. PQ creates segments to reduce the number of dimensions, and segments are stored as 8 bit integers instead of 32 bit floats. Compared to dimensions, there are fewer segments and each segment is much smaller than a single dimension. The PQ compression algorithm is [configurable](../reference-configuration/indexing-vector-index.md#pq-parameters). You control the number of segments, segment granularity, and the size of the training set. - **[Binary Quantization (BQ)](../how-to-configure-weaviate/compression-bq-compression.md)**\ BQ reduces the size of each vector dimension to a single bit. This compression algorithm works best for vectors with high dimensionality. - **[Scalar Quantization (SQ)](../how-to-configure-weaviate/compression-sq-compression.md)**\ SQ reduces the size of each vector dimension from 32 bits to 8 bits. SQ trains on your data to create custom buckets for each dimension. This training helps SQ to preserve data characteristics when it maps information from the 32 bit dimensions into 8 bit buckets. #### Flat indexes [Flat indexes](../indexing/vector-index.md#flat-index) are memory-efficient. They are disk based indexes that perform brute-force vector searches. These searches are fast for small data sets, but the search speed increases linearly as the number of indexed objects grows. Flat indexes are best suited for cases where the number of objects is low and will not grow significantly. [Binary quantization (BQ)](../how-to-configure-weaviate/compression-bq-compression.md) can improve flat indexes' search speeds. BQ improves search time by reducing the amount of data to read, and speeding up time taken to calculate the distance between vectors. #### Dynamic indexes :::callout{intent="info" title="Dynamic index requires `ASYNC_INDEXING`"} Dynamic indexes require asynchronous indexing. To enable asynchronous indexing in a self-hosted Weaviate instance, set the `ASYNC_INDEXING` [environment variable](../database-configuration/overview.md#general) to `true`. If your instance is hosted in Weaviate Cloud, use the Weaviate Cloud console to enable asynchronous indexing. ::: [Dynamic indexes](../indexing/vector-index.md#dynamic-index) offer a flexible approach to indexing. A dynamic index starts as a flat index and converts automatically to an HNSW index when the object count reaches a threshold value. In multi-tenant configurations where different tenants have different object counts, dynamic indexes are a good index choice. Collections with dynamic indexes have less overhead since tenants can use flat indexes when the HNSW index isn't needed. The default index conversion threshold is 10,000 objects. You can configure the threshold value when you create the dynamic index. This table shows how a dynamic index changes as the number of objects in a collection grows. The example configuration is for a dynamic index with the following properties: - A conversion threshold of 10,000 objects. - Flat index with BQ configured. - HNSW index with SQ configured - The training threshold for SQ is 100,000 objects. | Number of objects | Index type | Compression | Notes | | :---------------- | :----------- | :---------- | :--------------------------------------------------------------- | | 0 - 9,999 | Flat index | BQ | Flat index and BQ are active. | | 10,000 | Flat -> HNSW | None | The index converts to HNSW. The index is stored in RAM. | | 100,000 | HNSW | Training | The collection's object count reaches the SQ training threshold. | | 100,001 | HNSW | SQ | HNSW and SQ are active. | :::callout{intent="info" title="Configure the flat index and the HNSW index"} Configure the flat index and the HNSW index when you define the dynamic index. The dynamic index uses the flat index initially, then switches to the HNSW index. Both indexes should be configured before they are used. ::: #### HFresh indexes :::callout{intent="info" title="Added in `v1.36`"} ::: [HFresh indexes](../indexing/vector-index.md#hfresh-index) are cluster-based vector indexes that use HNSW for the centroid index. They partition vectors into posting lists (clusters) and search only the most relevant clusters for a query. HFresh indexes are well-suited when memory efficiency is a priority, especially with high-dimensional vectors. They use mandatory 1-bit rotational quantization (RQ) for postings and 8-bit RQ for centroids. Only the compressed centroid index is kept in memory. The posting lists live on disk, so memory usage stays low even as the collection grows. The trade-off is lower peak query throughput than HNSW, so HFresh is a good fit when you can tolerate higher query latency in exchange for smaller memory requirements. :::callout{intent="tip" title="Tuning"} Start with the default parameters. If recall is too low, increase `searchProbe` or the RQ `rescoreLimit`. Both take effect at runtime. ::: :::callout{intent="note"} HFresh only supports `cosine` and `l2-squared` distance metrics. ::: For configuration details, see [HFresh index parameters](../reference-configuration/indexing-vector-index.md#hfresh-index-parameters). ### Asynchronous vector indexing Building an HNSW index can be resource-intensive and slow down the time it takes to import objects into Weaviate. [Asynchronous vector indexing](../reference-configuration/indexing-vector-index.md#asynchronous-indexing) allows you to import objects into Weaviate without waiting for the HNSW index to be built. This allows decoupling of data ingestion from vector index creation. With asynchronous vector indexing, you can import objects into Weaviate without waiting for the HNSW index to finish building. You should be aware that vector searches during this time may be based on an incomplete index, which can affect search results. ## Inverted indexes Weaviate uses inverted indexes for keyword searches as well as filtering. There are multiple types of inverted indexes, including: - `indexSearchable`: for keyword searches - `indexFilterable`: for faster filtering - `indexRangeFilters`: for faster filtering on numerical ranges Aspects of these indexes can be configured at the collection and property level. ### Collection level configuration The inverted index is configurable on a collection level. The collection level settings determine BM25 parameters, and what metadata is indexed for filtering. For example, you can configure whether timestamps, null state, or property lengths are indexed. For details, see [set inverted index parameters](../how-to-manage-collections/inverted-index.md#set-inverted-index-parameters) ### Property level configuration Property level configuration is more limited. Individual indexes can be turned on or off at the property level, and keyword tokenization options can be set. `indexSearchable` determines whether a property is indexed for keyword searches. `indexFilterable` determines whether a property is indexed to speed up match-based filtering. `indexRangeFilters` determines whether a property is indexed for numerical range filters. For more on filters, see [Filtering](../concepts/filtering.md). ## Related pages For more information, see these documentation pages and blog posts. ### Documentation pages To configure indexing, follow the steps on these pages: - [Configure vector indexes](../reference-configuration/indexing-vector-index.md) - [Configure inverted indexes](../how-to-manage-collections/inverted-index.md#set-inverted-index-parameters) For more documentation details, see: - [Vector indexing](../indexing/vector-index.md) - [Inverted indexes](../indexing/index.md) ## Questions and feedback Have a question or feedback? Here's how to reach us. ::::card-grid :::card{title="Community Forum" href="https://forum.weaviate.io/c/support" icon="messages-square"} Ask questions and connect with other developers on our **Community forum**. ::: :::card{title="Support" href="/guides/support-overview" icon="life-buoy"} Weaviate Cloud user or customer? Find the right channel on the **Support page**. ::: :::: ## Related pages - [Agents](./agents-index.md) - [AI-assisted Weaviate code generation](./ai-assisted-vibe-coding-index.md) - [APIs](./apis-index.md) - [Authorization and authentication](./authorization-and-authentication-index.md) - [Benchmarks](./benchmarks-index.md) - [Best practices](./best-practices-index.md) - [Client libraries](./clients-index.md) - [Client Libraries / SDKs](./client-libraries-index.md) - [Cloud](./cloud-index.md) - [Cloud account management](./cloud-account-management-index.md) # Agent Instructions This portal answers questions programmatically. To receive a synthesized, source-cited answer instead of crawling page by page, append the `?ask=` query parameter to any page URL on this site: /guides/quickstart?ask=how+do+I+authenticate Optional parameters: - `&goal=` steers the answer toward your objective (e.g. `&goal=write+a+python+client`). - `&version=