Select & configure vector indexes

Each named vector in a Weaviate collection can have its own index configuration. This setting is then used to configure the index for that specific vector. In a multi-tenant collection, each tenant inherits the index configuration of the collection.

This tutorial provides a hands-on guide to configuring and tuning these vector indexes. While the defaults are a great starting point, understanding how to adjust index parameters can help you optimize for search speed, accuracy, and memory usage.

We will cover how to configure the HNSW, Flat, Dynamic, and HFresh index types.

The Hierarchical Navigable Small World (HNSW) index is the default in Weaviate. It's designed for large-scale datasets where you need fast and accurate similarity searches. HNSW builds a multi-layered graph of your vectors, which allows it to find approximate nearest neighbors very efficiently.

If you create a collection without specifying a vectorIndexConfig, Weaviate will use HNSW with its default settings. You can, however, explicitly define it.

from weaviate.classes.config import Configureclient.collections.create(    name=collection_name,    # ... other parameters    vector_config=Configure.Vectors.text2vec_weaviate(        vector_index_config=Configure.VectorIndex.hnsw()    ),)

The key to HNSW is balancing the trade-offs between search speed, recall (accuracy), and import/build time. You can tune this balance by adjusting its parameters.

The most important parameters are:

• maxConnections: The number of connections each node in the graph will have. More connections lead to higher accuracy but use more memory and can slow down searches.
• efConstruction: The size of the dynamic list used during index construction. A higher value creates a more accurate graph, improving search performance at the cost of longer import times.
• ef: The size of the dynamic list used during a search. This is one of the most critical parameters for tuning. A higher ef value leads to better recall but slower searches.

Let's create a collection with custom HNSW parameters to optimize for high recall.

from weaviate.classes.config import Configure, VectorDistancesclient.collections.create(    name=collection_name,    # ... other parameters    vector_config=Configure.Vectors.text2vec_weaviate(        vector_index_config=Configure.VectorIndex.hnsw(            # Distance metric            distance_metric=VectorDistances.COSINE,            # Parameters for HNSW index construction            ef_construction=256,    # Dynamic list size during construction            max_connections=128,    # Maximum number of connections per node            quantizer=Configure.VectorIndex.Quantizer.bq(), # Quantizer configuration            # Parameters for HNSW search            ef=-1,                  # Dynamic list size during search; -1 enables dynamic Ef            dynamic_ef_factor=15,   # Multiplier for dynamic Ef            dynamic_ef_min=200,     # Minimum threshold for dynamic Ef            dynamic_ef_max=1000,    # Maximum threshold for dynamic Ef        )    ),)

For a deeper dive into all available parameters, see the HNSW configuration reference.

The Flat index performs a brute-force search by comparing a query vector to every single vector in the index. This guarantees perfect recall but does not scale well for large datasets.

It's an excellent choice for:

• Small indexes (e.g., under 10,000-20,000 objects).
• Multi-tenant use cases where each tenant has a small, isolated index.

The main benefit of the Flat index is its extremely low memory overhead, as it doesn't need to store a complex graph structure.

Here’s how to configure a collection to use the Flat index. You can also enable quantization to speed up the brute-force search.

from weaviate.classes.config import Configure, VectorDistancesclient.collections.create(    name=collection_name,    # ... other parameters    vector_config=Configure.Vectors.text2vec_weaviate(        vector_index_config=Configure.VectorIndex.flat(            distance_metric=VectorDistances.COSINE,                     # Distance metric            quantizer=Configure.VectorIndex.Quantizer.bq(cache=True),   # Quantizer configuration            vector_cache_max_objects=1000000,                           # Maximum number of objects in the cache        )    ),)

For more details, see the Flat index configuration reference.

The Dynamic index is a powerful feature for use cases where the size of a collection (or a tenant's data) can vary unpredictably.

It works by:

1. Starting with a Flat index, which is efficient for small numbers of objects.
2. Automatically converting to an HNSW index once the number of objects crosses a specified threshold (default is 10,000).

This is particularly useful in multi-tenant environments, as small tenants can use the memory-efficient Flat index, while large tenants automatically get the performance benefits of HNSW.

Here's how to set up a collection with a Dynamic index using default settings.

from weaviate.classes.config import Configureclient.collections.create(    name=collection_name,    # ... other parameters    vector_config=Configure.Vectors.text2vec_weaviate(        vector_index_config=Configure.VectorIndex.dynamic()    ),    multi_tenancy_config=Configure.multi_tenancy(enabled=True), # Dynamic index works well with multi-tenancy set-ups)

You can customize both the HNSW and Flat configurations that the Dynamic index will use, as well as the threshold for switching.

from weaviate.classes.config import Configure, VectorDistancesclient.collections.create(    name=collection_name,    # ... other parameters    vector_config=Configure.Vectors.text2vec_weaviate(        vector_index_config=Configure.VectorIndex.dynamic(            distance_metric=VectorDistances.COSINE,                     # Distance metric            threshold=25000,                                            # Threshold for switching to dynamic index            hnsw=Configure.VectorIndex.hnsw(                # Your preferred HNSW configuration            ),            flat=Configure.VectorIndex.flat(                # Your preferred flat configuration            ),        )    ),    multi_tenancy_config=Configure.multi_tenancy(   # Dynamic index works well with multi-tenancy set-ups        enabled=True,        auto_tenant_creation=True,        auto_tenant_activation=True,    ),)

For more details, see the Dynamic index configuration reference.

The HFresh index is designed for large-scale datasets where you need fast approximate nearest neighbor searches with low memory overhead. It uses a posting list structure where each vector is added to multiple posting lists, enabling efficient parallel search.

To create a collection with the HFresh index using default settings, simply pass Configure.VectorIndex.hfresh() without any arguments. This uses the following defaults:

• distance_metric: cosine
• replicas: 4 (number of posting lists per vector)
• search_probe: 256 (number of posting lists to search)
• max_posting_size_kb: 48

The search_probe default is 256 in v1.36.20, v1.37.10, v1.38.2 and later. Earlier releases on each of those lines default to 64.

from weaviate.classes.config import Configureclient.collections.create(    name=collection_name,    # ... other parameters    vector_config=Configure.Vectors.text2vec_weaviate(        vector_index_config=Configure.VectorIndex.hfresh()    ),)

You can tune HFresh by adjusting its parameters:

• max_posting_size_kb: The maximum allowed size for a posting list in KB. Default: 48.
• replicas: The number of posting lists in which a vector is stored. A higher value increases recall but uses more storage.
• search_probe: The number of posting lists to search during a query. A higher value increases recall but slows down searches. The example below sets 64, which is below the current default, to favor speed over recall.

from weaviate.classes.config import Configure, VectorDistancesclient.collections.create(    name=collection_name,    # ... other parameters    vector_config=Configure.Vectors.text2vec_weaviate(        vector_index_config=Configure.VectorIndex.hfresh(            distance_metric=VectorDistances.COSINE,     # Distance metric            max_posting_size_kb=1024,                    # Max posting list size in KB            replicas=4,                                  # Number of posting lists per vector            search_probe=64,                             # Number of posting lists to search        )    ),)

You now have a practical understanding of how to configure Weaviate's different vector index types.

• For more about the concepts behind these indexes, visit the Vector Index Concepts page.
• To learn about reducing memory usage even further, check out the concepts page on Vector Quantization.