Inverted index

An inverted index is a data structure in Weaviate that enables efficient text search and filtering operations.

Additional information

In Weaviate, the inverted index supports search capabilities such as keyword search, filtering, and range queries. An inverted index maps from terms (tokens) back to the objects that contain them. This mapping allows Weaviate to quickly identify which objects contain specific terms or match certain criteria during search queries.

You can enable inverted indexes on properties and adjust various parameters that control indexing behavior and tokenization strategies. Proper configuration of these parameters is crucial for optimizing both search performance and storage efficiency.

Enable inverted index for keyword searches and filtering

Inverted index parameters control how individual properties are indexed for search and filtering operations. These parameters determine whether specific properties can be searched, filtered, or used in range queries.

Enabling inverted index

The inverted index in Weaviate can be enabled through parameters at the property level. The names below are the REST and camelCase client spellings; the Python client uses the snake_case equivalent, so indexFilterable is index_filterable, indexSearchable is index_searchable and indexRangeFilters is index_range_filters.

indexFilterable - Controls whether a property can be used in where filters. When set to true, the property values are indexed for efficient filtering operations. Disable this for properties that don't need filtering to save storage space.

indexSearchable - Determines whether a property participates in keyword search queries. When true, the property's text content is tokenized and indexed for search. Set to false for properties that shouldn't be searchable to improve performance.

indexRangeFilters - Enables range filtering capabilities (greater than, less than, etc.) for numerical and date properties. When enabled, additional indexing structures are created to support efficient range queries.

Python

from weaviate.classes.config import Configure, Property, DataTypeclient.collections.create(    "Article",    # Additional settings not shown    properties=[        Property(            name="title",            data_type=DataType.TEXT,            index_filterable=True,            index_searchable=True,        ),        Property(            name="chunk",            data_type=DataType.TEXT,            index_filterable=True,            index_searchable=True,        ),        Property(            name="chunk_number",            data_type=DataType.INT,            index_range_filters=True,        ),    ],)

JavaScript/TypeScript

await client.collections.create({  name: 'Article',  properties: [    {      name: 'title',      dataType: dataType.TEXT,      indexFilterable: true,      indexSearchable: true,    },    {      name: 'chunk',      dataType: dataType.TEXT,      indexFilterable: true,      indexSearchable: true,    },    {      name: 'chunk_number',      dataType: dataType.INT,      indexRangeFilters: true,    },  ],})

vTrue := true
vFalse := false

articleClass := &models.Class{
  Class:       "Article",
  Description: "Collection of articles",
  Properties: []*models.Property{
    {
      Name:            "title",
      DataType:        schema.DataTypeText.PropString(),
      Tokenization:    "lowercase",
      IndexFilterable: &vTrue,
      IndexSearchable: &vFalse,
    },
    {
      Name:            "chunk",
      DataType:        schema.DataTypeText.PropString(),
      Tokenization:    "word",
      IndexFilterable: &vTrue,
      IndexSearchable: &vTrue,
    },
    {
      Name:              "chunk_no",
      DataType:          schema.DataTypeInt.PropString(),
      IndexRangeFilters: &vTrue,
    },
  },
  InvertedIndexConfig: &models.InvertedIndexConfig{
    Bm25: &models.BM25Config{
      B:  0.7,
      K1: 1.25,
    },
    IndexNullState:      true,
    IndexPropertyLength: true,
    IndexTimestamps:     true,
  },
}

Java

client.collections.create("Article", col -> col
    .properties(
        Property.text("title",
            p -> p.indexFilterable(true).indexSearchable(true)),
        Property.text("chunk",
            p -> p.indexFilterable(true).indexSearchable(true)),
        Property.integer("chunk_number", p -> p.indexRangeFilters(true)))
    .invertedIndex(idx -> idx.bm25(b -> b.b(1).k1(2))
        .indexNulls(true)
        .indexPropertyLength(true)
        .indexTimestamps(true)));

await client.Collections.Create(
    new CollectionCreateParams
    {
        Name = "Article",
        Properties =
        [
            Property.Text("title", indexFilterable: true, indexSearchable: true),
            Property.Text("chunk", indexFilterable: true, indexSearchable: true),
            Property.Int("chunk_number", indexRangeFilters: true),
        ],
        InvertedIndexConfig = new InvertedIndexConfig
        {
            Bm25 = new BM25Config { B = 1, K1 = 2 },
            IndexNullState = true,
            IndexPropertyLength = true,
            IndexTimestamps = true,
        },
    }
);

Set inverted index parameters

Inverted index parameters control the overall behavior of the inverted index for an entire collection. These parameters affect ranking algorithms, null value handling, and timestamp indexing across all properties in the collection.

Inverted index parameters

The inverted index in Weaviate can be configured through various parameters at the collection level. The names below are the REST and camelCase client spellings. In REST, b and k1 are members of the bm25 object. The Python client uses the snake_case equivalent, so they are bm25_b, bm25_k1, index_null_state, index_property_length and index_timestamps.

bm25: b - Controls the degree of normalization by document length in the BM25 ranking algorithm. Values range from 0 to 1, where 0 means no length normalization and 1 means full normalization. Higher values favor shorter documents.

bm25: k1 - Controls term frequency saturation in BM25. Higher values make term frequency more important, while lower values reduce the impact of term frequency on scoring.

indexNullState - Determines whether null values are indexed. When enabled, you can filter for objects that have null values in specific properties.

indexPropertyLength - Controls whether the length of text properties is indexed. When enabled, allows filtering based on text length and can improve certain ranking algorithms.

indexTimestamps - Enables indexing of creation and update timestamps for objects, allowing filtering and sorting operations.

Python

from weaviate.classes.config import Configure, Property, DataTypeclient.collections.create(    "Article",    # Additional settings not shown    inverted_index_config=Configure.inverted_index(        bm25_b=0.7,        bm25_k1=1.25,        index_null_state=True,        index_property_length=True,        index_timestamps=True,    ),)

JavaScript/TypeScript

import { dataType } from 'weaviate-client';

vTrue := true
vFalse := false

articleClass := &models.Class{
  Class:       "Article",
  Description: "Collection of articles",
  Properties: []*models.Property{
    {
      Name:            "title",
      DataType:        schema.DataTypeText.PropString(),
      Tokenization:    "lowercase",
      IndexFilterable: &vTrue,
      IndexSearchable: &vFalse,
    },
    {
      Name:            "chunk",
      DataType:        schema.DataTypeText.PropString(),
      Tokenization:    "word",
      IndexFilterable: &vTrue,
      IndexSearchable: &vTrue,
    },
    {
      Name:              "chunk_no",
      DataType:          schema.DataTypeInt.PropString(),
      IndexRangeFilters: &vTrue,
    },
  },
  InvertedIndexConfig: &models.InvertedIndexConfig{
    Bm25: &models.BM25Config{
      B:  0.7,
      K1: 1.25,
    },
    IndexNullState:      true,
    IndexPropertyLength: true,
    IndexTimestamps:     true,
  },
}

Java

client.collections.create("Article", col -> col
    .properties(
        Property.text("title",
            p -> p.indexFilterable(true).indexSearchable(true)),
        Property.text("chunk",
            p -> p.indexFilterable(true).indexSearchable(true)),
        Property.integer("chunk_number", p -> p.indexRangeFilters(true)))
    .invertedIndex(idx -> idx.bm25(b -> b.b(1).k1(2))
        .indexNulls(true)
        .indexPropertyLength(true)
        .indexTimestamps(true)));

await client.Collections.Create(
    new CollectionCreateParams
    {
        Name = "Article",
        Properties =
        [
            Property.Text("title", indexFilterable: true, indexSearchable: true),
            Property.Text("chunk", indexFilterable: true, indexSearchable: true),
            Property.Int("chunk_number", indexRangeFilters: true),
        ],
        InvertedIndexConfig = new InvertedIndexConfig
        {
            Bm25 = new BM25Config { B = 1, K1 = 2 },
            IndexNullState = true,
            IndexPropertyLength = true,
            IndexTimestamps = true,
        },
    }
);

Drop an inverted index

Drop (delete) an inverted index from a property. This is a destructive operation: the index data is removed from disk. To use the index again, it must be regenerated.

The following index types can be dropped: searchable, filterable, rangeFilters.

Python

collection = client.collections.get("Article")# Drop the searchable inverted index from the "title" propertycollection.config.delete_property_index("title", "searchable")# Drop the filterable inverted index from the "title" propertycollection.config.delete_property_index("title", "filterable")# Drop the range filter index from the "chunk_number" propertycollection.config.delete_property_index("chunk_number", "rangeFilters")

JavaScript/TypeScript

const article = client.collections.use('Article')// Drop the searchable inverted index from the "title" propertyawait article.config.dropInvertedIndex('title', 'searchable')// Drop the filterable inverted index from the "title" propertyawait article.config.dropInvertedIndex('title', 'filterable')// Drop the range filter index from the "chunk_number" propertyawait article.config.dropInvertedIndex('chunk_number', 'rangeFilters')

collection := client.Schema()// Drop the searchable inverted index from the "title" propertyerr = collection.PropertyIndexDeleter().  WithClassName("Article").  WithPropertyName("title").  WithSearchable().  Do(ctx)// Drop the filterable inverted index from the "title" propertyerr = collection.PropertyIndexDeleter().  WithClassName("Article").  WithPropertyName("title").  WithFilterable().  Do(ctx)// Drop the range filter index from the "chunk_no" propertyerr = collection.PropertyIndexDeleter().  WithClassName("Article").  WithPropertyName("chunk_no").  WithRangeFilters().  Do(ctx)

Java

var collection = client.collections.use("Article");// Drop the searchable inverted index from the "title" propertycollection.config.dropPropertyIndex("title", PropertyIndexType.SEARCHABLE);// Drop the filterable inverted index from the "title" propertycollection.config.dropPropertyIndex("title", PropertyIndexType.FILTERABLE);// Drop the range filter index from the "chunk_number" propertycollection.config.dropPropertyIndex("chunk_number", PropertyIndexType.RANGE_FILTERS);

Set tokenization type for property

Configure a tokenization method for each property individually.

Tokenization methods

Tokenization determines how text content is broken down into individual terms that can be indexed and searched. Weaviate supports several tokenization strategies:

word - The default tokenization that splits text on whitespace and punctuation, converting to lowercase. Best for general text search where you want to match individual words.

lowercase - Splits text on whitespace only, then lowercases each token. Preserves symbols (like &, @, _) that word tokenization would strip. Good for case-insensitive matching where punctuation is meaningful, such as code snippets or email addresses.

whitespace - Splits text only on whitespace characters, preserving punctuation and case. Good when punctuation is meaningful for search.

field - Treats the entire property value as a single token without any processing. Use for exact matching of complete field values like IDs, email addresses, or URLs.

trigram - Breaks text into overlapping 3-character sequences. Enables fuzzy matching and is useful for handling typos or partial matches.

gse - Language-aware tokenization for Chinese and Japanese text. Disabled by default. Enable with the ENABLE_TOKENIZER_GSE environment variable. For Korean text, see the kagome_kr option.

For the full list of supported tokenizers (including kagome_ja, kagome_kr, and the per-property text-analyzer options), see the tokenization reference.

Python

from weaviate.classes.config import Configure, Property, DataType, Tokenizationclient.collections.create(    "Article",    vector_config=Configure.Vectors.text2vec_cohere(),    properties=[        Property(            name="title",            data_type=DataType.TEXT,            tokenization=Tokenization.LOWERCASE,  # Use "lowercase" tokenization            description="The title of the article.",  # Optional description        ),        Property(            name="body",            data_type=DataType.TEXT,            tokenization=Tokenization.WHITESPACE,  # Use "whitespace" tokenization        ),    ],)

JavaScript/TypeScript

const newCollection = await client.collections.create({  name: 'Article',  vectorizers: vectors.text2VecHuggingFace(),  properties: [    {      name: 'title',      dataType: dataType.TEXT,      tokenization: tokenization.LOWERCASE    },    {      name: 'body',      dataType: dataType.TEXT,      tokenization: tokenization.WHITESPACE    },  ],})

vTrue := true
vFalse := false

articleClass := &models.Class{
  Class:       "Article",
  Description: "Collection of articles",
  Properties: []*models.Property{
    {
      Name:            "title",
      DataType:        schema.DataTypeText.PropString(),
      Tokenization:    "lowercase",
      IndexFilterable: &vTrue,
      IndexSearchable: &vFalse,
      ModuleConfig: map[string]interface{}{
        "text2vec-cohere": map[string]interface{}{
          "vectorizePropertyName": true,
        },
      },
    },
    {
      Name:            "body",
      DataType:        schema.DataTypeText.PropString(),
      Tokenization:    "whitespace",
      IndexFilterable: &vTrue,
      IndexSearchable: &vTrue,
      ModuleConfig: map[string]interface{}{
        "text2vec-cohere": map[string]interface{}{
          "vectorizePropertyName": false,
        },
      },
    },
  },
  Vectorizer: "text2vec-cohere",
}

Java

client.collections.create("Article",
    col -> col.properties(
        Property.text("title",
            p -> p.description("The title of the article.")
                .tokenization(Tokenization.LOWERCASE)
                .vectorizePropertyName(false)),
        Property.text("body", p -> p.skipVectorization(true)
            .tokenization(Tokenization.WHITESPACE))));

await client.Collections.Create(
    new CollectionCreateParams
    {
        Name = "Article",
        Properties =
        [
            Property.Text("title", tokenization: PropertyTokenization.Lowercase),
            Property.Text("body", tokenization: PropertyTokenization.Whitespace),
        ],
    }
);

Further resources

Questions and feedback

Have a question or feedback? Here's how to reach us.

Community Forum

Ask questions and connect with other developers on our Community forum.

Support

Weaviate Cloud user or customer? Find the right channel on the Support page.