Inverted index

The inverted index maps values (like words or numbers) to the objects that contain them. It is the backbone for all attribute-based filtering (where filters) and keyword searching (bm25, hybrid).

Multiple inverted index types are available in Weaviate. Not all inverted index types are available for all data types. The available inverted index types are:

• Enable one or both of indexFilterable and indexRangeFilters to index a property for faster filtering.
  • If only one is enabled, the respective index is used for filtering.
  • If both are enabled, indexRangeFilters is used for operations involving comparison operators, and indexFilterable is used for equality and inequality operations.

These parameters are set within the invertedIndexConfig object in your collection definition.

This code example shows how to configure inverted index parameters through a client library:

from weaviate.classes.config import (    Configure,    DataType,    Property,    StopwordsPreset,    Tokenization,)client.collections.create(    "Article",    # Additional settings not shown    properties=[  # properties configuration is optional        Property(            name="title",            data_type=DataType.TEXT,            index_filterable=True,            index_searchable=True,            tokenization=Tokenization.WORD,        ),        Property(            name="chunk",            data_type=DataType.TEXT,            index_filterable=True,            index_searchable=True,            tokenization=Tokenization.FIELD,        ),        Property(            name="chunk_number",            data_type=DataType.INT,            index_range_filters=True,        ),    ],    inverted_index_config=Configure.inverted_index(  # Optional        bm25_b=0.7,        bm25_k1=1.25,        index_null_state=True,        index_property_length=True,        index_timestamps=True,        stopwords_preset=StopwordsPreset.EN,        stopwords_additions=["example", "stopword"],        stopwords_removals=["the", "and"],    ),)

import { dataType } from 'weaviate-client';

vTrue := true
vFalse := false

articleClass := &models.Class{
  Class:       "Article",
  Description: "Collection of articles",
  Properties: []*models.Property{
    {
      Name:            "title",
      DataType:        schema.DataTypeText.PropString(),
      Tokenization:    "lowercase",
      IndexFilterable: &vTrue,
      IndexSearchable: &vFalse,
    },
    {
      Name:            "chunk",
      DataType:        schema.DataTypeText.PropString(),
      Tokenization:    "word",
      IndexFilterable: &vTrue,
      IndexSearchable: &vTrue,
    },
    {
      Name:              "chunk_no",
      DataType:          schema.DataTypeInt.PropString(),
      IndexRangeFilters: &vTrue,
    },
  },
  InvertedIndexConfig: &models.InvertedIndexConfig{
    Bm25: &models.BM25Config{
      B:  0.7,
      K1: 1.25,
    },
    IndexNullState:      true,
    IndexPropertyLength: true,
    IndexTimestamps:     true,
  },
}

Part of invertedIndexConfig. The settings for BM25 are the free parameters k1 and b, and they are optional. The defaults (k1 = 1.2 and b = 0.75) work well for most cases.

They can be configured per collection, and can optionally be overridden per property.

{
  "class": "Article",
  // Configuration of the sparse index
  "invertedIndexConfig": {
    "bm25": {
      "b": 0.75,
      "k1": 1.2
    }
  },
  "properties": [
    {
      "name": "title",
      "description": "title of the article",
      "dataType": ["text"],
      // Property-level settings override the collection-level settings
      "invertedIndexConfig": {
        "bm25": {
          "b": 0.75,
          "k1": 1.2
        }
      },
      "indexFilterable": true,
      "indexSearchable": true
    }
  ]
}

Part of invertedIndexConfig. text properties may contain words that are very common and don't contribute to search results. Ignoring them speeds up queries that contain stopwords, as they can be automatically removed from queries as well. This speedup is very notable on scored searches, such as BM25.

The stopword configuration uses a preset system. You can select a preset to use the most common stopwords for a particular language (e.g. "en" preset). If you need more fine-grained control, you can add additional stopwords or remove stopwords that you believe should not be part of the list. Alternatively, you can create your custom stopword list by starting with an empty ("none") preset and adding all your desired stopwords as additions.

  "invertedIndexConfig": {
    "stopwords": {
      "preset": "en",
      "additions": ["star", "nebula"],
      "removals": ["a", "the"]
    }
  }

This configuration allows stopwords to be configured by collection. If not set, these values are set to the following defaults:

As of v1.18, stopwords are indexed. Thus stopwords are included in the inverted index, but not in the tokenized query. As a result, when the BM25 algorithm is applied, stopwords are ignored in the input for relevance ranking but will affect the score.

Stopwords can now be configured at runtime. You can use the RESTful API to update the list of stopwords after your data has been indexed.

Part of invertedIndexConfig. Defines named stopword presets at the collection level. Each preset is a flat list of words. Properties can then reference a preset by name via textAnalyzer.stopwordPreset.

A preset name that matches a built-in ("en", "none") fully replaces the built-in for properties of this collection. Preset names must not be empty or whitespace-only; each word list must contain at least one word; individual words must not be empty or whitespace-only.

"invertedIndexConfig": {
  "stopwordPresets": {
    "fr": ["le", "la", "les", "un", "une", "des", "du", "de", "et"],
    "de": ["der", "die", "das", "und", "oder", "aber"]
  }
}

The existing stopwords configuration remains as the default for properties that do not specify a textAnalyzer.stopwordPreset override. For extending a built-in preset with additions/removals, use stopwords instead. It is the only stopword config that accepts that object form.

Part of a property definition (not invertedIndexConfig). Configures text analysis behavior for individual text properties. The accent-folding options (asciiFold, asciiFoldIgnore) are supported on properties with tokenization word, lowercase, whitespace, field, or trigram. They are not supported on the language-specific tokenizers (gse, gse_ch, kagome_ja, and kagome_kr). The stopwordPreset option is only supported on properties with tokenization: "word".

{
  "name": "description",
  "dataType": ["text"],
  "tokenization": "word",
  "textAnalyzer": {
    "asciiFold": true,
    "asciiFoldIgnore": ["é"],
    "stopwordPreset": "fr"
  }
}

Part of invertedIndexConfig. To perform queries that are filtered by timestamps, configure the target collection to maintain an inverted index based on the objects' internal timestamps. Currently the timestamps include creationTimeUnix and lastUpdateTimeUnix.

To configure timestamp based indexing, set indexTimestamps to true in the invertedIndexConfig object.

Part of invertedIndexConfig. To perform queries that filter on null, configure the target collection to maintain an inverted index that tracks null values for each property in a collection .

To configure null based indexing, setting indexNullState to true in the invertedIndexConfig object.

Part of invertedIndexConfig. To perform queries that filter by the length of a property, configure the target collection to maintain an inverted index based on the length of the properties.

To configure indexing based on property length, set indexPropertyLength to true in the invertedIndexConfig object.

You can drop (delete) an inverted index from a property. This is a destructive operation: the index data is removed from disk. To use the index again, it must be regenerated.

The following index types can be dropped: searchable, filterable, rangeFilters.

See How-to: Drop an inverted index for code examples.

Weaviate creates separate inverted indexes for each property and each index type. For example, if you have a title property that is both searchable and filterable, Weaviate will create two separate inverted indexes for that property - one optimized for search operations and another for filtering operations. Find out more in Concepts: Inverted index.

Adding a property after importing objects can lead to limitations in inverted-index related behavior, such as filtering by the new property's length or null status.

This is caused by the inverted index being built at import time. If you add a property after importing objects, the inverted index for metadata such as the length or the null status will not be updated to include the new properties. This means that the new property will not be indexed for existing objects. This can lead to unexpected behavior when querying.

To avoid this, you can either:

• Add the property before importing objects.
• Delete the collection, re-create it with the new property and then re-import the data.

We are working on a re-indexing API to allow you to re-index the data after adding a property. This will be available in a future release.

For text properties, Weaviate first tokenizes the text before creating inverted index entries. Tokenization is the process of breaking text into individual tokens (words, phrases, or characters) that can be indexed and searched.

See the related concepts page for more details.

Two REST endpoints let you test tokenization without modifying your schema.

POST /v1/tokenize tokenizes arbitrary text with an explicit tokenizer and analyzer config.

Request body:

Example:

import weaviate
from weaviate.classes.config import Tokenization, Configure

client = weaviate.connect_to_local()

# Ad-hoc tokenization with custom config
result = client.tokenization.text(
    text="The organic café crème blend",
    tokenization=Tokenization.WORD,
    analyzer_config=Configure.text_analyzer(
        ascii_fold=True,
        stopword_preset="en",
    ),
)

print(f"indexed: {result.indexed}")
print(f"query:   {result.query}")

curl -X POST http://localhost:8080/v1/tokenize -d '{
  "text": "The organic café crème blend",
  "tokenization": "word",
  "analyzerConfig": { "asciiFold": true, "stopwordPreset": "en" }
}'

Response:

{
  "indexed": ["the", "organic", "cafe", "creme", "blend"],
  "query": ["organic", "cafe", "creme", "blend"]
}

• indexed: tokens as stored in the inverted index
• query: tokens after stopword filtering (what BM25 scores at search time)

Example with a custom stopword preset:

Define a named preset on the request via stopwordPresets and reference it from analyzerConfig.stopwordPreset. This is useful for previewing a non-English preset before adding it to a collection.

# Define a named "fr" preset and reference it from analyzer_config.
# stopword_presets is mutually exclusive with stopwords — pass at most one.
result = client.tokenization.text(
    text="La Tasse Bleue et le Bol",
    tokenization=Tokenization.WORD,
    analyzer_config=Configure.text_analyzer(stopword_preset="fr"),
    stopword_presets={
        "fr": ["le", "la", "les", "un", "une", "des", "du", "de", "et"],
    },
)

print(f"indexed: {result.indexed}")
print(f"query:   {result.query}")

curl -X POST http://localhost:8080/v1/tokenize -d '{
  "text": "La Tasse Bleue et le Bol",
  "tokenization": "word",
  "analyzerConfig": { "stopwordPreset": "fr" },
  "stopwordPresets": {
    "fr": ["le", "la", "les", "un", "une", "des", "du", "de", "et"]
  }
}'

Response:

{
  "indexed": ["la", "tasse", "bleue", "et", "le", "bol"],
  "query": ["tasse", "bleue", "bol"]
}

POST /v1/schema/{className}/properties/{propertyName}/tokenize resolves the full analyzer config from an existing property. The property's tokenization method, textAnalyzer settings, and the collection's stopword configuration are applied automatically. Nothing else needs to be passed.

Request body:

The response format is the same as freeform tokenization. Class and property names are case-insensitive, and collection aliases are resolved automatically.

Example:

# Tokenize using an existing property's configuration
result = client.tokenization.for_property(
    collection="TokenizeDemo",
    property_name="name_fr",
    text="La Tasse Bleue et le Bol",
)

print(f"indexed: {result.indexed}")
print(f"query:   {result.query}")

curl -X POST http://localhost:8080/v1/schema/TokenizeDemo/properties/name_fr/tokenize -d '{
  "text": "La Tasse Bleue et le Bol"
}'

See the tokenization tutorial for worked examples.

• Concepts: Inverted index
• How-to: Set inverted index parameters
• Reference: Tokenization options - Learn about different tokenization methods and how they affect text indexing

Have a question or feedback? Here's how to reach us.