The **[inverted index](../indexing/inverted-index.md)** maps values (like words or numbers) to the objects that contain them. It is the backbone for all attribute-based filtering (`where` filters) and keyword searching (`bm25`, `hybrid`). ## Inverted index types Multiple [inverted index types](../indexing/inverted-index.md) are available in Weaviate. Not all inverted index types are available for all data types. The available inverted index types are: | Inverted index type | Description | Applicable data types | Default | Availability | | ------------------- | ---------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | ------- | ------------ | | `indexSearchable` | A searchable index for BM25-suitable Map index for BM25 or hybrid searching. | `text`, `text[]`, | `true` | `v1.19` | | `indexFilterable` | A Roaring Bitmap index for match-based filtering. | Everything except `blob`, `geoCoordinates`, `object` and `phoneNumber` data types including arrays thereof | `true` | `v1.19` | | `indexRangeFilters` | A Roaring Bitmap index for numerical range-based filtering. | `int`, `number` and `date` only | `false` | `v1.26` | - Enable one or both of `indexFilterable` and `indexRangeFilters` to index a property for faster filtering. - If only one is enabled, the respective index is used for filtering. - If both are enabled, `indexRangeFilters` is used for operations involving comparison operators, and `indexFilterable` is used for equality and inequality operations. ## Inverted index parameters These parameters are set within the `invertedIndexConfig` object in your collection definition. | Parameter | Type | Default | Details | | :-------------------------------------------- | :-------- | :------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------- | | [`bm25`](#bm25) | `object` | `{ "k1": 1.2, "b": 0.75 }` | Sets the `k1` and `b` parameters for the BM25 ranking algorithm. Can be overridden at the property level. See [**BM25 Configuration**](#bm25) below. | | [`stopwords`](#stopwords) | `object` | (Varies) | Defines the stopword list to exclude common words from search queries. See [**Stopwords Configuration**](#stopwords) below. | | [`indexTimestamps`](#indextimestamps) | `boolean` | `false` | If `true`, indexes object creation and update timestamps, enabling filtering by `creationTimeUnix` and `lastUpdateTimeUnix`. | | [`indexNullState`](#indexnullstate) | `boolean` | `false` | If `true`, indexes the null/non-null state of each property, enabling filtering for `null` values. | | [`indexPropertyLength`](#indexpropertylength) | `boolean` | `false` | If `true`, indexes the length of each property, enabling filtering by property length. | :::callout{intent="warning" title="Performance Impact"} Enabling `indexTimestamps`, `indexNullState`, or `indexPropertyLength` adds overhead as these additional indexes must be created and maintained. Only enable them if you require these specific filtering capabilities. ::: #### Code example This code example shows how to configure inverted index parameters through a client library: :::code-group{sync="languages"} ```python title="Python" {16-18,23-25,30,33-42} from weaviate.classes.config import ( Configure, DataType, Property, StopwordsPreset, Tokenization, ) client.collections.create( "Article", # Additional settings not shown properties=[ # properties configuration is optional Property( name="title", data_type=DataType.TEXT, index_filterable=True, index_searchable=True, tokenization=Tokenization.WORD, ), Property( name="chunk", data_type=DataType.TEXT, index_filterable=True, index_searchable=True, tokenization=Tokenization.FIELD, ), Property( name="chunk_number", data_type=DataType.INT, index_range_filters=True, ), ], inverted_index_config=Configure.inverted_index( # Optional bm25_b=0.7, bm25_k1=1.25, index_null_state=True, index_property_length=True, index_timestamps=True, stopwords_preset=StopwordsPreset.EN, stopwords_additions=["example", "stopword"], stopwords_removals=["the", "and"], ), ) ``` ```typescript title="JavaScript/TypeScript" import { dataType } from 'weaviate-client'; ``` ```go title="Go" vTrue := true vFalse := false articleClass := &models.Class{ Class: "Article", Description: "Collection of articles", Properties: []*models.Property{ { Name: "title", DataType: schema.DataTypeText.PropString(), Tokenization: "lowercase", IndexFilterable: &vTrue, IndexSearchable: &vFalse, }, { Name: "chunk", DataType: schema.DataTypeText.PropString(), Tokenization: "word", IndexFilterable: &vTrue, IndexSearchable: &vTrue, }, { Name: "chunk_no", DataType: schema.DataTypeInt.PropString(), IndexRangeFilters: &vTrue, }, }, InvertedIndexConfig: &models.InvertedIndexConfig{ Bm25: &models.BM25Config{ B: 0.7, K1: 1.25, }, IndexNullState: true, IndexPropertyLength: true, IndexTimestamps: true, }, } ``` ::: *** #### `bm25` Part of `invertedIndexConfig`. The settings for BM25 are the [free parameters `k1` and `b`](https://en.wikipedia.org/wiki/Okapi_BM25#The_ranking_function), and they are optional. The defaults (`k1` = 1.2 and `b` = 0.75) work well for most cases. They can be configured per collection, and can optionally be overridden per property. :::accordion{title="Example bm25 configuration - JSON object"} An example of a complete collection object with `bm25` configuration: ```json { "class": "Article", // Configuration of the sparse index "invertedIndexConfig": { "bm25": { "b": 0.75, "k1": 1.2 } }, "properties": [ { "name": "title", "description": "title of the article", "dataType": ["text"], // Property-level settings override the collection-level settings "invertedIndexConfig": { "bm25": { "b": 0.75, "k1": 1.2 } }, "indexFilterable": true, "indexSearchable": true } ] } ``` ::: #### `stopwords` Part of `invertedIndexConfig`. `text` properties may contain words that are very common and don't contribute to search results. Ignoring them speeds up queries that contain stopwords, as they can be automatically removed from queries as well. This speedup is very notable on scored searches, such as `BM25`. The stopword configuration uses a preset system. You can select a preset to use the most common stopwords for a particular language (e.g. [`"en"` preset](https://github.com/weaviate/weaviate/blob/main/adapters/repos/db/inverted/stopwords/presets.go)). If you need more fine-grained control, you can add additional stopwords or remove stopwords that you believe should not be part of the list. Alternatively, you can create your custom stopword list by starting with an empty (`"none"`) preset and adding all your desired stopwords as additions. :::accordion{title="Example stopwords configuration - JSON object"} An example of a complete collection object with `stopwords` configuration: ```json "invertedIndexConfig": { "stopwords": { "preset": "en", "additions": ["star", "nebula"], "removals": ["a", "the"] } } ``` ::: This configuration allows stopwords to be configured by collection. If not set, these values are set to the following defaults: | Parameter | Default value | Acceptable values | | ------------- | ------------- | -------------------------- | | `"preset"` | `"en"` | `"en"`, `"none"` | | `"additions"` | `[]` | _any list of custom words_ | | `"removals"` | `[]` | _any list of custom words_ | :::callout{intent="note"} - If `preset` is `none`, then the collection only uses stopwords from the `additions` list. - If the same item is included in both `additions` and `removals`, Weaviate returns an error. ::: As of `v1.18`, stopwords are indexed. Thus stopwords are included in the inverted index, but not in the tokenized query. As a result, when the BM25 algorithm is applied, stopwords are ignored in the input for relevance ranking but will affect the score. Stopwords can now be configured at runtime. You can use the RESTful API to [update](/weaviate/api/rest#tag/schema/put/schema/%7BclassName%7D) the list of stopwords after your data has been indexed. :::callout{intent="info"} Stopwords are only removed when [tokenization](collections.md#tokenization) is set to `word`. ::: #### `stopwordPresets` :::callout{intent="warning" title="Preview — added in `v1.37`"} This is a preview feature. The API may change in future releases. ::: Part of `invertedIndexConfig`. Defines named stopword presets at the collection level. Each preset is a flat list of words. Properties can then reference a preset by name via [`textAnalyzer.stopwordPreset`](#textanalyzer). A preset name that matches a built-in (`"en"`, `"none"`) fully replaces the built-in for properties of this collection. Preset names must not be empty or whitespace-only; each word list must contain at least one word; individual words must not be empty or whitespace-only. :::accordion{title="Example stopwordPresets configuration - JSON object"} ```json "invertedIndexConfig": { "stopwordPresets": { "fr": ["le", "la", "les", "un", "une", "des", "du", "de", "et"], "de": ["der", "die", "das", "und", "oder", "aber"] } } ``` ::: The existing [`stopwords`](#stopwords) configuration remains as the default for properties that do not specify a `textAnalyzer.stopwordPreset` override. For extending a built-in preset with `additions`/`removals`, use [`stopwords`](#stopwords) instead. It is the only stopword config that accepts that object form. #### `textAnalyzer` :::callout{intent="warning" title="Preview — added in `v1.37`"} This is a preview feature. The API may change in future releases. ::: Part of a **property definition** (not `invertedIndexConfig`). Configures text analysis behavior for individual `text` properties. The accent-folding options (`asciiFold`, `asciiFoldIgnore`) are supported on properties with tokenization `word`, `lowercase`, `whitespace`, `field`, or `trigram`. They are not supported on the language-specific tokenizers (`gse`, `gse_ch`, `kagome_ja`, and `kagome_kr`). The `stopwordPreset` option is only supported on properties with `tokenization: "word"`. | Parameter | Type | Default | Details | | :---------------- | :--------- | :------ | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `asciiFold` | `boolean` | `false` | Normalizes accented Latin characters to ASCII equivalents during indexing and querying. Uses Unicode NFD decomposition. **Immutable** after the property is created. | | `asciiFoldIgnore` | `string[]` | `[]` | Characters exempt from ASCII folding. Each entry must be a single character. **Immutable** after the property is created. | | `stopwordPreset` | `string` | (none) | Name of a built-in (`en`, `none`) or collection-level stopword preset to use for this property, overriding the default `stopwords` config. **Only supported on properties with `tokenization: "word"`**. Schema validation rejects it on other tokenizers. | :::accordion{title="Example textAnalyzer configuration - JSON object"} ```json { "name": "description", "dataType": ["text"], "tokenization": "word", "textAnalyzer": { "asciiFold": true, "asciiFoldIgnore": ["é"], "stopwordPreset": "fr" } } ``` ::: :::callout{intent="note"} `asciiFoldIgnore` changes which tokens are written to disk. It cannot be modified after the property is created. Schema updates that change the ignore list are rejected. To change it, create a new property and reindex. ::: #### `indexTimestamps` Part of `invertedIndexConfig`. To perform queries that are filtered by timestamps, configure the target collection to maintain an inverted index based on the objects' internal timestamps. Currently the timestamps include `creationTimeUnix` and `lastUpdateTimeUnix`. To configure timestamp based indexing, set `indexTimestamps` to `true` in the `invertedIndexConfig` object. #### `indexNullState` Part of `invertedIndexConfig`. To perform queries that filter on `null`, configure the target collection to maintain an inverted index that tracks `null` values for each property in a collection . To configure `null` based indexing, setting `indexNullState` to `true` in the `invertedIndexConfig` object. #### `indexPropertyLength` Part of `invertedIndexConfig`. To perform queries that filter by the length of a property, configure the target collection to maintain an inverted index based on the length of the properties. To configure indexing based on property length, set `indexPropertyLength` to `true` in the `invertedIndexConfig` object. :::callout{intent="note"} Using these features requires more resources, as the additional inverted indexes must be created and maintained. ::: ## Drop an inverted index You can drop (delete) an inverted index from a property. This is a destructive operation: the index data is removed from disk. To use the index again, it must be regenerated. The following index types can be dropped: `searchable`, `filterable`, `rangeFilters`. See [How-to: Drop an inverted index](../how-to-manage-collections/inverted-index.md#drop-an-inverted-index) for code examples. ## How Weaviate creates inverted indexes Weaviate creates **separate inverted indexes for each property and each index type**. For example, if you have a `title` property that is both searchable and filterable, Weaviate will create two separate inverted indexes for that property - one optimized for search operations and another for filtering operations. Find out more in [Concepts: Inverted index](../indexing/inverted-index.md#how-weaviate-creates-inverted-indexes). ### Adding a property after collection creation Adding a property after importing objects can lead to limitations in inverted-index related behavior, such as filtering by the new property's length or null status. This is caused by the inverted index being built at import time. If you add a property after importing objects, the inverted index for metadata such as the length or the null status will not be updated to include the new properties. This means that the new property will not be indexed for existing objects. This can lead to unexpected behavior when querying. To avoid this, you can either: - Add the property before importing objects. - Delete the collection, re-create it with the new property and then re-import the data. We are working on a re-indexing API to allow you to re-index the data after adding a property. This will be available in a future release. ## How tokenization affects inverted indexing For `text` properties, Weaviate first **[tokenizes](collections.md#tokenization)** the text before creating inverted index entries. Tokenization is the process of breaking text into individual tokens (words, phrases, or characters) that can be indexed and searched. See the related [concepts page](../indexing/inverted-index.md#tokenization) for more details. ## Tokenize endpoint :::callout{intent="warning" title="Preview — added in `v1.37`"} This is a preview feature. The API may change in future releases. ::: Two REST endpoints let you test tokenization without modifying your schema. ### Freeform tokenization `POST /v1/tokenize` tokenizes arbitrary text with an explicit tokenizer and analyzer config. **Request body:** | Parameter | Type | Required | Details | | :---------------- | :------- | :------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `text` | `string` | yes | The text to tokenize. Maximum length 10,000 characters. | | `tokenization` | `string` | yes | Tokenization method (`word`, `lowercase`, `whitespace`, `field`, `trigram`, `gse`, `gse_ch`, `kagome_ja`, `kagome_kr`). | | `analyzerConfig` | `object` | no | Analyzer settings: `asciiFold` (`boolean`), `asciiFoldIgnore` (`string[]`), `stopwordPreset` (`string`). | | `stopwords` | `object` | no | Fallback stopword configuration (same shape as [`invertedIndexConfig.stopwords`](#stopwords)). Applied when `analyzerConfig.stopwordPreset` is not set. With `word` tokenization, defaults to preset `en` when omitted. | | `stopwordPresets` | `object` | no | Named stopword presets (same shape as [`invertedIndexConfig.stopwordPresets`](#stopwordpresets)). Reference one via `analyzerConfig.stopwordPreset`. | :::callout{intent="note"} `stopwords` and `stopwordPresets` are mutually exclusive. Pass one or the other, not both. Use `stopwords` for a single preset optionally tweaked with additions/removals; use `stopwordPresets` to define named presets and select one via `analyzerConfig.stopwordPreset`. ::: **Example:** :::code-group{sync="languages"} ```python title="Python" import weaviate from weaviate.classes.config import Tokenization, Configure client = weaviate.connect_to_local() # Ad-hoc tokenization with custom config result = client.tokenization.text( text="The organic café crème blend", tokenization=Tokenization.WORD, analyzer_config=Configure.text_analyzer( ascii_fold=True, stopword_preset="en", ), ) print(f"indexed: {result.indexed}") print(f"query: {result.query}") ``` ```bash title="cURL" curl -X POST http://localhost:8080/v1/tokenize -d '{ "text": "The organic café crème blend", "tokenization": "word", "analyzerConfig": { "asciiFold": true, "stopwordPreset": "en" } }' ``` ::: **Response:** ```json { "indexed": ["the", "organic", "cafe", "creme", "blend"], "query": ["organic", "cafe", "creme", "blend"] } ``` - `indexed`: tokens as stored in the inverted index - `query`: tokens after stopword filtering (what BM25 scores at search time) **Example with a custom stopword preset:** Define a named preset on the request via `stopwordPresets` and reference it from `analyzerConfig.stopwordPreset`. This is useful for previewing a non-English preset before adding it to a collection. :::code-group{sync="languages"} ```python title="Python" # Define a named "fr" preset and reference it from analyzer_config. # stopword_presets is mutually exclusive with stopwords — pass at most one. result = client.tokenization.text( text="La Tasse Bleue et le Bol", tokenization=Tokenization.WORD, analyzer_config=Configure.text_analyzer(stopword_preset="fr"), stopword_presets={ "fr": ["le", "la", "les", "un", "une", "des", "du", "de", "et"], }, ) print(f"indexed: {result.indexed}") print(f"query: {result.query}") ``` ```bash title="cURL" curl -X POST http://localhost:8080/v1/tokenize -d '{ "text": "La Tasse Bleue et le Bol", "tokenization": "word", "analyzerConfig": { "stopwordPreset": "fr" }, "stopwordPresets": { "fr": ["le", "la", "les", "un", "une", "des", "du", "de", "et"] } }' ``` ::: **Response:** ```json { "indexed": ["la", "tasse", "bleue", "et", "le", "bol"], "query": ["tasse", "bleue", "bol"] } ``` ### Property-based tokenization `POST /v1/schema/{className}/properties/{propertyName}/tokenize` resolves the full analyzer config from an existing property. The property's tokenization method, `textAnalyzer` settings, and the collection's stopword configuration are applied automatically. Nothing else needs to be passed. **Request body:** | Parameter | Type | Required | Details | | :-------- | :------- | :------- | :------------------- | | `text` | `string` | yes | The text to tokenize | The response format is the same as freeform tokenization. Class and property names are case-insensitive, and collection aliases are resolved automatically. **Example:** :::code-group{sync="languages"} ```python title="Python" # Tokenize using an existing property's configuration result = client.tokenization.for_property( collection="TokenizeDemo", property_name="name_fr", text="La Tasse Bleue et le Bol", ) print(f"indexed: {result.indexed}") print(f"query: {result.query}") ``` ```bash title="cURL" curl -X POST http://localhost:8080/v1/schema/TokenizeDemo/properties/name_fr/tokenize -d '{ "text": "La Tasse Bleue et le Bol" }' ``` ::: See the [tokenization tutorial](../guides-tutorials/tokenization.md#example-6-inspecting-tokenization-with-the-tokenize-endpoint) for worked examples. ## Further resources - [Concepts: Inverted index](../indexing/inverted-index.md) - [How-to: Set inverted index parameters](../how-to-manage-collections/inverted-index.md#set-inverted-index-parameters) - [Reference: Tokenization options](collections.md#tokenization) - Learn about different tokenization methods and how they affect text indexing ## Questions and feedback Have a question or feedback? Here's how to reach us. ::::card-grid :::card{title="Community Forum" href="https://forum.weaviate.io/c/support" icon="messages-square"} Ask questions and connect with other developers on our **Community forum**. ::: :::card{title="Support" href="/guides/support-overview" icon="life-buoy"} Weaviate Cloud user or customer? Find the right channel on the **Support page**. ::: :::: ## Related pages - [Agents](./agents-index.md) - [AI-assisted Weaviate code generation](./ai-assisted-vibe-coding-index.md) - [APIs](./apis-index.md) - [Authorization and authentication](./authorization-and-authentication-index.md) - [Benchmarks](./benchmarks-index.md) - [Best practices](./best-practices-index.md) - [Client libraries](./clients-index.md) - [Client Libraries / SDKs](./client-libraries-index.md) - [Cloud](./cloud-index.md) - [Cloud account management](./cloud-account-management-index.md) # Agent Instructions This portal answers questions programmatically. To receive a synthesized, source-cited answer instead of crawling page by page, append the `?ask=` query parameter to any page URL on this site: /guides/quickstart?ask=how+do+I+authenticate Optional parameters: - `&goal=` steers the answer toward your objective (e.g. `&goal=write+a+python+client`). - `&version=