# Inverted indexes Inverted indexes in Weaviate map values (like words or numbers) to the objects that contain them, enabling fast keyword search and filtering operations. ## How Weaviate creates inverted indexes Understanding Weaviate's indexing architecture is crucial for optimizing performance and resource usage. Weaviate creates **individual inverted indexes for each property and each index type**. This means: - Each property in your collection gets its own dedicated inverted index(es) - Meta properties (like creation timestamps) also get their own separate inverted indexes - A single property can have multiple inverted indexes if it supports multiple index types - All aggregations and combinations across properties happen at query time, not at index time **Example**: A `title` property with both `indexFilterable: true` and `indexSearchable: true` will result in two separate inverted indexes - one optimized for search operations and another for filtering operations. This architecture provides flexibility and performance optimization but also means that enabling multiple index types increases storage requirements and indexing overhead. For `text` properties specifically, the indexing process follows these steps: 1. **Tokenization**: The text is first tokenized according to the [tokenization method](#tokenization) configured for that property. 2. **Index entry creation**: Each processed token gets an entry in the inverted index, pointing to the object containing it. This process ensures that your text searches and filters can quickly locate relevant objects based on the tokens they contain. :::accordion{title="Performance improvements added in Oct 2024"} In Weaviate versions `v1.24.26`, `v1.25.20`, `v1.26.6` and `v1.27.0`, we introduced performance improvements and bugfixes for the BM25F scoring algorithm: - The BM25 segment merging algorithm was made faster - Improved WAND algorithm to remove exhausted terms from score computation and only do a full sort when necessary - Solved a bug in BM25F multi-prop search that could lead to not summing all the query term score for all segments - The BM25 scores are now calculated concurrently for multiple segments As always, we recommend upgrading to the latest version of Weaviate to benefit from improvements such as these. ::: ## BlockMax WAND algorithm :::callout{intent="info" title="Added in `v1.30`"} ::: The BlockMax WAND algorithm is a variant of the WAND algorithm that is used to speed up BM25 and hybrid searches. It organizes the inverted index in blocks to enable skipping over blocks that are not relevant to the query. This can significantly reduce the number of documents that need to be scored, improving search performance. If you are experiencing slow BM25 (or hybrid) searches and use a Weaviate version prior to `v1.30`, try migrating to a newer version that uses the BlockMax WAND algorithm to see if it improves performance. If you need to migrate existing data from a previous version of Weaviate, follow the [v1.30 migration guide](../deploy-migration/weaviate-1-30.md). :::callout{intent="note" title="Scoring changes with BlockMax WAND"} Due to the nature of the BlockMax WAND algorithm, the scoring of BM25 and hybrid searches may differ slightly from the default WAND algorithm. Additionally BlockMax WAND scores on single and multiple property search may be different due to different IDF and property length normalization calculations. This is expected behavior and is not a bug. ::: ## Configure inverted indexes There are three inverted index types in Weaviate: - `indexSearchable` - a searchable index for BM25 or hybrid search - `indexFilterable` - a match-based index for fast [filtering](../concepts/filtering.md) by matching criteria - `indexRangeFilters` - a range-based index for [filtering](../concepts/filtering.md) by numerical ranges Each inverted index can be set to `true` (on) or `false` (off) on a property level. The `indexSearchable` and `indexFilterable` indexes are on by default, while the `indexRangeFilters` index is off by default. The filterable indexes are only capable of [filtering](../concepts/filtering.md), while the searchable index can be used for both searching and filtering (though not as fast as the filterable index). So, setting `"indexFilterable": false` and `"indexSearchable": true` (or not setting it at all) will have the trade-off of worse filtering performance but faster imports (due to only needing to update one index) and lower disk usage. See the [related how-to section](../how-to-manage-collections/vector-config.md#property-level-settings) to learn how to enable or disable inverted indexes on a property level. A rule of thumb to follow when determining whether to switch off indexing is: _if you will never perform queries based on this property, you can turn it off._ #### Inverted index types summary | Inverted index type | Description | Applicable data types | Default | Availability | | ------------------- | ---------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | ------- | ------------ | | `indexSearchable` | A searchable index for BM25-suitable Map index for BM25 or hybrid searching. | `text`, `text[]`, | `true` | `v1.19` | | `indexFilterable` | A Roaring Bitmap index for match-based filtering. | Everything except `blob`, `geoCoordinates`, `object` and `phoneNumber` data types including arrays thereof | `true` | `v1.19` | | `indexRangeFilters` | A Roaring Bitmap index for numerical range-based filtering. | `int`, `number` and `date` only | `false` | `v1.26` | - Enable one or both of `indexFilterable` and `indexRangeFilters` to index a property for faster filtering. - If only one is enabled, the respective index is used for filtering. - If both are enabled, `indexRangeFilters` is used for operations involving comparison operators, and `indexFilterable` is used for equality and inequality operations. This chart shows which filter makes the comparison when one or both index type is `true` for an applicable property. | Operator | `indexRangeFilters` only | `indexFilterable` only | Both enabled | | :----------------- | :----------------------- | :--------------------- | :------------------ | | Equal | `indexRangeFilters` | `indexFilterable` | `indexFilterable` | | Not equal | `indexRangeFilters` | `indexFilterable` | `indexFilterable` | | Greater than | `indexRangeFilters` | `indexFilterable` | `indexRangeFilters` | | Greater than equal | `indexRangeFilters` | `indexFilterable` | `indexRangeFilters` | | Less than | `indexRangeFilters` | `indexFilterable` | `indexRangeFilters` | | Less than equal | `indexRangeFilters` | `indexFilterable` | `indexRangeFilters` | #### Inverted index for timestamps You can also enable an inverted index to search [based on timestamps](../reference-configuration/indexing-inverted-index.md#indextimestamps). Timestamps are currently indexed using the `indexFilterable` index. ## Collections without indexes If you don't want to set an index at all, this is possible too. To create a collection without any indexes, skip indexing on the collection and on the properties. :::accordion{title="Example collection configuration without inverted indexes - JSON object"} An example of a complete collection object without inverted indexes: ```js { "class": "Author", "description": "A description of this collection, in this case, it's about authors", "vectorIndexConfig": { "skip": true // <== disable vector index }, "properties": [ { "indexFilterable": false, // <== disable filterable index for this property "indexSearchable": false, // <== disable searchable index for this property "dataType": [ "text" ], "description": "The name of the Author", "name": "name" }, { "indexFilterable": false, // <== disable filterable index for this property "dataType": [ "int" ], "description": "The age of the Author", "name": "age" }, { "indexFilterable": false, // <== disable filterable index for this property "dataType": [ "date" ], "description": "The date of birth of the Author", "name": "born" }, { "indexFilterable": false, // <== disable filterable index for this property "dataType": [ "boolean" ], "description": "A boolean value if the Author won a nobel prize", "name": "wonNobelPrize" }, { "indexFilterable": false, // <== disable filterable index for this property "indexSearchable": false, // <== disable searchable index for this property "dataType": [ "text" ], "description": "A description of the author", "name": "description" } ] } ``` ::: ## Tokenization Tokenization is the process of breaking text into smaller units called tokens. This process is fundamental to how inverted indexes work - the tokens produced determine what can be searched and how matching occurs. ### How tokenization works When you add an object to Weaviate, text in each property is tokenized according to that property's configured tokenization method. For example, the text: '"Ankh-Morpork's police captain"' could be tokenized using different tokenization methods: 1. `'word'`: `["ankh", "morpork", "s", "police", "captain"]` - splits on non-alphanumeric characters, lowercased 2. `'lowercase'`: `["ankh-morpork's", "police", "captain"]` - splits on whitespace only, lowercased 3. `'whitespace'`: `["Ankh-Morpork's", "police", "captain"]` - splits on whitespace, preserves case 4. `'field'`: `["Ankh-Morpork's police captain"]` - treats entire text as single token Each tokenization method serves different use cases and directly impacts search and filter behavior. ### Tokenization and the inverted index The inverted index maps each token to the objects containing it. When you perform a keyword search or filter: 1. Your query/filter text is tokenized using the **same method** as the indexed property 2. The inverted index looks up which objects contain those tokens 3. For searches, BM25f ranks results based on token matches 4. For filters, exact token matches determine inclusion This means the tokenization method controls the "granularity" of matching. For example, with `word` tokenization, searching for `"clark"` will match an object containing `"Clark:"` because both tokenize to `["clark"]`. With `field` tokenization, only exact matches succeed. ### Available tokenization methods Weaviate provides several tokenization methods optimized for different data types: **Standard methods:** - **`word`** (default): Splits on non-alphanumeric characters, lowercases. Best for typical text. - **`lowercase`**: Splits on whitespace, lowercases. Preserves symbols like `@`, `_`, `-`. - **`whitespace`**: Splits on whitespace, preserves case and symbols. For case-sensitive data. - **`field`**: No splitting - entire value is one token. For exact matching. **Language-specific methods** (for languages without word boundaries): - **`gse`**: Japanese text segmentation using the [`gse`](https://pkg.go.dev/github.com/go-ego/gse) tokenizer (Japanese dictionary) - **`gse_ch`**: Chinese text segmentation using the same `gse` tokenizer with a Chinese dictionary - **`trigram`**: Splits into character trigrams for CJK languages - **`kagome_ja`**: Japanese morphological analysis - **`kagome_kr`**: Korean morphological analysis These language-specific tokenizers are not loaded by default. Enable them with the corresponding environment variables (`ENABLE_TOKENIZER_GSE`, `ENABLE_TOKENIZER_GSE_CH`, `ENABLE_TOKENIZER_KAGOME_JA`, `ENABLE_TOKENIZER_KAGOME_KR`). See the [tokenization configuration reference](../reference-configuration/collections.md#tokenization) for detailed specifications and behavior examples. ### Accent folding :::callout{intent="warning" title="Preview — added in `v1.37`"} This is a preview feature. The API may change in future releases. ::: Text properties can opt in to **accent folding** via the `textAnalyzer` block. When `asciiFold` is set to `true`, the analyzer normalizes accented Latin characters and any other character carrying combining marks or diacritics to their ASCII equivalents during both indexing and querying. A document containing "Café Crème" becomes searchable as "cafe creme", and vice versa. The same normalization is applied to filters (`Equal`, `Like`), so what you can search for is exactly what you can filter on. ```json { "name": "description", "dataType": ["text"], "tokenization": "word", "textAnalyzer": { "asciiFold": true } } ``` The implementation uses [Unicode NFD decomposition](https://unicode.org/reports/tr15/) (covering acute, grave, circumflex, tilde, dieresis, caron, cedilla, ogonek, macron, breve, ring, and more) plus an explicit replacement table for single-codepoint letters that do not decompose, such as `ł`, `æ`, `ø`, `ð`, `þ`, `đ`, and `ß`. Together this covers 20+ Latin-script languages, including French, Portuguese, Spanish, German, Polish, Czech, Croatian, and Icelandic. Accent folding composes with every tokenization method: `word`, `lowercase`, `whitespace`, `field`, and `trigram`. #### Per-character exceptions If you want most accents folded but need to preserve specific characters (for example, an `é` that distinguishes two product names) use `asciiFoldIgnore`: ```json { "name": "name", "dataType": ["text"], "tokenization": "word", "textAnalyzer": { "asciiFold": true, "asciiFoldIgnore": ["é", "Ł"] } } ``` Because `asciiFoldIgnore` changes which tokens are written to disk, it is **immutable** after the property is created. Schema updates that change the ignore list are rejected. To change it, create a new property and reindex. See the [accent folding tutorial](../guides-tutorials/tokenization.md#example-4-accent-folding) for a worked example and the [textAnalyzer configuration reference](../reference-configuration/indexing-inverted-index.md#textanalyzer) for all options. ### Impact on search and filtering #### Filters Filters perform binary matching - an object either matches or doesn't. Tokenization determines what counts as a match: | Query | Indexed text | `word` | `lowercase` | `whitespace` | `field` | | ----------------- | --------------------- | ------ | ----------- | ------------ | ------- | | `"clark"` | `"Clark:"` | ✅ | ❌ | ❌ | ❌ | | `"variable_name"` | `"variable_name"` | ✅ | ✅ | ✅ | ✅ | | `"variable_name"` | `"variable_new_name"` | ✅ | ❌ | ❌ | ❌ | With `word` tokenization, `"variable_name"` matches `"variable_new_name"` because both contain the tokens `["variable", "name"]`. #### Keyword searches Keyword searches use BM25f to rank results. Tokenization affects: 1. **Result inclusion**: Only objects with matching tokens appear 2. **Ranking scores**: More matching tokens = higher scores For example, searching for `"lois clark"` with `word` tokenization will rank objects containing both words higher than those with just one. ### Stop words Stop words are common words (like "a", "the", "is") that are typically ignored during search. By default, Weaviate uses a standard English stop words list. After tokenization, stop words in queries behave as if they're not present for matching purposes: - Filter for `"a computer mouse"` behaves like `"computer mouse"` - Stop words still affect BM25f ranking scores You can [configure custom stop words](../reference-configuration/indexing-inverted-index.md#stopwords) in your collection definition. **Note**: With `field` tokenization, stop words don't apply since the entire field is one token. #### Custom stopword presets :::callout{intent="warning" title="Preview — added in `v1.37`"} This is a preview feature. The API may change in future releases. ::: Beyond the built-in `en` and `none` presets, you can declare custom stopword presets on the collection's `invertedIndexConfig.stopwordPresets`. Each preset has a name and a flat word list. A preset name that matches a built-in (`en`, `none`) replaces the built-in for this collection. ```json { "invertedIndexConfig": { "stopwordPresets": { "fr": ["le", "la", "les", "un", "une", "des", "du", "de", "et"], "de": ["der", "die", "das", "und", "oder", "aber"] } } } ``` #### Per-property stopword overrides Each text property can override the collection-level stopword behavior via `textAnalyzer.stopwordPreset`. This is useful for multilingual collections where different properties contain text in different languages. The override is only supported on properties with `tokenization: "word"`. Schema validation rejects it on other tokenizers. ```json "properties": [ { "name": "name_en", "dataType": ["text"], "tokenization": "word", "textAnalyzer": { "stopwordPreset": "en" } }, { "name": "name_fr", "dataType": ["text"], "tokenization": "word", "textAnalyzer": { "stopwordPreset": "fr" } } ] ``` Stopwords are still **indexed**: they are only filtered at query time. Changing the stopword configuration does **not** require reindexing your data. See the [custom stopwords tutorial](../guides-tutorials/tokenization.md#example-5-custom-and-per-property-stopword-presets) for a worked example and the [stopwordPresets configuration reference](../reference-configuration/indexing-inverted-index.md#stopwordpresets) for all options. ### Choosing a tokenization method The choice of tokenization method should match your data characteristics and search requirements. Here are some general guidelines: - **General text** (articles, descriptions): Use `word` (default) - **Technical data with symbols** (code, emails): Use `lowercase` - **Case-sensitive data** (names, acronyms): Use `whitespace` - **Unique identifiers** (URLs, IDs): Use `field` - **CJK languages**: Use language-specific methods For detailed guidance and practical examples, see the [tokenization tutorial](../guides-tutorials/tokenization.md). ## Further resources :::callout{intent="info" title="Related pages"} - [Configuration: Inverted index](../reference-configuration/indexing-inverted-index.md) - [How-to: Configure collections](../how-to-manage-collections/vector-config.md#property-level-settings) - [Configuration: Tokenization](../reference-configuration/collections.md#tokenization) - [Tutorial: Configure tokenization](../guides-tutorials/tokenization.md) ::: ## Questions and feedback Have a question or feedback? Here's how to reach us. ::::card-grid :::card{title="Community Forum" href="https://forum.weaviate.io/c/support" icon="messages-square"} Ask questions and connect with other developers on our **Community forum**. ::: :::card{title="Support" href="/guides/support-overview" icon="life-buoy"} Weaviate Cloud user or customer? Find the right channel on the **Support page**. ::: :::: ## Related pages - [Agents](./agents-index.md) - [AI-assisted Weaviate code generation](./ai-assisted-vibe-coding-index.md) - [APIs](./apis-index.md) - [Authorization and authentication](./authorization-and-authentication-index.md) - [Benchmarks](./benchmarks-index.md) - [Best practices](./best-practices-index.md) - [Client libraries](./clients-index.md) - [Client Libraries / SDKs](./client-libraries-index.md) - [Cloud](./cloud-index.md) - [Cloud account management](./cloud-account-management-index.md) # Agent Instructions This portal answers questions programmatically. To receive a synthesized, source-cited answer instead of crawling page by page, append the `?ask=` query parameter to any page URL on this site: /guides/quickstart?ask=how+do+I+authenticate Optional parameters: - `&goal=` steers the answer toward your objective (e.g. `&goal=write+a+python+client`). - `&version=