## What you'll learn In this tutorial, you'll learn how to configure tokenization in Weaviate and see how different tokenization methods impact keyword search and filtering results. By the end of this tutorial, you'll understand: - How to configure tokenization for a collection property - How tokenization affects filter matching - How tokenization impacts keyword search ranking - How to choose the right tokenization method for your use case - How accent folding normalizes accented characters for matching - How to define and use custom stopword presets per property - How to use the tokenize endpoint to test configurations ## Prerequisites - A running Weaviate instance - Python Weaviate client installed - Basic familiarity with Weaviate collections ## Create a demo collection We'll create a collection with multiple properties, each using a different tokenization method. This allows us to compare how the same text behaves under different tokenization strategies. ```python import weaviate from weaviate.classes.config import Property, DataType, Tokenization, Configure ``` Note that we do not add object vectors in this case, as we are only interested in the impact of tokenization on filters and keyword searches. ## Add test data We'll use a small, custom dataset for demonstration purposes. ```python collection = client.collections.use("TokenizationDemo") ``` Now, add objects to the collection, repeating text objects across properties with different tokenization methods. ```python import weaviate ``` ## Example 1: Punctuation and case sensitivity Let's see how tokenization handles messy text with punctuation and mixed cases. We'll filter for various combinations of substrings from the TV show title `"Lois & Clark: The New Adventures of Superman"`. ### Setup the filter function We'll create a reusable function to filter objects based on query strings. Remember that a filter is binary: it either matches or it doesn't. ```python import weaviate from weaviate.classes.query import Filter from weaviate.collections import Collection from typing import List ``` ### Test "Clark:" vs "clark" ```python filter_demo(collection, property_names, ["clark", "Clark", "clark:", "Clark:", "lois clark", "clark lois"]) ``` The results show whether the query matched the title: | | `word` | `lowercase` | `whitespace` | `field` | | -------------- | ------ | ----------- | ------------ | ------- | | `"clark"` | ✅ | ❌ | ❌ | ❌ | | `"Clark"` | ✅ | ❌ | ❌ | ❌ | | `"clark:" ` | ✅ | ✅ | ❌ | ❌ | | `"Clark:" ` | ✅ | ✅ | ✅ | ❌ | | `"lois clark"` | ✅ | ❌ | ❌ | ❌ | | `"clark lois"` | ✅ | ❌ | ❌ | ❌ | :::accordion{title="Example output"} ```text ======================================== Hits for: 'clark' ======================================== >> 'text_word' matches Lois & Clark: The New Adventures of Superman ======================================== Hits for: 'Clark' ======================================== >> 'text_word' matches Lois & Clark: The New Adventures of Superman ======================================== Hits for: 'clark:' ======================================== >> 'text_word' matches Lois & Clark: The New Adventures of Superman >> 'text_lowercase' matches Lois & Clark: The New Adventures of Superman ======================================== Hits for: 'Clark:' ======================================== >> 'text_word' matches Lois & Clark: The New Adventures of Superman >> 'text_lowercase' matches Lois & Clark: The New Adventures of Superman >> 'text_whitespace' matches Lois & Clark: The New Adventures of Superman ======================================== Hits for: 'lois clark' ======================================== >> 'text_word' matches Lois & Clark: The New Adventures of Superman ======================================== Hits for: 'clark lois' ======================================== >> 'text_word' matches Lois & Clark: The New Adventures of Superman ``` ::: **Key observations:** - `word` tokenization consistently matches regardless of case or punctuation - `lowercase` and `whitespace` require more exact matches - Users typically don't include punctuation in queries, making `word` a good default ## Example 2: Stop words Here, we filter for variants of the phrase "computer mouse", where some queries include additional words like "a" or "the". ```python filter_demo(collection, property_names, ["computer mouse", "a computer mouse", "the computer mouse", "blue computer mouse"]) ``` **Matches for `"computer mouse"`** | | `word` | `lowercase` | `whitespace` | `field` | | ------------------------ | ------ | ----------- | ------------ | ------- | | `"computer mouse"` | ✅ | ✅ | ✅ | ✅ | | `"a computer mouse"` | ✅ | ✅ | ✅ | ❌ | | `"the computer mouse:" ` | ✅ | ✅ | ✅ | ❌ | | `"blue computer mouse" ` | ❌ | ❌ | ❌ | ❌ | **Matches for `"a computer mouse"`** | | `word` | `lowercase` | `whitespace` | `field` | | ------------------------ | ------ | ----------- | ------------ | ------- | | `"computer mouse"` | ✅ | ✅ | ✅ | ❌ | | `"a computer mouse"` | ✅ | ✅ | ✅ | ✅ | | `"the computer mouse:" ` | ✅ | ✅ | ✅ | ❌ | | `"blue computer mouse" ` | ❌ | ❌ | ❌ | ❌ | :::accordion{title="Example output"} ```text ======================================== Hits for: 'computer mouse' ======================================== >> 'text_word' matches computer mouse Computer Mouse mouse computer computer mouse pad a computer mouse >> 'text_lowercase' matches computer mouse Computer Mouse mouse computer computer mouse pad a computer mouse >> 'text_whitespace' matches computer mouse mouse computer computer mouse pad a computer mouse >> 'text_field' matches computer mouse ======================================== Hits for: 'a computer mouse' ======================================== >> 'text_word' matches computer mouse Computer Mouse mouse computer computer mouse pad a computer mouse >> 'text_lowercase' matches computer mouse Computer Mouse mouse computer computer mouse pad a computer mouse >> 'text_whitespace' matches computer mouse mouse computer computer mouse pad a computer mouse >> 'text_field' matches a computer mouse ======================================== Hits for: 'the computer mouse' ======================================== >> 'text_word' matches computer mouse Computer Mouse mouse computer computer mouse pad a computer mouse >> 'text_lowercase' matches computer mouse Computer Mouse mouse computer computer mouse pad a computer mouse >> 'text_whitespace' matches computer mouse mouse computer computer mouse pad a computer mouse ======================================== Hits for: 'blue computer mouse' ======================================== ``` ::: **Key observations:** - Stop words like "a" and "the" are ignored in `word`, `lowercase`, and `whitespace` tokenization - `field` tokenization treats the entire string as one token, so stop words matter - Adding non-stop words like "blue" prevents matches ## Example 3: Symbols and underscores The `word` tokenization is a good default, but may not work for data with meaningful symbols. Let's test different variants of `"variable_name"`. ```python filter_demo(collection, property_names, ["variable_name"]) ``` | | `word` | `lowercase` | `whitespace` | `field` | | ---------------------- | ------ | ----------- | ------------ | ------- | | `"variable_name"` | ✅ | ✅ | ✅ | ✅ | | `"Variable_Name:" ` | ✅ | ✅ | ❌ | ❌ | | `"Variable Name:" ` | ✅ | ❌ | ❌ | ❌ | | `"a_variable_name"` | ✅ | ❌ | ❌ | ❌ | | `"the_variable_name"` | ✅ | ❌ | ❌ | ❌ | | `"variable_new_name" ` | ✅ | ❌ | ❌ | ❌ | :::accordion{title="Example output"} ```text ======================================== Hits for: 'variable_name' ======================================== >> 'text_word' matches variable_name Variable_Name Variable Name a_variable_name the_variable_name variable_new_name >> 'text_lowercase' matches variable_name Variable_Name >> 'text_whitespace' matches variable_name >> 'text_field' matches variable_name ``` ::: **Key observations:** - `word` tokenization treats underscores as separators, which may be too permissive - For code, email addresses, or data where symbols are meaningful, use `lowercase` or `whitespace` - Consider whether `"variable_new_name"` should match `"variable_name"` in your use case ## Example 4: Accent folding :::callout{intent="warning" title="Preview — added in `v1.37`"} This is a preview feature. The API may change in future releases. ::: Suppose you run a multilingual product catalog with names like "Café Crème Bio", "Łódź Ceramics", and "São Paulo Sandals". Without accent folding, a user searching for "cafe creme" or "lodz" would not find these products because the accented and unaccented forms produce different tokens. ### Create a collection with accent folding :::code-group{sync="languages"} ```python title="Python" import weaviate from weaviate.classes.config import Property, DataType, Tokenization, Configure ``` ```typescript title="JavaScript/TypeScript" import weaviate from 'weaviate-client'; // Instantiate your client (not shown). e.g.: // const client = await weaviate.connectToWeaviateCloud(...) or // const client = await weaviate.connectToLocal(); ``` ```java title="Java v6" // START CustomStopwordsCreate import io.weaviate.client6.v1.api.WeaviateClient; import io.weaviate.client6.v1.api.collections.Property; import io.weaviate.client6.v1.api.collections.TextAnalyzer; import io.weaviate.client6.v1.api.collections.Tokenization; import io.weaviate.client6.v1.api.collections.VectorConfig; ``` ```csharp title="C#" await client.Collections.Create( new CollectionCreateParams { Name = "AccentFoldingDemo", Properties = [ new Property { Name = "text_default", DataType = DataType.Text, PropertyTokenization = PropertyTokenization.Word, }, new Property { Name = "text_folded", DataType = DataType.Text, PropertyTokenization = PropertyTokenization.Word, TextAnalyzer = new TextAnalyzerConfig { AsciiFold = new AsciiFoldConfig(), }, }, new Property { Name = "text_folded_keep_e", DataType = DataType.Text, PropertyTokenization = PropertyTokenization.Word, TextAnalyzer = new TextAnalyzerConfig { AsciiFold = new AsciiFoldConfig(Ignore: ["é"]), }, }, ], VectorConfig = new VectorConfigList { Configure.Vector("default", v => v.SelfProvided()), }, } ); ``` ::: We create three properties: one without folding (`text_default`), one with full folding (`text_folded`), and one that preserves `é` (`text_folded_keep_e`). ### Add test data :::code-group{sync="languages"} ```python title="Python" products = client.collections.get("AccentFoldingDemo") test_strings = [ "Café Crème Bio", "Łódź Ceramics", "São Paulo Sandals", "Müller Bräu", ] for text in test_strings: products.data.insert( properties={ "text_default": text, "text_folded": text, "text_folded_keep_e": text, } ) ``` ```typescript title="JavaScript/TypeScript" const products = client.collections.use('AccentFoldingDemo'); const testStrings = ['Café Crème Bio', 'Łódź Ceramics', 'São Paulo Sandals', 'Müller Bräu']; for (const text of testStrings) { await products.data.insert({ text_default: text, text_folded: text, text_folded_keep_e: text, }); } ``` ```java title="Java v6" CollectionHandle> products = client.collections.use(COLLECTION); List testStrings = List.of( "Café Crème Bio", "Łódź Ceramics", "São Paulo Sandals", "Müller Bräu"); for (String text : testStrings) { products.data.insert(Map.of( "text_default", text, "text_folded", text, "text_folded_keep_e", text)); } ``` ```csharp title="C#" var products = client.Collections.Use("AccentFoldingDemo"); string[] testStrings = [ "Café Crème Bio", "Łódź Ceramics", "São Paulo Sandals", "Müller Bräu", ]; foreach (var text in testStrings) { await products.Data.Insert( new { text_default = text, text_folded = text, text_folded_keep_e = text, } ); } ``` ::: ### Filter with accent folding :::code-group{sync="languages"} ```python title="Python" from weaviate.classes.query import Filter queries = ["cafe", "Café", "lodz", "sao paulo", "muller"] properties = ["text_default", "text_folded", "text_folded_keep_e"] for query in queries: print(f'\nQuery: "{query}"') for prop in properties: response = products.query.fetch_objects( filters=Filter.by_property(prop).equal(query), ) matches = [o.properties[prop] for o in response.objects] print(f" {prop}: {matches if matches else 'no match'}") ``` ```typescript title="JavaScript/TypeScript" import { Filters } from 'weaviate-client'; const queries = ['cafe', 'Café', 'lodz', 'sao paulo', 'muller']; const properties = ['text_default', 'text_folded', 'text_folded_keep_e']; for (const query of queries) { console.log(`\nQuery: "${query}"`); for (const prop of properties) { const response = await products.query.fetchObjects({ filters: products.filter.byProperty(prop).equal(query), }); const matches = response.objects.map((o) => o.properties[prop] as string); console.log(` ${prop}: ${matches.length ? JSON.stringify(matches) : 'no match'}`); } } ``` ```java title="Java v6" String[] queries = {"cafe", "Café", "lodz", "sao paulo", "muller"}; String[] properties = {"text_default", "text_folded", "text_folded_keep_e"}; for (String query : queries) { System.out.println("\nQuery: \"" + query + "\""); for (String prop : properties) { var response = products.query.fetchObjects( q -> q.filters(Filter.property(prop).eq(query))); var matches = response.objects().stream() .map(o -> (String) o.properties().get(prop)) .toList(); System.out.println( " " + prop + ": " + (matches.isEmpty() ? "no match" : matches)); } } ``` ```csharp title="C#" string[] queries = ["cafe", "Café", "lodz", "sao paulo", "muller"]; string[] properties = ["text_default", "text_folded", "text_folded_keep_e"]; foreach (var query in queries) { Console.WriteLine($"\nQuery: \"{query}\""); foreach (var prop in properties) { var response = await products.Query.FetchObjects( filters: Filter.Property(prop).IsEqual(query) ); var matches = response .Objects.Select(o => (string)o.Properties[prop]) .ToList(); Console.WriteLine( $" {prop}: {(matches.Count == 0 ? "no match" : string.Join(", ", matches))}" ); } } ``` ::: | Query | `text_default` (no folding) | `text_folded` | `text_folded_keep_e` | | ------------- | --------------------------- | ------------------- | -------------------- | | `"cafe"` | ❌ | ✅ Café Crème Bio | ❌ (`é` preserved) | | `"Café"` | ✅ | ✅ | ✅ | | `"lodz"` | ❌ | ✅ Łódź Ceramics | ✅ | | `"sao paulo"` | ❌ | ✅ São Paulo Sandals | ✅ | | `"muller"` | ❌ | ✅ Müller Bräu | ✅ | :::accordion{title="Example output"} ```text """ Query: "cafe" text_default: no match text_folded: ['Café Crème Bio'] text_folded_keep_e: no match Query: "Café" text_default: ['Café Crème Bio'] text_folded: ['Café Crème Bio'] text_folded_keep_e: ['Café Crème Bio'] Query: "lodz" text_default: no match text_folded: ['Łódź Ceramics'] text_folded_keep_e: ['Łódź Ceramics'] Query: "sao paulo" text_default: no match text_folded: ['São Paulo Sandals'] text_folded_keep_e: ['São Paulo Sandals'] Query: "muller" text_default: no match text_folded: ['Müller Bräu'] text_folded_keep_e: ['Müller Bräu'] """ ``` ::: **Key observations:** - Without folding, only exact accented forms match - With `asciiFold: true`, both accented and unaccented queries match - `asciiFoldIgnore` lets you preserve specific characters: `"cafe"` no longer matches `"Café"` when `é` is ignored - `asciiFoldIgnore` is immutable after property creation ## Example 5: Custom and per-property stopword presets :::callout{intent="warning" title="Preview — added in `v1.37`"} This is a preview feature. The API may change in future releases. ::: The default stopword presets are `en` and `none`. For a French property, neither is appropriate: `la`, `le`, and `et` should be filtered, but they are not in the English list. Define a custom preset on the collection and assign it to specific properties. ### Create a collection with custom stopwords :::code-group{sync="languages"} ```python title="Python" import weaviate from weaviate.classes.config import Property, DataType, Tokenization, Configure ``` ```typescript title="JavaScript/TypeScript" import weaviate from 'weaviate-client'; // Instantiate your client (not shown). e.g.: // const client = await weaviate.connectToWeaviateCloud(...) or // const client = await weaviate.connectToLocal(); ``` ```java title="Java v6" import io.weaviate.client6.v1.api.WeaviateClient; import io.weaviate.client6.v1.api.collections.Property; import io.weaviate.client6.v1.api.collections.TextAnalyzer; import io.weaviate.client6.v1.api.collections.Tokenization; import io.weaviate.client6.v1.api.collections.VectorConfig; // END AccentFoldingCreateCollection ``` ```csharp title="C#" var presets = new Dictionary> { ["fr"] = ["le", "la", "les", "un", "une", "des", "du", "de", "et"], }; await client.Collections.Create( new CollectionCreateParams { Name = "StopwordsDemo", InvertedIndexConfig = new InvertedIndexConfig { StopwordPresets = presets }, Properties = [ new Property { Name = "name_en", DataType = DataType.Text, PropertyTokenization = PropertyTokenization.Word, TextAnalyzer = new TextAnalyzerConfig { StopwordPreset = "en" }, }, new Property { Name = "name_fr", DataType = DataType.Text, PropertyTokenization = PropertyTokenization.Word, TextAnalyzer = new TextAnalyzerConfig { StopwordPreset = "fr" }, }, ], VectorConfig = new VectorConfigList { Configure.Vector("default", v => v.SelfProvided()), }, } ); ``` ::: ### Add test data :::code-group{sync="languages"} ```python title="Python" products = client.collections.get("StopwordsDemo") products.data.insert_many([ { "name_en": "The Blue Cup and the Bowl", "name_fr": "La Tasse Bleue et le Bol", }, { "name_en": "A Red Plate with the Saucer", "name_fr": "Une Assiette Rouge avec la Soucoupe", }, ]) ``` ```typescript title="JavaScript/TypeScript" const products = client.collections.use('StopwordsDemo'); await products.data.insertMany([ { name_en: 'The Blue Cup and the Bowl', name_fr: 'La Tasse Bleue et le Bol', }, { name_en: 'A Red Plate with the Saucer', name_fr: 'Une Assiette Rouge avec la Soucoupe', }, ]); ``` ```java title="Java v6" CollectionHandle> products = client.collections.use(COLLECTION); products.data.insert(Map.of( "name_en", "The Blue Cup and the Bowl", "name_fr", "La Tasse Bleue et le Bol")); products.data.insert(Map.of( "name_en", "A Red Plate with the Saucer", "name_fr", "Une Assiette Rouge avec la Soucoupe")); ``` ```csharp title="C#" var products = client.Collections.Use("StopwordsDemo"); await products.Data.Insert( new { name_en = "The Blue Cup and the Bowl", name_fr = "La Tasse Bleue et le Bol", } ); await products.Data.Insert( new { name_en = "A Red Plate with the Saucer", name_fr = "Une Assiette Rouge avec la Soucoupe", } ); ``` ::: ### Search with per-property stopwords :::code-group{sync="languages"} ```python title="Python" from weaviate.classes.query import MetadataQuery # Search the French property — "la" and "et" are French stopwords response = products.query.bm25( query="la tasse bleue et le bol", query_properties=["name_fr"], return_metadata=MetadataQuery(score=True), ) print("French property search:") for o in response.objects: print(f" {o.properties['name_fr']} (score: {o.metadata.score})") # Same words on the English property — "la", "et", "le" are NOT English stopwords response = products.query.bm25( query="la tasse bleue et le bol", query_properties=["name_en"], return_metadata=MetadataQuery(score=True), ) print("\nEnglish property search:") for o in response.objects: print(f" {o.properties['name_en']} (score: {o.metadata.score})") ``` ```typescript title="JavaScript/TypeScript" // Search the French property — "la" and "et" are French stopwords let response = await products.query.bm25('la tasse bleue et le bol', { queryProperties: ['name_fr'], returnMetadata: ['score'], }); console.log('French property search:'); for (const o of response.objects) { console.log(` ${o.properties.name_fr} (score: ${o.metadata?.score})`); } // Same words on the English property — "la", "et", "le" are NOT English stopwords response = await products.query.bm25('la tasse bleue et le bol', { queryProperties: ['name_en'], returnMetadata: ['score'], }); console.log('\nEnglish property search:'); for (const o of response.objects) { console.log(` ${o.properties.name_en} (score: ${o.metadata?.score})`); } ``` ```java title="Java v6" var responseFr = products.query.bm25( "la tasse bleue et le bol", q -> q.queryProperties("name_fr").returnMetadata(Metadata.SCORE)); System.out.println("French property search:"); for (var o : responseFr.objects()) { System.out.println( " " + o.properties().get("name_fr") + " (score: " + o.queryMetadata().score() + ")"); } var responseEn = products.query.bm25( "la tasse bleue et le bol", q -> q.queryProperties("name_en").returnMetadata(Metadata.SCORE)); System.out.println("\nEnglish property search:"); for (var o : responseEn.objects()) { System.out.println( " " + o.properties().get("name_en") + " (score: " + o.queryMetadata().score() + ")"); } ``` ```csharp title="C#" var responseFr = await products.Query.BM25( query: "la tasse bleue et le bol", searchFields: ["name_fr"], returnMetadata: MetadataOptions.Score ); Console.WriteLine("French property search:"); foreach (var o in responseFr.Objects) { Console.WriteLine( $" {o.Properties["name_fr"]} (score: {o.Metadata.Score})" ); } var responseEn = await products.Query.BM25( query: "la tasse bleue et le bol", searchFields: ["name_en"], returnMetadata: MetadataOptions.Score ); Console.WriteLine("\nEnglish property search:"); foreach (var o in responseEn.Objects) { Console.WriteLine( $" {o.Properties["name_en"]} (score: {o.Metadata.Score})" ); } ``` ::: :::accordion{title="Example output"} ```text """ French property search: La Tasse Bleue et le Bol (score: 0.95) English property search: (no results — "tasse", "bleue", "bol" are not in the English data) """ ``` ::: **Key observations:** - The `fr` preset filters out `la`, `le`, and `et` from BM25 scoring on the French property - The same words are not filtered on the English property (they are not English stopwords) - Stopwords are still indexed (they are only filtered at query time), so changing presets does not require reindexing - A preset name that matches a built-in (`en`, `none`) replaces the built-in for this collection. To tweak a built-in with `additions`/`removals`, use the collection-level `invertedIndexConfig.stopwords` field instead ## Example 6: Inspecting tokenization with the tokenize endpoint :::callout{intent="warning" title="Preview — added in `v1.37`"} This is a preview feature. The API may change in future releases. ::: Tuning an analyzer is much easier when you can see what it does. Two REST endpoints make the tokenization process visible. ### Ad-hoc tokenization `POST /v1/tokenize` tokenizes arbitrary text with an explicit tokenizer and analyzer config. Use this to test configurations before committing them to a schema. :::code-group{sync="languages"} ```python title="Python" import weaviate from weaviate.classes.config import Tokenization, Configure client = weaviate.connect_to_local() # Ad-hoc tokenization with custom config result = client.tokenization.text( text="The organic café crème blend", tokenization=Tokenization.WORD, analyzer_config=Configure.text_analyzer( ascii_fold=True, stopword_preset="en", ), ) print(f"indexed: {result.indexed}") print(f"query: {result.query}") ``` ```typescript title="JavaScript/TypeScript" import weaviate from 'weaviate-client'; const client = await weaviate.connectToLocal(); // Ad-hoc tokenization with custom config let result = await client.tokenize.text('The organic café crème blend', 'word', { analyzerConfig: { asciiFold: true, stopwordPreset: 'en', }, }); console.log(`indexed: ${JSON.stringify(result.indexed)}`); console.log(`query: ${JSON.stringify(result.query)}`); ``` ```java title="Java v6" import io.weaviate.client6.v1.api.tokenize.TokenizeResponse; ``` ```csharp title="C#" // Ad-hoc tokenization with custom config var result = await client.Tokenize.Text( text: "The organic café crème blend", tokenization: PropertyTokenization.Word, analyzerConfig: new TextAnalyzerConfig { AsciiFold = new AsciiFoldConfig(), StopwordPreset = "en", } ); Console.WriteLine($"indexed: [{string.Join(", ", result.Indexed)}]"); Console.WriteLine($"query: [{string.Join(", ", result.Query)}]"); ``` ::: ```json { "indexed": ["the", "organic", "cafe", "creme", "blend"], "query": ["organic", "cafe", "creme", "blend"] } ``` The response distinguishes **`indexed`** tokens (what is stored in the inverted index) from **`query`** tokens (what BM25 actually scores after stopword filtering). ### Property-specific tokenization `POST /v1/schema/{className}/properties/{propertyName}/tokenize` resolves the analyzer config from an existing property, so you can see what a specific property would do with a given input: :::code-group{sync="languages"} ```python title="Python" # Tokenize using an existing property's configuration result = client.tokenization.for_property( collection="TokenizeDemo", property_name="name_fr", text="La Tasse Bleue et le Bol", ) print(f"indexed: {result.indexed}") print(f"query: {result.query}") ``` ```typescript title="JavaScript/TypeScript" // Tokenize using an existing property's configuration const propResult = await client.tokenize.forProperty('TokenizeDemo', 'name_fr', 'La Tasse Bleue et le Bol'); console.log(`indexed: ${JSON.stringify(propResult.indexed)}`); console.log(`query: ${JSON.stringify(propResult.query)}`); ``` ```java title="Java v6" // Tokenize using an existing property's configuration TokenizeResponse result = client.tokenize.forProperty( "La Tasse Bleue et le Bol", COLLECTION, "name_fr"); System.out.println("indexed: " + result.indexed()); System.out.println("query: " + result.query()); ``` ```csharp title="C#" // Tokenize using an existing property's configuration var collection = client.Collections.Use(COLLECTION); var result = await collection.Tokenize.Property( propertyName: "name_fr", text: "La Tasse Bleue et le Bol" ); Console.WriteLine($"indexed: [{string.Join(", ", result.Indexed)}]"); Console.WriteLine($"query: [{string.Join(", ", result.Query)}]"); ``` ::: ```json { "indexed": ["la", "tasse", "bleue", "et", "le", "bol"], "query": ["tasse", "bleue", "bol"] } ``` **Notes:** - The endpoint resolves collection aliases to the underlying class - Class and property names are case-insensitive - All tokenizers are supported, including the optional APAC tokenizers (`gse`, `kagome_ja`, `kagome_kr`) when enabled ## Keyword searches vs filters Tokenization impacts keyword searches similarly to filters, but with important differences. ### Setup the search function ```python import weaviate from weaviate.classes.query import MetadataQuery from weaviate.collections import Collection from typing import List ``` ### Keyword search differences Keyword searches use the BM25f algorithm to rank results. Tokenization has two effects: 1. **Inclusion**: Determines whether a result appears at all 2. **Ranking**: Affects the score based on matching tokens Let's revisit the "Clark" example with keyword search: ```python search_demo(collection, property_names, ["clark", "Clark", "clark:", "Clark:", "lois clark", "clark lois"]) ``` | | `word` | `lowercase` | `whitespace` | `field` | | -------------- | ------ | ----------- | ------------ | ------- | | `"clark"` | 0.613 | ❌ | ❌ | ❌ | | `"Clark"` | 0.613 | ❌ | ❌ | ❌ | | `"clark:" ` | 0.613 | 0.48 | ❌ | ❌ | | `"Clark:" ` | 0.613 | 0.48 | 0.48 | ❌ | | `"lois clark"` | 1.226 | 0.48 | ❌ | ❌ | | `"clark lois"` | 1.226 | 0.48 | ❌ | ❌ | **Key observations:** - More matching tokens = higher scores (e.g., "lois clark" scores higher than "clark") - Keyword search returns objects matching ANY token (not just ALL tokens) - Scores vary based on token matching frequency ### Stop words in keyword search ```python search_demo(collection, property_names, ["computer mouse", "a computer mouse", "the computer mouse", "blue computer mouse"]) ``` **Matches for `"computer mouse"`** | | `word` | `lowercase` | `whitespace` | `field` | | ----------------------- | ------ | ----------- | ------------ | ------- | | `"computer mouse"` | 0.889 | 0.819 | 1.01 | 0.982 | | `"Computer Mouse"` | 0.889 | 0.819 | ❌ | ❌ | | `"a computer mouse"` | 0.764 | 0.764 | 0.849 | ❌ | | `"computer mouse pad" ` | 0.764 | 0.764 | 0.849 | ❌ | **Matches for `"a computer mouse"`** | | `word` | `lowercase` | `whitespace` | `field` | | ----------------------- | ------ | ----------- | ------------ | ------- | | `"computer mouse"` | 0.889 | 0.819 | 1.01 | ❌ | | `"Computer Mouse"` | 0.889 | 0.819 | ❌ | ❌ | | `"a computer mouse"` | 0.764 | 1.552 | 1.712 | 0.982 | | `"computer mouse pad" ` | 0.764 | 0.688 | 0.849 | ❌ | **Key observations:** - Stop words don't prevent matches, but affect ranking - Scores differ for objects with/without stop words - `lowercase` and `whitespace` don't remove stop words from queries, giving users more control ## Choosing your tokenization method Based on what we've learned, here's guidance for choosing a tokenization method: ### Use `word` (default) when: - Working with typical text data (articles, descriptions, names) - Users won't include exact punctuation in queries - Case-insensitivity is desired - You want forgiving search behavior ### Use `lowercase` when: - Symbols like `&`, `@`, `_`, `-` are meaningful - Working with code snippets, email addresses, or technical notation - You want case-insensitivity but need to preserve symbols ### Use `whitespace` when: - Case sensitivity is important (entity names, acronyms) - Symbols are meaningful - You can handle case-sensitivity in your query construction ### Use `field` when: - Exact matches are required - Working with unique identifiers (URLs, IDs, exact email addresses) - You'll use wildcard filters for partial matches - Note: Can be slow with wildcards; use judiciously ### Hybrid searches A hybrid search combines keyword search and vector search results. Tokenization only impacts the keyword search portion; the vector search part uses the model's built-in tokenization. ## Summary You've learned how to: - Configure different tokenization methods for collection properties - Test and compare tokenization behavior with filters and searches - Understand the trade-offs between different tokenization methods - Choose the appropriate tokenization method for your use case - Configure accent folding for multilingual text matching - Define custom stopword presets and assign them per property - Use the tokenize endpoint to preview tokenization behavior The key takeaway: **tokenization is a core part of your search strategy**. Start with `word` as a sensible default, but adjust based on your data characteristics and user expectations. ## Next steps - Read more about tokenization in the [Concepts page](../indexing/inverted-index.md#tokenization) - Configure tokenization in your schema: [Configuration reference](../reference-configuration/collections.md#tokenization) - Learn about stop words: [Stopwords configuration](../reference-configuration/indexing-inverted-index.md#stopwords) - Understand the inverted index: [Inverted index concepts](../indexing/inverted-index.md) ## Questions and feedback Have a question or feedback? Here's how to reach us. ::::card-grid :::card{title="Community Forum" href="https://forum.weaviate.io/c/support" icon="messages-square"} Ask questions and connect with other developers on our **Community forum**. ::: :::card{title="Support" href="/guides/support-overview" icon="life-buoy"} Weaviate Cloud user or customer? Find the right channel on the **Support page**. ::: :::: ## Related pages - [Agents](./agents-index.md) - [AI-assisted Weaviate code generation](./ai-assisted-vibe-coding-index.md) - [APIs](./apis-index.md) - [Authorization and authentication](./authorization-and-authentication-index.md) - [Benchmarks](./benchmarks-index.md) - [Best practices](./best-practices-index.md) - [Client libraries](./clients-index.md) - [Client Libraries / SDKs](./client-libraries-index.md) - [Cloud](./cloud-index.md) - [Cloud account management](./cloud-account-management-index.md) # Agent Instructions This portal answers questions programmatically. To receive a synthesized, source-cited answer instead of crawling page by page, append the `?ask=` query parameter to any page URL on this site: /guides/quickstart?ask=how+do+I+authenticate Optional parameters: - `&goal=` steers the answer toward your objective (e.g. `&goal=write+a+python+client`). - `&version=