Weaviate's integration with TwelveLabs' APIs allows you to access their models' capabilities directly from Weaviate. [Configure a Weaviate vector index](#configure-the-vectorizer) to use a TwelveLabs embedding model, and Weaviate will generate embeddings for various operations using the specified model and your TwelveLabs API key. This feature is called the _vectorizer_. At [import time](#data-import), Weaviate generates multimodal object embeddings and saves them into the index. For [vector](#vector-near-text-search) and [hybrid](#hybrid-search) search operations, Weaviate converts text queries into embeddings. [Multimodal search operations](#vector-near-image-search) are also supported. :::callout{intent="warning" title="Text and images only"} TwelveLabs is best known for video understanding, but this integration vectorizes **text and images only**. Video and audio are not supported. The vectorizer reads only the `textFields` and `imageFields` settings, and only `nearText` and `nearImage` search operations are available. Weaviate stores a `videoFields` entry if you add one to a collection definition, but nothing reads it, so it has no effect. ::: ![Embedding integration illustration](/assets/docs/weaviate/model-providers/_includes/integration_twelvelabs_embedding.png) ## Requirements ### Weaviate configuration Your Weaviate instance must be configured with the TwelveLabs vectorizer integration (`multi2vec-twelvelabs`) module. :::callout{intent="info" title="Added in `v1.38.9`"} This integration is available in Weaviate `v1.38.9`, `v1.39.0` and later. ::: :::accordion{title="For Weaviate Cloud (WCD) users"} This integration is enabled by default on Weaviate Cloud (WCD) instances. ::: :::accordion{title="For self-hosted users"} - Check the [cluster metadata](../monitoring-and-logging/status.md#cluster-metadata) to verify if the module is enabled. - Follow the [how-to configure modules](../how-to-configure-weaviate/modules.md) guide to enable the module in Weaviate. ::: ### API credentials You must provide a valid TwelveLabs API key to Weaviate for this integration. Go to [TwelveLabs](https://www.twelvelabs.io/) to sign up and obtain an API key. Provide the API key to Weaviate using one of the following methods: - Set the `TWELVELABS_APIKEY` environment variable that is available to Weaviate. - Provide the API key at runtime, as shown in the examples below. :::code-group{sync="languages"} ```python title="Python" # Recommended: save sensitive data as environment variables twelvelabs_key = os.getenv("TWELVELABS_APIKEY") ``` ```bash title="cURL" curl http://localhost:8080/v1/graphql \ -H "Content-Type: application/json" \ -H "X-Twelvelabs-Api-Key: $TWELVELABS_APIKEY" \ -H "X-Twelvelabs-Baseurl: https://api.twelvelabs.io/v1.3" \ -d '{"query": "{ Get { DemoCollection(nearText: {concepts: [\"A holiday film\"]}, limit: 2) { title } } }"}' ``` ::: The `X-Twelvelabs-Baseurl` header is optional. It overrides the base URL that is set in the collection definition for the duration of the request. ## Configure the vectorizer [Configure a Weaviate index](../how-to-manage-collections/vector-config.md#specify-a-vectorizer) as follows to use a TwelveLabs embedding model. Name the properties that hold your text in `textFields`, and the properties that hold your base64 encoded images in `imageFields`. Set at least one of the two. A collection that names no fields cannot produce a vector, and inserts into it fail with a `more than one embedding found for object` error. :::code-group{sync="languages"} ```python title="Python" {5-20} from weaviate.classes.config import Configure, DataType, Multi2VecField, Property client.collections.create( "DemoCollection", properties=[ Property(name="title", data_type=DataType.TEXT), Property(name="poster", data_type=DataType.BLOB), ], vector_config=[ Configure.Vectors.multi2vec_twelvelabs( name="title_vector", # Define the fields to be used for the vectorization - using image_fields, text_fields image_fields=[ Multi2VecField(name="poster", weight=0.9) ], text_fields=[ Multi2VecField(name="title", weight=0.1) ], ) ], # Additional parameters not shown ) ``` ```bash title="cURL" curl -X POST http://localhost:8080/v1/schema \ -H "Content-Type: application/json" \ -d '{ "class": "DemoCollection", "properties": [ {"name": "title", "dataType": ["text"]}, {"name": "poster", "dataType": ["blob"]} ], "vectorConfig": { "title_vector": { "vectorizer": { "multi2vec-twelvelabs": { "textFields": ["title"], "imageFields": ["poster"], "weights": { "textFields": [0.1], "imageFields": [0.9] } } }, "vectorIndexType": "hnsw" } } }' ``` ::: :::callout{intent="info" title="Client availability"} A typed configuration API for this integration is currently available in the Python client only. If `Configure.Vectors.multi2vec_twelvelabs` is missing from your installation, upgrade to the latest Python client version. With any other client library, configure the vectorizer by sending the collection definition as shown in the cURL example above, or by passing the equivalent module configuration map that your client accepts. Data import and search operations are not specific to this integration and work with every client library. ::: ### Select a model You can specify one of the [available models](#available-models) for the vectorizer to use, as shown in the following configuration example. :::code-group{sync="languages"} ```python title="Python" {5-21} from weaviate.classes.config import Configure, DataType, Multi2VecField, Property client.collections.create( "DemoCollection", properties=[ Property(name="title", data_type=DataType.TEXT), Property(name="poster", data_type=DataType.BLOB), ], vector_config=[ Configure.Vectors.multi2vec_twelvelabs( name="title_vector", model="marengo3.0", # Define the fields to be used for the vectorization - using image_fields, text_fields image_fields=[ Multi2VecField(name="poster", weight=0.9) ], text_fields=[ Multi2VecField(name="title", weight=0.1) ], ) ], # Additional parameters not shown ) ``` ```bash title="cURL" curl -X POST http://localhost:8080/v1/schema \ -H "Content-Type: application/json" \ -d '{ "class": "DemoCollection", "properties": [ {"name": "title", "dataType": ["text"]}, {"name": "poster", "dataType": ["blob"]} ], "vectorConfig": { "title_vector": { "vectorizer": { "multi2vec-twelvelabs": { "textFields": ["title"], "imageFields": ["poster"], "model": "marengo3.0" } }, "vectorIndexType": "hnsw" } } }' ``` ::: You can [specify](#vectorizer-parameters) one of the [available models](#available-models) for Weaviate to use. The [default model](#available-models) is used if no model is specified. :::accordion{title="Vectorization behavior"} Weaviate follows the collection configuration and a set of predetermined rules to vectorize objects. Unless specified otherwise in the collection definition, the default behavior is to: - Only vectorize properties that use the `text` or `text[]` data type (unless [skipped](../how-to-manage-collections/vector-config.md#property-level-settings)) - Sort properties in alphabetical (a-z) order before concatenating values - If `vectorizePropertyName` is `true` (`false` by default) prepend the property name to each property value - Join the (prepended) property values with spaces - Prepend the class name (unless `vectorizeClassName` is `false`) - Convert the produced string to lowercase ::: ### Vectorizer parameters The following examples show how to configure TwelveLabs-specific options. :::code-group{sync="languages"} ```python title="Python" {5-23} from weaviate.classes.config import Configure, DataType, Multi2VecField, Property client.collections.create( "DemoCollection", properties=[ Property(name="title", data_type=DataType.TEXT), Property(name="poster", data_type=DataType.BLOB), ], vector_config=[ Configure.Vectors.multi2vec_twelvelabs( name="title_vector", # Define the fields to be used for the vectorization - using image_fields, text_fields image_fields=[ Multi2VecField(name="poster", weight=0.9) ], text_fields=[ Multi2VecField(name="title", weight=0.1) ], # Further options # model="marengo3.0", # base_url="https://api.twelvelabs.io/v1.3", ) ], # Additional parameters not shown ) ``` ```bash title="cURL" curl -X POST http://localhost:8080/v1/schema \ -H "Content-Type: application/json" \ -d '{ "class": "DemoCollection", "properties": [ {"name": "title", "dataType": ["text"]}, {"name": "poster", "dataType": ["blob"]} ], "vectorConfig": { "title_vector": { "vectorizer": { "multi2vec-twelvelabs": { "textFields": ["title"], "imageFields": ["poster"], "weights": { "textFields": [0.1], "imageFields": [0.9] }, "model": "marengo3.0", "baseURL": "https://api.twelvelabs.io/v1.3" } }, "vectorIndexType": "hnsw" } } }' ``` ::: The collection definition accepts the following settings: | Setting | Description | | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `textFields` | Names of the `text` and `text[]` properties to vectorize. Each element of a `text[]` property is vectorized separately. | | `imageFields` | Names of the properties that hold base64 encoded images, typically `blob` properties. A `text[]` property listed here is ignored. | | `weights` | Relative weights for combining the field vectors, given as `textFields` and `imageFields` arrays. Each array must have the same number of entries as the field list it weights. The weights are normalized so that they sum to 1. If no weights are set, all fields are weighted equally. | | `model` | The model to use. The default is `marengo3.0`. | | `baseURL` | The base URL of the TwelveLabs API. The default is `https://api.twelvelabs.io/v1.3`. | In the Python client, these settings are named `text_fields`, `image_fields`, `model` and `base_url`. Weights are set per field with `Multi2VecField(name=..., weight=...)`. :::callout{intent="note" title="Settings that have no effect"} Weaviate writes a `vectorizeClassName` setting into every collection that uses this integration, but this integration does not read it. Its value does not change the vectors that are produced, and the collection name is never included in the vectorized text. The per-property `skip` and `vectorizePropertyName` settings also have no effect here. Property selection is determined only by `textFields` and `imageFields` membership, and property names are never vectorized. ::: For further details on model parameters, see the [TwelveLabs documentation](https://docs.twelvelabs.io/). ## Data import After configuring the vectorizer, [import data](../how-to-manage-objects/import.md) into Weaviate. Weaviate generates embeddings for text and image objects using the specified model. Provide image data as a base64 encoded string. A `data:;base64,` prefix is accepted and stripped before decoding. A property that is listed in `imageFields` but does not hold valid base64 data fails the import with a `decode base64 image` error. :::code-group{sync="languages"} ```python title="Python" {11-15} collection = client.collections.use("DemoCollection") with collection.batch.fixed_size(batch_size=200) as batch: for src_obj in source_objects: poster_b64 = url_to_base64(src_obj["poster_path"]) weaviate_obj = { "title": src_obj["title"], "poster": poster_b64 # Add the image in base64 encoding } # The model provider integration will automatically vectorize the object batch.add_object( properties=weaviate_obj, # vector=vector # Optionally provide a pre-obtained vector ) ``` ::: :::callout{intent="warning" title="This integration is not rate limited"} Weaviate does not throttle its requests to TwelveLabs for this integration, and it does not read rate limit response headers. It sends one request per text value and one request per image, and it processes batches of ten objects in parallel, so a large import can produce a high request rate. If TwelveLabs rejects a request, the error surfaces as a failed import for that object and is not retried. Pace large imports from the client side, for example by importing in smaller batches. ::: :::callout{intent="tip" title="Re-use existing vectors"} If you already have a compatible model vector available, you can provide it directly to Weaviate. This can be useful if you have already generated embeddings using the same model and want to use them in Weaviate, such as when migrating data from another system. ::: ## Searches Once the vectorizer is configured, Weaviate will perform vector and hybrid search operations using the specified TwelveLabs model. ![Embedding integration at search illustration](/assets/docs/weaviate/model-providers/_includes/integration_twelvelabs_embedding_search.png) The examples below use the Python client. Search operations are not specific to this integration, so see the [How-to: Query & Search](../how-to-query-search/index.md) guides for the equivalent examples in the other client libraries. ### Vector (near text) search When you perform a [vector search](../how-to-query-search/similarity.md#search-with-text), Weaviate converts the text query into an embedding using the specified model and returns the most similar objects from the database. The query below returns the `n` most similar objects from the database, set by `limit`. :::code-group{sync="languages"} ```python title="Python" {3-6} collection = client.collections.use("DemoCollection") response = collection.query.near_text( query="A holiday film", # The model provider integration will automatically vectorize the query limit=2 ) for obj in response.objects: print(obj.properties["title"]) ``` ::: ### Hybrid search :::callout{intent="info" title="What is a hybrid search?"} A hybrid search performs a vector search and a keyword (BM25) search, before [combining the results](../how-to-query-search/hybrid.md#change-the-fusion-method) to return the best matching objects from the database. ::: When you perform a [hybrid search](../how-to-query-search/hybrid.md), Weaviate converts the text query into an embedding using the specified model and returns the best scoring objects from the database. The query below returns the `n` best scoring objects from the database, set by `limit`. :::code-group{sync="languages"} ```python title="Python" {3-6} collection = client.collections.use("DemoCollection") response = collection.query.hybrid( query="A holiday film", # The model provider integration will automatically vectorize the query limit=2 ) for obj in response.objects: print(obj.properties["title"]) ``` ::: ### Vector (near image) search When you perform a [near image search](../how-to-query-search/similarity.md#search-with-image), Weaviate converts the query into an embedding using the specified model and returns the most similar objects from the database. To perform a near image search, convert the image query into a base64 string and pass it to the search query. The query below returns the `n` most similar objects to the input image from the database, set by `limit`. :::code-group{sync="languages"} ```python title="Python" def url_to_base64(url): import requests import base64 image_response = requests.get(url) content = image_response.content return base64.b64encode(content).decode("utf-8") ``` ::: ## References ### Available models The default model is `marengo3.0`, which produces 512-dimensional vectors. Weaviate does not validate the model name, so you can set any model that the TwelveLabs embedding endpoint accepts for your account. Weaviate does not publish the list of accepted names; see the [TwelveLabs documentation on creating embeddings](https://docs.twelvelabs.io/v1.3/docs/guides/create-embeddings) for the models that are currently available. ## Further resources ### Code examples Once the integrations are configured at the collection, the data management and search operations in Weaviate work identically to any other collection. See the following model-agnostic examples: - The [How-to: Manage collections](../how-to-manage-collections/index.md) and [How-to: Manage objects](../how-to-manage-objects/index.md) guides show how to perform data operations (i.e. create, read, update, delete collections and objects within them). - The [How-to: Query & Search](../how-to-query-search/index.md) guides show how to perform search operations (i.e. vector, keyword, hybrid) as well as retrieval augmented generation. ### External resources - [TwelveLabs documentation](https://docs.twelvelabs.io/) ## Questions and feedback Have a question or feedback? Here's how to reach us. ::::card-grid :::card{title="Community Forum" href="https://forum.weaviate.io/c/support" icon="messages-square"} Ask questions and connect with other developers on our **Community forum**. ::: :::card{title="Support" href="/guides/support-overview" icon="life-buoy"} Weaviate Cloud user or customer? Find the right channel on the **Support page**. ::: :::: ## Related pages - [Agents](./agents-index.md) - [AI-assisted Weaviate code generation](./ai-assisted-vibe-coding-index.md) - [APIs](./apis-index.md) - [Authorization and authentication](./authorization-and-authentication-index.md) - [Benchmarks](./benchmarks-index.md) - [Best practices](./best-practices-index.md) - [Client libraries](./clients-index.md) - [Client Libraries / SDKs](./client-libraries-index.md) - [Cloud](./cloud-index.md) - [Cloud account management](./cloud-account-management-index.md) # Agent Instructions This portal answers questions programmatically. To receive a synthesized, source-cited answer instead of crawling page by page, append the `?ask=` query parameter to any page URL on this site: /guides/quickstart?ask=how+do+I+authenticate Optional parameters: - `&goal=` steers the answer toward your objective (e.g. `&goal=write+a+python+client`). - `&version=