Weaviate's integration with [Google Gemini API](https://ai.google.dev/?utm_source=weaviate\&utm_medium=referral\&utm_campaign=partnerships\&utm_content=) and [Google Vertex AI](https://cloud.google.com/vertex-ai) APIs allows you to access their models' capabilities directly from Weaviate. :::callout{intent="note" title="Gemini API multimodal support"} The `gemini-embedding-2` model supports multimodal embeddings (text, images, PDFs, and audio) and is available via both Vertex AI and Google AI Studio (Gemini API). Audio is only supported through the Gemini API. The `multimodalembedding@001` model remains available for Vertex AI users only. ::: [Configure a Weaviate vector index](#configure-the-vectorizer) to use a Google embedding model, and Weaviate will generate embeddings for various operations using the specified model and your Google API key. This feature is called the _vectorizer_. At [import time](#data-import), Weaviate generates multimodal object embeddings and saves them into the index. For [vector](#vector-near-text-search) and [hybrid](#hybrid-search) search operations, Weaviate converts queries of one or more modalities into embeddings. [Multimodal search operations](#vector-near-media-search) are also supported. ![Embedding integration illustration](/assets/docs/weaviate/model-providers/_includes/integration_google_embedding.png) ## Requirements ### Weaviate configuration Your Weaviate instance must be configured with the Google multimodal vectorizer integration (`multi2vec-google`) module. :::callout{intent="info" title="Module name change"} `multi2vec-google` was called `multi2vec-palm` in Weaviate versions prior to `v1.27`. ::: :::accordion{title="For Weaviate Cloud (WCD) users"} This integration is enabled by default on Weaviate Cloud (WCD) instances. ::: :::accordion{title="For self-hosted users"} - Check the [cluster metadata](../monitoring-and-logging/status.md#cluster-metadata) to verify if the module is enabled. - Follow the [how-to configure modules](../how-to-configure-weaviate/modules.md) guide to enable the module in Weaviate. ::: ### API credentials You must provide valid API credentials to Weaviate for the appropriate integration. #### Google AI Studio (Gemini API) 1. Go to [Google AI Studio](https://aistudio.google.com/app/apikey/?utm_source=weaviate\&utm_medium=referral\&utm_campaign=partnerships\&utm_content=) 2. In the "API Keys" section create a new API key 3. Use the `X-Goog-Studio-Api-Key` header to provide your API key to Weaviate #### Vertex AI This is called an `access token` in Google Cloud. ##### Automatic token generation :::callout{intent="info" title="Not available on Weaviate Cloud instances"} This feature is not available on Weaviate cloud instances. ::: You can save your Google Vertex AI credentials and have Weaviate generate the necessary tokens for you. This enables use of IAM service accounts in private deployments that can hold Google credentials. To do so: - Set `USE_GOOGLE_AUTH` [environment variable](../database-configuration/overview.md#module-specific) to `true`. - Have the credentials available in one of the following locations. Once appropriate credentials are found, Weaviate uses them to generate an access token and authenticates itself against Vertex AI. Upon token expiry, Weaviate generates a replacement access token. In a containerized environment, you can mount the credentials file to the container. For example, you can mount the credentials file to the `/etc/weaviate/` directory and set the `GOOGLE_APPLICATION_CREDENTIALS` environment variable to `/etc/weaviate/google_credentials.json`. :::accordion{title="Search locations for Google Vertex AI credentials"} Once `USE_GOOGLE_AUTH` is set to `true`, Weaviate will look for credentials in the following places, preferring the first location found: 1. A JSON file whose path is specified by the `GOOGLE_APPLICATION_CREDENTIALS` environment variable. For workload identity federation, refer to [this link](https://cloud.google.com/iam/docs/how-to#using-workload-identity-federation) on how to generate the JSON configuration file for on-prem/non-Google cloud platforms. 2. A JSON file in a location known to the `gcloud` command-line tool. On Windows, this is `%APPDATA%/gcloud/application_default_credentials.json`. On other systems, `$HOME/.config/gcloud/application_default_credentials.json`. 3. On Google App Engine standard first generation runtimes (<= Go 1.9) it uses the appengine.AccessToken function. 4. On Google Compute Engine, Google App Engine standard second generation runtimes (>= Go 1.11), and Google App Engine flexible environment, it fetches credentials from the metadata server. ::: ##### Manual token retrieval If you have the [Google Cloud CLI tool](https://cloud.google.com/cli) installed and set up, you can view your token by running the following command: ```shell gcloud auth print-access-token ``` #### Token expiry for Vertex AI users :::callout{intent="warning" title="Important"} ::: By default, Google Cloud's OAuth 2.0 access tokens have a lifetime of 1 hour. You can create tokens that last up to 12 hours. To create longer lasting tokens, follow the instructions in the [Google Cloud IAM Guide](https://cloud.google.com/iam/docs/create-short-lived-credentials-direct#rest_2). Since the OAuth token is only valid for a limited time, you **must** periodically replace the token with a new one. After you generate the new token, you have to re-instantiate your Weaviate client to use it. You can update the OAuth token manually, but manual updates may not be appropriate for your use case. You can also automate the OAth token update. Weaviate does not control the OAth token update procedure. However, here are some automation options: :::accordion{title="With Google Cloud CLI"} If you are using the Google Cloud CLI, write a script to periodically update the token and extract the results. Python code to extract the token looks like this: ```python client = re_instantiate_weaviate() ``` This is the `re_instantiate_weaviate` function: ```python import subprocess import weaviate def refresh_token() -> str: result = subprocess.run(["gcloud", "auth", "print-access-token"], capture_output=True, text=True) if result.returncode != 0: print(f"Error refreshing token: {result.stderr}") return None return result.stdout.strip() def re_instantiate_weaviate() -> weaviate.Client: token = refresh_token() client = weaviate.Client( url = "https://WEAVIATE_INSTANCE_URL", # Replace WEAVIATE_INSTANCE_URL with the URL additional_headers = { "X-Goog-Vertex-Api-Key": token, } ) return client # Run this every ~60 minutes client = re_instantiate_weaviate() ``` ::: :::accordion{title="With google-auth"} Another way is through Google's own authentication library `google-auth`. See the links to `google-auth` in [Python](https://google-auth.readthedocs.io/en/master/index.html) and [Node.js](https://cloud.google.com/nodejs/docs/reference/google-auth-library/latest) libraries. You can, then, periodically the `refresh` function ([see Python docs](https://google-auth.readthedocs.io/en/master/reference/google.oauth2.service_account.html#google.oauth2.service_account.Credentials.refresh)) to obtain a renewed token, and re-instantiate the Weaviate client. For example, you could periodically run: ```python client = re_instantiate_weaviate() ``` Where `re_instantiate_weaviate` is something like: ```python from google.auth.transport.requests import Request from google.oauth2.service_account import Credentials import weaviate import os def get_credentials() -> Credentials: credentials = Credentials.from_service_account_file( "path/to/your/service-account.json", scopes=[ "https://www.googleapis.com/auth/generative-language", "https://www.googleapis.com/auth/cloud-platform", ], ) request = Request() credentials.refresh(request) return credentials def re_instantiate_weaviate() -> weaviate.Client: from weaviate.classes.init import Auth weaviate_api_key = os.environ["WEAVIATE_API_KEY"] credentials = get_credentials() token = credentials.token client = weaviate.connect_to_weaviate_cloud( # e.g. if you use the Weaviate Cloud Service cluster_url="https://WEAVIATE_INSTANCE_URL", # Replace WEAVIATE_INSTANCE_URL with the URL auth_credentials=Auth.api_key(weaviate_api_key), # Replace with your Weaviate Cloud key headers={ "X-Goog-Vertex-Api-Key": token, }, ) return client # Run this every ~60 minutes client = re_instantiate_weaviate() ``` The service account key shown above can be generated by following [this guide](https://cloud.google.com/iam/docs/keys-create-delete). ::: #### Provide the API key Provide the API key to Weaviate at runtime, as shown in the examples below. :::accordion{title="API key headers"} From `v1.27.7`, `v1.26.12` and `v1.25.27`, `X-Goog-Vertex-Api-Key` and `X-Goog-Studio-Api-Key` headers are supported for Vertex AI users and Gemini API respectively. We recommend these headers for highest compatibility. Consider `X-Google-Vertex-Api-Key`, `X-Google-Studio-Api-Key`, `X-Google-Api-Key` and `X-PaLM-Api-Key` deprecated. ::: :::code-group{sync="languages"} ```python title="Python" # Recommended: save sensitive data as environment variables vertex_key = os.getenv("VERTEX_API_KEY") ``` ```typescript title="JavaScript/TypeScript" const vertexApiKey = process.env.VERTEX_API_KEY || ''; // Replace with your inference API key ``` ::: ## Configure the vectorizer [Configure a Weaviate index](../how-to-manage-collections/vector-config.md#specify-a-vectorizer) as follows to use a Google embedding model: :::code-group{sync="languages"} ```python title="Python" {5-21} from weaviate.classes.config import Property, DataType, Configure, Multi2VecField client.collections.create( "DemoCollection", properties=[ Property(name="title", data_type=DataType.TEXT), Property(name="poster", data_type=DataType.BLOB), ], vector_config=Configure.Vectors.multi2vec_google( name="title_vector", # Define the fields to be used for the vectorization - using image_fields, text_fields, video_fields image_fields=[ Multi2VecField(name="poster", weight=0.9) ], text_fields=[ Multi2VecField(name="title", weight=0.1) ], # video_fields=[], project_id="", # Required for Vertex AI location="", # Required for Vertex AI ), # Additional parameters not shown ) ``` ```typescript title="JavaScript/TypeScript" {13-31} await client.collections.create({ name: 'DemoCollection', properties: [ { name: 'title', dataType: 'text' as const, }, { name: 'poster', dataType: 'blob' as const, }, ], vectorizers: [ weaviate.configure.vectors.multi2VecGoogle({ name: 'title_vector', location: '', projectId: '', imageFields: [ { name: 'poster', weight: 0.9, }, ], textFields: [ { name: 'title', weight: 0.1, }, ], }), ], // Additional parameters not shown }); ``` ::: You can [specify](#vectorizer-parameters) one of the [available models](#available-models) for the vectorizer to use. The [default model](#available-models) is used if no model is specified. :::accordion{title="Vectorization behavior"} Weaviate follows the collection configuration and a set of predetermined rules to vectorize objects. Unless specified otherwise in the collection definition, the default behavior is to: - Only vectorize properties that use the `text` or `text[]` data type (unless [skipped](../how-to-manage-collections/vector-config.md#property-level-settings)) - Sort properties in alphabetical (a-z) order before concatenating values - If `vectorizePropertyName` is `true` (`false` by default) prepend the property name to each property value - Join the (prepended) property values with spaces - Prepend the class name (unless `vectorizeClassName` is `false`) - Convert the produced string to lowercase ::: ### Vectorizer parameters The following examples show how to configure Google-specific options. - `location` (Required): e.g. `"us-central1"` - `projectId` (Only required if using Vertex AI): e.g. `cloud-large-language-models` - `apiEndpoint` (Optional): e.g. `us-central1-aiplatform.googleapis.com` - `modelId` (Optional): e.g. `gemini-embedding-2`, `multimodalembedding@001` - `dimensions` (Optional): For `multimodalembedding@001`: `128`, `256`, `512`, or `1408` (default `1408`). For `gemini-embedding-2`: `3072` (default). :::callout{intent="info" title="Audio support (added in `v1.37`)"} The `multi2vec-google` module supports audio as a fourth modality (alongside text, images, and videos) via the `audioFields` property. Configure it the same way as `imageFields` or `videoFields`. Audio is only supported through the **Gemini API** (Google AI Studio). Vertex AI does not support audio embeddings. ::: :::code-group{sync="languages"} ```python title="Python" {5-23} from weaviate.classes.config import Configure, DataType, Multi2VecField, Property client.collections.create( "DemoCollection", properties=[ Property(name="title", data_type=DataType.TEXT), Property(name="description", data_type=DataType.TEXT), Property(name="poster", data_type=DataType.BLOB), ], vector_config=Configure.Vectors.multi2vec_google( project_id="", # Required for Vertex AI location="us-central1", # model_id="", # dimensions=512, image_fields=[ Multi2VecField(name="poster", weight=0.9) ], text_fields=[ Multi2VecField(name="title", weight=0.1) ], # video_fields=[] # video_interval_seconds=20 ), # Additional parameters not shown ) ``` ```typescript title="JavaScript/TypeScript" {17-39} await client.collections.create({ name: 'DemoCollection', properties: [ { name: 'title', dataType: 'text' as const, }, { name: 'description', dataType: 'text' as const, }, { name: 'poster', dataType: 'blob' as const, }, ], vectorizers: [ weaviate.configure.vectors.multi2VecGoogle({ name: 'title_vector', projectId: '', model: '', location: '', dimensions: 512, imageFields: [ { name: 'poster', weight: 0.9, }, ], textFields: [ { name: 'title', weight: 0.1, }, ], // videoFields: [] // video_interval_seconds: 20 }), ], // Additional parameters not shown }); ``` ::: ## Data import After configuring the vectorizer, [import data](../how-to-manage-objects/import.md) into Weaviate. Weaviate generates embeddings for the objects using the specified model. :::code-group{sync="languages"} ```python title="Python" {11-15} collection = client.collections.use("DemoCollection") with collection.batch.fixed_size(batch_size=200) as batch: for src_obj in source_objects: poster_b64 = url_to_base64(src_obj["poster_path"]) weaviate_obj = { "title": src_obj["title"], "poster": poster_b64 # Add the image in base64 encoding } # The model provider integration will automatically vectorize the object batch.add_object( properties=weaviate_obj, # vector=vector # Optionally provide a pre-obtained vector ) ``` ```typescript title="JavaScript/TypeScript" const collectionName = 'DemoCollection' const myCollection = client.collections.use(collectionName) ``` ::: :::callout{intent="tip" title="Re-use existing vectors"} If you already have a compatible model vector available, you can provide it directly to Weaviate. This can be useful if you have already generated embeddings using the same model and want to use them in Weaviate, such as when migrating data from another system. ::: ## Searches Once the vectorizer is configured, Weaviate will perform vector and hybrid search operations using the specified Google model. ![Embedding integration at search illustration](/assets/docs/weaviate/model-providers/_includes/integration_google_embedding_search.png) ### Vector (near text) search When you perform a [vector search](../how-to-query-search/similarity.md#search-with-text), Weaviate converts the text query into an embedding using the specified model and returns the most similar objects from the database. The query below returns the `n` most similar objects from the database, set by `limit`. :::code-group{sync="languages"} ```python title="Python" {3-6} collection = client.collections.use("DemoCollection") response = collection.query.near_text( query="A holiday film", # The model provider integration will automatically vectorize the query limit=2 ) for obj in response.objects: print(obj.properties["title"]) ``` ```typescript title="JavaScript/TypeScript" const collectionName = 'DemoCollection' const myCollection = client.collections.use(collectionName) ``` ::: ### Hybrid search :::callout{intent="info" title="What is a hybrid search?"} A hybrid search performs a vector search and a keyword (BM25) search, before [combining the results](../how-to-query-search/hybrid.md) to return the best matching objects from the database. ::: When you perform a [hybrid search](../how-to-query-search/hybrid.md), Weaviate converts the text query into an embedding using the specified model and returns the best scoring objects from the database. The query below returns the `n` best scoring objects from the database, set by `limit`. :::code-group{sync="languages"} ```python title="Python" {3-6} collection = client.collections.use("DemoCollection") response = collection.query.hybrid( query="A holiday film", # The model provider integration will automatically vectorize the query limit=2 ) for obj in response.objects: print(obj.properties["title"]) ``` ```typescript title="JavaScript/TypeScript" const collectionName = 'DemoCollection' const myCollection = client.collections.use(collectionName) ``` ::: ### Vector (near media) search When you perform a media search such as a [near image search](../how-to-query-search/similarity.md#search-with-image), Weaviate converts the query into an embedding using the specified model and returns the most similar objects from the database. To perform a near media search such as near image search, convert the media query into a base64 string and pass it to the search query. The query below returns the `n` most similar objects to the input image from the database, set by `limit`. :::code-group{sync="languages"} ```python title="Python" def url_to_base64(url): import requests import base64 image_response = requests.get(url) content = image_response.content return base64.b64encode(content).decode("utf-8") ``` ```typescript title="JavaScript/TypeScript" const base64String = 'SOME_BASE_64_REPRESENTATION'; result = await myCollection.query.nearImage( base64String, // The model provider integration will automatically vectorize the query { limit: 2, } ) console.log(JSON.stringify(result.objects, null, 2)); ``` ::: ## References ### Available models - `gemini-embedding-2` (Vertex AI and Gemini API, added in 1.36.13). Supports text, images, PDFs, and audio (Gemini API only, up to 180 seconds); `3072` dimensions - `multimodalembedding@001` (default, Vertex AI only). Supports text, images, and video; dimensions: `128`, `256`, `512`, `1408` ## Further resources ### Other integrations - [Google text embedding models + Weaviate](google-embeddings.md) - [Google AI generative models + Weaviate](google-generative.md) ### Code examples Once the integrations are configured at the collection, the data management and search operations in Weaviate work identically to any other collection. See the following model-agnostic examples: - The [How-to: Manage collections](../how-to-manage-collections/index.md) and [How-to: Manage objects](../how-to-manage-objects/index.md) guides show how to perform data operations (i.e. create, read, update, delete collections and objects within them). - The [How-to: Query & Search](../how-to-query-search/index.md) guides show how to perform search operations (i.e. vector, keyword, hybrid) as well as retrieval augmented generation. ### External resources - [Google Vertex AI](https://cloud.google.com/vertex-ai) - [Google Gemini API](https://ai.google.dev/?utm_source=weaviate\&utm_medium=referral\&utm_campaign=partnerships\&utm_content=) ## Questions and feedback Have a question or feedback? Here's how to reach us. ::::card-grid :::card{title="Community Forum" href="https://forum.weaviate.io/c/support" icon="messages-square"} Ask questions and connect with other developers on our **Community forum**. ::: :::card{title="Support" href="/guides/support-overview" icon="life-buoy"} Weaviate Cloud user or customer? Find the right channel on the **Support page**. ::: :::: ## Related pages - [Agents](./agents-index.md) - [AI-assisted Weaviate code generation](./ai-assisted-vibe-coding-index.md) - [APIs](./apis-index.md) - [Authorization and authentication](./authorization-and-authentication-index.md) - [Benchmarks](./benchmarks-index.md) - [Best practices](./best-practices-index.md) - [Client libraries](./clients-index.md) - [Client Libraries / SDKs](./client-libraries-index.md) - [Cloud](./cloud-index.md) - [Cloud account management](./cloud-account-management-index.md) # Agent Instructions This portal answers questions programmatically. To receive a synthesized, source-cited answer instead of crawling page by page, append the `?ask=` query parameter to any page URL on this site: /guides/quickstart?ask=how+do+I+authenticate Optional parameters: - `&goal=` steers the answer toward your objective (e.g. `&goal=write+a+python+client`). - `&version=