Multimodal Embeddings

Weaviate's integration with Google Gemini API and Google Vertex AI APIs allows you to access their models' capabilities directly from Weaviate.

Configure a Weaviate vector index to use a Google embedding model, and Weaviate will generate embeddings for various operations using the specified model and your Google API key. This feature is called the vectorizer.

At import time, Weaviate generates multimodal object embeddings and saves them into the index. For vector and hybrid search operations, Weaviate converts queries of one or more modalities into embeddings. Multimodal search operations are also supported.

Your Weaviate instance must be configured with the Google multimodal vectorizer integration (multi2vec-google) module.

You must provide valid API credentials to Weaviate for the appropriate integration.

1. Go to Google AI Studio
2. In the "API Keys" section create a new API key
3. Use the X-Goog-Studio-Api-Key header to provide your API key to Weaviate

This is called an access token in Google Cloud.

You can save your Google Vertex AI credentials and have Weaviate generate the necessary tokens for you. This enables use of IAM service accounts in private deployments that can hold Google credentials.

To do so:

• Set USE_GOOGLE_AUTH environment variable to true.
• Have the credentials available in one of the following locations.

Once appropriate credentials are found, Weaviate uses them to generate an access token and authenticates itself against Vertex AI. Upon token expiry, Weaviate generates a replacement access token.

In a containerized environment, you can mount the credentials file to the container. For example, you can mount the credentials file to the /etc/weaviate/ directory and set the GOOGLE_APPLICATION_CREDENTIALS environment variable to /etc/weaviate/google_credentials.json.

If you have the Google Cloud CLI tool installed and set up, you can view your token by running the following command:

gcloud auth print-access-token

By default, Google Cloud's OAuth 2.0 access tokens have a lifetime of 1 hour. You can create tokens that last up to 12 hours. To create longer lasting tokens, follow the instructions in the Google Cloud IAM Guide.

Since the OAuth token is only valid for a limited time, you must periodically replace the token with a new one. After you generate the new token, you have to re-instantiate your Weaviate client to use it.

You can update the OAuth token manually, but manual updates may not be appropriate for your use case.

You can also automate the OAth token update. Weaviate does not control the OAth token update procedure. However, here are some automation options:

client = re_instantiate_weaviate()

import subprocess
import weaviate

def refresh_token() -> str:
    result = subprocess.run(["gcloud", "auth", "print-access-token"], capture_output=True, text=True)
    if result.returncode != 0:
        print(f"Error refreshing token: {result.stderr}")
        return None
    return result.stdout.strip()

def re_instantiate_weaviate() -> weaviate.Client:
    token = refresh_token()

    client = weaviate.Client(
      url = "https://WEAVIATE_INSTANCE_URL",  # Replace WEAVIATE_INSTANCE_URL with the URL
      additional_headers = {
        "X-Goog-Vertex-Api-Key": token,
      }
    )
    return client

# Run this every ~60 minutes
client = re_instantiate_weaviate()

client = re_instantiate_weaviate()

from google.auth.transport.requests import Request
from google.oauth2.service_account import Credentials
import weaviate
import os


def get_credentials() -> Credentials:
    credentials = Credentials.from_service_account_file(
        "path/to/your/service-account.json",
        scopes=[
            "https://www.googleapis.com/auth/generative-language",
            "https://www.googleapis.com/auth/cloud-platform",
        ],
    )
    request = Request()
    credentials.refresh(request)
    return credentials


def re_instantiate_weaviate() -> weaviate.Client:
    from weaviate.classes.init import Auth

    weaviate_api_key = os.environ["WEAVIATE_API_KEY"]
    credentials = get_credentials()
    token = credentials.token

    client = weaviate.connect_to_weaviate_cloud(  # e.g. if you use the Weaviate Cloud Service
        cluster_url="https://WEAVIATE_INSTANCE_URL",  # Replace WEAVIATE_INSTANCE_URL with the URL
        auth_credentials=Auth.api_key(weaviate_api_key),  # Replace with your Weaviate Cloud key
        headers={
            "X-Goog-Vertex-Api-Key": token,
        },
    )
    return client


# Run this every ~60 minutes
client = re_instantiate_weaviate()

Provide the API key to Weaviate at runtime, as shown in the examples below.

# Recommended: save sensitive data as environment variables
vertex_key = os.getenv("VERTEX_API_KEY")

const vertexApiKey = process.env.VERTEX_API_KEY || '';  // Replace with your inference API key

Configure a Weaviate index as follows to use a Google embedding model:

from weaviate.classes.config import Property, DataType, Configure, Multi2VecFieldclient.collections.create(    "DemoCollection",    properties=[        Property(name="title", data_type=DataType.TEXT),        Property(name="poster", data_type=DataType.BLOB),    ],    vector_config=Configure.Vectors.multi2vec_google(        name="title_vector",        # Define the fields to be used for the vectorization - using image_fields, text_fields, video_fields        image_fields=[            Multi2VecField(name="poster", weight=0.9)        ],        text_fields=[            Multi2VecField(name="title", weight=0.1)        ],        # video_fields=[],        project_id="<google-cloud-project-id>",  # Required for Vertex AI        location="<google-cloud-location>",  # Required for Vertex AI    ),    # Additional parameters not shown)

await client.collections.create({  name: 'DemoCollection',  properties: [    {      name: 'title',      dataType: 'text' as const,    },    {      name: 'poster',      dataType: 'blob' as const,    },  ],  vectorizers: [    weaviate.configure.vectors.multi2VecGoogle({      name: 'title_vector',      location: '<google-cloud-location>',      projectId: '<google-cloud-project-id>',      imageFields: [        {          name: 'poster',          weight: 0.9,        },      ],      textFields: [        {          name: 'title',          weight: 0.1,        },      ],    }),  ],  // Additional parameters not shown});

You can specify one of the available models for the vectorizer to use. The default model is used if no model is specified.

The following examples show how to configure Google-specific options.

• location (Required): e.g. "us-central1"
• projectId (Only required if using Vertex AI): e.g. cloud-large-language-models
• apiEndpoint (Optional): e.g. us-central1-aiplatform.googleapis.com
• modelId (Optional): e.g. gemini-embedding-2, multimodalembedding@001
• dimensions (Optional): For multimodalembedding@001: 128, 256, 512, or 1408 (default 1408). For gemini-embedding-2: 3072 (default).

from weaviate.classes.config import Configure, DataType, Multi2VecField, Propertyclient.collections.create(    "DemoCollection",    properties=[        Property(name="title", data_type=DataType.TEXT),        Property(name="description", data_type=DataType.TEXT),        Property(name="poster", data_type=DataType.BLOB),    ],    vector_config=Configure.Vectors.multi2vec_google(        project_id="<google-cloud-project-id>",  # Required for Vertex AI        location="us-central1",        # model_id="<google-model-id>",        # dimensions=512,        image_fields=[            Multi2VecField(name="poster", weight=0.9)        ],        text_fields=[            Multi2VecField(name="title", weight=0.1)        ],        # video_fields=[]        # video_interval_seconds=20    ),    # Additional parameters not shown)

await client.collections.create({  name: 'DemoCollection',  properties: [    {      name: 'title',      dataType: 'text' as const,    },    {      name: 'description',      dataType: 'text' as const,    },    {      name: 'poster',      dataType: 'blob' as const,    },  ],  vectorizers: [    weaviate.configure.vectors.multi2VecGoogle({      name: 'title_vector',      projectId: '<google-cloud-project-id>',      model: '<google-model-id>',      location: '<google-cloud-location>',      dimensions: 512,      imageFields: [        {          name: 'poster',          weight: 0.9,        },      ],      textFields: [        {          name: 'title',          weight: 0.1,        },      ],      // videoFields: []      // video_interval_seconds: 20    }),  ],  // Additional parameters not shown});

After configuring the vectorizer, import data into Weaviate. Weaviate generates embeddings for the objects using the specified model.

collection = client.collections.use("DemoCollection")with collection.batch.fixed_size(batch_size=200) as batch:    for src_obj in source_objects:        poster_b64 = url_to_base64(src_obj["poster_path"])        weaviate_obj = {            "title": src_obj["title"],            "poster": poster_b64  # Add the image in base64 encoding        }        # The model provider integration will automatically vectorize the object        batch.add_object(            properties=weaviate_obj,            # vector=vector  # Optionally provide a pre-obtained vector        )

const collectionName = 'DemoCollection'
const myCollection = client.collections.use(collectionName)

Once the vectorizer is configured, Weaviate will perform vector and hybrid search operations using the specified Google model.

When you perform a vector search, Weaviate converts the text query into an embedding using the specified model and returns the most similar objects from the database.

The query below returns the n most similar objects from the database, set by limit.

collection = client.collections.use("DemoCollection")response = collection.query.near_text(    query="A holiday film",  # The model provider integration will automatically vectorize the query    limit=2)for obj in response.objects:    print(obj.properties["title"])

const collectionName = 'DemoCollection'
const myCollection = client.collections.use(collectionName)

When you perform a hybrid search, Weaviate converts the text query into an embedding using the specified model and returns the best scoring objects from the database.

The query below returns the n best scoring objects from the database, set by limit.

collection = client.collections.use("DemoCollection")response = collection.query.hybrid(    query="A holiday film",  # The model provider integration will automatically vectorize the query    limit=2)for obj in response.objects:    print(obj.properties["title"])

const collectionName = 'DemoCollection'
const myCollection = client.collections.use(collectionName)

When you perform a media search such as a near image search, Weaviate converts the query into an embedding using the specified model and returns the most similar objects from the database.

To perform a near media search such as near image search, convert the media query into a base64 string and pass it to the search query.

The query below returns the n most similar objects to the input image from the database, set by limit.

def url_to_base64(url):
    import requests
    import base64

    image_response = requests.get(url)
    content = image_response.content
    return base64.b64encode(content).decode("utf-8")

const base64String = 'SOME_BASE_64_REPRESENTATION';

result = await myCollection.query.nearImage(
  base64String,  // The model provider integration will automatically vectorize the query
  {
    limit: 2,
  }
)

console.log(JSON.stringify(result.objects, null, 2));

• gemini-embedding-2 (Vertex AI and Gemini API, added in 1.36.13). Supports text, images, PDFs, and audio (Gemini API only, up to 180 seconds); 3072 dimensions
• multimodalembedding@001 (default, Vertex AI only). Supports text, images, and video; dimensions: 128, 256, 512, 1408

• Google text embedding models + Weaviate
• Google AI generative models + Weaviate

Once the integrations are configured at the collection, the data management and search operations in Weaviate work identically to any other collection. See the following model-agnostic examples:

• The How-to: Manage collections and How-to: Manage objects guides show how to perform data operations (i.e. create, read, update, delete collections and objects within them).
• The How-to: Query & Search guides show how to perform search operations (i.e. vector, keyword, hybrid) as well as retrieval augmented generation.

• Google Vertex AI
• Google Gemini API

Have a question or feedback? Here's how to reach us.