Multimedia search
Multimedia search uses images, video, audio and other multimedia formats as a search input to perform vector similarity search via the near_media operator.
How to configure Weaviate to use multimedia search?
Configure multimedia search
To use images, video, or audio as search inputs, configure a multi-modal vectorizer integration that supports these media types for your collection.
For example, Google's multi2vec-google with the gemini-embedding-2 model supports image, video, and audio inputs. See the model provider integrations page for available options.
Collection configuration
The collection must be configured with the appropriate media fields. For example:
from weaviate.classes.config import Configure, Property, DataTypeclient.collections.delete("MediaExample")client.collections.create( name="MediaExample", properties=[ Property(name="title", data_type=DataType.TEXT), Property(name="mediaType", data_type=DataType.TEXT), ], vector_config=Configure.Vectors.multi2vec_google_gemini( model="gemini-embedding-2", text_fields=["title"], # video_fields=["video"], # image_fields=["image"], ),)By local file path
Section titled “By local file path”Use the near_media operator to search by providing a file path to a video.
from pathlib import Pathfrom weaviate.classes.query import NearMediaTypecollection = client.collections.use("MediaExample")response = collection.query.near_media( media=Path("./videos/dog.mp4"), media_type=NearMediaType.VIDEO, return_properties=["title", "mediaType"], limit=5,)for obj in response.objects: print(obj.properties)By base64 representation
Section titled “By base64 representation”You can also provide a base64-encoded video string:
import base64from weaviate.classes.query import NearMediaTypewith open("./videos/butterfly.mp4", "rb") as f: video_base64 = base64.b64encode(f.read()).decode("utf-8")collection = client.collections.use("MediaExample")response = collection.query.near_media( media=video_base64, media_type=NearMediaType.VIDEO, return_properties=["title", "mediaType"], limit=5,)for obj in response.objects: print(obj.properties)Set a maximum distance
Section titled “Set a maximum distance”Set a maximum distance to filter results and return the distance metadata:
import base64from weaviate.classes.query import MetadataQuery, NearMediaTypewith open("./videos/cat.mp4", "rb") as f: video_base64 = base64.b64encode(f.read()).decode("utf-8")collection = client.collections.use("MediaExample")response = collection.query.near_media( media=video_base64, media_type=NearMediaType.VIDEO, distance=0.8, # Maximum accepted distance return_metadata=MetadataQuery(distance=True), return_properties=["title", "mediaType"], limit=5,)for obj in response.objects: print(f"{obj.properties} - Distance: {obj.metadata.distance}")With a filter
Section titled “With a filter”Combine near media searches with filters to narrow results:
import base64from weaviate.classes.query import NearMediaTypefrom weaviate.classes.query import Filterwith open("./videos/dog.mp4", "rb") as f: video_base64 = base64.b64encode(f.read()).decode("utf-8")collection = client.collections.use("MediaExample")response = collection.query.near_media( media=video_base64, media_type=NearMediaType.VIDEO, filters=Filter.by_property("mediaType").equal("animals"), return_properties=["title", "mediaType"], limit=5,)for obj in response.objects: print(obj.properties)Other media types
Section titled “Other media types”The near_media operator supports other media types such as audio. Set the media_type parameter to the appropriate NearMediaType value:
import base64from weaviate.classes.query import NearMediaTypecollection = client.collections.use("MediaExample")response = collection.query.near_media( media=Path("./audio/dog.wav"), media_type=NearMediaType.AUDIO, return_properties=["title", "mediaType"], limit=5,)for obj in response.objects: print(obj.properties)Supported media types
Section titled “Supported media types”The NearMediaType enum supports the following media types. Available types depend on the vectorizer module and model used.
| Media type | Enum value | Description |
|---|---|---|
| Audio | NearMediaType.AUDIO |
Audio files (e.g., .wav, .mp3) |
| Image | NearMediaType.IMAGE |
Image files (e.g., .jpg, .png). See also Image search. |
| Video | NearMediaType.VIDEO |
Video files (e.g., .mp4, .avi) |
| Depth | NearMediaType.DEPTH |
Depth map data |
| Thermal | NearMediaType.THERMAL |
Thermal image data |
| IMU | NearMediaType.IMU |
Inertial measurement unit data |
Further resources
Section titled “Further resources”- How-to: Query & Search - Image search
- How-to: Query & Search - Vector similarity search
- Model provider integrations - Google multimodal embeddings
Questions and feedback
Section titled “Questions and feedback”Have a question or feedback? Here's how to reach us.