Tenant Offloading

Tenants can be offloaded to cold storage to reduce memory and disk usage, and onloaded back when needed.

This page explains how to configure tenant offloading in Weaviate. For information on how to offload and onload tenants, see How-to: manage tenant states.

To use tenant offloading in Weaviate, you need enable a relevant offloading module.

The offload-s3 module enables you to offload or onload tenants to/from an S3 bucket.

To use the offload-s3 module, add offload-s3 to the ENABLE_MODULES in your docker-compose file as shown below.

services:  weaviate:    environment:      ENABLE_MODULES: 'offload-s3' # plus other modules you may need      OFFLOAD_S3_BUCKET: 'weaviate-offload' # the name of the S3 bucket      OFFLOAD_S3_BUCKET_AUTO_CREATE: 'true' # create the bucket if it does not exist

If the target S3 bucket does not exist, the OFFLOAD_S3_BUCKET_AUTO_CREATE must be set to true so that Weaviate can create the bucket automatically.

For kubernetes users, enable the relevant offload service in the helm chart values file, and set the necessary environment variables:

# Configure offload providers
offload:
  s3:
    enabled: true  # Set this value to true to enable the offload-s3 module
    envconfig:
      OFFLOAD_S3_BUCKET: weaviate-offload  # the name of the S3 bucket
      OFFLOAD_S3_BUCKET_AUTO_CREATE: true  # create the bucket if it does not exist

The offload-s3 module reads the following environment variables:

You must provide Weaviate with AWS authentication details. You can choose between access-key or ARN-based authentication:

The backup module will first try to authenticate itself using AWS IAM. If the authentication fails then it will try to authenticate with Option 2.

• Configure: Modules
• How-to: Manage tenant states
• Guide: Manage tenant states

Have a question or feedback? Here's how to reach us.