Tenants can be offloaded to cold storage to reduce memory and disk usage, and onloaded back when needed. This page explains how to configure tenant offloading in Weaviate. For information on how to offload and onload tenants, see [How-to: manage tenant states](../how-to-manage-collections/tenant-states.md). ## Tenant offload modules :::callout{intent="info" title="Offloading: AWS S3 only"} As of Weaviate `v1.26.0`, tenants can only be offloaded to cold storage in AWS S3. Additional storage options may be added in future releases. To offload a tenant, use the `offload-s3` module. ::: To use tenant offloading in Weaviate, you need enable a relevant offloading [module](../how-to-configure-weaviate/modules.md). ## `offload-s3` module The `offload-s3` module enables you to [offload or onload tenants](../how-to-manage-collections/tenant-states.md) to/from an S3 bucket. To use the `offload-s3` module, add `offload-s3` to the `ENABLE_MODULES` in your docker-compose file as shown below. ```yaml {4-6} services: weaviate: environment: ENABLE_MODULES: 'offload-s3' # plus other modules you may need OFFLOAD_S3_BUCKET: 'weaviate-offload' # the name of the S3 bucket OFFLOAD_S3_BUCKET_AUTO_CREATE: 'true' # create the bucket if it does not exist ``` If the target S3 bucket does not exist, the `OFFLOAD_S3_BUCKET_AUTO_CREATE` must be set to `true` so that Weaviate can create the bucket automatically. For kubernetes users, enable the relevant offload service in the helm chart values file, and set the necessary environment variables: ```yaml # Configure offload providers offload: s3: enabled: true # Set this value to true to enable the offload-s3 module envconfig: OFFLOAD_S3_BUCKET: weaviate-offload # the name of the S3 bucket OFFLOAD_S3_BUCKET_AUTO_CREATE: true # create the bucket if it does not exist ``` ### Environment variables The `offload-s3` module reads the following environment variables: | Env Var | Description | Default Value | | ------------------------------- | ------------------------------------------------------------------------------ | ------------------ | | `OFFLOAD_S3_BUCKET` | The destination S3 bucket to offload tenants. | `weaviate-offload` | | `OFFLOAD_S3_BUCKET_AUTO_CREATE` | When `true`, Weaviate automatically creates an S3 bucket if it does not exist. | `false` | | `OFFLOAD_S3_CONCURRENCY` | The number of concurrent offload operations. | `25` | | `OFFLOAD_TIMEOUT` | The timeout for offloading operations (create bucket, upload, download). | `120` (in seconds) | :::callout{intent="info" title="Timeout"} - Offload operations are asynchronous. As a result, the timeout is for the operation to start, not to complete. - Each operation will retry up to 10 times on timeouts, except on authentication/authorization errors. ::: ### AWS permissions :::callout{intent="tip" title="Requirements"} The Weaviate instance must have the [necessary permissions to access the S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-policy-language-overview.html). - The provided AWS identity must be able to write to the bucket. - If `OFFLOAD_S3_BUCKET_AUTO_CREATE` is set to `true`, the AWS identity must have permission to create the bucket. ::: You must provide Weaviate with AWS authentication details. You can choose between access-key or ARN-based authentication: #### Option 1: With IAM and ARN roles The backup module will first try to authenticate itself using AWS IAM. If the authentication fails then it will try to authenticate with `Option 2`. #### Option 2: With access key and secret access key | Environment variable | Description | | ----------------------- | ---------------------------------------------------------------------------------------------- | | `AWS_ACCESS_KEY_ID` | The id of the AWS access key for the desired account. | | `AWS_SECRET_ACCESS_KEY` | The secret AWS access key for the desired account. | | `AWS_REGION` | (Optional) The AWS Region. If not provided, the module will try to parse `AWS_DEFAULT_REGION`. | ## Related pages - [Configure: Modules](../how-to-configure-weaviate/modules.md) - [How-to: Manage tenant states](../how-to-manage-collections/tenant-states.md) - [Guide: Manage tenant states](../starter-guides/managing-resources-tenant-states.md) ## Questions and feedback Have a question or feedback? Here's how to reach us. ::::card-grid :::card{title="Community Forum" href="https://forum.weaviate.io/c/support" icon="messages-square"} Ask questions and connect with other developers on our **Community forum**. ::: :::card{title="Support" href="/guides/support-overview" icon="life-buoy"} Weaviate Cloud user or customer? Find the right channel on the **Support page**. ::: :::: ## Related pages - [Agents](./agents-index.md) - [AI-assisted Weaviate code generation](./ai-assisted-vibe-coding-index.md) - [APIs](./apis-index.md) - [Authorization and authentication](./authorization-and-authentication-index.md) - [Benchmarks](./benchmarks-index.md) - [Best practices](./best-practices-index.md) - [Client libraries](./clients-index.md) - [Client Libraries / SDKs](./client-libraries-index.md) - [Cloud](./cloud-index.md) - [Cloud account management](./cloud-account-management-index.md) # Agent Instructions This portal answers questions programmatically. To receive a synthesized, source-cited answer instead of crawling page by page, append the `?ask=` query parameter to any page URL on this site: /guides/quickstart?ask=how+do+I+authenticate Optional parameters: - `&goal=` steers the answer toward your objective (e.g. `&goal=write+a+python+client`). - `&version=