Backups

Weaviate's Backup feature is designed to work natively with cloud technology. Most notably, it allows:

• Seamless integration with widely-used cloud blob storage, such as AWS S3, GCS, or Azure Storage
• Backup and Restore between different storage providers
• Single-command backup and restore
• Choice of backing up an entire instance, or selected collections only
• Incremental backups that only store changed data, reducing backup and speeding up backup times
• Easy migration to new environments

This quickstart demonstrates using backups in Weaviate using the local filesystem as a backup provider, which is suitable for development and testing environments.

Add these environment variables to your Weaviate configuration (e.g. Docker or Kubernetes configuration file):

# Enable the filesystem backup module
ENABLE_MODULES=backup-filesystem

# Set backup location (e.g. within a Docker container or on a Kubernetes pod)
BACKUP_FILESYSTEM_PATH=/var/lib/weaviate/backups

Restart Weaviate to apply the new configuration. Then, you are ready to start a backup:

from weaviate.classes.backup import BackupLocation

import weaviate from 'weaviate-client';

const client = await weaviate.connectToLocal();

let result = await client.backup.create({
  backupId: 'my-very-first-backup',
  backend: 'filesystem',
  // Optional parameters
  // waitForCompletion: true,
  // includeCollections: ['Article', 'Publication'],
  // excludeCollections: ['Author'],
})

console.log(result.status);

createResponse, err := client.Backup().Creator().
  WithIncludeClassNames("Article", "Publication").
  WithBackend(backup.BACKEND_FILESYSTEM).
  WithBackupID("my-very-first-backup").
  WithWaitForCompletion(true).
  Do(context.Background())

curl \
    -X POST \
    -H "Content-Type: application/json" \
    -d '{
         "id": "my-very-first-backup",
         "include": ["Article", "Publication"]
        }' \
    http://localhost:8080/v1/backups/filesystem

That's all there is to getting started with backups in Weaviate. The backup will be stored in the specified location on the local filesystem.

You can also:

• Restore the backup to a Weaviate instance
• Check the status of the backup (if you did not wait for completion)
• Cancel the backup if needed

Note that local backups are not suitable for production environments. For production, use a cloud provider like S3, GCS, or Azure Storage.

The following sections provide more details on how to configure and use backups in Weaviate.

Weaviate supports four backup storage options:

To use any provider:

1. Enable the module
   • Add the module name to the ENABLE_MODULES environment variable
   • On Weaviate Cloud instances, a relevant default module is enabled
2. Configure the required modules
   • Option 1: Set the necessary environment variables
   • Option 2 (Kubernetes): Configure the Helm chart values

Note multiple providers can be enabled simultaneously.

• Works with Amazon S3 and S3-compatible object stores (for example, MinIO)
• Supports multi-node deployments
• Recommended for production use

To configure backup-s3, you need to enable the module and provide the necessary configuration.

Add backup-s3 to the ENABLE_MODULES environment variable. For example, to enable the S3 module along with the text2vec-cohere module, set:

ENABLE_MODULES=backup-s3,text2vec-cohere

This configuration applies to any S3-compatible backend.

For Amazon S3, provide Weaviate with authentication details. You can choose between AWS IAM/ARN-based authentication or access-key authentication. For S3-compatible object stores, use access-key authentication.

The backup-s3 module resolves credentials with its own logic rather than the AWS shared configuration chain. Profiles in ~/.aws/credentials and the AWS_PROFILE variable are not read. Credentials are resolved in this order:

1. An external authentication broker, if BACKUP_S3_AUTH_PROXY_ENDPOINT is set. This advanced option takes precedence over everything else, including access keys.
2. The access key and secret access key, if both are set in Weaviate's environment.
3. AWS IAM (an IRSA or EC2 instance role), used only when no access key and secret access key are set.

Weaviate uses AWS IAM only when no access key and secret access key are set in its environment. No additional variables are required; the module uses the IRSA or EC2 instance role attached to the workload.

These environment variables are named for AWS, but they also apply to S3-compatible object stores that issue access-key credentials. Both the key and the secret must be set; if only one is present, Weaviate falls back to IAM.

To use an S3-compatible object store, set BACKUP_S3_ENDPOINT to the provider's S3 endpoint host, optionally including a port, and authenticate with AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. Set AWS_REGION if your provider requires one.

BACKUP_S3_BUCKET=weaviate-backups
BACKUP_S3_ENDPOINT=your-s3-endpoint.example.com   # host[:port], no scheme
BACKUP_S3_USE_SSL=true
AWS_ACCESS_KEY_ID=<your-access-key-id>
AWS_SECRET_ACCESS_KEY=<your-secret-access-key>
# Optional: set if your provider requires a region.
# AWS_REGION=<your-region>

• Works with Google Cloud Storage
• Supports multi-node deployments
• Recommended for production use

To configure backup-gcs, you need to enable the module and provide the necessary configuration.

Add backup-gcs to the ENABLE_MODULES environment variable. For example, to enable the S3 module along with the text2vec-cohere module, set:

ENABLE_MODULES=backup-gcs,text2vec-cohere

The backup-gcs module follows the Google Application Default Credentials best-practices. This means that credentials can be discovered through the environment, through a local Google Cloud CLI setup, or through an attached service account.

This makes it easy to use the same module in different setups. For example, you can use the environment-based approach in production, and the CLI-based approach on your local machine. This way you can easily pull a backup that was created in a remote environment to your local system. This can be helpful in debugging an issue, for example.

• Works with Microsoft Azure Storage
• Supports multi-node deployments
• Recommended for production use

To configure backup-azure, you need to enable the module and provide the necessary configuration.

Add backup-azure to the ENABLE_MODULES environment variable. For example, to enable the Azure module along with the text2vec-cohere module, set:

ENABLE_MODULES=backup-azure,text2vec-cohere

In addition to enabling the module, you need to configure it using environment variables. There are container-related variables, as well as credential-related variables.

There are two different ways to authenticate against Azure with backup-azure. You can use either:

1. An Azure Storage connection string, or
2. An Azure Storage account name and key.

Both options can be implemented using environment variables as follows:

If both of AZURE_STORAGE_CONNECTION_STRING and AZURE_STORAGE_ACCOUNT are provided, Weaviate will use AZURE_STORAGE_CONNECTION_STRING to authenticate.

• Works with the local filesystem and cloud providers
• Supports single-node deployments only
• Not recommended for production use

To configure backup-filesystem, you need to enable the module and provide the necessary configuration.

Add backup-filesystem to the ENABLE_MODULES environment variable. For example, to enable the S3 module along with the text2vec-cohere module, set:

ENABLE_MODULES=backup-filesystem,text2vec-cohere

In addition to enabling the module, you need to configure it using environment variables:

If you are missing your desired backup module, you can open a feature request on the Weaviate GitHub repository. We are also open to community contributions for new backup modules.

For REST API documentation, see the Backups section.

Once the modules are enabled and the configuration is provided, you can start a backup on any running instance with a single request.

You can choose to include or exclude specific collections in the backup. If you do not specify any collections, all collections are included by default.

The include and exclude options are mutually exclusive. You can set none or exactly one of those.

The include and exclude options support wildcard patterns to match multiple collections at once. Wildcard matching is case sensitive.

The * character matches any sequence of characters. For example, Article* matches Article, ArticleV1, and ArticleV2, but not article or Publication.

from weaviate.classes.backup import BackupLocation

import weaviate from 'weaviate-client';

const client = await weaviate.connectToLocal();

let result = await client.backup.create({
  backupId: 'my-very-first-backup',
  backend: 'filesystem',
  // Optional parameters
  // waitForCompletion: true,
  // includeCollections: ['Article', 'Publication'],
  // excludeCollections: ['Author'],
})

console.log(result.status);

createResponse, err := client.Backup().Creator().
  WithIncludeClassNames("Article", "Publication").
  WithBackend(backup.BACKEND_FILESYSTEM).
  WithBackupID("my-very-first-backup").
  WithWaitForCompletion(true).
  Do(context.Background())

var createResult = client.backup
    .create(backupId, backend,
        backup -> backup.includeCollections("Article", "Publication"))
    .waitForCompletion(client); // Replicates wait_for_completion=True

System.out.println(createResult);

curl \
    -X POST \
    -H "Content-Type: application/json" \
    -d '{
         "id": "my-very-first-backup",
         "include": ["Article", "Publication"]
        }' \
    http://localhost:8080/v1/backups/filesystem

While you are waiting for a backup to complete, Weaviate stays available.

Where zstd compression is available, choose one of: ZstdDefaultCompression, ZstdBestSpeed and ZstdBestCompression.

Otherwise, choose one of the standard gzip compression options: DefaultCompression, BestSpeed and BestCompression.

You can also explicitly disable compression by setting the level to NoCompression.

All client implementations have a "wait for completion" option which will poll the backup status in the background and only return once the backup has completed (successfully or unsuccessfully).

If you set the "wait for completion" option to false, you can also check the status yourself using the Backup Creation Status API.

GET /v1/backups/{backend}/{backup_id}

The response contains a "status" field. If the status is SUCCESS, the backup is complete. If the status is FAILED, an additional error is provided.

from weaviate.classes.backup import BackupLocation

let backupStatus = await client.backup.getCreateStatus({
  backupId: 'my-very-first-backup',
  backend: 'filesystem',
})

console.log(backupStatus);

statusCreateResponse, err := client.Backup().CreateStatusGetter().
  WithBackend(backup.BACKEND_FILESYSTEM).
  WithBackupID("my-very-first-backup").
  Do(context.Background())

Optional<Backup> createStatus =
    client.backup.getCreateStatus(backupId, backend);

System.out.println(createStatus.orElse(null));

curl http://localhost:8080/v1/backups/filesystem/my-very-first-backup

Incremental backups reduce backup size and duration by only storing data that has changed since a previous backup. Instead of copying all files again, an incremental backup references unchanged files from a base backup.

This can result in dramatically smaller backups and much faster backup times.

When creating a backup, Weaviate packs a shard's files into chunks. During an incremental backup, Weaviate compares each file against the base backup. Files that haven't changed are stored as pointers to the base backup rather than being copied again. On restore, Weaviate automatically fetches the referenced files from the base backup.

The base backup can itself be an incremental backup, so you can build a chain of incremental backups that ends at a full backup. Weaviate walks the whole chain to find unchanged files, so every backup in the chain must remain available.

Only a file large enough to get a chunk of its own can be referenced individually, so the way Weaviate groups files into chunks determines how much an incremental backup can reuse. For how Weaviate decides which files get their own chunk, and the environment variables that control chunking, see Chunking and file splitting.

First, create a regular backup that will serve as the base:

result = client.backup.create(
    backup_id="base-backup",
    backend="filesystem",
    include_collections=["Article", "Publication"],
    wait_for_completion=True,
)

print(result)

To create an incremental backup, pass the incremental_base_backup_id parameter with the ID of the base backup:

result = client.backup.create(    backup_id="incremental-backup-1",    backend="filesystem",    include_collections=["Article", "Publication"],    wait_for_completion=True,    incremental_base_backup_id="base-backup",)print(result)

You can chain incremental backups by using a previous incremental backup as the base. Weaviate will walk the chain back to the original full backup to find unchanged files.

result = client.backup.create(    backup_id="incremental-backup-2",    backend="filesystem",    include_collections=["Article", "Publication"],    wait_for_completion=True,    incremental_base_backup_id="incremental-backup-1",)print(result)

Restoring an incremental backup works the same as restoring any other backup. Weaviate automatically resolves the chain and fetches files from previous backups as needed.

result = client.backup.restore(
    backup_id="incremental-backup-2",
    backend="filesystem",
    wait_for_completion=True,
)

print(result)

You can list the backups that are stored in a backup backend. The listing reports the status of each backup, the collections it holds, its size, and, for an incremental backup, the backup it was built on. This is how you inspect an existing chain of incremental backups and confirm that every backup the chain depends on is still present.

GET /v1/backups/{backend}

The listing only includes backups whose collections you are authorized to read. Backups you have no read access to are left out rather than causing an error.

backups = client.backup.list_backups(
    backend="filesystem",
    sort_by_starting_time_asc=True,
)

for backup in backups:
    print(backup.backup_id, backup.status, backup.incremental_base_backup_id)

base-backup BackupStatus.SUCCESS None
incremental-backup-1 BackupStatus.SUCCESS base-backup
incremental-backup-2 BackupStatus.SUCCESS incremental-backup-1

An ongoing backup can be cancelled at any time. The backup process will be stopped, and the backup will be marked as CANCELLED.

from weaviate.classes.backup import BackupLocation

let cancelStatus = await client.backup.cancel({
  backupId: 'my-very-first-backup',
  backend: 'filesystem'
})

console.log(cancelStatus);

// Note: The cancel() method is called on the Backup object
backupToCancel.cancel(client);

err = client.Backup().Canceler().
  WithBackend(backup.BACKEND_FILESYSTEM).
  WithBackupID("some-unwanted-backup").
  Do(context.Background())

This operation is particularly useful if you have started a backup by accident, or if you would like to stop a backup that is taking too long.

You can restore any backup to any machine as long as the name and number of nodes between source and target are identical. The backup does not need to be created on the same instance. Once a backup backend is configured, you can restore a backup with a single request.

As with backup creation, the include and exclude options are mutually exclusive. You can set none or exactly one of those. In a restore operation, include and exclude are relative to the collections contained in the backup. The restore process is not aware of collections that existed on the source machine if they were not part of the backup.

Note that a restore fails if any of the collections already exist on this instance.

from weaviate.classes.backup import BackupLocation

import weaviate from 'weaviate-client';

const client = await weaviate.connectToLocal();

let result = await client.backup.restore({
  backupId: 'my-very-first-backup',
  backend: 'filesystem',
  // Optional parameters
  // waitForCompletion: true,
  // includeCollections: ['Article', 'Publication'],
  // excludeCollections: ['Author'],
})

console.log(result.status);

restoreResponse, err := client.Backup().Restorer().
  WithExcludeClassNames("Article").
  WithBackend(backup.BACKEND_FILESYSTEM).
  WithBackupID("my-very-first-backup").
  WithWaitForCompletion(true).
  // The following two methods can be used to restore roles and users separately
  // .WithRBACRoles(rbac.RBACAll).
  // .WithRBACUsers(rbac.UserAll).
  WithRBACAndUsers(). // This is a convenience method to restore both roles and users
  Do(context.Background())

var restoreResult = client.backup.restore(backupId, backend,
    restore -> restore.excludeCollections("Article")
// Note: roles_restore and users_restore syntax not provided
).waitForCompletion(client); // Replicates wait_for_completion=True

System.out.println(restoreResult);

curl \
    -X POST \
    -H "Content-Type: application/json" \
    -d '{
         "id": "my-very-first-backup",
         "exclude": ["Article"]
        }' \
    http://localhost:8080/v1/backups/filesystem/my-very-first-backup/restore

All client implementations have a "wait for completion" option which will poll the restore status in the background and only return once the restore has completed (successfully or unsuccessfully).

If you set the "wait for completion" option to false, you can also check the status yourself using the Backup Restore Status API.

The response contains a "status" field. If the status is SUCCESS, the restore is complete. If the status is FAILED, an additional error is provided.

from weaviate.classes.backup import BackupLocation

let restoreStatus = await client.backup.getRestoreStatus({
  backupId: 'my-very-first-backup',
  backend: 'filesystem',
})

console.log(restoreStatus);

statusRestoreResponse, err := client.Backup().RestoreStatusGetter().
  WithBackend(backup.BACKEND_FILESYSTEM).
  WithBackupID("my-very-first-backup").
  Do(context.Background())

Optional<Backup> restoreStatus =
    client.backup.getRestoreStatus(backupId, backend);

System.out.println(restoreStatus.orElse(null));

curl http://localhost:8080/v1/backups/filesystem/my-very-first-backup/restore

An ongoing restore operation can be cancelled before it reaches the FINALIZING phase. Cancellation is not possible once schema changes are being applied via Raft, as this could leave the cluster in an inconsistent state.

A restore goes through the following phases:

from weaviate.classes.backup import BackupLocation

let cancelRestoreStatus = await client.backup.cancel({
  backupId: 'my-very-first-backup',
  backend: 'filesystem',
  operation: 'restore',
})

console.log(cancelRestoreStatus);

client.backup.cancelRestore(backupId, backend);

curl -X DELETE http://localhost:8080/v1/backups/filesystem/my-very-first-backup/restore

When running Weaviate on Kubernetes, you can configure the backup provider using Helm chart values.

These values are available under the backups key in the values.yaml file. Refer to the inline documentation in the values.yaml file for more information.

The backup process is designed to be minimally invasive to a running setup. Even on very large setups, where terabytes of data need to be copied, Weaviate stays available during backup. It even accepts write requests while a backup process is running. This sections explains how backups work under the hood and why Weaviate can safely accept writes while a backup is copied.

Weaviate uses a custom LSM Store for its object store and inverted index. LSM stores are a hybrid of immutable disk segments and an in-memory structure called a memtable that accepts all writes (including updates and deletes). Most of the time, files on disk are immutable, there are only three situations where files are changed:

1. Anytime a memtable is flushed. This creates a new segment. Existing segments are not changed.
2. Any write into the memtable is also written into a Write-Ahead-Log (WAL). The WAL is only needed for disaster-recovery. Once a segment has been orderly flushed, the WAL can be discarded.
3. There is an async background process called Compaction that optimizes existing segments. It can merge two small segments into a single larger segment and remove redundant data as part of the process.

Weaviate's Backup implementation makes use of the above properties in the following ways:

1. Weaviate first flushes all active memtables to disk. This process takes in the 10s or 100s of milliseconds. Any pending write requests simply waits for a new memtable to be created without any failing requests or substantial delays.
2. Now that the memtables are flushed, there is a guarantee: All data that should be part of the backup is present in the existing disk segments. Any data that will be imported after the backup request ends up in new disk segments. The backup references a list of immutable files.
3. To prevent a compaction process from changing the files on disk while they are being copied, compactions are temporarily paused until all files have been copied. They are automatically resumed right after.

This way the backup process can guarantee that the files that are transferred to the remote backend are immutable (and thus safe to copy) even with new writes coming in. Even if it takes minutes or hours to backup a very large setup, Weaviate stays available without any user impact while the backup process is running.

It is not just safe - but even recommended - to create backups on live production instances while they are serving user requests.

The backup API is built in a way that no long-running network requests are required. The request to create a new backup returns immediately. It does some basic validation, then returns to the user. The backup is now in status STARTED. To get the status of a running backup you can poll the status endpoint. This makes the backup itself resilient to network or client failures.

If you would like your application to wait for the background backup process to complete, you can use the "wait for completion" feature that is present in all language clients. The clients will poll the status endpoint in the background and block until the status is either SUCCESS or FAILED. This makes it easy to write simple synchronous backup scripts, even with the async nature of the API.

On every backup, full or incremental, Weaviate packs each shard's files into chunks:

• Each of the shard's biggest files gets a chunk of its own.
• A file larger than BACKUP_SPLIT_FILE_SIZE is split into parts, and each part gets a chunk of its own that holds nothing else.
• The remaining smaller files are packed together into shared chunks of roughly BACKUP_CHUNK_TARGET_SIZE.

Chunking matters because it determines how much a later incremental backup can reuse:

• Only a file with a chunk of its own can be referenced from the base backup instead of copied again, and only if Weaviate also treats the file as immutable.
• A big file that Weaviate keeps rewriting has its own chunk but is still re-uploaded on every backup.
• Shared chunks are re-uploaded on every incremental backup, even if nothing in them changed.

A file gets its own chunk when it reaches the qualifying size: the larger of BACKUP_MIN_CHUNK_SIZE and the size of the shard's Nth largest file, where N is BACKUP_MAX_INDIVIDUAL_FILES, reduced on an incremental backup by the number of files already reused from the base backup. This budget is shared across a whole backup chain rather than renewed for each backup in it. Because the larger value wins, BACKUP_MIN_CHUNK_SIZE is only a floor. Lowering it never reduces how many files qualify. Which knob qualifies more files depends on the shard. With N or more files above the floor, raise BACKUP_MAX_INDIVIDUAL_FILES. With fewer, lower BACKUP_MIN_CHUNK_SIZE.

flowchart TD
    File["Shard file"]
    Qualifies{"Reaches the<br>qualifying size?"}
    OverSplit{"Larger than<br>the split size?"}
    Own["Own chunk"]
    Parts["Split into parts,<br>one chunk per part"]
    Shared["Packed into a<br>shared chunk"]
    Immutable{"Immutable<br>file?"}
    Reusable["Reusable by later<br>incremental backups"]
    Reuploaded["Re-uploaded on<br>every backup"]

    File --> Qualifies
    Qualifies -->|"No"| Shared
    Qualifies -->|"Yes"| OverSplit
    OverSplit -->|"No"| Own
    OverSplit -->|"Yes"| Parts
    Own --> Immutable
    Parts --> Immutable
    Immutable -->|"Yes"| Reusable
    Immutable -->|"No"| Reuploaded
    Shared --> Reuploaded

    style File fill:#ffffff,stroke:#B9C8DF,color:#130C49
    style Qualifies fill:#ffffff,stroke:#B9C8DF,color:#130C49
    style OverSplit fill:#ffffff,stroke:#B9C8DF,color:#130C49
    style Own fill:#ffffff,stroke:#B9C8DF,color:#130C49
    style Parts fill:#ffffff,stroke:#B9C8DF,color:#130C49
    style Shared fill:#ffffff,stroke:#B9C8DF,color:#130C49
    style Immutable fill:#ffffff,stroke:#B9C8DF,color:#130C49
    style Reusable fill:#ffffff,stroke:#B9C8DF,color:#130C49
    style Reuploaded fill:#ffffff,stroke:#B9C8DF,color:#130C49

The parts of a split file are sized by BACKUP_SPLIT_FILE_SIZE, not by the chunk target: Weaviate divides the file into roughly equal parts that are each at least half and at most the full split size. At the defaults, a chunk carrying a split part is therefore up to 50GiB. That is far larger than the 10MiB chunk target, not smaller.

The three size variables accept a plain number of bytes or a number with a case-sensitive unit suffix (B, KB, MB, GB, TB, KiB, MiB, GiB, TiB), for example 4MiB. Decimal and binary suffixes are distinct: MB is 1,000,000 bytes while MiB is 1,048,576 bytes. They also accept unlimited or nolimit, which is how you disable file splitting through BACKUP_SPLIT_FILE_SIZE. All three are read at startup, so changing them requires a restart.

When a cloud backup backend (backup-s3, backup-gcs, or backup-azure) initializes, Weaviate verifies that the configured credentials can write to and delete from the target bucket. It does this by writing a temporary access-check object and then removing it. This probe fails on immutable (write-once / WORM) buckets, or with least-privilege credentials that are not permitted to delete objects.

Set BACKUP_SKIP_ACCESS_CHECK=true to skip this probe. The variable applies to all cloud backup backends, defaults to false, and is applied at startup (a restart is required to change it).

The flexibility around backup providers opens up new use cases. Besides using the backup & restore feature for disaster recovery, you can also use it for duplicating environments or migrating between clusters.

For example, consider the following situation: You would like to do a load test on production data. If you would do the load test in production it might affect users. An easy way to get meaningful results without affecting uses it to duplicate your entire environment. Once the new production-like "loadtest" environment is up, create a backup from your production environment and restore it into your "loadtest" environment. This even works if the production environment is running on a completely different cloud provider than the new environment.

• Single node backup is available starting in Weaviate v1.15. Multi-node backups is available starting in v1.16.
• In some cases, backups can take a long time, or get "stuck", causing Weaviate to be unresponsive. If this happens, you can cancel the backup and try again.
• If a backup module is misconfigured, such as having an invalid backup path, it can cause Weaviate to not start. Review the system logs for any errors.
• RBAC roles and users are not restored by default. You need to enable them manually through the configuration properties when restoring a backup.

• References: REST API: Backups

Have a question or feedback? Here's how to reach us.