:::callout{intent="warning" title="Preview (added in `v1.37`)"} This is a preview feature. The API may change in future releases. ::: Export collections from Weaviate to cloud storage in [Apache Parquet](https://parquet.apache.org/) format. Exports are point-in-time snapshots, writes that occur during an export do not affect the exported data. Only one export at a time per node is possible. The export feature is **disabled by default**. To use it: 1. [Enable the export API](#environment-variables) and configure a storage bucket. 2. [Configure cloud storage credentials](#backend-configuration) for your backend (S3, GCS, or Azure). 3. [Create an export](#create-a-collection-export) via the client or REST API. ## Environment variables Set these [environment variables](../database-configuration/overview.md) to enable and configure exports: | Environment Variable | Default | Description | | :------------------------- | :--------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `EXPORT_ENABLED` | `false` | Enable the export API. | | `EXPORT_DEFAULT_BUCKET` | (empty) | Storage bucket name. Required for S3, GCS, and Azure backends. | | `EXPORT_DEFAULT_PATH` | `""` | Optional base path prefix for exported files within the bucket. Defaults to an empty string (no prefix). _Changed in `v1.37.1`: previously required to be explicitly set._ | | `EXPORT_PARALLELISM` | `0` (GOMAXPROCS) | Number of concurrent scan workers. | | `EXPORT_SKIP_ACCESS_CHECK` | `false` | Skip the write-and-delete access check that runs when the export backend initializes. Set to `true` for immutable (write-once / WORM) buckets or least-privilege credentials that cannot delete objects. _Added in `v1.37.8`._ | `EXPORT_ENABLED`, `EXPORT_DEFAULT_BUCKET`, `EXPORT_DEFAULT_PATH`, and `EXPORT_PARALLELISM` are [runtime-configurable](../database-configuration/runtime-config.md) and can be changed without restarting Weaviate. `EXPORT_SKIP_ACCESS_CHECK` is applied at startup and requires a restart to change. :::callout{intent="note" title="Weaviate Cloud"} The collection export feature is not available in Weaviate Cloud. ::: ## Backend configuration Exports support three cloud storage backends and the [local filesystem](backups.md#filesystem). Each cloud storage backend uses the same credential environment variables as [backups](backups.md#configuration): | Backend | Value | Credential env vars | | :---------------------------------------------------------- | :------ | :-------------------------------------------------------------------------------- | | [Amazon S3](backups.md#s3-aws-or-s3-compatible) | `s3` | `AWS_REGION`, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` | | [Google Cloud Storage](backups.md#gcs-google-cloud-storage) | `gcs` | `GOOGLE_APPLICATION_CREDENTIALS` | | [Azure Blob Storage](backups.md#azure-storage) | `azure` | `AZURE_STORAGE_ACCOUNT`, `AZURE_STORAGE_KEY` or `AZURE_STORAGE_CONNECTION_STRING` | :::callout{intent="warning" title="Use a separate bucket for exports"} Do not export to backup buckets. Backup buckets may have immutability policies that cause export operations to fail. Use a dedicated bucket for exports. ::: ## Create a collection export Specify an export ID, backend, file format, and optionally which collections to include or exclude. If neither `include` nor `exclude` is specified, all collections are exported. :::code-group{sync="languages"} ```python title="Python" # Export specific collections result = client.export.create( export_id="my-export-include", backend=ExportStorage.FILESYSTEM, file_format=ExportFileFormat.PARQUET, include_collections=["Articles", "Products"], wait_for_completion=True, ) print(result.status) # ExportStatus.SUCCESS print(result.collections) # ['Articles', 'Products'] # Or exclude specific collections (exports everything else) result = client.export.create( export_id="my-export-exclude", backend=ExportStorage.FILESYSTEM, file_format=ExportFileFormat.PARQUET, exclude_collections=["TempData"], wait_for_completion=True, ) ``` ```csharp title="C#" // Export specific collections var includeResult = await client.Export.CreateSync( new ExportCreateRequest( Id: "my-export-include", Backend: ExportBackend.Filesystem(), FileFormat: ExportFileFormat.Parquet, IncludeCollections: ["Articles", "Products"] ), timeout: TimeSpan.FromMinutes(2) ); Console.WriteLine(includeResult.Status); // ExportStatus.Success Console.WriteLine(string.Join(", ", includeResult.Collections ?? [])); // Articles, Products // Or exclude specific collections (exports everything else) var excludeResult = await client.Export.CreateSync( new ExportCreateRequest( Id: "my-export-exclude", Backend: ExportBackend.Filesystem(), FileFormat: ExportFileFormat.Parquet, ExcludeCollections: ["TempData"] ), timeout: TimeSpan.FromMinutes(2) ); ``` ```bash title="cURL" curl -X POST http://localhost:8080/v1/export/filesystem \ -H "Content-Type: application/json" \ -d '{ "id": "my-export-2024", "file_format": "parquet", "include": ["Articles", "Products"] }' ``` ::: ### Request parameters | Field | Required | Description | | :------------ | :------- | :-------------------------------------------------------------------------- | | `id` | Yes | Unique export ID. Must match `^[a-z0-9_-]+$`, max 128 characters. | | `file_format` | Yes | Output format. Currently only `parquet` is supported. | | `include` | No | Collections to export. Cannot be used together with `exclude`. | | `exclude` | No | Collections to exclude from export. Cannot be used together with `include`. | ## Check collection export status Exports run asynchronously. Poll the status endpoint to track progress. :::code-group{sync="languages"} ```python title="Python" status = client.export.get_status( export_id=async_export_id, backend=ExportStorage.FILESYSTEM, ) print(status.status) # e.g. ExportStatus.TRANSFERRING print(status.collections) # ['Articles'] print(status.shard_status) # Per-shard progress details ``` ```csharp title="C#" var status = await client.Export.GetStatus( backend: ExportBackend.Filesystem(), id: asyncId ); Console.WriteLine(status.Status); // e.g. Transferring Console.WriteLine(string.Join(", ", status.Collections ?? [])); // Articles // status.ShardStatus has per-shard progress details (collection -> shard -> ShardProgress) ``` ```bash title="cURL" curl http://localhost:8080/v1/export/filesystem/my-async-export ``` ::: ### Export states | State | Description | | :------------- | :--------------------------------------------- | | `STARTED` | Export has been created and is initializing. | | `TRANSFERRING` | Data is being written to cloud storage. | | `SUCCESS` | Export completed successfully. | | `FAILED` | Export failed. Check shard status for details. | | `CANCELED` | Export was canceled by the user. | ### Shard states Each shard within an export has its own status: | State | Description | | :------------- | :------------------------------------------ | | `TRANSFERRING` | Shard data is being written. | | `SUCCESS` | Shard export completed. | | `FAILED` | Shard export failed. | | `SKIPPED` | Shard was skipped (e.g., offloaded tenant). | ## Cancel a collection export :::code-group{sync="languages"} ```python title="Python" client.export.cancel( export_id=cancel_id, backend=ExportStorage.FILESYSTEM, ) ``` ```csharp title="C#" await client.Export.Cancel( backend: ExportBackend.Filesystem(), id: cancelId ); ``` ```bash title="cURL" curl -X DELETE http://localhost:8080/v1/export/filesystem/my-async-export ``` ::: ## Output format Exports produce [Apache Parquet](https://parquet.apache.org/) files with Zstd compression. Each file contains: | Column | Type | Description | | :-------------- | :----- | :------------------------------------- | | `id` | string | Object UUID | | `creation_time` | int64 | Creation timestamp (nanoseconds) | | `update_time` | int64 | Last update timestamp (nanoseconds) | | `vector` | bytes | Primary vector (little-endian float32) | | `named_vectors` | bytes | JSON-encoded named vectors | | `multi_vectors` | bytes | JSON-encoded multi-vectors | | `properties` | bytes | Raw JSON of object properties | Files are named `{collection}_{shard}_{rangeIndex}.parquet`. Collection and tenant names are stored as Parquet file-level metadata. ## Multi-tenancy | Tenant state | Behavior | | :----------- | :---------------------------------------------------------------------- | | HOT | Exported from live data. | | COLD | Exported directly from disk without loading into memory (remains COLD). | | OFFLOADED | Skipped. The skip reason is recorded in the shard status. | The tenant list is snapshotted when the export is created. Tenants created during the export are not included. ## Permissions Export uses the backups permission `manage_backups` for [RBAC authorization](../authorization-and-authentication/weaviate-configuration-rbac.md). ## Further resources - [REST API endpoint](/weaviate/api/rest) ## Questions and feedback Have a question or feedback? Here's how to reach us. ::::card-grid :::card{title="Community Forum" href="https://forum.weaviate.io/c/support" icon="messages-square"} Ask questions and connect with other developers on our **Community forum**. ::: :::card{title="Support" href="/guides/support-overview" icon="life-buoy"} Weaviate Cloud user or customer? Find the right channel on the **Support page**. ::: :::: ## Related pages - [Agents](./agents-index.md) - [AI-assisted Weaviate code generation](./ai-assisted-vibe-coding-index.md) - [APIs](./apis-index.md) - [Authorization and authentication](./authorization-and-authentication-index.md) - [Benchmarks](./benchmarks-index.md) - [Best practices](./best-practices-index.md) - [Client libraries](./clients-index.md) - [Client Libraries / SDKs](./client-libraries-index.md) - [Cloud](./cloud-index.md) - [Cloud account management](./cloud-account-management-index.md) # Agent Instructions This portal answers questions programmatically. To receive a synthesized, source-cited answer instead of crawling page by page, append the `?ask=` query parameter to any page URL on this site: /guides/quickstart?ask=how+do+I+authenticate Optional parameters: - `&goal=` steers the answer toward your objective (e.g. `&goal=write+a+python+client`). - `&version=