Migrate data

Follow these examples to migrate data manually when using a backup is not possible. They cover all permutations between:

a single-tenancy collection (Collection), and
a tenant in a multi-tenancy collection (Tenant).

Additional information

The examples use two different Weaviate instances, exposed through different ports. The same process can be used for two different instances as well.

Cross-references in Weaviate are properties. As such, you can retrieve cross-reference as a part of the object.

What about cross-references?

These scripts should migrate cross-references as well.

Cross-references are properties. As such, these cursor-based exports will include them. During restoration, restore the cross-referenced (i.e. "to") object first, then the object that contains the cross-reference (i.e. "from" object).

Collection → Collection

Step 1: Create the target collection(s)

Create a collection (e.g. WineReview) at the target instance, matching the collection (e.g. WineReview) at the source instance.

Python

import weaviate
import weaviate.classes as wvc
from weaviate.collections import Collection
from weaviate.client import WeaviateClient

JavaScript/TypeScript

import weaviate, { Collection, WeaviateClient } from 'weaviate-client'

let client_src: WeaviateClient,client_tgt: WeaviateClient;
let reviews_mt_tgt, reviews_mt_src;

Java

private static CollectionHandle<Map<String, Object>> createCollection(
    WeaviateClient clientIn, String collectionName, boolean enableMt)
    throws IOException {

private static async Task<CollectionClient> CreateCollection(
    WeaviateClient clientIn,
    string collectionName,
    bool enableMt
)
{

Step 2: Migrate the data

Migrate:

The source collection data in the client_src instance
to target collection in the client_tgt instance

Python

def migrate_data(collection_src: Collection, collection_tgt: Collection):

JavaScript/TypeScript

let reviews_tgt, reviews_src;

Java

private void migrateData(CollectionHandle<Map<String, Object>> collectionSrc,
    CollectionHandle<Map<String, Object>> collectionTgt) {
  System.out.println("Starting data migration...");
  List<WeaviateObject<Map<String, Object>>> sourceObjects = StreamSupport
      .stream(collectionSrc.paginate(p -> p.includeVector()).spliterator(),
          false)
      .map(readObj -> WeaviateObject
          .<Map<String, Object>>of(c -> c.properties(readObj.properties())
              .uuid(readObj.uuid())
              .vectors(readObj.vectors())))
      .collect(Collectors.toList());

  collectionTgt.data.insertMany(sourceObjects);

  System.out.println("Data migration complete.");
}

private async Task MigrateData<T>(
    CollectionClient collectionSrc,
    CollectionClient collectionTgt
)
{
    Console.WriteLine("Starting data migration...");

    // Fetch source objects
    var response = await collectionSrc.Query.FetchObjects(limit: 10000, includeVectors: true);

    // Map to Strong Type List
    var sourceObjects = new List<T>();
    foreach (var obj in response.Objects)
    {
        // Deserialize the inner properties Dictionary to the POCO type
        var json = JsonSerializer.Serialize(obj.Properties);
        var typedObj = JsonSerializer.Deserialize<T>(json);
        if (typedObj != null)
        {
            sourceObjects.Add(typedObj);
        }
    }

    // InsertMany using Strong Types
    await collectionTgt.Data.InsertMany(sourceObjects.ToArray());

    Console.WriteLine($"Data migration complete. Migrated {sourceObjects.Count} objects.");
}

Collection → Tenant

Step 1: Create the target collection(s)

Create a collection (e.g. WineReview) at the target instance, matching the collection (e.g. WineReview) at the source instance, and enable multi-tenancy for the target collection.

Python

import weaviate
import weaviate.classes as wvc
from weaviate.collections import Collection
from weaviate.client import WeaviateClient

JavaScript/TypeScript

import weaviate, { Collection, WeaviateClient } from 'weaviate-client'

let client_src: WeaviateClient,client_tgt: WeaviateClient;
let reviews_mt_tgt, reviews_mt_src;

Java

private static CollectionHandle<Map<String, Object>> createCollection(
    WeaviateClient clientIn, String collectionName, boolean enableMt)
    throws IOException {

private static async Task<CollectionClient> CreateCollection(
    WeaviateClient clientIn,
    string collectionName,
    bool enableMt
)
{

Step 2: Create the tenant(s)

Add tenants at the target instance before adding data objects.

Python

tenants_tgt = [wvc.tenants.Tenant(name="tenantA"), wvc.tenants.Tenant(name="tenantB")]

reviews_mt_tgt = client_tgt.collections.get("WineReviewMT")
reviews_mt_tgt.tenants.create(tenants_tgt)

JavaScript/TypeScript

let tenantsTgt = [
    { name: 'tenantA'},
    { name: 'tenantB'}
  ]

reviews_mt_tgt = client_tgt.collections.use("WineReviewMT")
reviews_mt_tgt.tenants.create(tenantsTgt)

Java

void createTenants() throws IOException {
  var reviewsMtTgt = clientTgt.collections.use("WineReviewMT");

  var tenantsTgt =
      List.of(Tenant.active("tenantA"), Tenant.active("tenantB"));
  reviewsMtTgt.tenants.create(tenantsTgt);
}

private async Task CreateTenants()
{
    var reviewsMtTgt = clientTgt.Collections.Use("WineReviewMT");

    var tenantsTgt = new[]
    {
        new Tenant { Name = "tenantA" },
        new Tenant { Name = "tenantB" },
    };
    await reviewsMtTgt.Tenants.Create(tenantsTgt);
}

Step 3: Migrate the data

Migrate:

The source collection data in the client_src instance
to target tenant data from target collection in the client_tgt instance

Python

def migrate_data(collection_src: Collection, collection_tgt: Collection):

JavaScript/TypeScript

let reviews_tgt, reviews_src;

Java

private void migrateData(CollectionHandle<Map<String, Object>> collectionSrc,
    CollectionHandle<Map<String, Object>> collectionTgt) {
  System.out.println("Starting data migration...");
  List<WeaviateObject<Map<String, Object>>> sourceObjects = StreamSupport
      .stream(collectionSrc.paginate(p -> p.includeVector()).spliterator(),
          false)
      .map(readObj -> WeaviateObject
          .<Map<String, Object>>of(c -> c.properties(readObj.properties())
              .uuid(readObj.uuid())
              .vectors(readObj.vectors())))
      .collect(Collectors.toList());

  collectionTgt.data.insertMany(sourceObjects);

  System.out.println("Data migration complete.");
}

private async Task MigrateData<T>(
    CollectionClient collectionSrc,
    CollectionClient collectionTgt
)
{
    Console.WriteLine("Starting data migration...");

    // Fetch source objects
    var response = await collectionSrc.Query.FetchObjects(limit: 10000, includeVectors: true);

    // Map to Strong Type List
    var sourceObjects = new List<T>();
    foreach (var obj in response.Objects)
    {
        // Deserialize the inner properties Dictionary to the POCO type
        var json = JsonSerializer.Serialize(obj.Properties);
        var typedObj = JsonSerializer.Deserialize<T>(json);
        if (typedObj != null)
        {
            sourceObjects.Add(typedObj);
        }
    }

    // InsertMany using Strong Types
    await collectionTgt.Data.InsertMany(sourceObjects.ToArray());

    Console.WriteLine($"Data migration complete. Migrated {sourceObjects.Count} objects.");
}

Tenant → Collection

Step 1: Create the target collection(s)

Create a collection (e.g. WineReview) at the target instance, matching the collection (e.g. WineReview) at the source instance, and enable multi-tenancy for the target collection.

Python

import weaviate
import weaviate.classes as wvc
from weaviate.collections import Collection
from weaviate.client import WeaviateClient

JavaScript/TypeScript

import weaviate, { Collection, WeaviateClient } from 'weaviate-client'

let client_src: WeaviateClient,client_tgt: WeaviateClient;
let reviews_mt_tgt, reviews_mt_src;

Java

private static CollectionHandle<Map<String, Object>> createCollection(
    WeaviateClient clientIn, String collectionName, boolean enableMt)
    throws IOException {

private static async Task<CollectionClient> CreateCollection(
    WeaviateClient clientIn,
    string collectionName,
    bool enableMt
)
{

Step 2: Migrate the data

Migrate:

The source tenant data from source collection in the client_src instance
to target collection in the client_tgt instance

Python

def migrate_data(collection_src: Collection, collection_tgt: Collection):

JavaScript/TypeScript

let reviews_tgt, reviews_src;

Java

private void migrateData(CollectionHandle<Map<String, Object>> collectionSrc,
    CollectionHandle<Map<String, Object>> collectionTgt) {
  System.out.println("Starting data migration...");
  List<WeaviateObject<Map<String, Object>>> sourceObjects = StreamSupport
      .stream(collectionSrc.paginate(p -> p.includeVector()).spliterator(),
          false)
      .map(readObj -> WeaviateObject
          .<Map<String, Object>>of(c -> c.properties(readObj.properties())
              .uuid(readObj.uuid())
              .vectors(readObj.vectors())))
      .collect(Collectors.toList());

  collectionTgt.data.insertMany(sourceObjects);

  System.out.println("Data migration complete.");
}

private async Task MigrateData<T>(
    CollectionClient collectionSrc,
    CollectionClient collectionTgt
)
{
    Console.WriteLine("Starting data migration...");

    // Fetch source objects
    var response = await collectionSrc.Query.FetchObjects(limit: 10000, includeVectors: true);

    // Map to Strong Type List
    var sourceObjects = new List<T>();
    foreach (var obj in response.Objects)
    {
        // Deserialize the inner properties Dictionary to the POCO type
        var json = JsonSerializer.Serialize(obj.Properties);
        var typedObj = JsonSerializer.Deserialize<T>(json);
        if (typedObj != null)
        {
            sourceObjects.Add(typedObj);
        }
    }

    // InsertMany using Strong Types
    await collectionTgt.Data.InsertMany(sourceObjects.ToArray());

    Console.WriteLine($"Data migration complete. Migrated {sourceObjects.Count} objects.");
}

Tenant → Tenant

Step 1: Create the target collection(s)

Create a collection (e.g. WineReview) at the target instance, matching the collection (e.g. WineReview) at the source instance including enabling multi-tenancy.

Python

import weaviate
import weaviate.classes as wvc
from weaviate.collections import Collection
from weaviate.client import WeaviateClient

JavaScript/TypeScript

import weaviate, { Collection, WeaviateClient } from 'weaviate-client'

let client_src: WeaviateClient,client_tgt: WeaviateClient;
let reviews_mt_tgt, reviews_mt_src;

Java

private static CollectionHandle<Map<String, Object>> createCollection(
    WeaviateClient clientIn, String collectionName, boolean enableMt)
    throws IOException {

private static async Task<CollectionClient> CreateCollection(
    WeaviateClient clientIn,
    string collectionName,
    bool enableMt
)
{

Step 2: Create the tenant(s)

Add tenants at the target instance before adding data objects.

Python

tenants_tgt = [wvc.tenants.Tenant(name="tenantA"), wvc.tenants.Tenant(name="tenantB")]

reviews_mt_tgt = client_tgt.collections.get("WineReviewMT")
reviews_mt_tgt.tenants.create(tenants_tgt)

JavaScript/TypeScript

let tenantsTgt = [
    { name: 'tenantA'},
    { name: 'tenantB'}
  ]

reviews_mt_tgt = client_tgt.collections.use("WineReviewMT")
reviews_mt_tgt.tenants.create(tenantsTgt)

Java

void createTenants() throws IOException {
  var reviewsMtTgt = clientTgt.collections.use("WineReviewMT");

  var tenantsTgt =
      List.of(Tenant.active("tenantA"), Tenant.active("tenantB"));
  reviewsMtTgt.tenants.create(tenantsTgt);
}

private async Task CreateTenants()
{
    var reviewsMtTgt = clientTgt.Collections.Use("WineReviewMT");

    var tenantsTgt = new[]
    {
        new Tenant { Name = "tenantA" },
        new Tenant { Name = "tenantB" },
    };
    await reviewsMtTgt.Tenants.Create(tenantsTgt);
}

Step 3: Migrate the data

Migrate:

The source tenant data from source collection in the client_src instance
to target tenant data from target collection in the client_tgt instance

Python

def migrate_data(collection_src: Collection, collection_tgt: Collection):

JavaScript/TypeScript

import weaviate, { Collection, WeaviateClient } from 'weaviate-client'

let client_src: WeaviateClient,client_tgt: WeaviateClient;
let reviews_mt_tgt, reviews_mt_src;

Java

private void migrateData(CollectionHandle<Map<String, Object>> collectionSrc,
    CollectionHandle<Map<String, Object>> collectionTgt) {
  System.out.println("Starting data migration...");
  List<WeaviateObject<Map<String, Object>>> sourceObjects = StreamSupport
      .stream(collectionSrc.paginate(p -> p.includeVector()).spliterator(),
          false)
      .map(readObj -> WeaviateObject
          .<Map<String, Object>>of(c -> c.properties(readObj.properties())
              .uuid(readObj.uuid())
              .vectors(readObj.vectors())))
      .collect(Collectors.toList());

  collectionTgt.data.insertMany(sourceObjects);

  System.out.println("Data migration complete.");
}

private async Task MigrateData<T>(
    CollectionClient collectionSrc,
    CollectionClient collectionTgt
)
{
    Console.WriteLine("Starting data migration...");

    // Fetch source objects
    var response = await collectionSrc.Query.FetchObjects(limit: 10000, includeVectors: true);

    // Map to Strong Type List
    var sourceObjects = new List<T>();
    foreach (var obj in response.Objects)
    {
        // Deserialize the inner properties Dictionary to the POCO type
        var json = JsonSerializer.Serialize(obj.Properties);
        var typedObj = JsonSerializer.Deserialize<T>(json);
        if (typedObj != null)
        {
            sourceObjects.Add(typedObj);
        }
    }

    // InsertMany using Strong Types
    await collectionTgt.Data.InsertMany(sourceObjects.ToArray());

    Console.WriteLine($"Data migration complete. Migrated {sourceObjects.Count} objects.");
}

Questions and feedback

Have a question or feedback? Here's how to reach us.

Community Forum

Ask questions and connect with other developers on our Community forum.

Support

Weaviate Cloud user or customer? Find the right channel on the Support page.

Collection → Collection

Step 1: Create the target collection(s)

Step 2: Migrate the data

Collection → Tenant

Step 1: Create the target collection(s)

Step 2: Create the tenant(s)

Step 3: Migrate the data

Tenant → Collection

Step 1: Create the target collection(s)

Step 2: Migrate the data

Tenant → Tenant

Step 1: Create the target collection(s)

Step 2: Create the tenant(s)

Step 3: Migrate the data

Related pages

Questions and feedback