# Data structure ## Data object concepts Each data object in Weaviate belongs to a `collection` and has one or more `properties`. Weaviate stores `data objects` in class-based collections. Data objects are represented as JSON-documents. Objects normally include a `vector` that is derived from a machine learning model. The vector is also called an `embedding` or a `vector embedding`. Each collection contains objects of the same `class`. The objects are defined by a common `schema`. ```mermaid flowchart LR subgraph Collection["πŸ—„οΈ Collection"] direction LR CollectionConfig["Collection Configuration

(e.g. data schema,
embedding model integration,
index configurations,
replication config, etc.)"] end subgraph search ["Indexes"] direction LR Indexes["Indexes"] subgraph vector ["Vector Search"] direction TB VectorIndex["Vector Index"] IndexStructure["Index Structure"] VectorCache["Vector Cache"] end subgraph text ["Filtering / Text Search"] direction LR InvertedIndex["Inverted Index"] BM25Index["BM25 Index"] FilterIndex["Filter Index"] end end subgraph storage ["Data Storage"] direction TB ObjectStore["Object Store"] ObjectData["Object Data / Metadata"] VectorData["Vector Data"] end %% Connections Collection --> Indexes Collection --> ObjectStore Indexes --> VectorIndex Indexes --> InvertedIndex VectorIndex --> IndexStructure VectorIndex --> VectorCache InvertedIndex --> BM25Index InvertedIndex --> FilterIndex ObjectStore --> ObjectData ObjectStore --> VectorData %% Style Collection node style Collection fill:#ffffff,stroke:#130C49,color:#130C49,stroke-width:2px %% Style Config components (purple color) style CollectionConfig fill:#f5f5f5,stroke:#9575CD,color:#130C49 %% Style Memory components (warm color) style Indexes fill:#FFF3E0,stroke:#FFB74D,color:#130C49 style VectorIndex fill:#FFF3E0,stroke:#FFB74D,color:#130C49 style InvertedIndex fill:#FFF3E0,stroke:#FFB74D,color:#130C49 style VectorCache fill:#FFF3E0,stroke:#FFB74D,color:#130C49 style IndexStructure fill:#FFF3E0,stroke:#FFB74D,color:#130C49 style BM25Index fill:#FFF3E0,stroke:#FFB74D,color:#130C49 style FilterIndex fill:#FFF3E0,stroke:#FFB74D,color:#130C49 %% Style Disk components (cool color) style ObjectStore fill:#E3F2FD,stroke:#64B5F6,color:#130C49 style ObjectData fill:#E3F2FD,stroke:#64B5F6,color:#130C49 style VectorData fill:#E3F2FD,stroke:#64B5F6,color:#130C49 %% Style subgraphs style search fill:#ffffff,stroke:#7AD6EB,stroke-width:2px,color:#130C49 style vector fill:#ffffff,stroke:#61BD73,stroke-width:2px,color:#130C49 style text fill:#ffffff,stroke:#61BD73,stroke-width:2px,color:#130C49 style storage fill:#ffffff,stroke:#7AD6EB,stroke-width:2px,color:#130C49 ``` :::callout{intent="note" title="Capitalization"} Weaviate follows GraphQL naming conventions. - Start collection names with an upper case letter. - Start property names with a lower case letter. If you use an initial upper case letter to define a property name, Weaviate changes it to a lower case letter internally. ::: ### JSON documents as objects Imagine we need to store information about an author named Alice Munro. In JSON format the data looks like this: ```json { "name": "Alice Munro", "age": 91, "born": "1931-07-10T00:00:00.0Z", "wonNobelPrize": true, "description": "Alice Ann Munro is a Canadian short story writer who won the Nobel Prize in Literature in 2013. Munro's work has been described as revolutionizing the architecture of short stories, especially in its tendency to move forward and backward in time." } ``` ### Vectors You can also attach `vector` representations to your data objects. Vectors are arrays of numbers that are stored under the `"vector"` property. In this example, the `Alice Munro` data object has a small vector. The vector is some information about Alice, maybe a story or an image, that a machine learning model has transformed into an array of numerical values. ```json { "id": "779c8970-0594-301c-bff5-d12907414002", "class": "Author", "properties": { "name": "Alice Munro", (...) }, "vector": [ -0.16147631, -0.065765485, -0.06546908 ] } ``` To generate vectors for your data, use one of Weaviate's vectorizer [modules](modules.md). You can also use your own vectorizer. ### Collections Collections are groups of objects that share a schema definition. In this example, the `Author` collection holds objects that represent different authors. The collection looks like this: ```json [{ "id": "dedd462a-23c8-32d0-9412-6fcf9c1e8149", "class": "Author", "properties": { "name": "Alice Munro", "age": 91, "born": "1931-07-10T00:00:00.0Z", "wonNobelPrize": true, "description": "Alice Ann Munro is a Canadian short story writer who won the Nobel Prize in Literature in 2013. Munro's work has been described as revolutionizing the architecture of short stories, especially in its tendency to move forward and backward in time." }, "vector": [ -0.16147631, -0.065765485, -0.06546908 ] }, { "id": "779c8970-0594-301c-bff5-d12907414002", "class": "Author", "properties": { "name": "Paul Krugman", "age": 69, "born": "1953-02-28T00:00:00.0Z", "wonNobelPrize": true, "description": "Paul Robin Krugman is an American economist and public intellectual, who is Distinguished Professor of Economics at the Graduate Center of the City University of New York, and a columnist for The New York Times. In 2008, Krugman was the winner of the Nobel Memorial Prize in Economic Sciences for his contributions to New Trade Theory and New Economic Geography." }, "vector": [ -0.93070928, -0.03782172, -0.56288009 ] }] ``` Every collection has its own vector space. This means that different collections can have different embeddings of the same object. ### UUIDs Every object stored in Weaviate has a [UUID](https://en.wikipedia.org/wiki/Universally_unique_identifier). The UUID guarantees uniqueness across all collections. You can [use a deterministic UUID](../how-to-manage-objects/import.md#specify-an-id-value) to ensure that the same object always has the same UUID. This is useful when you want to update an object without changing its UUID. If you don't specify an ID, Weaviate generates a random UUID for you. In requests without any other ordering specified, Weaviate processes them in ascending UUID order. This means that requests to [list objects](../how-to-query-search/basics.md#list-objects), use of the [cursor API](../how-to-manage-objects/read-all-objects.md), or requests to [delete objects](../how-to-manage-objects/delete.md#delete-multiple-objects-by-id), without any other ordering specified, will be processed in ascending UUID order. ### Cross-references :::callout{intent="warning" title="Cross-references and query performance"} Queries involving cross-references can be slower than queries that do not involve cross-references, especially at scale such as for multiple objects or complex queries. At the first instance, we strongly encourage you to consider whether you can avoid using cross-references in your data schema. As a scalable AI database, Weaviate is well-placed to perform complex queries with vector, keyword and hybrid searches involving filters. You may benefit from rethinking your data schema to avoid cross-references where possible. For example, instead of creating separate "Author" and "Book" collections with cross-references, consider embedding author information directly in Book objects and using searches and filters to find books by author characteristics. ::: If data objects are related, you can use [cross-references](../how-to-manage-collections/cross-references.md) to represent the relationships. Cross-references in Weaviate are like links that help you retrieve related information. Cross-references capture relationships, but they do not change the vectors of the underlying objects. To create a reference, use a property from one collection to specify the value of a related property in the other collection. #### Cross-reference example For example, _"Paul Krugman writes for the New York Times"_ describes a relationship between Paul Krugman and the New York Times. To capture that relationship, create a cross-reference between the `Publication` object that represents the New York Times and the `Author` object that represents Paul Krugman. The New York Times `Publication` object looks like this. Note the UUID in the `"id"` field: ```json { "id": "32d5a368-ace8-3bb7-ade7-9f7ff03eddb6", "class": "Publication", "properties": { "name": "The New York Times" }, "vector": [...] } ``` The Paul Krugman `Author` object adds a new property, `writesFor`, to capture the relationship. ```json {7-12} { "id": "779c8970-0594-301c-bff5-d12907414002", "class": "Author", "properties": { "name": "Paul Krugman", ... "writesFor": [ { "beacon": "weaviate://localhost/32d5a368-ace8-3bb7-ade7-9f7ff03eddb6", "href": "/v1/objects/32d5a368-ace8-3bb7-ade7-9f7ff03eddb6" } ], }, "vector": [...] } ``` The value of the `beacon` sub-property is the `id` value from the New York Times `Publication` object. Cross-reference relationships are directional. To make the link bi-directional, update the `Publication` collection to add a `hasAuthors` property points back to the `Author` collection. ### Multiple vector embeddings (named vectors) Collections can have multiple [named vectors](../reference-configuration/collections.md#multiple-vector-embeddings-named-vectors). The vectors in a collection can have their own configurations. Each vector space can set its own index, its own compression algorithm, and its own vectorizer. This means you can use different vectorization models, and apply different distance metrics, to the same object. To work with named vectors, adjust your queries to specify a target vector for [vector search](../how-to-query-search/similarity.md#named-vectors) or [hybrid search](../how-to-query-search/hybrid.md#named-vectors) queries. #### Adding a named vector after collection creation :::callout{intent="info" title="Added in `v1.31`"} ::: A named vector can be added to an existing collection definition after collection creation. This allows you to add new vector representations for objects without having to delete and recreate the collection. When you add a new named vector to an existing collection definition, it's important to understand that **existing objects' new named vector will remain unpopulated**. Only objects created after the named vector addition will receive these new vector embeddings. This prevents any unintended side effects, such as incurring large vectorization time or costs for all existing objects in a collection. If you want to populate the new named vector for existing objects, delete and reinsert the objects manually. This will trigger the vectorization process for the new named vector. :::callout{intent="warning" title="Not available for legacy (unnamed) vectorizers"} The ability to add a named vector after collection creation is only available for collections configured with named vectors. ::: ### Time to live (TTL) :::callout{intent="info" title="Added in `v1.36`"} ::: Objects can be optionally set to expire after a predetermined amount of time using the Time to Live (TTL) feature. A TTL can be set at the collection level. The expiration time can be defined in relation to: - The time of object creation - The time of the last object update - A Weaviate `DATE` property value (a date & time property) The TTL value is specified in seconds. The TTL must be positive, except for those relative to a `DATE` property, which can also be negative to allow for expiration before the specified date. Expired objects are automatically deleted by Weaviate at a set of predetermined intervals. The default value can be overridden using the `OBJECTS_TTL_DELETE_SCHEDULE` environment variable or `objects_ttl_delete_schedule` configuration in the helm chart. Expired, but yet undeleted, objects can optionally be excluded from query results to prevent erroneous data retrieval before the deletion process runs. Note that for multi-tenant collections, deletions can only occur for active tenants. Deletion operations will be skipped for inactive or offloaded tenants; and the deletion will occur only when the tenant becomes active again. ## Data Schema Weaviate requires a data schema before you add data. However, you don't have to create a data schema manually. If you don't provide one, Weaviate generates a schema based on the incoming data. Weaviate's schema defines its data structure in a formal language. In other words, it is a blueprint of how to organize and store the data. The schema defines data classes (i.e. collections of objects), the properties within each class (name, type, description, settings), possible graph links between data objects ([cross-references](data.md#cross-references)), and the vectorizer module (if any) to be used for the class, as well as settings such as the vectorizer module, and index configurations. :::callout{intent="note" title="Schema vs. Taxonomy"} A Weaviate data schema is slightly different from a taxonomy. A taxonomy has a hierarchy. Read more about how taxonomies, ontologies and schemas are related in this Weaviate [blog post](https://medium.com/semi-technologies/taxonomies-ontologies-and-schemas-how-do-they-relate-to-weaviate-9f76739fc695). ::: Schemas fulfill several roles: 1. Schemas define collections and properties. 2. Schemas define cross-references that link collections, even collections that use different embeddings. 3. Schemas let you configure module behavior, ANN index settings, reverse indexes, and other features on a collection level. For details on configuring your schema, see the [schema tutorial](../starter-guides/managing-collections.md) or [How-to: Manage collections](../how-to-manage-collections/index.md). ## Multi-tenancy To separate data within a cluster, use multi-tenancy. Weaviate partitions the cluster into shards. Each shard holds data for a single tenant. ```mermaid %%{init: {'theme': 'base', 'themeVariables': { 'background': '#f5f5f5' }}}%% flowchart TB subgraph MultiDB ["Multi-Tenant"] direction LR subgraph MTCollection["πŸ—„οΈ Collection"] direction LR MTCollectionConfig["Collection Configuration

(e.g. data schema,
embedding model integration,
index configurations,
replication config, etc.)"] end ShardA["Tenant A Shard"] IndexA["Indexes"] StoreA["Object Store"] ShardB["Tenant B Shard"] IndexB["Indexes"] StoreB["Object Store"] ShardC["Tenant C Shard"] IndexC["Indexes"] StoreC["Object Store"] MTCollection --> ShardA MTCollection --> ShardB MTCollection --> ShardC ShardA --> IndexA ShardA --> StoreA ShardB --> IndexB ShardB --> StoreB ShardC --> IndexC ShardC --> StoreC end subgraph SingleDB ["Single Collection"] direction LR subgraph SingleCollection["πŸ—„οΈ Collection"] direction LR SingleCollectionConfig["Collection Configuration

(e.g. data schema,
embedding model integration,
index configurations,
replication config, etc.)"] end SingleIndexes["Indexes"] SingleStore["Object Store"] SingleCollection --> SingleIndexes SingleCollection --> SingleStore end %% Style nodes - Single tenant style SingleCollection fill:#ffffff,stroke:#130C49,color:#130C49,stroke-width:2px style SingleIndexes fill:#FFF3E0,stroke:#FFB74D,color:#130C49 style SingleStore fill:#E3F2FD,stroke:#64B5F6,color:#130C49 %% Style nodes - Multi tenant style MTCollection fill:#ffffff,stroke:#130C49,color:#130C49,stroke-width:2px style ShardA fill:#ffffff,stroke:#130C49,color:#130C49,stroke-width:2px style ShardB fill:#ffffff,stroke:#130C49,color:#130C49,stroke-width:2px style ShardC fill:#ffffff,stroke:#130C49,color:#130C49,stroke-width:2px %% Style tenant resources style IndexA fill:#FFF3E0,stroke:#FFB74D,color:#130C49 style IndexB fill:#FFF3E0,stroke:#FFB74D,color:#130C49 style IndexC fill:#FFF3E0,stroke:#FFB74D,color:#130C49 style StoreA fill:#E3F2FD,stroke:#64B5F6,color:#130C49 style StoreB fill:#E3F2FD,stroke:#64B5F6,color:#130C49 style StoreC fill:#E3F2FD,stroke:#64B5F6,color:#130C49 %% Style subgraphs style SingleDB fill:transparent,stroke:#7AD6EB,stroke-width:2px,color:#130C49 style MultiDB fill:transparent,stroke:#7AD6EB,stroke-width:2px,color:#130C49 style MTCollectionConfig fill:#f5f5f5,stroke:#130C49,color:#130C49,stroke-width:2px style SingleCollectionConfig fill:#f5f5f5,stroke:#130C49,color:#130C49,stroke-width:2px ``` Sharding has several benefits: - Data isolation - Fast, efficient querying - Easy and robust setup and clean up Tenant shards are more lightweight. You can easily have 50,000, or more, active shards per node. This means that you can support 1M concurrently active tenants with just 20 or so nodes. Multi-tenancy is especially useful when you want to store data for multiple customers, or when you want to store data for multiple projects. :::callout{intent="warning" title="Tenant deletion == Tenant data deletion"} Deleting a tenant deletes the associated shard. As a result, deleting a tenant also deletes all of its objects. ::: ### Tenant states Tenants have an activity status (also called a tenant state) that reflects their availability and storage location. A tenant can be `ACTIVE`, `INACTIVE`, `OFFLOADED`, `OFFLOADING`, or `ONLOADING`. - `ACTIVE` tenants are loaded and available for read and write operations. - In all other states, the tenant is not available for read or write access. Access attempts return an error message. - `INACTIVE` tenants are stored on local disk storage for quick activation. - `OFFLOADED` tenants are stored on cloud storage. This status is useful for long-term storage for tenants that are not frequently accessed. - `OFFLOADING` tenants are being moved to cloud storage. This is a transient status, and therefore not user-specifiable. - `ONLOADING` tenants are being loaded from cloud storage. This is a transient status, and therefore not user-specifiable. An `ONLOADING` tenant may be being warmed to a `ACTIVE` status or a `INACTIVE` status. For more details on managing tenants, see [Multi-tenancy operations](../how-to-manage-collections/multi-tenancy.md). | Status | Available | Description | User-specifiable | | :----------- | :-------- | :------------------------------------------------------------------------------------------------ | :--------------- | | `ACTIVE` | Yes | Loaded and available for read/write operations. | Yes | | `INACTIVE` | No | On local disk storage, no read / write access. Access attempts return an error message. | Yes | | `OFFLOADED` | No | On cloud storage, no read / write access. Access attempts return an error message. | Yes | | `OFFLOADING` | No | Being moved to cloud storage, no read / write access. Access attempts return an error message. | No | | `ONLOADING` | No | Being loaded from cloud storage, no read / write access. Access attempts return an error message. | No | :::callout{intent="info" title="Tenant status renamed in `v1.26`"} In `v1.26`, the `HOT` status was renamed to `ACTIVE` and the `COLD` status was renamed to `INACTIVE`. ::: :::callout{intent="info" title="Tenant state propagation"} A tenant state change may take some time to propagate across a cluster, especially a multi-node cluster. For example, data may not be immediately available after reactivating an offloaded tenant. Similarly, data may not be immediately unavailable after offloading a tenant. This is because the [tenant states are eventually consistent](../replication-architecture/consistency.md#tenant-states-and-data-objects), and the change must be propagated to all nodes in the cluster. ::: #### Offloaded tenants :::callout{intent="info" title="Offloading: AWS S3 only"} As of Weaviate `v1.26.0`, tenants can only be offloaded to cold storage in AWS S3. Additional storage options may be added in future releases. To offload a tenant, use the `offload-s3` module. ::: Offloading tenants requires the relevant `offload-` module to be [enabled](../how-to-configure-weaviate/modules.md) in the Weaviate cluster. When a tenant is offloaded, the entire tenant shard is moved to cloud storage. This is useful for long-term storage of tenants that are not frequently accessed. Offloaded tenants are not available for read or write operations until they are loaded back into the cluster. ### Backups :::callout{intent="warning" title="Backups do not include offloaded tenants"} Backups of multi-tenant collections include both `active` and `inactive` tenants. Inactive tenants are read directly from disk, so you do not need to activate them before creating a backup. `Offloaded` tenants are not included, because their data is not held on local disk. The tenant status is preserved, but the tenant's data is not part of the backup. [Activate offloaded tenants](../how-to-manage-collections/multi-tenancy.md#manage-tenant-states) before creating a backup to include their data. Support for backing up `inactive` tenants was added in `v1.37.0`, and backported to `v1.35.17` and `v1.36.10`. In earlier releases, backups include only `active` tenants. ::: ### Tenancy and IDs Each tenancy is like a namespace, so different tenants could, in theory, have objects with the same IDs. To avoid naming problems, object IDs in multi-tenant clusters combine the tenant ID and the object ID to create an ID that is unique across tenants. ### Tenancy and cross-references Multi-tenancy supports some cross-references. Cross-references like these are supported: - From a multi-tenancy object to a non-multi-tenancy object. - From a multi-tenancy object to another multi-tenancy object, as long as they belong to the same tenant. Cross-references like these are not supported: - From a non-multi-tenancy object to a multi-tenancy object. - From a multi-tenancy object to another multi-tenancy object if they belong to different tenants. ### Key features - Each tenant has a dedicated, high-performance vector index. Dedicated indexes mean faster query speeds. Instead of searching a shared index space, each tenant responds as if it was the only user on the cluster. - Each tenant's data is isolated on a dedicated shard. This means that deletes are fast and do not affect other tenants. - To scale out, add a new node to your cluster. Weaviate does not redistribute existing tenants, however Weaviate adds new tenants to the node with the least resource usage. :::callout{intent="info" title="Related pages"} * [How-to: Manage Data | Multi-tenancy operations](../how-to-manage-collections/multi-tenancy.md) * [Multi-tenancy blog](https://weaviate.io/blog/multi-tenancy-vector-search) ::: ### Monitoring metrics To group tenants together for monitoring, set [`PROMETHEUS_MONITORING_GROUP = true`](../database-configuration/overview.md) in your system configuration file. ### Number of tenants per node The number of tenants per node is limited by operating system constraints. The number of tenants cannot exceed the Linux open file limit per process. For example, a 9-node test cluster built on `n1-standard-8` machines holds around 170k active tenants. There are 18,000 to 19,000 tenants per node. Note that these numbers relate to active tenants only. If you [set unused tenants as `inactive`](../how-to-manage-collections/multi-tenancy.md#manage-tenant-states), the open file per process limit does not apply. ## Related pages For more information, see the following: - [How-to: Multi-tenancy operations](../how-to-manage-collections/multi-tenancy.md) - [References: REST API: Schema](/weaviate/api/rest#tag/schema) - [How-to: Manage collections](../how-to-manage-collections/index.md) ## Summary - The schema defines collections and properties. - Collections contain data objects that are describe in JSON documents. - Data objects can contain a vector and properties. - Vectors come from machine learning models. - Different collections represent different vector spaces. - Cross-references link objects between schemas. - Multi-tenancy isolates data for each tenant. ## Questions and feedback Have a question or feedback? Here's how to reach us. ::::card-grid :::card{title="Community Forum" href="https://forum.weaviate.io/c/support" icon="messages-square"} Ask questions and connect with other developers on our **Community forum**. ::: :::card{title="Support" href="/guides/support-overview" icon="life-buoy"} Weaviate Cloud user or customer? Find the right channel on the **Support page**. ::: :::: ## Related pages - [Agents](./agents-index.md) - [AI-assisted Weaviate code generation](./ai-assisted-vibe-coding-index.md) - [APIs](./apis-index.md) - [Authorization and authentication](./authorization-and-authentication-index.md) - [Benchmarks](./benchmarks-index.md) - [Best practices](./best-practices-index.md) - [Client libraries](./clients-index.md) - [Client Libraries / SDKs](./client-libraries-index.md) - [Cloud](./cloud-index.md) - [Cloud account management](./cloud-account-management-index.md) # Agent Instructions This portal answers questions programmatically. To receive a synthesized, source-cited answer instead of crawling page by page, append the `?ask=` query parameter to any page URL on this site: /guides/quickstart?ask=how+do+I+authenticate Optional parameters: - `&goal=` steers the answer toward your objective (e.g. `&goal=write+a+python+client`). - `&version=