Collection definitions (schemas)
This tutorial will guide you through the process of defining a schema for your data, including commonly used settings and key considerations.
Collection definition: An Introduction
Section titled “Collection definition: An Introduction”The collection definition specifies how data is stored, organized and retrieved in Weaviate.
If auto-schema is enabled, Weaviate can infer missing elements and add them to the collection definition. However, it is a best practice to manually define as much of the data schema as possible since manual definition gives you the most control.
Let's begin with a simple example before diving into the details.
Basic collection creation
Section titled “Basic collection creation”This example creates a collection called Question. The collection has three properties, answer, question, and category. The definition specifies the text2vec-openai vectorizer and the generative-cohere module for RAG.
questions = client.collections.create(
name="Question",
vector_config=wvc.config.Configure.Vectors.text2vec_openai(), # Set the vectorizer to "text2vec-openai" to use the OpenAI API for vector-related operations
generative_config=wvc.config.Configure.Generative.cohere(), # Set the generative module to "generative-cohere" to use the Cohere API for RAG
properties=[
wvc.config.Property(
name="question",
data_type=wvc.config.DataType.TEXT,
),
wvc.config.Property(
name="answer",
data_type=wvc.config.DataType.TEXT,
),
wvc.config.Property(
name="category",
data_type=wvc.config.DataType.TEXT,
)
]
)
print(questions.config.get(simple=False))Optional<CollectionConfig> questionsConfig =
client.collections.create("Question",
col -> col.vectorConfig(VectorConfig.text2vecWeaviate()) // Set the vectorizer to use the OpenAI API for vector-related operations
.generativeModule(Generative.cohere()) // Set the generative module to use the Cohere API for RAG
.properties(Property.text("question"), Property.text("answer"),
Property.text("category"))).config.get();
System.out.println(questionsConfig);var questionsCollection = await client.Collections.Create(
new CollectionCreateParams
{
Name = "Question",
VectorConfig = Configure.Vector("default", v => v.Text2VecWeaviate()),
GenerativeConfig = Configure.Generative.Cohere(), // Set the generative module
Properties =
[
Property.Text("question"),
Property.Text("answer"),
Property.Text("category"),
],
}
);
Console.WriteLine(questionsCollection);{/*
package main
import (
"context"
"fmt"
"github.com/weaviate/weaviate-go-client/v5/weaviate"
"github.com/weaviate/weaviate/entities/models"
)
func main() {
cfg := weaviate.Config{
Host: "WEAVIATE_INSTANCE_URL/", // Replace with the URL
Scheme: "https",
}
client, err := weaviate.NewClient(cfg)
if err != nil {
panic(err)
}
// we will create the class "Question"
classObj := &models.Class{
Class: "Question",
Description: "Information from a Jeopardy! question", // description of the class
Properties: []*models.Property{
{
DataType: []string{"string"},
Description: "The question",
Name: "question",
},
{
DataType: []string{"string"},
Description: "The answer",
Name: "answer",
},
},
}
// add the schema
err := client.Schema().ClassCreator().WithClass(classObj).Do(context.Background())
if err != nil {
panic(err)
}
// get the schema
schema, err := client.Schema().Getter().Do(context.Background())
if err != nil {
panic(err)
}
// print the schema
fmt.Printf("%v", schema)
}/} {/
# Edit ${WEAVIATE_INSTANCE_URL} to provide your instance URL.
curl \
-X POST \
-H "Content-Type: application/json" \
-d '{
"class": "Question",
"description": "Information from a Jeopardy! question",
"properties": [
{
"dataType": ["text"],
"description": "The question",
"name": "question"
},
{
"dataType": ["text"],
"description": "The answer",
"name": "answer"
}
]
}' \
https://${WEAVIATE_INSTANCE_URL}/v1/schema
curl https://${WEAVIATE_INSTANCE_URL}/v1/schema*/}
The returned configuration looks similar to this:
See the returned collection definition
{
"class": "Question",
"invertedIndexConfig": {
"bm25": {
"b": 0.75,
"k1": 1.2
},
"cleanupIntervalSeconds": 60,
"stopwords": {
"additions": null,
"preset": "en",
"removals": null
},
"usingBlockMaxWAND": true
},
"moduleConfig": {
"generative-cohere": {}
},
"multiTenancyConfig": {
"autoTenantActivation": false,
"autoTenantCreation": false,
"enabled": false
},
"properties": [
{
"dataType": [
"text"
],
"indexFilterable": true,
"indexRangeFilters": false,
"indexSearchable": true,
"moduleConfig": {
"text2vec-openai": {
"skip": false,
"vectorizePropertyName": false
}
},
"name": "question",
"tokenization": "word"
},
{
"dataType": [
"text"
],
"indexFilterable": true,
"indexRangeFilters": false,
"indexSearchable": true,
"moduleConfig": {
"text2vec-openai": {
"skip": false,
"vectorizePropertyName": false
}
},
"name": "answer",
"tokenization": "word"
},
{
"dataType": [
"text"
],
"indexFilterable": true,
"indexRangeFilters": false,
"indexSearchable": true,
"moduleConfig": {
"text2vec-openai": {
"skip": false,
"vectorizePropertyName": false
}
},
"name": "category",
"tokenization": "word"
}
],
"shardingConfig": {
"actualCount": 1,
"actualVirtualCount": 128,
"desiredCount": 1,
"desiredVirtualCount": 128,
"function": "murmur3",
"key": "_id",
"strategy": "hash",
"virtualPerPhysical": 128
},
"vectorConfig": {
"default": {
"vectorIndexConfig": {
"bq": {
"enabled": false
},
"cleanupIntervalSeconds": 300,
"distance": "cosine",
"dynamicEfFactor": 8,
"dynamicEfMax": 500,
"dynamicEfMin": 100,
"ef": -1,
"efConstruction": 128,
"filterStrategy": "acorn",
"flatSearchCutoff": 40000,
"maxConnections": 32,
"multivector": {
"aggregation": "maxSim",
"enabled": false,
"muvera": {
"dprojections": 16,
"enabled": false,
"ksim": 4,
"repetitions": 10
}
},
"pq": {
"bitCompression": false,
"centroids": 256,
"enabled": false,
"encoder": {
"distribution": "log-normal",
"type": "kmeans"
},
"segments": 0,
"trainingLimit": 100000
},
"rq": {
"bits": 8,
"enabled": false,
"rescoreLimit": 20
},
"skip": false,
"skipDefaultQuantization": false,
"sq": {
"enabled": false,
"rescoreLimit": 20,
"trainingLimit": 100000
},
"trackDefaultQuantization": false,
"vectorCacheMaxObjects": 1000000000000
},
"vectorIndexType": "hnsw",
"vectorizer": {
"text2vec-openai": {
"baseURL": "https://api.openai.com",
"isAzure": false,
"model": "text-embedding-3-small",
"vectorizeClassName": true
}
}
}
},
"replicationConfig": {
"deletionStrategy": "TimeBasedResolution",
"factor": 1,
"asyncEnabled": false
}
}Although we only specified the collection name, its properties, the vectorizer and the generative module, the returned definition includes much more information.
This is because Weaviate infers the definition based on the data schema and default settings. Each of these options can be specified manually at collection creation time.
Collection definitions in detail
Section titled “Collection definitions in detail”Conceptually, it may be useful to think of each Weaviate instance as consisting of multiple collections, each of which is a set of objects that share a common structure.
For example, you might have a movie database with Movie and Actor collections, each with their own properties. Or you might have a news database with Article, Author and Publication collections.
Available settings
Section titled “Available settings”For the most part, each collection should be thought of as isolated from the others (in fact, they are!). Accordingly, they can be configured independently. Each collection has:
- A set of
propertiesspecifying the object data structure. - Multi-tenancy settings.
- Vectorizer and generative modules.
- Index settings (for vector and inverted indexes).
- Replication and sharding settings.
And depending on your needs, you might want to change any number of these.
Properties
Section titled “Properties”Each property has a number of settings that can be configured, such as the dataType, tokenization, and vectorizePropertyName. You can read more about these in the collection definition reference.
So for example, you might specify a collection definition like the one below, with additional options for the question and answer properties:
questions = client.collections.create( name="Question", vector_config=wvc.config.Configure.Vectors.text2vec_openai(), # Set the vectorizer to "text2vec-openai" to use the OpenAI API for vector-related operations generative_config=wvc.config.Configure.Generative.cohere(), # Set the generative module to "generative-cohere" to use the Cohere API for RAG properties=[ wvc.config.Property( name="question", data_type=wvc.config.DataType.TEXT, vectorize_property_name=True, # Include the property name ("question") when vectorizing tokenization=wvc.config.Tokenization.LOWERCASE # Use "lowecase" tokenization ), wvc.config.Property( name="answer", data_type=wvc.config.DataType.TEXT, vectorize_property_name=False, # Skip the property name ("answer") when vectorizing tokenization=wvc.config.Tokenization.WHITESPACE # Use "whitespace" tokenization ), ])client.collections.create("Question", col -> col.vectorConfig(VectorConfig.text2vecWeaviate()) .generativeModule(Generative.cohere()) .properties(Property.text("question", p -> p .vectorizePropertyName(true) // Include the property name ("question") when vectorizing .tokenization(Tokenization.LOWERCASE) // Use "lowercase" tokenization ), Property.text("answer", p -> p .vectorizePropertyName(false) // Skip the property name ("answer") when vectorizing .tokenization(Tokenization.WHITESPACE) // Use "whitespace" tokenization )));await client.Collections.Create(
new CollectionCreateParams
{
Name = "Question",
VectorConfig = Configure.Vector("default", v => v.Text2VecWeaviate()),
GenerativeConfig = Configure.Generative.Cohere(),
Properties =
[
Property.Text("question", tokenization: PropertyTokenization.Lowercase),
Property.Text("answer", tokenization: PropertyTokenization.Whitespace),
],
}
);import weaviate from 'weaviate-client';
const client: WeaviateClient = await weaviate.connectToWeaviateCloud(
'WEAVIATE_INSTANCE_URL', { // Replace WEAVIATE_INSTANCE_URL with your instance URL
authCredentials: new weaviate.ApiKey('WEAVIATE_INSTANCE_API_KEY'),
}
)
// Define the 'Question' collection
const collectionObj = {
name: 'Question',
properties: [
{
name: 'question',
dataType: 'text' as const,
description: 'Category of the question' as const,
tokenization: 'lowercase' as const,
vectorizePropertyName: true,
},
{
name: 'answer',
dataType: 'text' as const,
description: 'The question',
tokenization: 'whitespace' as const,
vectorizePropertyName: false,
}
],
vectorizers: weaviate.configure.vectorizer.text2VecOpenAI(),
generative: weaviate.configure.generative.openAI()
}
// Add the class to the schema
const newCollection = await client.collections.create(collectionObj){/*
package main
import (
"context"
"fmt"
"github.com/weaviate/weaviate-go-client/v5/weaviate"
"github.com/weaviate/weaviate/entities/models"
)
func main() {
cfg := weaviate.Config{
Host: "WEAVIATE_INSTANCE_URL/", // Replace with your URL
Scheme: "https",
}
client, err := weaviate.NewClient(cfg)
if err != nil {
panic(err)
}
// we will create the class "Question"
classObj := &models.Class{
Class: "Question",
Description: "Information from a Jeopardy! question", // description of the class
Properties: []*models.Property{
{
DataType: []string{"string"},
Description: "The question",
Name: "question",
},
{
DataType: []string{"string"},
Description: "The answer",
Name: "answer",
},
},
}
// add the schema
err := client.Schema().ClassCreator().WithClass(classObj).Do(context.Background())
if err != nil {
panic(err)
}
// get the schema
schema, err := client.Schema().Getter().Do(context.Background())
if err != nil {
panic(err)
}
// print the schema
fmt.Printf("%v", schema)
}/} {/
# Replace ${WEAVIATE_INSTANCE_URL} with your instance URL.
curl \
-X POST \
-H "Content-Type: application/json" \
-d '{
"class": "Question",
"description": "Information from a Jeopardy! question",
"properties": [
{
"dataType": ["text"],
"description": "The question",
"name": "question"
},
{
"dataType": ["text"],
"description": "The answer",
"name": "answer"
}
]
}' \
https://${WEAVIATE_INSTANCE_URL}/v1/schema
curl https://${WEAVIATE_INSTANCE_URL}/v1/schema*/}
Cross-references
Section titled “Cross-references”This is also where you would specify cross-references, which are a special type of property that links to another collection.
Cross-references can be very useful for creating relationships between objects. For example, you might have a Movie collection with a withActor cross-reference property that points to the Actor collection. This will allow you to retrieve relevant actors for each movie.
However, cross-references can be costly in terms of performance. Use them sparingly. Additionally, cross-reference properties do not affect the object's vector. So if you want the related properties to be considered in a vector search, they should be included in the object's vectorized properties.
You can find examples of how to define and use cross-references here.
Vectorizer and generative modules
Section titled “Vectorizer and generative modules”Each collection can be configured with a vectorizer and a generative module. The vectorizer is used to generate vectors for each object and also for any un-vectorized queries, and the generative module is used to perform retrieval augmented generation (RAG) queries.
If you are not sure where to start, modules that integrate with popular API-based model providers such as Cohere or OpenAI are good starting points. You can find a list of available model integrations here.
Multi-tenancy settings
Section titled “Multi-tenancy settings”Starting from version v1.20.0, each collection can be configured as a multi-tenancy collection. This allows separation of data between tenants, typically end-users, at a much lower overhead than creating separate collections for each tenant.
This is useful if you want to use Weaviate as a backend for a multi-tenant (e.g. SaaS) application, or if data isolation is required for any other reason.
questions = client.collections.create( name="Question", vector_config=wvc.config.Configure.Vectors.text2vec_openai(), # Set the vectorizer to "text2vec-openai" to use the OpenAI API for vector-related operations generative_config=wvc.config.Configure.Generative.cohere(), # Set the generative module to "generative-cohere" to use the Cohere API for RAG properties=[ wvc.config.Property( name="question", data_type=wvc.config.DataType.TEXT, ), wvc.config.Property( name="answer", data_type=wvc.config.DataType.TEXT, ), ], multi_tenancy_config=wvc.config.Configure.multi_tenancy(enabled=True), # Enable multi-tenancy)client.collections.create("Question", col -> col.vectorConfig(VectorConfig.text2vecWeaviate()) .generativeModule(Generative.cohere()) .properties(Property.text("question"), Property.text("answer")) .multiTenancy(c -> c.autoTenantCreation(true)) // Enable multi-tenancy);await client.Collections.Create( new CollectionCreateParams { Name = "Question", VectorConfig = Configure.Vector("default", v => v.Text2VecWeaviate()), GenerativeConfig = Configure.Generative.Cohere(), Properties = [Property.Text("question"), Property.Text("answer")], MultiTenancyConfig = new MultiTenancyConfig { Enabled = true, AutoTenantCreation = true, }, // Enable multi-tenancy });import weaviate from 'weaviate-client';
const client: WeaviateClient = await weaviate.connectToWeaviateCloud(
'WEAVIATE_INSTANCE_URL', { // Replace WEAVIATE_INSTANCE_URL with your instance URL
authCredentials: new weaviate.ApiKey('WEAVIATE_INSTANCE_API_KEY'),
}
)
// Define the 'Question' class
const collectionObj = {
name: 'Question',
properties: [
{
name: 'question',
dataType: 'text' as const,
description: 'Category of the question',
tokenization: 'lowercase' as const,
vectorizePropertyName: true,
},
{
name: 'answer',
dataType: 'text' as const,
description: 'The question',
tokenization: 'whitespace' as const,
vectorizePropertyName: false,
}
],
vectorizers: weaviate.configure.vectorizer.text2VecOpenAI(),
generative: weaviate.configure.generative.openAI(),
multiTenancy: weaviate.configure.multiTenancy({enabled: true})
}
// Add the class to the schema
const newCollection = await client.collections.create(collectionObj){/*
package main
import (
"context"
"fmt"
"github.com/weaviate/weaviate-go-client/v5/weaviate"
"github.com/weaviate/weaviate/entities/models"
)
func main() {
cfg := weaviate.Config{
Host: "WEAVIATE_INSTANCE_URL/", // Replace WEAVIATE_INSTANCE_URL with your instance URL
Scheme: "https",
}
client, err := weaviate.NewClient(cfg)
if err != nil {
panic(err)
}
// we will create the class "Question"
classObj := &models.Class{
Class: "Question",
Description: "Information from a Jeopardy! question", // description of the class
Properties: []*models.Property{
{
DataType: []string{"string"},
Description: "The question",
Name: "question",
},
{
DataType: []string{"string"},
Description: "The answer",
Name: "answer",
},
},
}
// add the schema
err := client.Schema().ClassCreator().WithClass(classObj).Do(context.Background())
if err != nil {
panic(err)
}
// get the schema
schema, err := client.Schema().Getter().Do(context.Background())
if err != nil {
panic(err)
}
// print the schema
fmt.Printf("%v", schema)
}/} {/
# Replace WEAVIATE_INSTANCE_URL with your instance URL
curl \
-X POST \
-H "Content-Type: application/json" \
-d '{
"class": "Question",
"description": "Information from a Jeopardy! question",
"properties": [
{
"dataType": ["text"],
"description": "The question",
"name": "question"
},
{
"dataType": ["text"],
"description": "The answer",
"name": "answer"
}
]
}' \
https://WEAVIATE_INSTANCE_URL/v1/schema
curl https://WEAVIATE_INSTANCE_URL/v1/schema*/}
Index settings
Section titled “Index settings”Weaviate uses two types of indexes: vector indexes and inverted indexes. Vector indexes are used to store and organize vectors for fast vector similarity-based searches. Inverted indexes are used to store data for fast filtering and keyword searches.
The default vector index type is HNSW. The other options are flat, which is suitable for small collections, such as those in a multi-tenancy collection, or dynamic, which starts as a flat index before switching to an HNSW index if its size grows beyond a predetermined threshold.
questions = client.collections.create( name="Question", vector_config=wvc.config.Configure.Vectors.text2vec_openai( name="default", # Set the name of the vector configuration # Configure the vector index vector_index_config=wvc.config.Configure.VectorIndex.hnsw( # Or `flat` or `dynamic` distance_metric=wvc.config.VectorDistances.COSINE, quantizer=wvc.config.Configure.VectorIndex.Quantizer.bq(), ), ), # Set the vectorizer to "text2vec-openai" to use the OpenAI API for vector-related operations generative_config=wvc.config.Configure.Generative.cohere(), # Set the generative module to "generative-cohere" to use the Cohere API for RAG properties=[ wvc.config.Property( name="question", data_type=wvc.config.DataType.TEXT, ), wvc.config.Property( name="answer", data_type=wvc.config.DataType.TEXT, ), ], # Configure the inverted index inverted_index_config=wvc.config.Configure.inverted_index( index_null_state=True, index_property_length=True, index_timestamps=True, ),)client.collections.create("Question", col -> col.vectorConfig(VectorConfig.text2vecWeaviate("default", // Set the name of the vector configuration vc -> vc .vectorIndex(Hnsw.of(hnsw -> hnsw.distance(Distance.COSINE))) // Configure the vector index .quantization(Quantization.rq()) // Enable vector compression (quantization) )) .generativeModule(Generative.cohere()) .properties(Property.text("question"), Property.text("answer")) // Configure the inverted index .invertedIndex(iic -> iic.indexNulls(true) .indexPropertyLength(true) .indexTimestamps(true)));await client.Collections.Create( new CollectionCreateParams { Name = "Question", VectorConfig = Configure.Vector( "default", v => v.Text2VecWeaviate(), new VectorIndex.HNSW { Distance = VectorIndexConfig.VectorDistance.Cosine, // Configure the vector index Quantizer = new VectorIndex.Quantizers.BQ(), // Enable vector compression (quantization) } ), GenerativeConfig = Configure.Generative.Cohere(), Properties = [Property.Text("question"), Property.Text("answer")], // Configure the inverted index InvertedIndexConfig = new InvertedIndexConfig { IndexNullState = true, IndexPropertyLength = true, IndexTimestamps = true, }, });import weaviate from 'weaviate-client';
import { vectorizer, generative, configure, dataType } from 'weaviate-client';
const client: WeaviateClient = await weaviate.connectToWeaviateCloud(
'WEAVIATE_INSTANCE_URL', { // Replace WEAVIATE_INSTANCE_URL with your instance URL
authCredentials: new weaviate.ApiKey('WEAVIATE_INSTANCE_API_KEY'),
}
)
// Define the 'Question' class
const collectionObj = {
name: 'Question',
properties: [
{
name: 'question',
dataType: 'text' as const,
description: 'Category of the question',
tokenization: 'lowercase' as const,
vectorizePropertyName: true,
},
{
name: 'answer',
dataType: 'text' as const,
description: 'The question',
tokenization: 'whitespace' as const,
vectorizePropertyName: false,
}
],
vectorizers: vectorizer.text2VecOpenAI({
vectorIndexConfig: configure.vectorIndex.hnsw({ // Or `flat` or `dynamic`
distanceMetric: 'cosine',
quantizer: configure.vectorIndex.quantizer.bq(),
})
}),
generative: generative.openAI(),
invertedIndex: configure.invertedIndex({
indexNullState: true,
indexPropertyLength: true,
indexTimestamps: true,
}),
}
// Add the class to the schema
const newCollection = await client.collections.create(collectionObj){/*
package main
import (
"context"
"fmt"
"github.com/weaviate/weaviate-go-client/v5/weaviate"
"github.com/weaviate/weaviate/entities/models"
)
func main() {
cfg := weaviate.Config{
Host: "WEAVIATE_INSTANCE_URL/", // Replace with the URL
Scheme: "https",
}
client, err := weaviate.NewClient(cfg)
if err != nil {
panic(err)
}
// we will create the class "Question"
classObj := &models.Class{
Class: "Question",
Description: "Information from a Jeopardy! question", // description of the class
Properties: []*models.Property{
{
DataType: []string{"string"},
Description: "The question",
Name: "question",
},
{
DataType: []string{"string"},
Description: "The answer",
Name: "answer",
},
},
}
// add the schema
err := client.Schema().ClassCreator().WithClass(classObj).Do(context.Background())
if err != nil {
panic(err)
}
// get the schema
schema, err := client.Schema().Getter().Do(context.Background())
if err != nil {
panic(err)
}
// print the schema
fmt.Printf("%v", schema)
}/} {/
# Replace ${WEAVIATE_INSTANCE_URL} with your instance URL.
curl \
-X POST \
-H "Content-Type: application/json" \
-d '{
"class": "Question",
"description": "Information from a Jeopardy! question",
"properties": [
{
"dataType": ["text"],
"description": "The question",
"name": "question"
},
{
"dataType": ["text"],
"description": "The answer",
"name": "answer"
}
]
}' \
https://${WEAVIATE_INSTANCE_URL}/v1/schema
curl https://${WEAVIATE_INSTANCE_URL}/v1/schema*/}
Replication and sharding settings
Section titled “Replication and sharding settings”Replication
Section titled “Replication”Replication settings determine how many copies of the data are stored. For example, a replication setting of 3 means that each object is stored on 3 different replicas. This is important for providing redundancy and fault tolerance in production. (The default replication factor is 1.)
This goes hand-in-hand with consistency settings, which determine how many replicas must respond before an operation is considered successful.
We recommend that you read the concepts page on replication for information on how replication works in Weaviate. To specify a replication factor, follow this how-to.
Sharding
Section titled “Sharding”Sharding settings determine how each collection is sharded and distributed across nodes. This is not a setting that is typically changed, but you can use it to control how many shards are created in a cluster, and how many virtual shards are created per physical shard (read more here).
Collection aliases
Section titled “Collection aliases”Collection aliases are alternative names (pointers) for Weaviate collections that allow you to reference a collection by multiple names. When you query using an alias, Weaviate automatically routes the request to the target collection. You can set up collection aliases programmatically through client libraries or by using the REST endpoints.
Collection & property names
Section titled “Collection & property names”Collection names always start with a capital letter. Properties always begin with a small letter. You can use PascalCase class names, and property names allow underscores. Read more here.
Further resources
Section titled “Further resources”The following resources include more detailed information on collection definition settings and how to use them:
- Reference: Configuration - Collection definition: A reference of all available collection definition settings.
- How-to: Manage collections: Code examples for creating and managing collections, including how to configure various settings using client libraries.
- Reference: REST - Schema : A reference of all available collection definition settings for the REST API.
Questions and feedback
Section titled “Questions and feedback”Have a question or feedback? Here's how to reach us.