:::callout{intent="info" title="Compression by Default"} Starting with `v1.33`, you can set a default quantization for new collections using the [`DEFAULT_QUANTIZATION`](../database-configuration/overview.md#DEFAULT_QUANTIZATION) environment variable. This variable is not set by default, meaning no quantization is applied unless you explicitly configure it. When set (e.g., to 8-bit [RQ quantization](compression-rq-compression.md)), all newly created collections will use that quantization setting. Note that once set on a collection, quantization can't be disabled. Default quantization won't be applied to a collection if the index type isn't supported (for example PQ and SQ aren't supported for the flat index). ::: [**Product quantization (PQ)**](../concepts/vector-quantization.md#product-quantization) is a form of data compression for vectors. PQ reduces the HNSW index's memory footprint so you can work with larger datasets. For a discussion of how PQ saves memory, see [Product quantization](../concepts/vector-quantization.md#product-quantization). PQ makes tradeoffs between recall, performance, and memory usage. This means a PQ configuration that reduces memory may also reduce recall. There are similar trade-offs when you use HNSW without PQ. If you use PQ compression, you should also tune HNSW so that they compliment each other. To configure HNSW, see [Configuration: Vector index](../reference-configuration/indexing-vector-index.md). ## Enable PQ compression PQ is configured at a collection level. There are two ways to enable PQ compression: - [Use AutoPQ to enable PQ compression](compression-pq-compression.md#configure-autopq). - [Manually enable PQ compression](compression-pq-compression.md#manually-configure-pq). ## Configure AutoPQ For new collections, use AutoPQ. AutoPQ automates triggering of the PQ training step based on the size of the collection. ### 1. Set the environment variable AutoPQ requires asynchronous indexing. - **Open-source Weaviate users**: To enable AutoPQ, set the environment variable `ASYNC_INDEXING=true` and restart your Weaviate instance. - [**Weaviate Cloud (WCD)**](/go/console?utm_content=howto/) users: Enable async indexing through the WCD Console and restart your Weaviate instance. ### 2. Configure PQ To configure PQ in a collection, use the [PQ parameters](compression-pq-compression.md#pq-parameters). :::code-group{sync="languages"} ```python title="Python" {7} from weaviate.classes.config import Configure client.collections.create( name="Question", vector_config=Configure.Vectors.text2vec_openai( name="default", quantizer=Configure.VectorIndex.Quantizer.pq(training_limit=50000), # Set the threshold to begin training ), ) ``` ```typescript title="JavaScript/TypeScript" import { configure } from 'weaviate-client'; ``` ```java title="Java" {3-4} client.collections.create("Question", col -> col.vectorConfig(VectorConfig.text2vecOpenAi("default", vc -> vc .quantization(Quantization.pq(pq -> pq.trainingLimit(50000))) // Set the threshold to begin training ))); ``` ```csharp title="C#" {10-18} await client.Collections.Create( new CollectionCreateParams { Name = "Question", VectorConfig = Configure.Vector( "default", v => v.Text2VecTransformers(), index: new VectorIndex.HNSW { Quantizer = new VectorIndex.Quantizers.PQ { TrainingLimit = 50000, // Set the threshold to begin training Encoder = new VectorIndex.Quantizers.PQ.EncoderConfig { Type = VectorIndex.Quantizers.EncoderType.Tile, Distribution = VectorIndex.Quantizers.DistributionType.Normal, }, }, } ), Properties = [ Property.Text("question"), Property.Text("answer"), Property.Text("category"), ], } ); ``` ::: ### 3. Load your data Load your data. You do not have to load an initial set of training data. AutoPQ creates the PQ codebook when the object count reaches the training limit. By default, the training limit is 100,000 objects per shard. ## Manually configure PQ You can manually enable PQ on an existing collection. After PQ is enabled, Weaviate trains the PQ codebook. Before you enable PQ, verify that the training set has 100,000 objects per shard. To manually enable PQ, follow these steps: - Phase One: Create a codebook - [Define a collection without PQ](compression-pq-compression.md#1-define-a-collection-without-pq) - [Load some training data](compression-pq-compression.md#2-load-training-data) - [Enable and train PQ](compression-pq-compression.md#3-enable-pq-and-create-the-codebook) - Phase Two: Load the rest of your data - [Load the rest of your data](compression-pq-compression.md#4-load-the-rest-of-your-data) :::callout{intent="tip" title="How large should the training set be?"} We suggest 10,000 to 100,000 objects per shard. ::: Weaviate [logs a message](#check-the-system-logs) when PQ is enabled and another message when vector compression is complete. Do not import the rest of your data until the initial training step is complete. Follow these steps to manually enable PQ. ### 1. Define a collection without PQ [Create a collection](../how-to-manage-collections/collection-operations.md#create-a-collection) without specifying a quantizer. :::code-group{sync="languages"} ```python title="Python" from weaviate.classes.config import Configure client.collections.create( name="Question", description="A Jeopardy! question", vector_config=Configure.Vectors.text2vec_openai( name="default", ), generative_config=Configure.Generative.openai(), ) ``` ```typescript title="JavaScript/TypeScript" const collection = await client.collections.create({ name: 'Question', vectorizer: weaviate.configure.vectors.text2VecOpenAI({ sourceProperties: ["title"], }) }) ``` ```go title="Go" // Create initial collection without PQ initialClass := &models.Class{ Class: className, Vectorizer: "text2vec-openai", Properties: []*models.Property{ {Name: "question", DataType: []string{"text"}}, {Name: "answer", DataType: []string{"text"}}, }, VectorIndexConfig: map[string]interface{}{ "distance": "cosine", }, } err = client.Schema().ClassCreator(). WithClass(initialClass). Do(context.Background()) ``` ```java title="Java" client.collections.create("Question", col -> col.description("A Jeopardy! question") .properties(Property.text("question"), Property.text("answer")) .vectorConfig(VectorConfig.text2vecOpenAi( vc -> vc.quantization(Quantization.uncompressed())))); ``` ```csharp title="C#" await client.Collections.Create( new CollectionCreateParams { Name = "Question", Description = "A Jeopardy! question", VectorConfig = Configure.Vector("default", v => v.Text2VecTransformers()), Properties = [ Property.Text("question"), Property.Text("answer"), Property.Text("category"), ], } ); ``` ::: ### 2. Load training data [Add objects](../how-to-manage-objects/import.md) that will be used to train PQ. Weaviate will use the greater of the training limit, or the collection size, to train PQ. We recommend loading a representative sample such that the trained centroids are representative of the entire dataset. From `v1.27.0`, Weaviate uses a sparse [Fisher-Yates algorithm](https://en.wikipedia.org/wiki/Fisher–Yates_shuffle) to select the training set from the available objects when PQ is enabled manually. Nonetheless, it is still recommended to load a representative sample of the data so that the trained centroids are representative of the entire dataset. ### 3. Enable PQ and create the codebook Update your collection definition to enable PQ. Once PQ is enabled, Weaviate trains the codebook using the training data. PQ relies on a codebook to compress the original vectors. The codebook defines "centroids" that are used to calculate the compressed vector. If you are not using [AutoPQ](compression-pq-compression.md#configure-autopq), you must have some vectors loaded before you enable PQ so Weaviate can define the centroids. We recommend a training set size of between 10,000 and 100,000 for each shard. To enable PQ, update your collection definition as shown below. For additional configuration options, see the [PQ parameter table](compression-pq-compression.md#pq-parameters). :::code-group{sync="languages"} ```python title="Python" from weaviate.classes.config import Reconfigure jeopardy = client.collections.use("Question") jeopardy.config.update( vector_config=Reconfigure.Vectors.update( name="default", vector_index_config=Reconfigure.VectorIndex.hnsw( quantizer=Reconfigure.VectorIndex.Quantizer.pq( training_limit=50000 # Default: 100000 ), ) ) ) ``` ```typescript title="JavaScript/TypeScript" const collection = client.collections.use(collectionName); await collection.config.update({ vectorizers: weaviate.reconfigure.vectors.update({ vectorIndexConfig: weaviate.reconfigure.vectorIndex.hnsw({ quantizer: weaviate.reconfigure.vectorIndex.quantizer.pq({ trainingLimit: 50000 }) }) }) }) ``` ```go title="Go" // Get the existing collection configuration class, err := client.Schema().ClassGetter(). WithClassName(className).Do(context.Background()) if err != nil { log.Fatalf("get class for vec idx cfg update: %v", err) } // Get the current vector index configuration cfg := class.VectorIndexConfig.(map[string]interface{}) // Add PQ configuration to enable product quantization cfg["pq"] = map[string]interface{}{ "enabled": true, "trainingLimit": 100000, // Optional: number of vectors to use for training "segments": 96, // Optional: number of segments for product quantization } // Update the class configuration class.VectorIndexConfig = cfg // Apply the updated configuration to the collection err = client.Schema().ClassUpdater(). WithClass(class).Do(context.Background()) if err != nil { log.Fatalf("update class to use pq: %v", err) } ``` ```java title="Java" collection.config .update(c -> c.vectorConfig(VectorConfig.text2vecOpenAi(vc -> vc .quantization(Quantization.pq(pq -> pq.trainingLimit(50000)))))); ``` ```csharp title="C#" await collection.Config.Update(c => { var vectorConfig = c.VectorConfig["default"]; vectorConfig.VectorIndexConfig.UpdateHNSW(h => h.Quantizer = new VectorIndex.Quantizers.PQ { TrainingLimit = 50000, Encoder = new VectorIndex.Quantizers.PQ.EncoderConfig { Type = VectorIndex.Quantizers.EncoderType.Tile, Distribution = VectorIndex.Quantizers.DistributionType.Normal, }, } ); }); ``` ::: ### 4. Load the rest of your data Once the [codebook has been trained](#3-enable-pq-and-create-the-codebook), you may continue to add data as per normal. Weaviate compresses the new data when it adds it to the database. If you already have data in your Weaviate instance when you create the codebook, Weaviate automatically compresses the remaining objects (the ones after the initial training set). ## PQ parameters You can configure PQ compression by setting the following parameters at the collection level. | Parameter | Type | Default | Details | | :-------------- | :------ | :----------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `enabled` | boolean | `false` | Enable PQ when `true`.

The Python client v4 does not use the `enabled` parameter. To enable PQ with the v4 client, set a `quantizer` in the collection definition. | | `trainingLimit` | integer | 100000 | The maximum number of objects, per shard, used to fit the centroids. Larger values increase the time it takes to fit the centroids. Larger values also require more memory. | | `segments` | integer | -- | The number of segments to use. The number of vector dimensions must be evenly divisible by the number of segments.

Starting in `v1.23`, Weaviate uses the number of dimensions to optimize the number of segments. | | `centroids` | integer | 256 | The number of centroids to use (max: 256).

We generally recommend you do not change this value.

Due to the data structure used, smaller centroid value will not result in smaller vectors, but may result in faster compression at cost of recall. | | `encoder` | string | `kmeans` | Encoder specification. There are two encoders. You can specify the `type` of encoder as either `kmeans` (default) or `tile`. | | `distribution` | string | `log-normal` | Encoder distribution type. Only used with the `tile` encoder. If you use the `tile` encoder, you can specify the `distribution` as `log-normal` (default) or `normal`. | ## Additional tools and considerations ### Change the codebook training limit For most use cases, 100,000 objects is an optimal training size. There is little benefit to increasing `trainingLimit`. If you do increase `trainingLimit`, the training period will take longer. You could also have memory problems if you set a high `trainingLimit`. If you have a small dataset and wish to enable compression, consider using [binary quantization (BQ)](compression-bq-compression.md). BQ is a simpler compression method that does not require training. ### Check the system logs When compression is enabled, Weaviate logs diagnostic messages like these. ```bash pq-conf-demo-1 | {"action":"compress","level":"info","msg":"switching to compressed vectors","time":"2023-11-13T21:10:52Z"} pq-conf-demo-1 | {"action":"compress","level":"info","msg":"vector compression complete","time":"2023-11-13T21:10:53Z"} ``` If you use `docker-compose` to run Weaviate, you can get the logs on the system console. ```bash docker compose logs -f --tail 10 weaviate ``` You can also view the log file directly. Check `docker` to get the file location. ```bash docker inspect --format='{{.LogPath}}' ``` ### Review the current `pq` configuration To review the current `pq` configuration, you can retrieve it as shown below. :::code-group{sync="languages"} ```python title="Python" jeopardy = client.collections.use("Question") config = jeopardy.config.get() pq_config = config.vector_config["default"].vector_index_config.quantizer # print some of the config properties print(f"Encoder: { pq_config.encoder }") print(f"Training: { pq_config.training_limit }") print(f"Segments: { pq_config.segments }") print(f"Centroids: { pq_config.centroids }") ``` ```typescript title="JavaScript/TypeScript" const collection = client.collections.use(collectionName); ``` ```go title="Go" // Verify the PQ configuration was applied updatedClass, err := client.Schema().ClassGetter(). WithClassName(className).Do(context.Background()) if err != nil { log.Fatalf("get class to verify vec idx cfg changes: %v", err) } cfg = updatedClass.VectorIndexConfig.(map[string]interface{}) log.Printf("pq config: %v", cfg["pq"]) ``` ```java title="Java" CollectionHandle> jeopardy = client.collections.use("Question"); Optional configOpt = jeopardy.config.get(); System.out.println(configOpt); ``` ```csharp title="C#" var jeopardy = client.Collections.Use("Question"); var config = await jeopardy.Config.Get(); Console.WriteLine( JsonSerializer.Serialize(config, new JsonSerializerOptions { WriteIndented = true }) ); ``` ::: ### Multiple vector embeddings (named vectors) Collections can have multiple [named vectors](../reference-configuration/collections.md#multiple-vector-embeddings-named-vectors). The vectors in a collection can have their own configurations, and compression must be enabled independently for each vector. Every vector is independent and can use [PQ](compression-pq-compression.md), [BQ](compression-bq-compression.md), [RQ](compression-rq-compression.md), [SQ](compression-sq-compression.md), or no compression. ### Multi-vector embeddings (ColBERT, ColPali, etc.) :::callout{intent="info" title="Added in `v1.30`"} ::: Multi-vector embeddings (implemented through models like ColBERT, ColPali, or ColQwen) represent each object or query using multiple vectors instead of a single vector. Just like with single vectors, multi-vectors support [PQ](compression-pq-compression.md), [BQ](compression-bq-compression.md), [RQ](compression-rq-compression.md), [SQ](compression-sq-compression.md), or no compression. During the initial search phase, compressed vectors are used for efficiency. However, when computing the `MaxSim` operation, uncompressed vectors are utilized to ensure more precise similarity calculations. This approach balances the benefits of compression for search efficiency with the accuracy of uncompressed vectors during final scoring. ## Further resources - [Starter guides: Compression](../starter-guides/managing-resources-compression.md) - [Reference: Vector index](../reference-configuration/indexing-vector-index.md) - [Concepts: Vector quantization](../concepts/vector-quantization.md) - [Concepts: Vector index](../indexing/vector-index.md) ## Questions and feedback Have a question or feedback? Here's how to reach us. ::::card-grid :::card{title="Community Forum" href="https://forum.weaviate.io/c/support" icon="messages-square"} Ask questions and connect with other developers on our **Community forum**. ::: :::card{title="Support" href="/guides/support-overview" icon="life-buoy"} Weaviate Cloud user or customer? Find the right channel on the **Support page**. ::: :::: ## Related pages - [Agents](./agents-index.md) - [AI-assisted Weaviate code generation](./ai-assisted-vibe-coding-index.md) - [APIs](./apis-index.md) - [Authorization and authentication](./authorization-and-authentication-index.md) - [Benchmarks](./benchmarks-index.md) - [Best practices](./best-practices-index.md) - [Client libraries](./clients-index.md) - [Client Libraries / SDKs](./client-libraries-index.md) - [Cloud](./cloud-index.md) - [Cloud account management](./cloud-account-management-index.md) # Agent Instructions This portal answers questions programmatically. To receive a synthesized, source-cited answer instead of crawling page by page, append the `?ask=` query parameter to any page URL on this site: /guides/quickstart?ask=how+do+I+authenticate Optional parameters: - `&goal=` steers the answer toward your objective (e.g. `&goal=write+a+python+client`). - `&version=