:::callout{intent="warning" title="🚧 To be updated 🚧"} This tutorial is currently being updated to reflect the latest features and improvements in Weaviate. We appreciate your patience and invite you to check back soon for the updated content. ::: This tutorial will show you how to import a large dataset (25k articles from Wikipedia) that already includes vectors (embeddings generated by OpenAI). We will, - download and unzip a CSV file that contains the Wikipedia articles - create a Weaviate instance - create a schema - parse the file and batch import the records, with Python and JavaScript code - make sure the data was imported correctly - run a few queries to demonstrate semantic search capabilities ## Prerequisites :::callout{intent="tip" title="Prerequisites"} If you haven't yet, we recommend going through the [**Quickstart tutorial**](../quickstart/index.md) first to get the most out of this section. ::: Before you start this tutorial, make sure to have: - An [OpenAI API key](https://platform.openai.com/api-keys). Even though we already have vector embeddings generated by OpenAI, we'll need an OpenAI key to vectorize search queries, and to recalculate vector embeddings for updated object contents. - Your preferred Weaviate [client library](../client-libraries/index.md) installed. :::::accordion{title="See how to delete data from previous tutorials (or previous runs of this tutorial)."} You can delete any unwanted collection(s), along with the data that they contain. :::callout{intent="warning" title="Deleting a collection also deletes its objects"} When you **delete a collection, you delete all associated objects**! Be very careful with deletes on a production database and anywhere else that you have important data. ::: This code deletes a collection and its objects. ::::tabs{sync="languages"} {/\* :::tab{title="GraphQL"} ```graphql ``` ::: \*/} :::tab{title="Python"} ```python # collection_name can be a string ("Article") or a list of strings (["Article", "Category"]) client.collections.delete( collection_name ) # THIS WILL DELETE THE SPECIFIED COLLECTION(S) AND THEIR OBJECTS # Note: you can also delete all collections in the Weaviate instance with: # client.collections.delete_all() ``` ::: :::tab{title="JavaScript/TypeScript"} ```ts // delete collection "Article" - THIS WILL DELETE THE COLLECTION AND ALL ITS DATA await client.collections.delete('Article') // you can also delete all collections of a cluster // await client.collections.deleteAll() ``` ::: :::tab{title="Go"} ```go className := "YourClassName" // delete the class if err := client.Schema().ClassDeleter().WithClassName(className).Do(context.Background()); err != nil { // Weaviate will return a 400 if the class does not exist, so this is allowed, only return an error if it's not a 400 if status, ok := err.(*fault.WeaviateClientError); ok && status.StatusCode != http.StatusBadRequest { panic(err) } } ``` ::: :::tab{title="Java"} ```java client.collections.delete(collectionName); ``` ::: :::tab{title="Curl"} ```bash curl \ -X DELETE \ https://WEAVIATE_INSTANCE_URL/v1/schema/YourClassName # Replace WEAVIATE_INSTANCE_URL with your instance URL ``` ::: :::tab{title="C#"} ```csharp await client.Collections.Delete(collectionName); ``` ::: :::: ::::: ## Download the dataset We will use this [Simple English](https://simple.wikipedia.org/wiki/Simple_English_Wikipedia) Wikipedia [dataset hosted by OpenAI](https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip) (\~700MB zipped, 1.7GB CSV file) that includes vector embeddings. These are the columns of interest, where `content_vector` is a [vector embedding](https://weaviate.io/blog/vector-embeddings-explained) with [1536 elements (dimensions)](https://openai.com/index/new-and-improved-embedding-model), generated using OpenAI's `text-embedding-ada-002` model: | id | url | title | text | content\_vector | | -- | --------------------------------------- | ----- | ------------------------------------------ | --------------------------------------- | | 1 | https://simple.wikipedia.org/wiki/April | April | "April is the fourth month of the year..." | \[-0.011034, -0.013401, ..., -0.009095] | If you haven't already, make sure to download the dataset and unzip the file. You should end up with `vector_database_wikipedia_articles_embedded.csv` in your working directory. The records are mostly (but not strictly) sorted by title. Download Wikipedia dataset ZIP ## Create a Weaviate instance We can create a Weaviate instance locally using the [embedded](../installation/installation-guides-embedded.md) option on Linux (transparent and fastest), Docker on any OS (fastest import and search), or in the cloud using the Weaviate Cloud (easiest setup, but importing may be slower due to the network speed). Each option is explained on its [Installation](../installation/index.md) page. :::callout{intent="warning" title="text2vec-openai"} If using the Docker option, make sure to select "With Modules" (instead of standalone), and the `text2vec-openai` module when using the Docker configurator, at the "Vectorizer & Retriever Text Module" step. At the "OpenAI Requires an API Key" step, you can choose to "provide the key with each request", as we'll do so in the next section. ::: ## Connect to the instance and OpenAI Add the OpenAI API key to the client so you can use the OpenAI vectorizer API when you send queries to Weaviate. The API key can be provided to Weaviate as an environment variable, or in the HTTP header with every request. This example adds the key to the client. The client sends the key with every request as a part of the HTTP request header. :::code-group{sync="languages"} ```python title="Python" import weaviate # Instantiate the client with the auth config client = weaviate.Client( url="https://WEAVIATE_INSTANCE_URL", # Replace with your Weaviate endpoint auth_client_secret=weaviate.auth.AuthApiKey(api_key="YOUR-WEAVIATE-API-KEY"), # Replace with your Weaviate instance API key additional_headers={ "X-OpenAI-Api-Key": "YOUR-OPENAI-API-KEY", }, ) ``` ```go title="Go" package main import ( "context" "fmt" "github.com/weaviate/weaviate-go-client/v5/weaviate" ) // Instantiate the client with the auth config cfg := weaviate.Config{ Host:"", // Replace WEAVIATE_INSTANCE_URL with your instance URL Scheme: "http", AuthConfig: auth.ApiKey{Value: "YOUR-WEAVIATE-API-KEY"}, // Replace with your Weaviate instance API key Headers: map[string]string{ "X-OpenAI-Api-Key": "YOUR-OPENAI-API-KEY", }, } client, err := weaviate.NewClient(cfg) if err != nil{ fmt.Println(err) } ``` ```bash title="Curl" # Replace WEAVIATE_INSTANCE_URL with your instance URL curl https://WEAVIATE_INSTANCE_URL/v1/meta \ -H 'Content-Type: application/json' \ -H "X-OpenAI-Api-Key: YOUR-OPENAI-API-KEY" \ -H "Authorization: Bearer YOUR-WEAVIATE-API-KEY" | jq ``` ::: ## Create the schema The [schema](../starter-guides/managing-collections.md) defines the data structure for objects in a given Weaviate class. We'll create a schema for a Wikipedia `Article` class mapping the CSV columns, and using the [text2vec-openai vectorizer](../how-to-manage-collections/vector-config.md#specify-a-vectorizer). The schema will have two properties: - `title` - article title, not vectorized - `content` - article content, corresponding to the `text` column from the CSV As of Weaviate 1.18, the `text2vec-openai` vectorizer uses by default the same model as the OpenAI dataset, `text-embedding-ada-002`. To make sure the tutorial will work the same way if this default changes (i.e. if OpenAI releases an even better-performing model and Weaviate switches to it as the default), we'll configure the schema vectorizer explicitly to use the same model: ```json { "moduleConfig": { "text2vec-openai": { "model": "ada", "modelVersion": "002", "type": "text" } } } ``` Another detail to be careful about is how exactly we store the `content_vector` embedding. [Weaviate vectorizes entire objects](../reference-configuration/indexing-vector-index.md#configure-semantic-indexing) (not properties), and it includes by default the class name in the string serialization of the object it will vectorize. Since OpenAI has provided embeddings only for the `text` (content) field, we need to make sure Weaviate vectorizes an `Article` object the same way. That means we need to disable including the class name in the vectorization, so we must set `vectorizeClassName: false` in the `text2vec-openai` section of the `moduleConfig`. Together, these schema settings will look like this: :::code-group{sync="languages"} ```python title="Python" # client.schema.delete_all() # ⚠️ uncomment to start from scratch by deleting ALL data # ===== Create Article class for the schema ===== article_class = { "class": "Article", "description": "An article from the Simple English Wikipedia data set", "vectorizer": "text2vec-openai", "moduleConfig": { # Match how OpenAI created the embeddings for the `content` (`text`) field "text2vec-openai": { "model": "ada", "modelVersion": "002", "type": "text", "vectorizeClassName": False } }, "properties": [ { "name": "title", "description": "The title of the article", "dataType": ["text"], # Don't vectorize the title "moduleConfig": {"text2vec-openai": {"skip": True}} }, { "name": "content", "description": "The content of the article", "dataType": ["text"], } ] } # Add the Article class to the schema client.schema.create_class(article_class) print('Created schema'); ``` ::: To quickly check that the schema was created correctly, you can navigate to `/v1/schema`. For example in the Docker installation scenario, go to `http://localhost:8080/v1/schema` or run, ```bash curl -s http://localhost:8080/v1/schema | jq ``` :::callout{intent="tip" title="jq"} The [`jq`](https://stedolan.github.io/jq/) command used after `curl` is a handy JSON preprocessor. When simply piping some text through it, `jq` returns the text pretty-printed and syntax-highlighted. ::: ## Import the articles We're now ready to import the articles. For maximum performance, we'll load the articles into Weaviate via [batch import](../how-to-manage-objects/import.md). :::code-group{sync="languages"} ```python title="Python" # ===== Import data ===== # Settings for displaying the import progress counter = 0 interval = 100 # print progress every this many records # Create a pandas dataframe iterator with lazy-loading, # so we don't load all records in RAM at once. import pandas as pd csv_iterator = pd.read_csv( 'vector_database_wikipedia_articles_embedded.csv', usecols=['id', 'url', 'title', 'text', 'content_vector'], chunksize=100, # number of rows per chunk # nrows=350 # optionally limit the number of rows to import ) # Iterate through the dataframe chunks and add each CSV record to the batch import ast client.batch.configure(batch_size=100) # Configure batch with client.batch as batch: for chunk in csv_iterator: for index, row in chunk.iterrows(): properties = { "title": row.title, "content": row.text, "url": row.url } # Convert the vector from CSV string back to array of floats vector = ast.literal_eval(row.content_vector) # Add the object to the batch, and set its vector embedding batch.add_data_object(properties, "Article", vector=vector) # Calculate and display progress counter += 1 if counter % interval == 0: print(f"Imported {counter} articles...") print(f"Finished importing {counter} articles.") ``` ::: ### Checking the import went correctly Two quick sanity checks that the import went as expected: 1. Get the number of articles 2. Get 5 articles If your instance runs in Weaviate Cloud, open the [Explorer tool](../other-tools/explorer-tool.md) in the console and select the `Article` collection. The Explorer tool shows the total number of objects in the collection, and lists the articles with their properties, including `title` and `url`. You can also run this GraphQL query against your instance: ```graphql query { Aggregate { Article { meta { count } } } Get { Article(limit: 5) { title url } } } ``` To run it against a locally hosted instance, post it to the `/v1/graphql` endpoint: ```bash curl -s -X POST http://localhost:8080/v1/graphql \ -H "Content-Type: application/json" \ -d '{"query": "{ Aggregate { Article { meta { count } } } Get { Article(limit: 5) { title url } } }"}' | jq ``` You should see the `Aggregate.Article.meta.count` field equal to the number of articles you've imported (e.g. 25,000), as well as five random articles with their `title` and `url` fields. ## Queries Now that we have the articles imported, let's run some queries! ### nearText The [`nearText` filter](../apis/graphql-search-operators.md#neartext) lets us search for objects close (in vector space) to the vector embedding of one or more concepts. For example, the vector for the query "modern art in Europe" would be close to the vector for the article [Documenta](https://simple.wikipedia.org/wiki/Documenta), which describes > "one of the most important exhibitions of modern art in the world... \[taking] place in Kassel, Germany". ::::tabs{sync="languages"} :::tab{title="GraphQL"} ```graphql { Get { Article( nearText: {concepts: ["modern art in Europe"]}, limit: 1 ) { title content } } } ``` ::: :::tab{title="Python"} ```python import weaviate import json client = weaviate.Client( url="https://WEAVIATE_INSTANCE_URL/", # replace with your Weaviate endpoint additional_headers={ "X-OpenAI-Api-Key": "YOUR-OPENAI-API-KEY" # Replace with your API key } ) nearText = {"concepts": ["modern art in Europe"]} result = ( client.query .get("Article", ["title", "content"]) .with_near_text(nearText) .with_limit(1) .do() ) print(json.dumps(result, indent=4)) ``` ::: {/\* :::tab{title="Go"} ```go ``` ::: \*/} :::tab{title="Curl"} ```bash echo '{ "query": "{ Get { Article( nearText: { concepts: [\"modern art in Europe\"], }, limit: 1 ) { title } } }" }' | curl \ -X POST \ -H 'Content-Type: application/json' \ -H 'Authorization: Bearer learn-weaviate' \ -H "X-OpenAI-Api-Key: $OPENAI_API_KEY" \ -d @- \ https://edu-demo.weaviate.network/v1/graphql ``` ::: :::: ### hybrid While `nearText` uses dense vectors to find objects similar in meaning to the search query, it does not perform very well on keyword searches. For example, a `nearText` search for "jackfruit" in this Simple English Wikipedia dataset, will find "cherry tomato" as the top result. For these (and indeed, most) situation, we can obtain better search results by using the [`hybrid` filter](../apis/graphql-search-operators.md#hybrid), which combines dense vector search with keyword search: ::::tabs{sync="languages"} :::tab{title="GraphQL"} ```graphql { Get { Article ( hybrid: { query: "jackfruit" alpha: 0.5 # default 0.75 } limit: 3 ) { title content _additional {score} } } } ``` ::: :::tab{title="Python"} ```python result = ( client.query .get("Article", ["title", "content"]) .with_hybrid("jackfruit", alpha=0.5) # default 0.75 .with_limit(3) .do() ) print(json.dumps(result, indent=4)) ``` ::: {/\* :::tab{title="Go"} ```go hybrid := &HybridArgumentBuilder{} hybrid.WithQuery("jackfruit").WithAlpha(0.5) query := builder.WithClassName("Article").WithHybrid(hybrid).build() ``` ::: \*/} :::tab{title="Curl"} ```bash echo '{ "query": "{ Get { Article ( hybrid: { query: \"jackfruit\" alpha: 0.5 } limit: 3 ) { title _additional {score} } } }" }' | curl \ -X POST \ -H 'Content-Type: application/json' \ -H 'Authorization: Bearer learn-weaviate' \ -H "X-OpenAI-Api-Key: $OPENAI_API_KEY" \ -d @- \ https://edu-demo.weaviate.network/v1/graphql ``` ::: :::: ## Recap In this tutorial, we've learned - how to efficiently import large datasets using Weaviate batching and CSV lazy loading with `pandas` / `csv-parser` - how to import existing vectors ("Bring Your Own Vectors") - how to quickly check that all records were imported - how to use `nearText` and `hybrid` searches ## Suggested reading - [Tutorial: Schemas in detail](../starter-guides/managing-collections.md) - [How-to: Query and search](../how-to-query-search/index.md) - [Tutorial: Introduction to modules](tutorials-modules.md) ## Questions and feedback Have a question or feedback? Here's how to reach us. ::::card-grid :::card{title="Community Forum" href="https://forum.weaviate.io/c/support" icon="messages-square"} Ask questions and connect with other developers on our **Community forum**. ::: :::card{title="Support" href="/guides/support-overview" icon="life-buoy"} Weaviate Cloud user or customer? Find the right channel on the **Support page**. ::: :::: ## Related pages - [Agents](./agents-index.md) - [AI-assisted Weaviate code generation](./ai-assisted-vibe-coding-index.md) - [APIs](./apis-index.md) - [Authorization and authentication](./authorization-and-authentication-index.md) - [Benchmarks](./benchmarks-index.md) - [Best practices](./best-practices-index.md) - [Client libraries](./clients-index.md) - [Client Libraries / SDKs](./client-libraries-index.md) - [Cloud](./cloud-index.md) - [Cloud account management](./cloud-account-management-index.md) # Agent Instructions This portal answers questions programmatically. To receive a synthesized, source-cited answer instead of crawling page by page, append the `?ask=` query parameter to any page URL on this site: /guides/quickstart?ask=how+do+I+authenticate Optional parameters: - `&goal=` steers the answer toward your objective (e.g. `&goal=write+a+python+client`). - `&version=