Wikipedia with custom vectors

This tutorial will show you how to import a large dataset (25k articles from Wikipedia) that already includes vectors (embeddings generated by OpenAI). We will,

• download and unzip a CSV file that contains the Wikipedia articles
• create a Weaviate instance
• create a schema
• parse the file and batch import the records, with Python and JavaScript code
• make sure the data was imported correctly
• run a few queries to demonstrate semantic search capabilities

Before you start this tutorial, make sure to have:

• An OpenAI API key. Even though we already have vector embeddings generated by OpenAI, we'll need an OpenAI key to vectorize search queries, and to recalculate vector embeddings for updated object contents.
• Your preferred Weaviate client library installed.

# collection_name can be a string ("Article") or a list of strings (["Article", "Category"])
client.collections.delete(
    collection_name
)  # THIS WILL DELETE THE SPECIFIED COLLECTION(S) AND THEIR OBJECTS

# Note: you can also delete all collections in the Weaviate instance with:
# client.collections.delete_all()

// delete collection "Article" - THIS WILL DELETE THE COLLECTION AND ALL ITS DATA
await client.collections.delete('Article')

// you can also delete all collections of a cluster
// await client.collections.deleteAll()

className := "YourClassName"

// delete the class
if err := client.Schema().ClassDeleter().WithClassName(className).Do(context.Background()); err != nil {
  // Weaviate will return a 400 if the class does not exist, so this is allowed, only return an error if it's not a 400
  if status, ok := err.(*fault.WeaviateClientError); ok && status.StatusCode != http.StatusBadRequest {
    panic(err)
  }
}

client.collections.delete(collectionName);

curl \
  -X DELETE \
  https://WEAVIATE_INSTANCE_URL/v1/schema/YourClassName  # Replace WEAVIATE_INSTANCE_URL with your instance URL

await client.Collections.Delete(collectionName);

We will use this Simple English Wikipedia dataset hosted by OpenAI (~700MB zipped, 1.7GB CSV file) that includes vector embeddings. These are the columns of interest, where content_vector is a vector embedding with 1536 elements (dimensions), generated using OpenAI's text-embedding-ada-002 model:

If you haven't already, make sure to download the dataset and unzip the file. You should end up with vector_database_wikipedia_articles_embedded.csv in your working directory. The records are mostly (but not strictly) sorted by title.

Download Wikipedia dataset ZIP

We can create a Weaviate instance locally using the embedded option on Linux (transparent and fastest), Docker on any OS (fastest import and search), or in the cloud using the Weaviate Cloud (easiest setup, but importing may be slower due to the network speed). Each option is explained on its Installation page.

Add the OpenAI API key to the client so you can use the OpenAI vectorizer API when you send queries to Weaviate.

The API key can be provided to Weaviate as an environment variable, or in the HTTP header with every request. This example adds the key to the client. The client sends the key with every request as a part of the HTTP request header.

import weaviate

# Instantiate the client with the auth config
client = weaviate.Client(
    url="https://WEAVIATE_INSTANCE_URL",  # Replace with your Weaviate endpoint
    auth_client_secret=weaviate.auth.AuthApiKey(api_key="YOUR-WEAVIATE-API-KEY"),  # Replace with your Weaviate instance API key
    additional_headers={
        "X-OpenAI-Api-Key": "YOUR-OPENAI-API-KEY",
    },
)

package main

import (
  "context"
  "fmt"
  "github.com/weaviate/weaviate-go-client/v5/weaviate"
)

// Instantiate the client with the auth config
cfg := weaviate.Config{
  Host:"",  // Replace WEAVIATE_INSTANCE_URL with your instance URL
  Scheme: "http",
  AuthConfig: auth.ApiKey{Value: "YOUR-WEAVIATE-API-KEY"}, // Replace with your Weaviate instance API key
  Headers: map[string]string{
    "X-OpenAI-Api-Key": "YOUR-OPENAI-API-KEY",
    },
}

client, err := weaviate.NewClient(cfg)
if err != nil{
  fmt.Println(err)
}

# Replace WEAVIATE_INSTANCE_URL with your instance URL

curl https://WEAVIATE_INSTANCE_URL/v1/meta \
-H 'Content-Type: application/json' \
-H "X-OpenAI-Api-Key: YOUR-OPENAI-API-KEY" \
-H "Authorization: Bearer YOUR-WEAVIATE-API-KEY" | jq

The schema defines the data structure for objects in a given Weaviate class. We'll create a schema for a Wikipedia Article class mapping the CSV columns, and using the text2vec-openai vectorizer. The schema will have two properties:

• title - article title, not vectorized
• content - article content, corresponding to the text column from the CSV

As of Weaviate 1.18, the text2vec-openai vectorizer uses by default the same model as the OpenAI dataset, text-embedding-ada-002. To make sure the tutorial will work the same way if this default changes (i.e. if OpenAI releases an even better-performing model and Weaviate switches to it as the default), we'll configure the schema vectorizer explicitly to use the same model:

{
  "moduleConfig": {
    "text2vec-openai": {
      "model": "ada",
      "modelVersion": "002",
      "type": "text"
    }
  }
}

Another detail to be careful about is how exactly we store the content_vector embedding. Weaviate vectorizes entire objects (not properties), and it includes by default the class name in the string serialization of the object it will vectorize. Since OpenAI has provided embeddings only for the text (content) field, we need to make sure Weaviate vectorizes an Article object the same way. That means we need to disable including the class name in the vectorization, so we must set vectorizeClassName: false in the text2vec-openai section of the moduleConfig. Together, these schema settings will look like this:

# client.schema.delete_all()  # ⚠️ uncomment to start from scratch by deleting ALL data

# ===== Create Article class for the schema =====
article_class = {
    "class": "Article",
    "description": "An article from the Simple English Wikipedia data set",
    "vectorizer": "text2vec-openai",
    "moduleConfig": {
        # Match how OpenAI created the embeddings for the `content` (`text`) field
        "text2vec-openai": {
            "model": "ada",
            "modelVersion": "002",
            "type": "text",
            "vectorizeClassName": False
        }
    },
    "properties": [
        {
            "name": "title",
            "description": "The title of the article",
            "dataType": ["text"],
            # Don't vectorize the title
            "moduleConfig": {"text2vec-openai": {"skip": True}}
        },
        {
            "name": "content",
            "description": "The content of the article",
            "dataType": ["text"],
        }
    ]
}

# Add the Article class to the schema
client.schema.create_class(article_class)
print('Created schema');

To quickly check that the schema was created correctly, you can navigate to <weaviate-endpoint>/v1/schema. For example in the Docker installation scenario, go to http://localhost:8080/v1/schema or run,

curl -s http://localhost:8080/v1/schema | jq

We're now ready to import the articles. For maximum performance, we'll load the articles into Weaviate via batch import.

# ===== Import data =====
# Settings for displaying the import progress
counter = 0
interval = 100  # print progress every this many records

# Create a pandas dataframe iterator with lazy-loading,
# so we don't load all records in RAM at once.
import pandas as pd
csv_iterator = pd.read_csv(
    'vector_database_wikipedia_articles_embedded.csv',
    usecols=['id', 'url', 'title', 'text', 'content_vector'],
    chunksize=100,  # number of rows per chunk
    # nrows=350  # optionally limit the number of rows to import
)

# Iterate through the dataframe chunks and add each CSV record to the batch
import ast
client.batch.configure(batch_size=100)  # Configure batch
with client.batch as batch:
  for chunk in csv_iterator:
      for index, row in chunk.iterrows():

          properties = {
              "title": row.title,
              "content": row.text,
              "url": row.url
          }

          # Convert the vector from CSV string back to array of floats
          vector = ast.literal_eval(row.content_vector)

          # Add the object to the batch, and set its vector embedding
          batch.add_data_object(properties, "Article", vector=vector)

          # Calculate and display progress
          counter += 1
          if counter % interval == 0:
              print(f"Imported {counter} articles...")
print(f"Finished importing {counter} articles.")

Two quick sanity checks that the import went as expected:

1. Get the number of articles
2. Get 5 articles

If your instance runs in Weaviate Cloud, open the Explorer tool in the console and select the Article collection. The Explorer tool shows the total number of objects in the collection, and lists the articles with their properties, including title and url.

You can also run this GraphQL query against your instance:

query {
  Aggregate { Article { meta { count } } }

  Get {
    Article(limit: 5) {
      title
      url
    }
  }
}

To run it against a locally hosted instance, post it to the /v1/graphql endpoint:

curl -s -X POST http://localhost:8080/v1/graphql \
  -H "Content-Type: application/json" \
  -d '{"query": "{ Aggregate { Article { meta { count } } } Get { Article(limit: 5) { title url } } }"}' | jq

You should see the Aggregate.Article.meta.count field equal to the number of articles you've imported (e.g. 25,000), as well as five random articles with their title and url fields.

Now that we have the articles imported, let's run some queries!

The nearText filter lets us search for objects close (in vector space) to the vector embedding of one or more concepts. For example, the vector for the query "modern art in Europe" would be close to the vector for the article Documenta, which describes

"one of the most important exhibitions of modern art in the world... [taking] place in Kassel, Germany".

{
  Get {
    Article(
      nearText: {concepts: ["modern art in Europe"]},
      limit: 1
    ) {
      title
      content
    }
  }
}

import weaviate
import json

client = weaviate.Client(
    url="https://WEAVIATE_INSTANCE_URL/",  # replace with your Weaviate endpoint
    additional_headers={
        "X-OpenAI-Api-Key": "YOUR-OPENAI-API-KEY"  # Replace with your API key
    }
)

nearText = {"concepts": ["modern art in Europe"]}

result = (
    client.query
    .get("Article", ["title", "content"])
    .with_near_text(nearText)
    .with_limit(1)
    .do()
)

print(json.dumps(result, indent=4))

echo '{
  "query": "{
    Get {
      Article(
        nearText: {
          concepts: [\"modern art in Europe\"],
        },
        limit: 1
      ) {
        title
      }
    }
  }"
}' | curl \
    -X POST \
    -H 'Content-Type: application/json' \
    -H 'Authorization: Bearer learn-weaviate' \
    -H "X-OpenAI-Api-Key: $OPENAI_API_KEY" \
    -d @- \
    https://edu-demo.weaviate.network/v1/graphql

While nearText uses dense vectors to find objects similar in meaning to the search query, it does not perform very well on keyword searches. For example, a nearText search for "jackfruit" in this Simple English Wikipedia dataset, will find "cherry tomato" as the top result. For these (and indeed, most) situation, we can obtain better search results by using the hybrid filter, which combines dense vector search with keyword search:

{
  Get {
    Article (
      hybrid: {
        query: "jackfruit"
        alpha: 0.5  # default 0.75
      }
      limit: 3
    ) {
      title
      content
      _additional {score}
    }
  }
}

result = (
    client.query
    .get("Article", ["title", "content"])
    .with_hybrid("jackfruit", alpha=0.5)  # default 0.75
    .with_limit(3)
    .do()
)

print(json.dumps(result, indent=4))

hybrid := &HybridArgumentBuilder{}
  hybrid.WithQuery("jackfruit").WithAlpha(0.5)

  query := builder.WithClassName("Article").WithHybrid(hybrid).build()

echo '{
  "query": "{
      Get {
        Article (
          hybrid: {
            query: \"jackfruit\"
            alpha: 0.5
          }
          limit: 3
        ) {
          title
          _additional {score}
      }
    }
  }"
}' | curl \
    -X POST \
    -H 'Content-Type: application/json' \
    -H 'Authorization: Bearer learn-weaviate' \
    -H "X-OpenAI-Api-Key: $OPENAI_API_KEY" \
    -d @- \
    https://edu-demo.weaviate.network/v1/graphql

In this tutorial, we've learned

• how to efficiently import large datasets using Weaviate batching and CSV lazy loading with pandas / csv-parser
• how to import existing vectors ("Bring Your Own Vectors")
• how to quickly check that all records were imported
• how to use nearText and hybrid searches

• Tutorial: Schemas in detail
• How-to: Query and search
• Tutorial: Introduction to modules

Have a question or feedback? Here's how to reach us.