Empty Embeddings when using Weaviate and Indexing API

[rlnasuti]

Hello!

Pertinent packages:

• Langchain v0.0.295
• Weaviate-client v3.24.1

I'm trying to use Weaviate with the Indexing API. I'm having an issue and was hoping someone might know whether this is a bug or a PEBCAK.

Basically, I setup a weaviate database locally and tried to use the indexing API to populate it. It worked - sort of - but I appear to be missing all of my vector embeddings. When I check the contents of the database I see entries like this:

{
  "class": "_index",
  "creationTimeUnix": 1695172478326,
  "id": "0053cef8-0720-5da0-a5a4-dd1566fe0f04",
  "lastUpdateTimeUnix": 1695172478326,
  "properties": {
    "page_content": "populated like i want",
    "metadata1": "good",
    "metadata2": "it's here",
    "metadata3": "all good",
    "metadata4": "yup"
  },
  "vectorWeights": null
}

What's confusing me is the vectorweights. I expected that to be OpenAI embeddings. They were retrieved - i saw the requests and responses coming out of the API - but it looks like they weren't stored.

Here's the code I'm using:

embeddings = OpenAIEmbeddings(
            openai_api_base=OPENAI_DEPLOYMENT_ENDPOINT,
            deployment_id=OPENAI_EMBEDDING_DEPLOYMENT_NAME,
            model=OPENAI_EMBEDDING_DEPLOYMENT_NAME,
            chunk_size=1,
            openai_api_type="azure",
        )

# create the vector store - url defined in .env
client = weaviate.Client(url=os.getenv("WEAVIATE_URL"))
vectorstore = Weaviate(
    client = client,
    index_name=collection_name, 
    text_key="page_content",
    embedding=embeddings
)

# create the record manager to track entries into the vectorstore
namespace = f"weaviate/{collection_name}"
record_manager = SQLRecordManager(
    namespace, db_url="sqlite:///record_manager_cache.sql"
)
record_manager.create_schema()

# Read the Parquet file
df = pd.read_parquet('raw_data/a_parquet_file.parq')

# Initialize an empty list to store the documents
documents = []

print("Extracting data from Parquet file")
# Loop through each row in the Parquet file
for idx, row in df.iterrows():
    # Check if all pertinent fields exist and 'IsValid' is True
    if pd.notna(row['HTMLContentDisplay']) and row['IsValid'] == 1:
        # Convert the HTML to text
        soup = BeautifulSoup(row['HTMLContentDisplay'], 'html.parser')
        content = soup.get_text()

        # Create the metadata
        metadata = {
            'metadata1': row['metadata1'],
            'metadata2': row['metadata2'],
            'metadata3': row['metadata3'],
            'metadata4': row['metadata4']
        }

        # Create the document
        doc = Document(page_content=content, metadata=metadata)

        # Add the document to the list
        documents.append(doc)

print("Updating vector database")
result = index(
    documents,
    record_manager,
    vectorstore,
    cleanup=None,
    source_id_key="source",
)

print(result)

Can anyone see something that I'm doing wrong that would result in the vector weights not being stored? Or have I stumbled across a bug specific to how the indexing API interacts with Weaviate? Any help would be very much appreciated.

[dosubot[bot]]

🤖

Hello,

Based on the context provided, it seems like the issue you're facing might be related to how the add_texts method in the Weaviate class is handling the case when self._embedding is None.

In the add_texts method, the embeddings are calculated only if self._embedding is not None. If self._embedding is None, the vector parameter in the batch.add_data_object method is set to None, which means no vector embeddings are stored in the Weaviate database.

Here is the relevant code:

params = {
    "uuid": _id,
    "data_object": data_properties,
    "class_name": index_name,
}
if embeddings is not None:
    params["vector"] = embeddings[i]

batch.add_data_object(**params)

To fix this issue, you need to ensure that self._embedding is not None when calling the add_texts method. If self._embedding is None, you should initialize it with an appropriate embedding model before calling add_texts.

Alternatively, if you want Weaviate to create the embeddings, make sure Weaviate is installed with a vectorizer module like text2vec-contextionary.

You can find more details in the Weaviate class source code.

As for your question about chunk_size=1, I couldn't find a specific reason within the repository. However, the chunk_size parameter typically controls the number of texts that are processed at once. Setting it to 1 means that each text is processed individually, which could potentially slow down the embedding process but shouldn't affect the storage of embeddings.

I hope this helps! Let me know if you have any other questions.

Sources

• Embedding Seems Unsuccessful for Chroma + OpenAI
• libs/langchain/langchain/vectorstores/weaviate.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[rlnasuti]

@dosu-bot I don't think that's it. Notice how I pass in an embeddings class to the weaviate vectorstore? This should ensure that self._embedding is not none.

Here's the pertinent code from weaviate.py:

def __init__(
    self,
    client: Any,
    index_name: str,
    text_key: str,
    embedding: Optional[Embeddings] = None,
    attributes: Optional[List[str]] = None,
    relevance_score_fn: Optional[
        Callable[[float], float]
    ] = _default_score_normalizer,
    by_text: bool = True,
):
    """Initialize with Weaviate client."""
    try:
        import weaviate
    except ImportError:
        raise ImportError(
            "Could not import weaviate python package. "
            "Please install it with `pip install weaviate-client`."
        )
    if not isinstance(client, weaviate.Client):
        raise ValueError(
            f"client should be an instance of weaviate.Client, got {type(client)}"
        )
    self._client = client
    self._index_name = index_name
    self._embedding = embedding

Note that self._embedding is set to the embedding parameter in the init method. I'm passing in an instance of OpenAIEmbeddings when I declare the Weaviate vectorstore.

[dosubot[bot]]

🤖

Hello,

Based on the information you've provided and the context from the LangChain repository, there are a few things that might be causing the issue.

Firstly, it's important to note that the 'None' value for the 'vectorWeights' field when printing the Weaviate database collection is actually expected behavior. The Weaviate database doesn't store the embeddings directly. Instead, it keeps a compressed representation of these embeddings. When you print the collection, it shows 'None' for the embeddings because the actual embeddings aren't directly accessible. As for the similarity search, it operates by comparing these compressed representations. When you execute a similarity search, Weaviate decompresses the stored representations to compute the similarity scores. So, even though you don't see the embeddings when you print the collection, they are there in a compressed form and are utilized for similarity searches. You can refer to this issue for more details.

Secondly, the OpenAIEmbeddings class in LangChain uses environment variables to configure the OpenAI API. If these environment variables are not set correctly, the embeddings might not be generated as expected. Please ensure that the following environment variables are set correctly:

• OPENAI_API_KEY: The API key for OpenAI.
• OPENAI_API_BASE: The base URL for the OpenAI API.
• OPENAI_API_TYPE: The type of OpenAI API. If it's set to "azure", "azure_ad", or "azuread", the default API version will be "2022-12-01" and the default chunk size will be 16. Otherwise, the default API version will be an empty string and the default chunk size will be 1000.
• OPENAI_PROXY: The proxy for the OpenAI API.
• OPENAI_API_VERSION: The version of the OpenAI API. If it's not provided, the default API version will be used.
• OPENAI_ORGANIZATION: The organization for the OpenAI API.

Also, make sure that the OpenAI Python package is installed. If it's not, an ImportError will be raised. You can install it with pip install openai.

You can refer to the source code of the OpenAIEmbeddings class in LangChain for more details: libs/langchain/langchain/embeddings/openai.py

I hope this helps! If you have any other questions or need further clarification, feel free to ask.

Sources

• Embedding Seems Unsuccessful for Chroma + OpenAI
• libs/langchain/langchain/vectorstores/weaviate.py
• libs/langchain/langchain/embeddings/openai.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.