How to read a milvus index that already contains my data? and how to add data to it?

[ywancit]

I can only find documnets about milvus like

storage_context = StorageContext.from_defaults(vector_store=vector_store)
index = VectorStoreIndex([Document(text="The number that is being searched for is ten.")], storage_context=storage_context)

But I want to just read the milvus index I created before without passing [Document(text="The number that is being searched for is ten.")], so I use from_vector_store

index = VectorStoreIndex.from_vector_store(vector_store=vector_store)
query_engine = index.as_query_engine()
response = query_engine.query("The number that is being searched for is?")
print(response)

It works when my index is created by llama_index, but when my milvus index created by my own code and data with fields other than
doc_id_field
embedding_field
but also desc , source ,
the query raise error
ValueError: Node content not found in metadata dict.

[dosubot[bot]]

Hey @ywancit! I'm here to assist you with any issues you may be experiencing. I can help troubleshoot bugs, answer questions, and guide you through contributing. Let's tackle this together!

To read a Milvus index that already contains your data and add data to it, you need to ensure that the metadata dictionary contains the necessary fields when converting the metadata back to a node. Specifically, the _node_content and _node_type fields must be present in the metadata dictionary.

Here's how you can handle this:

1. Ensure Metadata Fields are Set When Adding Nodes:

   When adding nodes to the Milvus index, make sure the _node_content and _node_type fields are included in the metadata dictionary. This can be done in the add method:

   def add(self, nodes: List[BaseNode], **add_kwargs: Any) -> List[str]:
       """Add the embeddings and their nodes into Milvus.
   
       Args:
           nodes (List[BaseNode]): List of nodes with embeddings to insert.
   
       Raises:
           MilvusException: Failed to insert data.
   
       Returns:
           List[str]: List of ids inserted.
       """
       insert_list = []
       insert_ids = []
   
       # Process the data we are going to insert
       for node in nodes:
           entry = node_to_metadata_dict(node)
           entry[MILVUS_ID_FIELD] = node.node_id
           entry[self.embedding_field] = node.embedding
           entry["_node_content"] = node.content  # Ensure _node_content is set
           entry["_node_type"] = node.node_type  # Ensure _node_type is set
   
           insert_ids.append(node.node_id)
           insert_list.append(entry)
   
       # Insert the data into Milvus
       self.collection.insert(insert_list)
       self.collection.flush()
       self._create_index_if_required()
       logger.debug(
           f"Successfully inserted embeddings into: {self.collection_name} "
           f"Num Inserted: {len(insert_list)}"
       )
       return insert_ids

2. Convert Metadata Back to Node:

   When reading the Milvus index, ensure that the metadata dictionary contains the necessary fields to convert it back to a node:

   nodes = []
   similarities = []
   ids = []
   # Parse the results
   for hit in res[0]:
       if not self.text_key:
           node = metadata_dict_to_node(
               {
                   "_node_content": hit.entity.get("_node_content"),
                   "_node_type": hit.entity.get("_node_type"),
               }
           )
       else:
           try:
               text = hit.entity.get(self.text_key)
           except Exception:
               raise ValueError(
                   "The passed in text_key value does not exist "
                   "in the retrieved entity."
               )
   
           metadata = {key: hit.entity.get(key) for key in self.output_fields}
           node = TextNode(text=text, metadata=metadata)
   
       nodes.append(node)
       similarities.append(hit.distance)
       ids.append(hit.id)
   
   return VectorStoreQueryResult(nodes=nodes, similarities=similarities, ids=ids)

3. Ensure Metadata Dictionary Contains Necessary Fields:

   The node_to_metadata_dict function ensures that the _node_content and _node_type fields are included in the metadata dictionary:

   def node_to_metadata_dict(
       node: BaseNode,
       remove_text: bool = False,
       text_field: str = DEFAULT_TEXT_KEY,
       flat_metadata: bool = False,
   ) -> Dict[str, Any]:
       """Common logic for saving Node data into metadata dict."""
       node_dict = node.dict()
       metadata: Dict[str, Any] = node_dict.get("metadata", {})
   
       if flat_metadata:
           _validate_is_flat_dict(metadata)
   
       # store entire node as json string - some minor text duplication
       if remove_text:
           node_dict[text_field] = ""
   
       # remove embedding from node_dict
       node_dict["embedding"] = None
   
       # dump remainder of node_dict to json string
       metadata["_node_content"] = json.dumps(node_dict)
       metadata["_node_type"] = node.class_name()
   
       # store ref doc id at top level to allow metadata filtering
       metadata["document_id"] = node.ref_doc_id or "None"
       metadata["doc_id"] = node.ref_doc_id or "None"
       metadata["ref_doc_id"] = node.ref_doc_id or "None"
   
       return metadata

By ensuring that the _node_content and _node_type fields are included in the metadata dictionary when adding nodes, you can avoid the "ValueError: Node content not found in metadata dict" error when querying the Milvus index ^[1][2][3].

To continue talking to Dosu, mention @dosu.