How load a Summary index from storage

[zapatacc]

Maybe it is a dummy question, but I'm facing a blocker.

I have two collections saved in a Weaviate vector Database. These collections were created with the Weaviate llamaindex integration using a simple SentenceSplitter. I added metadata to the nodes (Summary, Questions, Title, Keywords, etc).

For query/retrieve time, I used a query engine built with a QueryEngine on top of a VectorStoreIndex, but now, I want to use too a SummaryIndex to have the possibility of query with specific Q&A questions or summary of the documents.

My questions are:

1. How should I store the nodes for SummaryIndex? What kind of Database can I use? Can I use Weaviate but without using the embedding field? Has llamaindex any integration for this end?
2. Once the nodes for summary are already stored in a DB, how can I connect the SummaryIndex object to this nodes? This object doesn'n have a from_vector_store or something similar to connect that DB with the SummaryIndex.

[dosubot[bot]]

To address your questions about using a SummaryIndex with nodes stored in a database like Weaviate, follow these steps:

1. Storing Nodes for SummaryIndex in Weaviate

You can store nodes in Weaviate without using the embedding field by creating objects for each node and including the necessary metadata. Here’s an example:

import weaviate

client = weaviate.Client("http://localhost:8080")

def store_nodes_in_weaviate(nodes):
    for node in nodes:
        client.data_object.create(
            {
                "node_id": node.node_id,
                "content": node.content,
                "summary": node.summary,
                "questions": node.questions,
                "title": node.title,
                "keywords": node.keywords,
                # Add other node attributes as needed
            },
            "Node"
        )

2. Connecting SummaryIndex to Nodes in Weaviate

To connect the SummaryIndex object to nodes stored in Weaviate, you need to implement methods to retrieve these nodes and modify the SummaryIndex class to use these methods during query time.

Retrieve Nodes from Weaviate

def get_nodes_from_weaviate(node_ids):
    nodes = []
    for node_id in node_ids:
        result = client.data_object.get_by_id(node_id)
        if result:
            node_data = result['properties']
            node = BaseNode(
                node_id=node_data['node_id'],
                content=node_data['content'],
                summary=node_data['summary'],
                questions=node_data['questions'],
                title=node_data['title'],
                keywords=node_data['keywords']
            )
            nodes.append(node)
    return nodes

Modify SummaryIndex Class

Modify the SummaryIndex class to fetch nodes from Weaviate during query time:

class SummaryIndex(BaseIndex[IndexList]):
    def _build_index_from_nodes(self, nodes, show_progress=False):
        index_struct = IndexList()
        for n in nodes:
            index_struct.add_node(n)
        return index_struct

    def _insert(self, nodes, **insert_kwargs):
        for n in nodes:
            self._index_struct.add_node(n)
            store_nodes_in_weaviate([n])  # Store node in Weaviate

    def _delete_node(self, node_id, **delete_kwargs):
        cur_node_ids = self._index_struct.nodes
        cur_nodes = get_nodes_from_weaviate(cur_node_ids)
        nodes_to_keep = [n for n in cur_nodes if n.node_id != node_id]
        self._index_struct.nodes = [n.node_id for n in nodes_to_keep]

    @property
    def ref_doc_info(self):
        node_doc_ids = self._index_struct.nodes
        nodes = get_nodes_from_weaviate(node_doc_ids)
        all_ref_doc_info = {}
        for node in nodes:
            ref_node = node.source_node
            if not ref_node:
                continue
            ref_doc_info = self.docstore.get_ref_doc_info(ref_node.node_id)
            if not ref_doc_info:
                continue
            all_ref_doc_info[ref_node.node_id] = ref_doc_info
        return all_ref_doc_info

Example Usage

Here’s how you can use the modified SummaryIndex with Weaviate:

# Assuming you have a list of nodes
nodes = [...]  # Your list of BaseNode objects

# Store nodes in Weaviate
store_nodes_in_weaviate(nodes)

# Create a SummaryIndex and connect it to Weaviate
summary_index = SummaryIndex(nodes=nodes)

# Now you can use summary_index for querying

For a practical example of using Weaviate with LlamaIndex, you can refer to the WeaviateDemo notebook ^[1].

This approach ensures that nodes are stored in Weaviate and can be retrieved and used by the SummaryIndex for querying.

Details

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

[zapatacc]

@logan-markewich thoughts?

[logan-markewich]

@zapatacc a summary index cannot use a vector store. You'd have to configure the index store and docstore in the storage context (there are many integrations here, redis, postgres, firestore, azure data tables, mongodb)