How to delete all nodes with specific metadata

[notnecessarilylost]

I am ingesting Google Docs from a target folder into an index and storing in qdrant running in local embedded mode.

As my user takes files out of the target Google Docs folder I want to delete any nodes that belong to the file that was removed, so they don't show up in future queries.

I store the doc_name and doc_id in metadata for each node. How do I return the nodes with a specific doc_id and then subsequently delete them?

[dosubot[bot]]

Hey @notnecessarilylost! I'm here to help you with any bugs, questions, or contributions you have. Let's solve this together!

To delete all nodes with a specific doc_id from Qdrant, ensuring they don't appear in future queries, you can use the delete method provided in the QdrantVectorStore class. Here's how you can do it:

def delete(self, ref_doc_id: str, **delete_kwargs: Any) -> None:
    """
    Delete nodes using with ref_doc_id.

    Args:
        ref_doc_id (str): The doc_id of the document to delete.

    """
    from qdrant_client.http import models as rest

    self._client.delete(
        collection_name=self.collection_name,
        points_selector=rest.Filter(
            must=[
                rest.FieldCondition(
                    key=DOCUMENT_ID_KEY, match=rest.MatchValue(value=ref_doc_id)
                )
            ]
        ),
    )

If you prefer to perform this operation asynchronously, you can use the adelete method:

async def adelete(self, ref_doc_id: str, **delete_kwargs: Any) -> None:
    """
    Asynchronous method to delete nodes using with ref_doc_id.

    Args:
        ref_doc_id (str): The doc_id of the document to delete.

    """
    await self._aclient.delete(
        collection_name=self.collection_name,
        points_selector=rest.Filter(
            must=[
                rest.FieldCondition(
                    key=DOCUMENT_ID_KEY, match=rest.MatchValue(value=ref_doc_id)
                )
            ]
        ),
    )

These methods delete nodes from the Qdrant collection by filtering them based on the doc_id. You need to provide the ref_doc_id of the document you want to delete. This ensures that nodes with the specified doc_id are removed and will not appear in future queries ^[1][2].

To continue talking to Dosu, mention @dosu.

[notnecessarilylost]

Thank you @dosu

Do I also need to delete the nodes from my Index in memory? How would I do that?

[dosubot[bot]]

Yes, you need to delete the nodes from your Index in memory if they are no longer needed. You can use the delete_nodes method to remove nodes from the in-memory index. Here's how you can do it:

def test_delete_nodes(
    docs: List[Document],
    index: DocumentSummaryIndex,
) -> None:
    """Test delete node."""
    nodes = list(index.index_struct.node_id_to_summary_id.keys())
    index.delete_nodes([nodes[0], nodes[1]])

    assert len(index.ref_doc_info) == 2
    assert len(index.index_struct.doc_id_to_summary_id) == 2
    assert len(index.index_struct.node_id_to_summary_id) == 2
    assert len(index.index_struct.summary_id_to_node_ids) == 2

    assert len(index.vector_store._data.embedding_dict) == 2  # type: ignore

This method will remove the specified nodes from the index. If you need to delete nodes based on metadata, such as doc_id, you can use the delete_nodes method with metadata filters:

def delete_nodes(
    self,
    node_ids: Optional[List[str]] = None,
    filters: Optional[MetadataFilters] = None,
    **delete_kwargs: Any,
) -> None:
    """Deletes nodes.

    Args:
        node_ids (Optional[List[str]], optional): IDs of nodes to delete. Defaults to None.
        filters (Optional[MetadataFilters], optional): Metadata filters. Defaults to None.
    """
    if not node_ids and not filters:
        return

    query = {"query": {"bool": {"filter": []}}}
    if node_ids:
        query["query"]["bool"]["filter"].append({"terms": {"_id": node_ids or []}})

    if filters:
        query["query"]["bool"]["filter"].extend(self._parse_filters(filters))

    self._os_client.delete_by_query(index=self._index, body=query, refresh=True)

You can use the filters argument to specify metadata conditions, such as doc_id, to delete the corresponding nodes ^[1][2].

To continue talking to Dosu, mention @dosu.