Why index clear all documents on vector db?

[kientv]

The docs_transformed is not change, just re-index with current data:

indexing_stats = index(
        docs_transformed,
        record_manager,
        vectorstore,
        cleanup="scoped_full",
        source_id_key="source",
        force_update=(os.environ.get("FORCE_UPDATE") or "false").lower() == "true",
    )
logging.info(f"Indexing stats: {indexing_stats}")

Result:
Indexing stats: {'num_added': 0, 'num_updated': 0, 'num_skipped': 23, 'num_deleted': 23}

[dosubot[bot]]

When using the cleanup="scoped_full" mode in the index function, it deletes documents associated with a specific source, as identified by the source_id_key. This means that during the indexing process, any existing documents with the same source identifier are removed and replaced with the new set of documents from that source. However, if the documents in docs_transformed have not changed, they should be classified as 'num_skipped', not 'num_deleted'.

In your case, it seems that all documents were deleted, which might indicate that the source_id_key is not correctly identifying the source, or there might be an issue with how the source IDs are being managed. Ensure that the source_id_key is correctly set and that the source identifiers in your documents are consistent with those in the vector store ^[1].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[dosubot[bot]]

When using clean mode: full, the behavior you're observing is expected. This mode is designed to delete all existing documents before re-adding them, even if the data hasn't changed. This is why, during the first indexing, all documents are deleted and skipped, and during the second indexing, they are deleted and then re-added. This behavior is confirmed by the test test_index_simple_delete_full, which shows that with cleanup="full", documents are first deleted and then re-added, resulting in the stats you see ^[1].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[kientv]

@dosu
When using clean mode: full, the behavior you're observing is expected. This mode is designed to delete all existing documents before re-adding them, even if the data hasn't changed. This is why, during the first indexing, all documents are deleted and skipped, and during the second indexing, they are deleted and then re-added. This behavior is confirmed by the test test_index_simple_delete_full, which shows that with cleanup="full", documents are first deleted and then re-added, resulting in the stats you see

But the problem is: My number vectors is zero after indexing.

[dosubot[bot]]

If your number of vectors is zero after indexing with clean mode: full, it could be because the indexing process did not add any new documents, and all existing documents were skipped. This can occur if the documents being indexed are identical to those already present in the vector store, resulting in no new additions.

In such cases, ensure that the documents are indeed being added by checking the indexing logic and verifying that the documents are not being skipped due to identical content and metadata. If you want to force updates even for unchanged documents, consider using the force_update parameter to ensure that all documents are re-indexed ^[1].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[kientv]

@dosu I have try cleanup="incremental", and run with only one document to ingest. But it's still delete after added

This is my logs:

INFO: Indexing stats: {'num_added': 1, 'num_updated': 0, 'num_skipped': 0, 'num_deleted': 1}
INFO: Number vectors: 0, Number docs: 1

I'm using from langchain.indexes import index, Maybe your indexing logic is wrong?

[dosubot[bot]]

When using cleanup="incremental", the expected behavior is to delete previous documents from the same source and add the new ones, ensuring only the latest version is retained ^[1][2]. If you're seeing that a document is added and then immediately deleted, it suggests there might be an issue with how the source IDs are being managed or identified.

Here are a few things to check:

1. Source ID Consistency: Ensure that the source_id_key is correctly extracting the source ID from the document's metadata. The source ID should be consistent and correctly match those in the vector store.

2. Document Hashing: Verify that the document's content and metadata have not changed in a way that affects their hash. Even a small change can result in a different hash, leading to unexpected deletions.

3. Indexing Logic: Review the logic in your indexing function to ensure it is correctly handling the documents and their source IDs. There might be an issue with how documents are being identified or updated.

4. Vector Store State: Check the state of the vector store before and after indexing to ensure that the documents are being indexed correctly and that the source IDs are being managed as expected.

If these checks don't resolve the issue, consider adding detailed logging to capture more information about the indexing process, which might help identify where the mismatch or error is occurring.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}