Index only new documents and avoid indexing entire document repository

[TheBugKing]

I want to understand how can I index only new documents and avoid indexing entire document repository.
as per #354, it states that only new documents are added to summaries but when I did it practically it does not seems to be the case when:

1. using a custom base_dir, example: "output/files/artifacts", everytime when indexer runs it ovewrites previous indexed data.
2. when using default paths "output/${timestamp}/artifacts", it creates new indexed folder by time stamps, which is file. Lets say when we had a indexed folder for previous run and and new indexed folder for new run. How do we perform the searched over the the entire available indexes, by default seaches only pick up the last modified index. which means it conply has data related to new documents only.

some help would be appriciated. I do not want to reindex entire documents every time a new document is added due to cost implications

[natoverse]

Tracking incremental indexing with #741