Should temp files be deleted after ImportDocumentAsync completed?

[luismanez]

I've been doing some tests with memory.ImportDocumentAsync (either using Local file system and Azure Blob storage). In both cases, the temp files are not deleted when the pipeline is successfully finished.
As they are temp files, shouldn't be deleted by default after pipeline complete?

According to the source code, the BaseOrchestrator class has a ``CleanUpAfterCompletionAsync``` method, but that method is only executed if you add the delete_document handler to the pipeline. Besides, the code in the orchestrators leads to misunderstanding:

if (pipeline.Complete)
        {
            this.Log.LogInformation("Pipeline '{0}/{1}' complete", pipeline.Index, pipeline.DocumentId);

            // Save the pipeline status. If this fails, the system should retry the current step.
            await this.UpdatePipelineStatusAsync(pipeline, cancellationToken).ConfigureAwait(false);

            await this.CleanUpAfterCompletionAsync(pipeline, cancellationToken).ConfigureAwait(false);
        }

It says that when the pipeline is complete, it will do a CleanUp, which is exactly what I'd want and expect. However, as said before, that method only runs when the DeleteDocument or DeleteIndex handlers are in the pipeline:

if (pipeline.IsDocumentDeletionPipeline())
// stuff

if (pipeline.IsIndexDeletionPipeline())
// stuff

Is there any way to really CleanUp those temp files?

Thanks!

[dluc]

hi @luismanez the files in storage, currently, are not considered "temp files". The plan (still under development) is to allow users to open the relevant files when receiving an "answer". The "citations" coming back with an answer will allow to reach and download/open the files stored. This is still TBD, e.g. it should be configurable, and we have to consider the impact of duplicating files. Depending on how one uses the service, the files could be considered temporary, an unwanted duplication, or useful.

About the "CleanUp" step, that has a different purpose, and it's mostly internal, I would suggest ignoring it and not relying on that. Its goal is keeping the storage consistent when a deletion is requested (in your case you're not asking for a deletion, but uploading information).

Considering we'd like to leave this open and configurable, please feel free to say how you would like to configure this and how you're using citations (if at all), so we can design accordingly :-)

Thanks!

[luismanez]

Hi @dluc, many thanks for the detailed explanation. You made a great point here. Let me explain our scenario (and I know this is not a "one size fits all", but I think is a pretty common one nowadays.

We are indexing documents coming from different sources (mainly SharePoint document libraries). Our solution runs on Azure, so we're using Azure Cognitive Search and Blob storage (our "Indexer" is an Azure Function). We are in your "unwanted duplication" point. For citations, we want to point to the "original" file in SharePoint, and to do so, we are indexing a custom Tag with the Full SP URL. We don't want to duplicate those files. It's not just about increasing cost, but also about Security and compliance. Docs remain in M365, though we index "chunks" in Azure Search, but Search service is better seen to the eyes of Security teams than Blob storages :)
BTW, we have a 1-1 relationship between Memory Document and File (i.e: we don't index 2 .pdf files in the same Memory document).

In this scenario, I'd like to have some flag in the library to say "hey, delete all the physical files once this document is indexed".
In the meantime, we will configure a policy in our blob storage to delete any file that was last updated a week ago (something like that).

Also, when deleting a document, files are deleted, but there´s still the pipeline_status.json file. I'd like to have another flag to delete that one.

Last comment not related with files. We also miss the chance to tune the final prompt is sent to OpenAI. For instance, currently the library assumes that you only want the answer based on the Facts, but this won't always the case), or what about sending the chat history...
It would also be nice to have a Stream option for the AskAsync method 😃
What we are doing is to call SearchAsync, compose the Prompt ourselves and calling OpenAI API with the stream option.

Let me know any question. Happy to share more details and help to improve the library.

Thanks!

[dluc]

thanks for the notes!

Also, when deleting a document, files are deleted, but there´s still the pipeline_status.json file. I'd like to have another flag to delete that one.

That's strange, is it happening with the latest code? Deleting that file is exactly the purpose of CleanUpAfterCompletionAsync, when deleting a document, although the implementation uses a best effort approach, without retry logic.

Last comment not related with files. We also miss the chance to tune the final prompt is sent to OpenAI. For instance, currently the library assumes that you only want the answer based on the Facts, but this won't always the case), or what about sending the chat history...

agreed, customizing prompts and other params is in our roadmap :-) (and using the Search API is the current workaround, glad to see you're on that path). Things will improve over time.

Thank you!

[dluc]

@luismanez FYI it's now possible to customize prompts via configuration, see #110

Let me know if we need to keep this issue open, I believe the issue with deletions was addressed in a recent bugfix. Thanks!

[luismanez]

Hi @dluc , thanks for the heads up! ... I like the IPromptProvider approach.

About this issue, it was more about having the option, so that when a Document is imported/indexed, all the generated files (extraction, chunks, summaries...) are deleted from the Storage (disk, azure storage...). As discussed, in our approach, the docs are stored in SharePoint, and the chunks are indexed in Az Search, so we don't need anything else, and having the generated physical files in Az Storage, is more a security/compliance concern than a benefit.

I think there's still no option to do so, but happy to prove me wrong 😄

[dluc]

#177 merged - thank you!!