Add support for queue prioritization or per-index queues

[westdavidr]

In our implementation of Kernel Memory, we generate numerous indexes from various sources such as SharePoint, Azure DevOps, web crawling, and in-chat documents.

We are facing challenges in sharing the queues for all these indexes, especially when they are under heavy load during synchronization tasks.

To address this, I propose adding a "PrefixConfiguration" to the queue options. This would allow us to set specific prefixes for different categories. For example, we could set a prefix for the chat documents index (chatdocs) to prepend the queue names with chat-, resulting in queue names like chat-extract, chat-gen_embeddings, etc.

What are your thoughts on this approach?

[dluc]

You should be able to do this today with some configuration, without code changes. During the ingestion you can provide a list of steps, which is a list of strings. Each string maps to a handler and to a queue. Via configuration, you should try registering the same handler with multiple names, so to have different queues.

For instance, this is the default config:

"extract": { "Assembly": "Microsoft.KernelMemory.Core.dll", "Class": "Microsoft.KernelMemory.Handlers.TextExtractionHandler" },
"partition": { "Assembly": "Microsoft.KernelMemory.Core.dll", "Class": "Microsoft.KernelMemory.Handlers.TextPartitioningHandler" },
"gen_embeddings": { "Assembly": "Microsoft.KernelMemory.Core.dll", "Class":"Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler" },
"save_records": {"Assembly": "Microsoft.KernelMemory.Core.dll","Class": "Microsoft.KernelMemory.Handlers.SaveRecordsHandler"},
"summarize": {"Assembly": "Microsoft.KernelMemory.Core.dll","Class": "Microsoft.KernelMemory.Handlers.SummarizationHandler"},

and you could change it to:

"chat-extract": { "Assembly": "Microsoft.KernelMemory.Core.dll", "Class": "Microsoft.KernelMemory.Handlers.TextExtractionHandler" },
"doc-extract": { "Assembly": "Microsoft.KernelMemory.Core.dll", "Class": "Microsoft.KernelMemory.Handlers.TextExtractionHandler" },
"email-extract": { "Assembly": "Microsoft.KernelMemory.Core.dll", "Class": "Microsoft.KernelMemory.Handlers.TextExtractionHandler" },
etc.

then set steps: [ "chat-extract", "chat-...", "chat-..." ] during the uploada