experimenting uploading files as stream

[markat1]

I tried see wether I could upload a simple pdf into my memory kernel service, but no luck so far.

Following is my approach + errors I'm getting in the memory kernel service - any thoughts on what I'm doing wrong?

I tried uploading the following file as a upload stream:
mydocs-NASA-news.pdf

[HttpPost]
public async Task<ActionResult<IList<UploadResult>>> PostFile([FromForm] IEnumerable<IFormFile> files)
{

    foreach (var file in files)
    {

        try
        {
            var document = new Document();
            document.AddStream("Test", file.OpenReadStream());

            await _kernelMemory.ImportDocumentAsync(document);

            while (!(await _kernelMemory.IsDocumentReadyAsync(document.Id)))
            {
                var t = "waiting";
            }

            var ppp = "Yay it worked!";

        }
        catch (Exception ex)
        {
            var t = ex;
            throw;
        }

Error received on the kernel memory service:

2024-09-14T15:25:41.4801407 [13:25:41.467] warn: Microsoft.KernelMemory.Pipeline.Queue.DevTools.SimpleQueues[0] Message '20240914.132541.3504114.a3d926a79f704dfd8899bca437fc9431' processing failed with exception, putting message back in the queue. Message content: {"index":"default","document_id":"6ea604ad91644b53a3cfa328f3a435d9202409140325399030475","execution_id":"e13be1eb4f3840b485d4a8f41eed2539","steps":["extract","partition","gen_embeddings","save_records"]} System.NotSupportedException: File type not supported: Test at Microsoft.KernelMemory.Pipeline.MimeTypesDetection.GetFileType(String filename) in /src/service/Abstractions/Pipeline/MimeTypes.cs:line 223 at Microsoft.KernelMemory.FileSystem.DevTools.VolatileFileSystem.ReadFileInfoAsync(String volume, String relPath, String fileName, CancellationToken cancellationToken) in /src/service/Core/FileSystem/DevTools/VolatileFileSystem.cs:line 242 at Microsoft.KernelMemory.DocumentStorage.DevTools.SimpleFileStorage.ReadFileAsync(String index, String documentId, String fileName, Boolean logErrIfNotFound, CancellationToken cancellationToken) in /src/service/Core/DocumentStorage/DevTools/SimpleFileStorage.cs:line 114 at Microsoft.KernelMemory.Pipeline.BaseOrchestrator.ReadFileAsStreamAsync(DataPipeline pipeline, String fileName, CancellationToken cancellationToken) in /src/service/Core/Pipeline/BaseOrchestrator.cs:line 237 at Microsoft.KernelMemory.Pipeline.BaseOrchestrator.ReadFileAsync(DataPipeline pipeline, String fileName, CancellationToken cancellationToken) in /src/service/Core/Pipeline/BaseOrchestrator.cs:line 251 at Microsoft.KernelMemory.Handlers.TextExtractionHandler.InvokeAsync(DataPipeline pipeline, CancellationToken cancellationToken) in /src/service/Core/Handlers/TextExtractionHandler.cs:line 73 at Microsoft.KernelMemory.Pipeline.DistributedPipelineOrchestrator.RunPipelineStepAsync(DataPipeline pipeline, IPipelineStepHandler handler, CancellationToken cancellationToken) in /src/service/Core/Pipeline/DistributedPipelineOrchestrator.cs:line 226 at Microsoft.KernelMemory.Pipeline.DistributedPipelineOrchestrator.<>c__DisplayClass5_0.<<AddHandlerAsync>b__0>d.MoveNext() in /src/service/Core/Pipeline/DistributedPipelineOrchestrator.cs:line 167 --- End of stack trace from previous location --- at Microsoft.KernelMemory.Pipeline.Queue.DevTools.SimpleQueues.<>c__DisplayClass19_0.<<OnDequeue>b__0>d.MoveNext() in /src/service/Core/Pipeline/Queue/DevTools/SimpleQueues.cs:line 153

[igrmak]

It has predefined types, based on that type different decoders to text involved. Just try to add "txt" So test.txt

[dluc]

as suggested by @igrmak, change this

document.AddStream("Test", file.OpenReadStream());

to

document.AddStream("Test.pdf", file.OpenReadStream());

that is because KM type detection uses the file extension to know how to open the file and extract the content.

[markat1]

nice! thanks @igrmak and @dluc