Adding support for IAsyncEnumerable result types to query API

[NameIsBad]

[virzak]

This would be extremely useful to read huge excel sheets

[michelebastione]

@virzak This feature is currently under the final stages of development and testing, if all goes according to plan it should make the next release or the following one.

That said the synchronous API is already very reliable, if you could share the details of your use case we might be able to help you out.

[virzak]

@michelebastione thanks for prompt response.

We have two lists of items in excel format which are somewhat similar to each other.
Both lists are about 700K rows each.
We are matching these items based on some rules which we have and returning the matching info for them.
Both lists are sorted in a way which only requires us to read them once - we'll never need to randomly jump to an arbitrary index.

The idea is to have a method with the following signature:

IAsyncEnumerable<MatchingInfo> MatchAsync(IAsyncEnumerable<ItemA> listOfAs, IAsyncEnumerable<ItemB> listOfBs, IProgress<float>? progress)
{
}

Since no other library produces IAsyncEnumerable, the current solution waits until everything is loaded into memory.

Would love to test it out, while it is still in development. I notice there's no PR for this. Perhaps you could share a location where the latest development is happening.

[michelebastione]

I will make a draft later if you wish to check it out, but I don't see why you require for this matching process to use IAsyncEnumerables. If your concerns are memory related, our querying api already lazily evaluates the rows one at a time, specifically to avoid loading alll data into memory. You could, as a crude example, do the following:

IEnumerable<MatchingInfo> Match(string pathToA, string pathToB)
{
    var queryA = MiniExcel.Query<ItemA>(pathToA);
    var queryB = MiniExcel.Query<ItemB>(pathToB);
    
    using var enumeratorA = queryA.GetEnumeratorA();
    using var enumeratorB = queryB.GetEnumeratorB();

    while (enumeratorA.MoveNext() && enumeratorB.MoveNext())
    {
        yield return MatchInfo(enumeratorA.Current, enumeratorB.Current);
    }
}

You could in fact even wrap the IEnumerable into a IAsyncEnumerable if you really need this part of your process to work as asynchronously as possible:

async IAsyncEnumerable<MatchingInfo> MatchAsync(string pathToA, string pathToB)
{
    var queryA = MiniExcel.Query<ItemA>(pathToA);
    var queryB = MiniExcel.Query<ItemB>(pathToB);
    
    using var enumeratorA = queryA.GetEnumeratorA();
    using var enumeratorB = queryB.GetEnumeratorB();

    while (enumeratorA.MoveNext() && enumeratorB.MoveNext())
    {
        var match = MatchInfo(enumeratorA.Current, enumeratorB.Current);
        yield return await Task.FromResult(match);
    }
}

Though be aware that this is asynchronous pretty much in name only as no actual I/O method calls are being awaited and it will most likely result in some overhead, but then again that depends on the specifics of your use case.

Let us know if this helps you, or feel free to share more details if this solution does not quite work for you.

[virzak]

Looks like the "Async" behaviour is simply wrapping Sync with Task.Run() in QueryAsync.

This is something That EPPlus did at some point but then reversed course.

Do you think that this library could benefit from actual asynchronous code? Here is a WIP PR #799

[michelebastione]

@virzak We are aware that the current asynchronous implementation is far from ideal, that's why this issue was pinned and I've been working on a true asynchronous API that makes use of IAsyncEnumerables. What I was suggesting in my previous comment was in fact just a workaround that uses the synchronous lazily evaluated IEnumerables, given that in the majority of cases we have dealt with before those were perfectly fine in terms of memory usage and performance.

[virzak]

Apologies if I overstepped here a bit. Since I can't see the PR, I didn't know it was implementing true asynchronous API.

[michelebastione]

No problem at all, just trying to make sure we're on the same page so that we can help you out. I should have made a PR but was delayed by issues with higher priority, we'll sort it out as soon as possible.