ToListAsync extension method for IAsyncEnumerable<T>

[beepboopbork]

The IAsyncEnumerable<T> interface is working pretty well for many things in the projects I've been working on. Seeing that it is still pretty new, I wouldn't expect the interface or extensions to be as complete as IEnumerable<T>, but I find myself writing my own extensions to make IAsyncEnumerable<T> usage work a little more like IEnumerable<T> in some cases.

In a similar sense that IEnumerable<T> has .ToList<T>() to immediately evaluate into a list, does it make sense to add a method or extension method to the C# libraries for IAsyncEnumerable<T> that does a similar thing for those times the enumerable needs to be immediately evaluated?

I like to think others may find this useful if built into the C# library, especially since IEnumerable<T>.ToList<T>() already exists. It has become especially useful in unit tests, and other situations where an async enumerator is useful but isn't always needed: as a simple example - in a method that reads a file async, yielding each line - sometimes each line can be read and processed one by one by the consuming code (and stopped without having to read the entire file) but other times all the lines need to be read before processing can begin.

I have created and have been using the following extension method to suit my needs:

public static async Task<List<T>> ToListAsync<T>(this IAsyncEnumerable<T> items)
{
    var evaluatedItems = new List<T>();
    await foreach (var item in items)
        evaluatedItems.Add(item);
    return evaluatedItems;
}

Example on usage, with a method doing something like this:

public async IAsyncEnumerable<string> ReadAsync(string path)
{
    await using (var fileStream = File.OpenRead(path))
    using (var streamReader = new StreamReader(fileStream))
    {
        yield return await streamReader.ReadLineAsync();
    }
}

it could be used either way (similar to what is already possible with IEnumerable<T>):

public async Task DoStuffLineByLineAsync()
{
    await foreach(var line in ReadAsync("some file path"))
        ProcessLine(line); // process each line
}

public async Task DoStuffAtOnceAsync()
{
    List<string> allLines = await ReadAsync("some file path").ToListAsync();
        CalculateSomething(allLines); // needed all the lines before processing 
}

Any thoughts on adding this, and possibly other IEnumerable<T> methods/extensions to IAsyncEnumerable<T>?

[sharwell]

I would not be surprised to see an AsyncEnumerable class appear (possibly as a third-party library) to help with two cases:

1. Inputs that are IAsyncEnumerable<T>
2. Predicates that return ValueTask<bool> and/or transformation functions that return ValueTask<TResult>

[HaloFour]

I'd suggest opening a discussion at the Runtime repository as they handle API discussions. This repository is more for evolution of the C# language itself.

[CyrusNajmabadi]

This is definitely a good question. However, @HaloFour is correct that it's out of scope for CSharpLang. I would recommend taking this to the Runtime repo as they manage adding new APIs.

[beepboopbork]

apologies to all, I'll open a discussion over there. Thanks for pointing me in the right direction!

[YairHalberstadt]

Check out System.Linq.Async by ReactiveExtensions: https://www.nuget.org/packages/System.Linq.Async

[MgSam]

For whatever reason, the BCL team is dead-set against including any extension methods for IAsyncEnumerable, instead delegating it all to Rx (which of course has no where near the level of support as the BCL)

dotnet/runtime#31580

[CyrusNajmabadi]

If LINQ had been shipped without BCL support, it would have been a failure; of this I have little doubt.

Nuget wasn't a thing at the time. Things have changed. Neither team believes that both must be in lockstep on a feature for something to be successful. Libraries can be released without specific language features for them. And language features can be released without specific in-box APIs for them.

[stephentoub]

I'm commenting on the fact that it apparently hasn't even occurred to you guys prior to now what the library situation around IAsyncEnumerable is or that you should have any input on it.

This was discussed extensively with @MadsTorgersen and in C# LDM meetings. It even shows up in the original proposal for the C# 8 feature.

[MgSam]

Because you are inferring lack of cooperation. That's only something you have come up with.

Cyrus, if you had been cooperating with the runtime team, you (C# team) would have already had this discussion internally with the runtime team a year+ ago and known their position that they do not want to support IAsyncEnumerable in the BCL. You wouldn't need a part-time community member to tell you about the decision. I don't enjoy getting into these back-and-forths with you but let's call a spade a spade here.

[jnm2]

@MgSam What makes you think Cyrus in particular was part of that discussion at that time? There are some strong assumptions on your part needed to hold up what you're saying. I don't see the point of challenging him about this.

[CyrusNajmabadi]

@MgSam My position has continued to be: this is conversation to have with the runtime team. Regardless of the decisions and discussions that have happened previously, that is the appropriate team to be involving to see if anything will change here. That has been entirely the messaging i've been adding to this discussion (both to the OP, and you). The runtime may continue to believe their stated reasoning so far (which i have no reason to believe they would change), or they may adjust based on the presence of this new feedback. In either case, that is the best and most appropriate venue to converse on this.

[HaloFour]

That said, if it's a question of visibility and discoverability that said MS NuGet packages exist, which team would that fall upon? Like say if a developer tried to do iae.ToList could VS/Roslyn suggest, "hey, looks like you're trying to use LINQ on an IAsyncEnumerable, would you like to automagickally add this package and import these namespaces?"

[CyrusNajmabadi]

Yes. That's tracked under this issue: dotnet/roslyn#8757

[beepboopbork]

@HaloFour - I think you nailed why I posted (on the wrong repo, sorry!). I didn't realize there was an official package due to Nuget not showing the dotnetframework or Microsoft as an owner like all the other packages I've pulled in to get extra "official" features. I've learned something today at least 😄