Allow iterator/async iterator lambdas/anonymous methods

[roji]

Champion issue: #9467

Currently, trying to use yield inside a lambda triggers CS1621: "The yield statement cannot be used inside an anonymous method or lambda expression".

One main motivator for this is being able to work with IAsyncEnumerable (and potentially IEnumerable) within ASP.NET Minimal API delegates:
 

app.MapGet("/search", async IAsyncEnumerable<Product> (string query, VectorStoreCollection<int, Product> collection) =>
{
    await foreach (var result in collection.SearchAsync(query, top: 5, new() { Filter = r => r.TenantId == 8 }))
    {
        yield return result.Record;
    }
})

Because of the current restriction, the body above must be split out to a named function, just in order to be able to use yield. As IAsyncEnumerable becomes more popular, there may be more cases where async iterator lambdas could be helpful for working with it.

Given that we can now explicitly specify the return value of lambdas, it may make more sense to allow this feature. This may also fit within the general move of making lambdas/anonymous functions support more features that regular methods do (static modifier, attributes, explicit return types...).

/cc @MadsTorgersen @stephentoub @davidfowl

[stephentoub]

cc: @jaredpar

[jaredpar]

Believe the last time this came up was in 2018. My memory, and the notes seem to reflect, is that the reception was fairly okay. Just lack of motivating sample to push it into the "go for it bucket".

[jaredpar]

Dropped it into the LDM meeting queue. Is there a larger issue / item we can look at for context?

[stephentoub]

IAsyncEnumerator is now used heavily in streaming APIs for AI services. It'd be great to enable its use in lambdas for C# 15.

[HaloFour]

How about non-async iterators?

[jaredpar]

The proposal title is "iterator / async iterator" which I'm interpreting as "or" so think that is already covered. From my perspective: I would assume we do both or neither.

[HaloFour]

Hah, yeah, missed that in the title. I do recall conversations a while ago about concerns around anonymous/lambda iterators so I'm curious to see how the discussion plays out.

[MadsTorgersen]

When we first rejected lambda iterators many years ago, we did not have local functions, and lambdas did not have return types or inferred delegate types. We were looking for statement-scoped ways of declaring iterators, and lambdas were too roundabout for that: First create a delegate object by explicitly converting the lambda to a clunky type name, then immediately invoke it to get the enumerable. Local functions ended up being the more straightforward design for this.

However, the scenarios now are compelling, and I think the design would be straightforward. I am certainly in favor.

[GabeSchaffer]

Am I wrong to believe that it there was something about the C version of the compiler that made the lambda iterator feature prohibitive to implement? I seem to recall that the VB.Net supports it (anonymous iterator functions), but the old C# compiler was a completely different code base and couldn't as easily support the feature, so it was disallowed in C#.

[HaloFour]

Nah, this was during the Roslyn era on one of the public repos. Must've been on CodePlex as the first discussion I can find on Roslyn seems to just rehash the conversation and push towards local functions. If I recall correctly, one of the concerns was the lack of an obvious way to indicate that the lambda was an iterator. C# requires async for async lambdas, and even VB.NET requires an Iterator keyword. I personally never really bought that argument given that normal iterator methods have the same problem. Happy to see that it's being reconsidered. :)

[GabeSchaffer]

I found Eric Lippert's blog post from 2009 explaining how iterator rewriting and anonymous function rewriting were the two most complicated parts of the compiler (the original C# compiler, written in C), and combining them would make things even more complicated: https://learn.microsoft.com/en-us/archive/blogs/ericlippert/iterator-blocks-part-seven-why-no-anonymous-iterators

C# got iterators and anonymous delegates in the 2.0 release back in 2005, while VB didn't get iterators until the async release in 2012. I'm guessing that it's probably not hard to generalize coroutines to both async and iterators, but by then C# already had the rule that iterators couldn't be in anonymous functions and there was little motivation to drop the restriction.

[jaredpar]

It was certainly one of the more complicated parts of the native compiler, but not so complicated that it would have stopped us from doing these type of features.