Feature Request: Syntactic Sugar to yield the whole of an IEnumerable directly.

[Brondahl]

See also #378 . This is separating out a syntactic sugar request, from a functional performance enhancement request.
This is NOT a duplicate!
Please consult @CyrusNajmabadi before Closing-As-Duplicate. :)

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

If you are writing a yield Iterator, and you end up with an IEnumerable<T> that represents some sub-set of the values you wish to return, then you are obliged to do this:

foreach (var x in someValuesToBeYielded) { yield return x; }

I (and many others) would like there to be a syntactic sugar to express that (even) more tersely:

return yield foreach someValuesToBeYielded;

This would be solely syntactic sugar - it's literally just a short hand for the explicit foreach version.
(Implementation wise ... we're talking about a Lowering Rewriter)
There would be no changes to performance in any sense.

To support my assertion that this is a syntactic sugar that is desired by C# users:

• There is a heavily viewed and liked SO question asking "how do you do this?". (Answer: you can't.)
• The syntax exists in Python, Javascript, and F#.
• The related-but-subtly-different request of "write a syntax for in recursive calls that fixes a performance issue with the long-hand version" is littered with people coming along and saying "but all I want is the syntactic sugar ... I don't care about improving the performance".

The possibility of this syntax as a pure sugar syntax came up in #378 , which relates to a very similar concept, but concerns an edge case involving recursive calls, in which the the long-hand code about causes quadratic overhead.
Over in that discussion, the argument presented against this syntax (primarily lead by @CyrusNajmabadi) claims that it is at risk of looking like it's doing something clever about performance, when it's not. i.e. that by including the sugar syntax, which is not confusing in the majority of cases, it will make it look like the recursive edge case has in some way been "fixed" so as to not have the quadratic performance cost.
(I think. ... Note that I don't fully understand this aspect ... @CyrusNajmabadi perhaps you could summarise better?)

The quadratic performance is not itself a concern of this discussion (since we're pitching this a sugar syntax, not claiming to change the performance). But the concern that the syntax could mislead some users does seem a legitimate concern.

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

(Note: the exact nature of the keyword is not important for this stage of the discussion. I happen to have proposed yield foreach , but it could be yield from, yield all yield many, yield*, yield! ... maybe even just yield with type analysis identifying that it's yielding an IEnumerable in a method that is declared to yield Ts. The particular keyword isn't important for now - this discussion is to establish whether such a syntax is appropriate/desirable.)

[TonyValenti]

I really hope we see this at some point. It has been asked for so.many.times.

[CyrusNajmabadi]

@TonyValenti the above concerns would need to be addressed in order for any movement to be made on this. Right now there is already a viable solution in the language to syntactically yield an entire enumerable. Given that, there will be practically no chance of a terser syntax being taken given the negatives outlined above.

[The-Futurist]

I've wondered about this too sometimes but I also now wonder if LINQ's SelectMany is sufficient for this?

[CyrusNajmabadi]

@Korporal can you expand on this? How would you use SelectMany to help out here? Thanks!

[foxesknow]

I don't see why we need this feature just to save writing an foreach loop to accomplish the same thing. This sort of pattern is not so common that it needs explicit language support, and adding it is more of a box ticking exercise more than anything else.

[Brondahl]

adding it is more of a box ticking exercise more than anything else

Yeah, definitely a box-ticking exercise and not, say, a response to:

• dozens of users actively requesting it on the other GH issue
• nearly 200 SO Likes on the question asking how to do it
• nearly 50 thousand views on the same SO question.
  /s

To assert that it's a bad idea, or that you personally don't need it, is perfectly reasonable.
To assert that there's been no desire for it expressed by the user-base is utterly absurd.

[sab39]

but concerns an edge case involving recursive calls, in which the the long-hand code about causes quadratic overhead.

I think the miscommunication here is in calling this an "edge case", for two reasons:

1. It doesn't actually only apply to recursive calls - the nasty performance behavior happens every time you use the longhand code, it just compounds the more nested functions are involved, and recursion is the most common way you get lots of nested functions.
2. Recursive code isn't an edge case, it's an incredibly common type of coding for anything that involves hierarchical data structures (ie Json, Xml/Html, any kind of source code, file systems, ...). This feature, if it existed, would be very widely used (that's the whole point of the "there's huge demand for it" argument), and a lot of that usage would surely be in recursive code.

In other words - the current longhand version of this is already a problem and you need to be careful about how you use it. Until that gets fixed, the syntax being inconvenient is a feature because it discourages use of the pattern and encourages looking for other approaches.

The argument isn't "adding syntactic sugar would make it seem like the issue has been fixed", it's "adding syntactic sugar would encourage people to write this kind of code, which they generally shouldn't be doing until the compiler can produce non-awful code for it".

[CyrusNajmabadi]

In any way, someone who's proficient enough to use yielding will certainly know performance drawbacks of that.

I don't believe that's going to be true. Even today, we run into many people who run into this pitfall, even with the explicit foreach. Making it easier to fall into is not something i'm in support of. As mentioned above:

In any way, someone who's proficient enough to use yielding will certainly know performance drawbacks of that.

[CyrusNajmabadi]

P.S. It's just ridiculous to hear this arguments of performance, when I use this yield* from TS on a daily basis,

Sure. The performance hit may be totally fine for you (And some subset of our audience). But we have to consider the entirety of our audience, and we cannot haphazardly impact performance for everyone just because some are ok with it.

And it do not stops neither devs of TS

Right... but these are different communities with different priorities. What is fine for one community may be fine for another. For example, many things taht C# has done for our community would not be right for C++ or Rust. Or, if they wanted to do something similar, they would need to do so in a way that didn't go against the goals of those languages.

it's just an excuse.

Yes. It's the reason and explanation for why this hasn't been able to move forward so far :)

All that we, the language users, should just take into account, is that the language team do things that make interest to them, not to us, the actual users.

We have millions of "actual users". We have thousands of requests for language improvements, and we have to consider the pros/cons for all users. Given limited time and resources, taht necessarily means making judicious choices as to where to allocate those resources. That means things that have problems, or do not seem to provide enough value will not make the cut.

[GrafGenerator]

we have to consider the entirety of our audience
What is fine for one community may be fine for another
making judicious choices

I understand the tradeoff, indeed:) Just to be clear, I don't want to be flaming here, I just wanted a clear and honest reason why:)

many people who run into this pitfall, even with the explicit foreach

Well you can't build a completely safe-to-use language, I'm pretty sure you understand it as well

Just to be on the same page, we're talking here about a pure syntactic sugar which will replace kind of "yield foreach" into "foreach (var x in y) yield return x", and this adds no performance overhead, however it can be more obscure for those who don't go even one step under the cover of the matter. And main point of language team is that potential value of this is just too small given the limited time and more priority features, is that correct?

If that is correct, then would the following be correct as well - if someone will make a PR with this syntactic sugar implemented, the language team will be ready to consider it for release?

[CyrusNajmabadi]

I just wanted a clear and honest reason why:)

I was being honest. :)

Well you can't build a completely safe-to-use language, I'm pretty sure you understand it as well

Yes. but we can also avoid lang features taht actively lead people astray. :)

if someone will make a PR with this syntactic sugar implemented, the language team will be ready to consider it for release?

Absolutely not. See https://github.com/dotnet/csharplang/#milestones. The proposal would need to at least be in the "any time" category for us to consider the PR for release.

For this issue, we would need an actual proposal that we signed off on that we were confident was the right lang change and would be the right thing for the community/ecosystem.

[CyrusNajmabadi]

Just to be on the same page, we're talking here about a pure syntactic sugar which will replace kind of "yield foreach" into "foreach (var x in y) yield return x", and this adds no performance overhead, however it can be more obscure for those who don't go even one step under the cover of the matter. And main point of language team is that potential value of this is just too small given the limited time and more priority features, is that correct?

Yes. Though i would also add: having a simpler way of expressing the exact same thing, with teh same perf overhead, but which looks like it would not have such overhead is something we feel is actively harmful in that we expect that people will see this and now think they can write these iterators without the high computation complexity, which would not be true. We would be open to a nice syntax form if it could both provide recursive iterators without that computation complexity explosion, and without overhead for users whose iterators do not make use of this functionality.

In other words, this should be "pay to play". You should only have a perf hit here if your iterator makes use of this capability. Existing code that does not should not be affected at all.

[sean-bloch]

Hello, I came here from the "I have done a search for similar discussions". I came about looking for a simpler syntax such as yield return each ... , when I encountered code that was hiding implementation details where the IEnumerable may have been a List, and further processing treated my result differently based on the underlying type. In my case, the "simpler" syntax feels like it would make it clearer that it was "all" i was attempting with the code I was writing...

The discussion I almost started until I found the existing ones is below. I understand now from reading that there may be some performance cases, but also sharing while I would use even just the "longer" syntax in a common case:

In short, this would allow shortcutting:

foreach(var i in items)
{
yield return i;
}

with simply:

yield return each items;

likewise, "items" could just as easily be a method result like:

yield return each GetItems();

It's small, but I ran into a situation with an ORM and abstractions of IEnumerables, where I wanted to only treat the IEnumerable as an IEnumerable. I found that a simple yield return helped to "guarantee" this.

The below shows a small sample of how an "external api" might hide some implementation behavior (outputs to console)

IEnumerable<SomeData> YieldReturnEach(IEnumerable<SomeData> items)
{
    // this below would be shortcut simply to:
    // yield return each items;
    foreach (var i in items)
    {
        yield return i;
    }    
}

void Main()
{
    var xApi = new ExternalAPIClass();
   
    //Note that the api surface hides that this IEnumerable is *also* a List...
    IEnumerable<SomeData> fromDataList  /**/ = xApi.aList().GetSomeEnumerableData();
    //The api surface returns this IEnumerable from a source that is *not* a List... 
    IEnumerable<SomeData> fromKeyedData /**/ = xApi.anEnumerable().GetSomeEnumerableData();
    
    

    Console.WriteLine($$"""
    {{nameof(fromDataList)}}
    ===
    One:               {{xApi.APITransformOne(fromDataList)}}
    
    // Despite being visible as an IEnumerable<> from the API surface.. it is treated as a List... 
    Two:               {{xApi.APITransformTwo(fromDataList)}}
    Two(AsEnumerable): {{xApi.APITransformTwo(fromDataList.AsEnumerable())}}
    Two(Cast):         {{xApi.APITransformTwo((IEnumerable<SomeData>)fromDataList)}} 
    Two(FuncResult):   {{xApi.APITransformTwo(((Func<IEnumerable<SomeData>>)(() => { return fromDataList.AsEnumerable(); })).Invoke())}}
    
    // by "wrapping" in a yield return IEnumerable, we can "guarantee" that our IEnumerable<> is treated *only* as an IEnumerable... 
    Two(ReturnEach):   {{xApi.APITransformTwo(YieldReturnEach(fromDataList))}}
    
    {{nameof(fromKeyedData)}}
    ===
    One: {{xApi.APITransformOne(fromKeyedData)}}
    Two: {{xApi.APITransformTwo(fromKeyedData)}}
    """
    );
    
}

public sealed class ExternalAPIClass()
{
    private string doOne(IEnumerable<SomeData> items) /**/ => $"1[ {string.Join(",", items.Select(i => i.d.ToString()))} ]";
    private string doOne(List<SomeData> items)        /**/ => $"1( {string.Join(",", items.Select(i => i.d.ToString()))} )";
    public string APITransformOne(IEnumerable<SomeData> items) => doOne(items);

    //This highlights a potential "hidden implementation problem"
    public string APITransformTwo(IEnumerable<SomeData> items) => items switch
    {
        List<SomeData> /**/=> $"2( {string.Join(",", items.Select(i => i.d.ToString()))} )",
        _              /**/=> $"2[ {string.Join(",", items.Select(i => i.d.ToString()))} ]",
    };
    
    List<SomeData> m_someDataList               /**/ = new([new(1),new(2)]);
    Dictionary<string,SomeData> m_someKeyedData /**/ = new([new("three",new(3)),new("four",new(4))]);
    public IAdherent aList()        => new Adherent_List(m_someDataList);
    public IAdherent anEnumerable() => new Adherent_Enumerable(m_someKeyedData);
}

public interface Id
{
    public int d { get; }
}

public record SomeData(int d) : Id; 

public interface IAdherent
{
    public IEnumerable<SomeData> GetSomeEnumerableData();
}

public class Adherent_List(List<SomeData> someDataList) : IAdherent
{
    public IEnumerable<SomeData> GetSomeEnumerableData() => someDataList;
}

public class Adherent_Enumerable(Dictionary<string,SomeData> someKeyedData) : IAdherent
{
    public IEnumerable<SomeData> GetSomeEnumerableData() => someKeyedData.Select(kv => kv.Value);
}

Expected Output:

fromDataList
===
One:               1[ 1,2 ]
Two:               2( 1,2 )
Two(AsEnumerable): 2( 1,2 )
Two(Cast):         2( 1,2 ) 
Two(FuncResult):   2( 1,2 )

// by "wrapping" in a yield return IEnumerable, we can "guarantee" that our IEnumerable<> is treated *only* as an IEnumerable... 
Two(ReturnEach):   2[ 1,2 ]

fromKeyedData
===
One: 1[ 3,4 ]
Two: 2[ 3,4 ]

Thanks,

[mrwensveen]

I would also like this feature, but I understand the reasons for not implementing this yet. However, I'd like to note that the performance problems with the current syntax is either undocumented, or very hard to find. This makes writing foreach (var x in someValuesToBeYielded) { yield return x; } less a "I know about the performance problems but I need to do this anyway" and more a "Ugh, why is it so ugly" kind of thing.

If and when syntax is implemented, I'd like yield foreach to be the syntax, because of the already existing await foreach syntax.

[TonyValenti]

I think that the performance problems with the current methodology are exactly a strong reason this feature should be implemented.
yield all items; would certainly be no worse performing that the current foreach loop people write and, if at some point down the road, the enumerable implementations are optimized, then things are future ready.

[dnarayan-orepath]

I understand that the reason this hasn't been implemented is because there's a quadratic complexity issue, and that not implementing this is meant to encourage devs to do something else.

But most devs are just going to write it out anyway, and then be annoyed that Javascript is nicer to use.

Might a better way be to actually implement it, detect possible recursion at compile time (is this possible?), and then yield a warning/error if they used recursion? Thus, actually pointing the dev towards the problem, while helping those who are not using it in a recursive context.

[TonyValenti]

To piggy-back on what @dnarayan-orepath said:

The current answer is "We don't want you to do this because there might be performance issues, so you should do something else."
This would be a fine answer is "something else" was a clear, repeatable, answer that was easy to articulate.
But it isn't. The answer is simply "We're not sure what you should do. Just don't do that."
So what do most people do?

They hand-write the loop because that's the straight forward thing to do.

[Brondahl]

@dnarayan-orepath & @TonyValenti
I believe (but defer to @CyrusNajmabadi ) that their point/instruction/recommendation is not:

"don't do [this call pattern that is or might be quadratic]"

but rather

"if you're going to do [this call pattern that is or might be quadratic] then we insist that you do so using a foreach, so the performance risk is more obvious."

As it happens, I don't personally agree that the sugar syntax would make it less obvious.
I'm not 100% certain I understand the risk case, but either:

• I do understand the risk case ... and IMO the sugar syntax doesn't hide that at all.
  or (more likely)
• I don't understand the risk case ... and writing the long-hand foreach version hasn't made the risk any clearer at all 😅

But I believe that is this is the core of the language devs' position w.r.t. "Sugar without underlying change".

[Brondahl]

Actually ... that does suggest a potential out for this conversation ...

@CyrusNajmabadi if the concern around the sugar syntax is "it hides the perf risk" ... then would you be open to a sugar syntax which actively highlights the perf risk?
Especially given that (I suspect) the perf risk is not actually obvious from the foreach version that devs will actually write?

[CyrusNajmabadi]

then would you be open to a sugar syntax which actively highlights the perf risk?

No clue. Would have to see what you're proposing.

That says, a foreach loop with a yield in it is trivial to write and read. So it's unclear to me what value is being provided that would make it worth it to introduce something else here to do the same with sugar.

But feel free to suggest. We can definitely look.

[dnarayan-orepath]

From what I gather, the problem is that foreach creates a new IEnumerable/IEnumerator, and to get the single next item, it must step forward in each nested foreach. If that's the case, a for loop wouldn't have this same issue (because it can't be nested in the same way) so what about:

int[] arr = [0,1,2];
var plusone = arr.Select(a=>a+1);

yield .. return arr; // Works - converted into a for loop, not a foreach loop.
yield .. return plusone; // Compile Error: Expected IReadOnlyList<T>. IEnumerable<T> not 
//supported due to significant performance implications. See <link>.

Developer sees error on the plusone line, but doesn't click on the link and types:
yield .. return plusone.ToArray();

Dev is then like "Wait, the performance implications of this are tragic. This is a bad idea. Maybe I better see why only a IReadOnlyList is supported."

The primary reason to do this improvement is less about avoiding a small foreach/for loop, but to help devs write better code by exposing them to this arcane implementation detail. I looked through my tree traversal code, and it's all n^2 - I had absolutely no idea.

[LWChris]

I was about to request this feature with the syntax yield foreach <enumerable>;.

My usecase is a splitting function where I either need to return an aggregate or all aggregates from the next recursion level:

public static IEnumerable<Split> SplitEntries(IEnumerable<Entry> entries, int max)
{
    return SplitEntriesRecursive(entries, SplitLevel.Year, max);
}

private enum SplitLevel
{
    Year,
    Month,
    Day,
    Hour,
    Minute,
    Second
}

private static IEnumerable<Split> SplitEntriesRecursive(IEnumerable<Entry> entries, SplitLevel level, int max)
{
    var splits = entries
        .GroupBy(e => new Split(e.DateTime, level))
        .ToDictionary(g => g.Key, g => g.ToList());

    foreach (var kvp in splits)
    {
        if (level == SplitLevel.Second || kvp.Value.Count <= max)
        {
            yield return kvp.Key;
        }
        else
        {
            foreach (var s in SplitEntriesRecursive(kvp.Value, level + 1, max))
            {
                yield return s;
            }
        }
    }
}

Regarding people might introduce performance issues by writing yield foreach x instead of return x:

Whereas yield return and yield break turn a function into an iterator function, yield foreach could simply not, and then we can decide:

• disallow yield foreach x unless iterator function (like disallow await unless asynchronous function), or
• warn that yield foreach x is slower than return x, or
• simply emit the IL for return x (like emit "Abcd" instead of "A" + "b" + "c" + "d") - no need to warn, just works.

[CyrusNajmabadi]

And, as multiple others have stated, the compiler could generate a warning

We're not going to add a language feature just to have it immediately warn you the moment you use it.

There's already a one-liner that performs badly for recursive enumeration. We don't need to add syntax to save a few tokens from that that will be just as bad.

[CyrusNajmabadi]

But what should I do instead?

Write a foreach loop if you really want to yield and you're ok with bad performance.

Manually write the iteration if you want the good performance.

[LWChris]

Manually write the iteration if you want the good performance.

@CyrusNajmabadi Can you show what that means in actual code? How do I "manually write the iteration" for

yield return 1;
foreach (var i in GetIEnumerableOfInt()) yield return i;

Sorry if this is stupid, but I've seen that mentioned in prosa a lot, but have yet to see what that practically means in code. I appreciate your time helping me/us to understand implications and mitigations.

[austinw-fineart]

I've see that exact foreach yield return code written in Microsoft docs so it must be the correct way.

[LWChris]

I mean if whatever "1" stands for is available beforehand and cheap to create, you can rewrite that as return GetIEnumerableOfInt().Prepend(1). Not sure how a huge chain of Prepend, Append, Concat performs.

Unfortunately Prepend and co. do not take a factory methods that are only executed on demand.