Custom iterator types (methods using yield return)

[Rekkonnect]

Summary

Enable the construction of custom iterator types that replace the need for explicitly using IEnumerable<T>.

Motivation

Only some interface types can be returned on a method returning with yield return:

• IEnumerable
• IEnumerable<T>
• IEnumerator
• IEnumerator<T>
• IAsyncEnumerable<T>
• IAsyncEnumerator<T>

This is discouraging for two reasons:

• We might not always want to return a reference type as the result, especially for cases where we return structs to enumerate the collection of our interest.
• We cannot use ref structs as type arguments, which is the main reason why many duck-typing features were released, in order to detach from the need to implement a specific interface to use some of the language's syntax sugar features.

Design

We would have an API similar to async method state machine builders, exposing the necessary attributes and classes for designing a type as capable of being returned in a method containing yield return.

The exact details are not reported here.

Examples

Using a custom enumerable would simply allow you to return a different type in the method, like in the below example:

public CustomType Method()
{
    yield return 1;
    yield return 2;
}

Effects

Potentially we can design custom types for enumerating nested collections, like two- or three-level nesting which would be the most common. For example:

public NestedEnumerable<T> Enumerate<T>(T single, IEnumerable<T> multiple)
{
    yield return single;
    yield return multiple;
}

// example expected use case:
var result = Enumerate(1, new[] { 2, 3, 4 })
   .ToArray();
// result = { 1, 2, 3, 4 };

[colejohnson66]

Is there a requirement that these user-defined types implement IEnumerable<T>? I'd assume so, considering your call to ToArray().

[Rekkonnect]

Due to not having the ability to provide ref structs as type arguments, this should not be a requirement. One of the main motives of this feature is to allow ref structs to be enumerated without having to implement that interface. That being said, it is not prohibited, and in that example, NestedEnumerable<T> would most likely implement IEnumerable<T>. Bear in mind that the result of the method is the type that will ultimately provide an enumeration mechanism either via duck-typing, or by implementing those said interfaces.

[colejohnson66]

How do you expect the code to be "lowered"? What is the compiler supposed to do when encountering a yield return if it's not generating an enumerable? Currently, a full state machine is implemented.

using System.Collections.Generic;

static IEnumerable<int> Test()
{
    yield return 1;
    yield return 2;
}

[CompilerGenerated]
internal class Program
{
    [CompilerGenerated]
    private sealed class <<<Main>$>g__Test|0_0>d : IEnumerable<int>, IEnumerable, IEnumerator<int>, IEnumerator, IDisposable
    {
        private int <>1__state;

        private int <>2__current;

        private int <>l__initialThreadId;

        int IEnumerator<int>.Current
        {
            [DebuggerHidden]
            get
            {
                return <>2__current;
            }
        }

        object IEnumerator.Current
        {
            [DebuggerHidden]
            get
            {
                return <>2__current;
            }
        }

        [DebuggerHidden]
        public <<<Main>$>g__Test|0_0>d(int <>1__state)
        {
            this.<>1__state = <>1__state;
            <>l__initialThreadId = Environment.CurrentManagedThreadId;
        }

        [DebuggerHidden]
        void IDisposable.Dispose()
        {
        }

        private bool MoveNext()
        {
            switch (<>1__state)
            {
                default:
                    return false;
                case 0:
                    <>1__state = -1;
                    <>2__current = 1;
                    <>1__state = 1;
                    return true;
                case 1:
                    <>1__state = -1;
                    <>2__current = 2;
                    <>1__state = 2;
                    return true;
                case 2:
                    <>1__state = -1;
                    return false;
            }
        }

        bool IEnumerator.MoveNext()
        {
            //ILSpy generated this explicit interface implementation from .override directive in MoveNext
            return this.MoveNext();
        }

        [DebuggerHidden]
        void IEnumerator.Reset()
        {
            throw new NotSupportedException();
        }

        [DebuggerHidden]
        [return: System.Runtime.CompilerServices.Nullable(1)]
        IEnumerator<int> IEnumerable<int>.GetEnumerator()
        {
            if (<>1__state == -2 && <>l__initialThreadId == Environment.CurrentManagedThreadId)
            {
                <>1__state = 0;
                return this;
            }
            return new <<<Main>$>g__Test|0_0>d(0);
        }

        [DebuggerHidden]
        [return: System.Runtime.CompilerServices.Nullable(1)]
        IEnumerator IEnumerable.GetEnumerator()
        {
            return ((IEnumerable<int>)this).GetEnumerator();
        }
    }

    private static void <Main>$(string[] args)
    {
    }

    [System.Runtime.CompilerServices.NullableContext(1)]
    [IteratorStateMachine(typeof(<<<Main>$>g__Test|0_0>d))]
    [CompilerGenerated]
    internal static IEnumerable<int> <<Main>$>g__Test|0_0()
    {
        return new <<<Main>$>g__Test|0_0>d(-2);
    }
}

What is the compiler supposed to do in your functions? If it's as simple as the following, I doubt the team would consider such a feature.

public CustomType Method()
{
    yield return 1;
    yield return 2;
}

public CustomType Method()
{
    CustomType result = new();
    result.AddYield(1);
    result.AddYield(2);
    return result;
}

[HaloFour]

It couldn't be that simple either given the compiler emitted state machine would have to call through the custom enumerable builder everytime it tried to advance. I suspect that the addition of general purpose machinery like that would end up being more expensive than performance gains to be had from using ref types. I personally would rather see support for value enumerables, or even better, value linq.