Syntax Suggestions for C# 8 Await Foreach and Extensions

[siegfriedpammer]

I just watched the video on Channel 9 about the new C# 8 features. I really like the new features!
I hope I am not too late, but there are just two things I would like to propose:

1. foreach await (var s in stream) looks a bit strange. How about foreach (await var s in stream)? This would make it more C#-like as we have only one keyword before the open parenthesis in all the other control structures.
2. extension A extends ClassA : ... is very similar to inheritance in other languages (Java), which might be confusing for other people. Plus, it's a new keyword. I would propose to use extension A for ClassA : ....
   Benefits include:
   a) Reuse of existing keyword.
   b) Similar to natural language.

[DavidArno]

@siegfriedpammer,

Could you share a link to the video, please? It helps those of us who haven't seem it and puts context to your suggestions.

[benaadams]

https://channel9.msdn.com/Blogs/Seth-Juarez/A-Preview-of-C-8-with-Mads-Torgersen

[flq]

1. Agree - Either that, or possibly foreach (var s in await stream)
2. Just pushing it out there, brainstorm style, what about extension A<Customer> { } ? Constrains the type of this to be the "generic argument" - and could allow defining generic extension modules :)

[FurkanKambay]

Maybe await foreach(string s in myStream)? I think it helps with readability; you know, like "await for each string s in myStream).

[mariusGundersen]

foreach await(string s in myStream) is similar to EcmaScript stage 3 proposal of for await(let s of myStream), so I would recommend keeping that syntax.

[siegfriedpammer]

@mariusGundersen Is it me, or is JavaScript creeping into C#?
@flq extension A<Customer> ... would work for me as well, I think using a keyword would make it more obvious, what's going on, though.

[mariusGundersen]

@siegfriedpammer, as Mads said in the video, they are stealing ideas from TypeScript, which is a superset of EcmaScript, so I guess ideas are creeping in both directions.

[siegfriedpammer]

@mariusGundersen I am fine with ideas and features, but I don't think C# should follow the example of JavaScript in terms of syntax.

[iam3yal]

@siegfriedpammer

extension A extends ClassA : ... is very similar to inheritance in other languages (Java), which might be confusing for other people. Plus, it's a new keyword. I would propose to use extension A for ClassA : ....
Benefits include:
a) Reuse of existing keyword.
b) Similar to natural language.

Just because the semantics for extends are different in other languages doesn't mean that C# shouldn't use it as a keyword and instead look for another confusing word such as for when it may imply that you extend a set of classes as opposed to a single one.

@mariusGundersen

foreach await(string s in myStream) is similar to EcmaScript stage 3 proposal of for await(let s of myStream), so I would recommend keeping that syntax.

I really don't like it, I don't think you await the whole thing and prefer to have foreach (string s await myStream).

I take it back, 😆

[MillKaDe]

@eyalsk
await should not placed right before the IEnumerable<T> myStream, because it is not the stream as a whole that is awaited. Instead each item pulled out (by some MoveNext-like method) is awaited. There is one await for each item, not one await for the whole stream of items.

For the same reason, await foreach (..) doesn't make sense. It is not the whole loop, that is awaited.

IMHO, the syntax should be one of

1. foreach await (var i in stream) ... (as proposed by @MadsTorgersen )
2. foreach (await var i in stream) ... (as proposed by @siegfriedpammer above)
3. foreach (var await i in stream) ...

[benaadams]

foreach (var i await in stream) ...

It is not the whole loop, that is awaited.

Is an async foreach Task returning?

e.g. could you do this?

Task t = foreach await (var i in stream) ....

// ...

await t;

[pentp]

foreach (await var s in stream) feels better that the alternatives because await is "executed" for each variable pulled from stream, not on the foreach as a whole. It doesn't make much sense between the variable type and name declarations and can't be after in because that's valid syntax today.

[benaadams]

Adding an cosmetic equals for variable assignment

foreach (var i = in stream) ... (current foreach)

1. foreach await (var i = in stream) ...
2. foreach (await var i = in stream) ...
3. foreach (var await i = in stream) ...
4. foreach (var i = await in stream) ...

Or adding a intermediate task var that is returned from the for

foreach (var i = await t in stream) ....
foreach (int i = await Task<int> t in stream) ....

I still think 4. foreach (var i await in stream) makes most sense as i is the awaited result not the task that you then await in the loop.

[MillKaDe]

@benaadams
Your option 4 makes my eyes bleed a little bit ... ;-P

But it is maybe the best syntax variant to avoid any misunderstandings about whats going on. The await happens when/while pulling the next item out of the stream, and when the await completes, the pulled item is assigned to i.

[benaadams]

Your option 4 makes my eyes bleed a little bit ...

@MillKaDe I assume there is an implicit await on the foreach otherwise the options are kinda

1. await foreach await (var i in stream) ...
2. await foreach (await var i in stream) ...
3. await foreach (var await i in stream) ...
4. await foreach (var i await in stream) ...

As you should be able to capture the overall sequence in a Task(-like)?

1. Task t = foreach await (var i in stream) ...
2. Task t = foreach (await var i in stream) ...
3. Task t = foreach (var await i in stream) ...
4. Task t = foreach (var i await in stream) ...

[gulshan]

@CyrusNajmabadi Can the naming be optional and a name will be auto-generated by convention when no name given? Something like-

//Name auto-generated to StringExtension or something like that
extension String
{
    ...
}

//Name not generated
static class MyStringExtension extension String
{
    ...
}

The if any conflict happens with convention based auto generated names, an error will be shown.

[Joe4evr]

If you let the compiler generate a name, then it's always an "unspeakable" name in order to avoid conflicts. I'd rather not change that behavior, since otherwise that would need an exact specification on how to generate a name and you're marrying yourself to that spec for the rest of time.

[artelk]

I've seen that the proposed design of the interfaces is like this:

public interface IAsyncEnumerable<T>
{
  public IAsyncEnumerator<T> GetEnumerator();
}

public interface IAsyncEnumerator<T>
{
  public T Current { get; }
  public Task<bool> MoveNextAsync();
}

I believe it is MUCH better to change the IAsuncEnumerator to this:

public interface IAsyncEnumerator<T>: IDisposable
{
  public Task<Optional<T>> MoveNextAsync();
}

And assume the the None value is returned at the end.
This would allow to parallelize the things.

Task<Something> MyOperationAsync(obj)
{
   var x = await DoSomethingSlowAsync(obj);
   await WriteToDBAsync(x);
   return new Something(obj, x);
}

Task<...> MySecondOperationAsync(...)
{
  ///...
}

var results = await GetStreamOfObjectFromRemoteService()
   .Select(o => MyOperationAsync(o))
   .Select(o => MySecondOperationAsync(o))
   .Parallelize(maxDegreeOfParallelizm: 20, maxPrefetchedItemsQueueSize: 1000) // this parallelizes everything above
   .ToListAsync();

Assuming that not only the MySecondOperationAsync is parallelized. I mean this should work the same as

var results = await GetStreamOfObjectFromRemoteService()
   .Select(o =>
               {
                   var a = await MyOperationAsync(o);
                    return await MySecondOperationAsync(a);
               })
   .Parallelize(maxDegreeOfParallelizm: 20, maxPrefetchedItemsQueueSize: 1000)
   .ToListAsync();

So each time you call MoveNextAsync() you potentially start a new task.
This could be very useful in practice to have such parallelization feature.
If the Current is a separate mutable property you wan't be able to use it in parallel.

Of course the enumerators generated by using async yield returns should put semaphore.WaitAsync() to prevent from parallelization. But the methods like Select should allow executing the continuations in parallel if the client called MoveNextAsync multiple times not awaiting the previous calls results.
@MadsTorgersen : please have a look at this :)

Btw, I think the Linq methods can be implemented directly for the IAsyncEnumerator.
Maybe IAsyncEnumerable is not needed at all.

[HaloFour]

@artelk

IIRC the problem with defining the interface like that is the loss of covariance.

[jnm2]

While we're at it, how about this?

public interface IAsyncEnumerator<T>: IDisposable
{
  public Task<IReadOnlyList<T>> MoveNextAsync();
}

Allows for batching. Empty list signals end of async enumeration.

[artelk]

@HaloFour
That's true. But maybe we would also like to use ValueTask instead of Task? So anyways we lose covariance.
We could implement an Upcast operation (safe version of Cast allowing only upcasts).

[artelk]

@jnm2
Is it supposed to preload all items before returning the list or return only currently fetched/available ones?
What is the marker of EndOfStream? Return null? please unread this

Yes, that is possible solution. But it would be more difficult to work with batches every time.
IOptional would solve the problem but that is an additional GC object.
I would prefer Optional to be a struct.

[wholroyd]

For the conversation around Extensions syntax, I don't know why we wouldn't try to limit the introduction of keywords. The addition of extension is understood, but not the addition of extends. If what I understood by the Channel9 video - extensions are classes on their own right (since they would be able to maintain their own state via statics), then the syntax and behavior of the extension class should try to follow the syntax of it just the same in my opinion.

Current syntax for extensions on a reference type...

public static class PersonExtensionClass
{
  public string FullName(this Person person)
  {
    return $"{person.FirstName} {person.MiddleName} {person.LastName}";
  }
}

Current proposed syntax for extensions on a reference type...

public extension PersonExtensionClass extends Person
{
  public string FullName()
  {
    return $"{this.FirstName} {this.MiddleName} {this.LastName}";
  }
}

My proposed syntax for extensions on a reference type is mainly around the class declaration itself, not the logic being suggested within the class definition itself. My main gripe is that the proposal is for extensions to become their own classes in itself, as they would also maintain their own state, aren't following the syntax of normal classes. If extensions should be able to extend other extensions, there is the requirement to inheir like normal classes.

public extension PersonExtensionClass<IPerson>
{
  // code
}

public extension PersonExtensionClassOfExtensionClass<IPerson> : PersonExtensionClass<IPerson>
{
  // code
}

public extension PersonExtensionClassOfExtensionClass<IPerson> : PersonExtensionClass<IPerson>, IDispose
{
  // code
}

Seems a bit more natural in this usage to me.

[jnm2]

Is it supposed to preload all items before returning the list or return only currently fetched/available ones?

The latter. It waits for at least one item and returns all the items it has at that time without waiting for more.

What is the marker of EndOfStream? Return null?

Like I said, empty list signals end of async enumeration. I think null is not a good idea.

[jnm2]

:D https://twitter.com/onovotny/status/900482078537654274

[artelk]

@HaloFour
Imagine our Extension feature is so powerful so you can specify an implicit cast from IAsyncEnumerator[TDerived] to IAsyncEnumerator[TBase] by wrapping it into an adapter.
Or you could even avoid the adapter and provide an implementation of IAsyncEnumerator[TBase] extending the IAsyncEnumerator[TDerived]. I mean something like this:

public extension IAsyncEnumeratorExtensions<TDerived, TBase> where TDerived: TBase for IAsyncEnumerator<TDerived>: IAsyncEnumerator<TBase>
{
    public Task<Optional<TBase>> MoveNextAsync()
    {
        Optional<TDerived> item = await this.MoveNextAsync();
        return item.IsNone ? Optional<TBase>.None : new Optional<TBase>(item.Value);
    }
}

So even Object.ReferenceEquals would return true.
This requires some changes in CLR. But they were mentioned in the presentation. Not sure if that supposed to be so flexible to support such generic extensions.

[IEvangelist]

I wanted to add to the conversation regarding the "extension everything" syntax. Current syntax adds two new keywords:

• extension
• extends

I think we really only need one. The extension keyword is really just an alias for static class that is intended to be an extension class. The real work horse it would seem is the extends keyword. I'd like to propose having this instead:

public class Foo extends Bar
{
   // ... extension everything stuff
}

It's really a tiny change, but avoids two new keywords when one will do. Now, wherever you see extends this is an extension class.

[artelk]

I would suggest the following design for IAsyncEnumerator:

public interface IAsyncEnumerator<out T> : IDisposable // covariant
{
    ValueTask<ResultToken?> MoveNextAsync();
    T FetchResult(ResultToken resultToken);
}

public struct ResultToken
{
    internal readonly object source;
    internal readonly long key;
    internal ResultToken(object source, long key)
    {
        this.source = source;
        this.key = key;
    }
}

public static class AsyncEnumerator
{
    public static async ValueTask<Optional<T>> FetchNextAsync<T>(this IAsyncEnumerator<T> source)
    {
        ResultToken? token = await source.MoveNextAsync().ConfigureAwait(false);
        return token.HasValue ? source.FetchResult(token.Value) : Optional<T>.None;
    }
}

public struct Optional<T>
{
    public static readonly Optional<T> None = default(Optional<T>);

    private readonly bool hasValue;
    private readonly T value;

    public Optional(T value)
    {
        this.value = value;
        this.hasValue = true;
    }

    public bool HasValue => this.hasValue;

    public T Value => this.hasValue ? this.value : throw new InvalidOperationException("Doesn't have value");

    public static implicit operator Optional<T>(T value) => new Optional<T>(value);
}

internal abstract class AsyncEnumeratorBase<T> : IAsyncEnumerator<T>
{
    private readonly ConcurrentDictionary<long, T> results = new ConcurrentDictionary<long, T>();
    private long lastKey;

    public async ValueTask<ResultToken?> MoveNextAsync()
    {
        Optional<T> result = await this.FetchNextAsync().ConfigureAwait(false); // virtual method
        if (!result.HasValue)
        {
            return null;
        }

        long key = Interlocked.Increment(ref this.lastKey);
        this.results[key] = result.Value;
        return new ResultToken(this, key);
    }

    public T FetchResult(ResultToken resultToken)
    {
        if (!ReferenceEquals(resultToken.source, this))
        {
            throw new ArgumentException("Token from different source");
        }

        return this.results.TryRemove(resultToken.key, out T result)
            ? result
            : throw new InvalidOperationException("Has been already fetched");
    }

    protected abstract ValueTask<Optional<T>> FetchNextAsync();

    public abstract void Dispose();
}

//Sample async enumerator
internal sealed class WhereAsyncEnumerator<T> : AsyncEnumeratorBase<T>
{
    private readonly IAsyncEnumerator<T> source;
    private readonly Func<T, bool> predicate;

    public WhereAsyncEnumerator(IAsyncEnumerator<T> source, Func<T, bool> predicate)
    {
        this.source = source;
        this.predicate = predicate;
    }

    protected override async ValueTask<Optional<T>> FetchNextAsync()
    {
        while (true)
        {
            Optional<T> result = await this.source.FetchNextAsync().ConfigureAwait(false); // extension method
            if (!result.HasValue || this.predicate(result.Value))
            {
                return result;
            }
        }
    }

    public override void Dispose()
    {
        source.Dispose();
    }
}

[kind-serge]

@artelk that design looks impractical. You can look at AsyncEnumerable library - the implementation can be more complex than that.

[btecu]

I'm guessing this can be closed as C# shipped with await foreach (...).