IRefEnumerable support to foreach and array

[Thaina]

Follow up from return ref feature. I would like to have IRefEnumerable<T> interface included in core C#

public interface IRefEnumerable<T>
{
    IRefEnumerator<T> GetEnumerator()
}

public interface IRefEnumerator<T>
{
    ref T Current { get; }
    bool MoveNext { get; }
}

Which should be implemented natively to array. It should support foreach too. And there should be whole set of Linq implementation support ref argument

But, well, ref linq may be need to ask in corefx (I was once create #7978). But aside from that, native array and foreach support on this feature would require roslyn support I think

[svick]

How would foreach work? Would it use the same syntax or would there be ref somewhere?

And is the goal to allow writing to the collection inside foreach? E.g.:

var array = new int[10];
foreach (ref int i in array)
{
    i = -1;
}

[Thaina]

@svick Yes, I wish it would work like that. And it would be used to manipulate collection of large struct like matrix. But well, writing collection like that in foreach would be neat too

I wish compiler for foreach would extend to look for IRefEnumerable<T> first then IEnumerable<T> and finally the plain IEnumerable. And array would implement IRefEnumerable<T> so foreach on array would always yield the ref item

Not sure we could use var to assume ref ?

var array = new int[10];
foreach (ref var i in array)
    i = -1;

foreach (var i in array) // possible?
    i = -1;

[svick]

I wish compiler for foreach would extend to look for IRefEnumerable<T> first then IEnumerable<T> and finally the plain IEnumerable.

That's not in line with how it works now. The compiler first looks for a GetEnumerator() method and only if it doesn't find one, it looks for an enumerable interface. Maybe for ref foreach, the method could be named GetRefEnuemrator()? That way, struct enumerators could work even with ref foreach.

[Thaina]

Oh yes, that's right. We actually need

public interface IRefEnumerable<T>
{
    IRefEnumerator<T> GetEnumerator()
}

public interface IRefEnumerator<T>
{
    ref T Current { get; }
    bool MoveNext { get; }
}

[alrz]

@svick

The compiler first looks for a GetEnumerator()

AFAIK, for arrays, foreach translates to a regular for, so I think your first example should work as expected because no enumerators is created by foreach.

[svick]

@alrz Right, arrays (and dynamic) are a special case mentioned in the specification.

[svick]

@Thaina I still think you can't reuse the GetEnumerator name, given that:

1. struct enumerators are an important optimization and will still be that even for ref enumerables.
2. Existing types already expose non-ref struct enumerables

Consider List<T>, the relevant parts of its implementation look like this (simplified):

public class List<T> : IEnumerable<T>
{
    public struct Enumerator : IEnumerator<T>
    {
        List<T> list;
        int index;

        internal Enumerator(List<T> list) { … }

        public T Current => list[index];

        …
    }

    public Enumerator GetEnumerator() => new Enumerator(this);

    IEnumerator<T> IEnumerable<T>.GetEnumerator() => new Enumerator(this);
}

With the GetRefEnumerator name, you could mirror the same scheme for ref enumerator:

public class List<T> : IRefEnumerable<T>
{
    public struct RefEnumerator : IRefEnumerator<T>
    {
        List<T> list;
        int index;

        internal Enumerator(List<T> list) { … }

        // assuming ref T Get() on List<T>
        public ref T Current => ref list.Get(index);

        …
    }

    public RefEnumerator GetRefEnumerator() => new RefEnumerator(this);

    IRefEnumerator<T> IRefEnumerable<T>.GetRefEnumerator() => new RefEnumerator(this);
}

I don't think you could do the same if the method was named IRefEnumerable<T>.GetEnumerator.

[Thaina]

Well, that's right. So I think we need to extend foreach to support anyway

[DavidArno]

@Thaina,

Can you suggest a use-case for these IRefEnumerable's types?

[Thaina]

@DavidArno Using foreach for collections of Vector2 3 4 up to Matrix4x4

Well, if you never handle large struct with bulk operation you would never understand

[DavidArno]

@Thaina,

I get the idea that some folk have the need to process large amounts of data in an effective way and ref's assist with that need. What I don't get is the desire for IRefEnumerable. A more intrinsically mutable RefArray<T> type seems more appropriate for collections of ref'ed data. Why would you need IRefEnumerable?

[Thaina]

@DavidArno for foreach that's why

I propose it as interface because we could make any kind of collections for our needed. And the main point I post this issue because foreach

for(var i = 0; i < LongNameStructArray.Length; i++)
{
    LongNameStructArray[i].Position = Vector3.Zero;
}

// vs

foreach(ref var obj in LongNameStructArray)
{
    obj.Position = Vector3.Zero;
}

[Thaina]

Oh I forgot that I should also propose ref yield

Is it possible for ref yield to enumerate things though?

[DavidArno]

@Thaina,

You can foreach over an array. Mutating an ref version of IEnumerable whilst enumerating it implies it will be re-enumerated, which breaks the cardinal rule for IEnumerable.

[Thaina]

@DavidArno Why you think it must be implied?

We always mutate reference type in foreach and it never need to be reenumerated

Changing value of a memory block, be it a reference type or ref of value type, is still the same memory block and it never need to reenumerate the same pointer

[DavidArno]

@Thaina,

What is the point of mutating those elements of the enumeration, if they are never read again?

[Thaina]

@DavidArno Why you think it never be read? After foreach loop that array could be read from anywhere. In game engine there would be tree of matrixes that loop to update each of them in hierarchy and then it will be read by physics / graphics / logic / network / etc / etc / etc / etc / etc

As I said, if you never handle large struct with bulk operation you would never understand

[DavidArno]

@Thaina

Why you think it never be read?

Because you said "We always mutate reference type in foreach and it never need to be reenumerated". But now you are saying that such collections are re-read by "physics / graphics / logic / network / etc / etc / etc / etc / etc".

So you are right, I do not understand (still) why you want IRefEnumerable, rather than RefArray even though the latter would seem a better fit for the scenarios you are describing.

[Thaina]

@DavidArno Man, reenumerated and another system would read it is not the same word

Why you pick the word reenumerated?

Really, have you ever mutate array of class in your life? Did you know you can mutate value of class in list when you foreach?

Need example?

class SomeData { public float x,y,z; }
var datas = new List<SomeData>() { new SomeData(),new SomeData(),new SomeData() };
foreach(var data in datas)
    data.x = Random();

See it obviously? Mutation in foreach is such normal things and could be done in any collection. I just want it to extend to collection for struct

Now I think you just misunderstand everything. With ref to struct it never change pointer in the array. Each item in array of struct is memory block of it own if you reread it would also be the same pointer

And why you are so attached to your RefArray? It's me who should ask why you still want your RefArray while IRefEnumerable is also work and it let you able to implement any kind of collection in the future, not just array

[HaloFour]

@Thaina

Collections like List<T> wouldn't be able to expose their internal storage as ref as it would be simply unsafe to use. That would be true of probably the majority of the collections. Arrays are of the few cases where it would be safe because the array itself can't change or move so the references to the elements will remain in scope for as long as the array does.

[Thaina]

@HaloFour I know. But with interface we could just implement our own structure ourself and that's what I ask for. I don't ask that we must implement IRefEnumerable to all collection currently exist

And foreach on ref collection does not move anything. It may change value in the block but it must not move or change the size of the collection while iterate. The array itself would not changed, only the value of each item can change

struct Matrix4x4 {}

SomeRefCollections<Matrix4x4> ms;
foreach(ref var m in ms)
    m.Position = Vector3.Zero;

// Changing value of m never change anything of the collection

[DavidArno]

@Thaina

And foreach on ref collection does not move anything. It may change value in the block but it must not move or change the size of the collection while iterate. The array itself would not changed, only the value of each item can change

With ref, the following would be possible:

struct Matrix4x4 {}

SomeRefCollections<Matrix4x4> ms;
foreach(ref var m in ms)
    m = new Matrix4x4();

But you appear to not want that functionality. It appears that all you want is for a foreach over a collection of structs to:

1. Not do a value copy, thus saving time, and
2. Allow you to break the "structs should be immutable" rule by modifying the contents of those structs within the loop.

Is that a correct summary of what you've been asking for here?

[Thaina]

@DavidArno Kinda and yes

I think I am fine if you new struct in foreach loop. Normally we would not write code like that but maybe it would be useful, I don't know

Something like

struct Matrix4x4 {}

SomeRefCollections<Matrix4x4> ms;
foreach(ref var m in ms)
    m = Matrix4x4.Identity;

But it has no reason to not be allowed if we could return ref

And yes for 1 and 2

[omariom]

It is supported for pattern based enumerables