Optional ref and out parameters, and pass-by-value for ref parameters

[dmchurch]

The language currently does not let one specify default values for ref or out parameters, and thus, ref/out parameters are always mandatory parameters. in parameters, on the other hand, do allow default parameters, and the compiler treats omitted arguments the same way it would a constant literal argument passed to the in parameter, by creating an unnamed temporary variable in the caller's stack frame and passing a ref to it.

It would be very handy to be able to omit both ref and out parameters from a method invocation, as would being able to pass a constant literal or evaluated expression as an argument for a ref parameter as you can for an in. The semantics seem straightforward, as they're essentially the same as for the case of an omitted in parameter: the compiler allocates space for the value in the caller's stack frame and, in the case of a defaulted or constant-literal ref parameter, assigns the value to that space. As an example, a dictionary that indexes on case-insensitive strings and maintains insertion order, and which allows inserting into an empty slot after a lookup fails:

public class CaseInsensitiveOrderPreservingStringDictionary<TValue>
{
    private List<string> keys = new();
    private List<TValue> values = new();

    public int Count => keys.Count;

    // A reference to a dictionary location that has just been looked up, can index this dictionary
    public readonly ref struct Slot
    {
        public required readonly string Key { get; init; }
        public required int Index { get; init; }
        public bool HasValue => Index >= 0;
    }

    public TValue this[string key]
    {
        get => TryGetSlot(in key, out Slot slot) ? GetFromSlot(slot) : throw new KeyNotFoundException();
        set => this[GetSlot(in key)] = value;
    }

    public TValue this[Slot slot]
    {
        get => GetFromSlot(slot);
        set => SetFromSlot(ref slot, value);
    }

    /// <summary>
    /// Extended implementation of IDictionary.ContainsKey. Checks whether the key is present, optionally
    /// normalizes the key itself, optionally beginning the search from a given index, and optionally returning
    /// the dictionary index of the key when found.
    /// </summary>
    /// <param name="key">The key to check for. If passed as a ref, it will be normalized before return.</param>
    /// <param name="index">When passed as ref or in, this is the index to start checking from.
    ///                     When passed as ref or out, it will contain the dictionary index of the key afterwards,
    ///                     or -1 if the key was not present.</param>
    /// <returns>
    /// True when the key was found, false otherwise.
    /// </returns>
    public bool ContainsKey(ref string key, ref int index = 0) => GetSlot(ref key, ref index).HasValue;

    /// <summary>
    /// Extended implementation of IDictionary.TryGetValue. Attempts to fetch the value for the given key, optionally
    /// normalizes the key, and optionally returns a Slot which can be used to set the value for that key.
    /// </summary>
    /// <param name="key">The key to look up. If passed as a ref, it will be normalized before return.</param>
    /// <param name="value">Set to the value found, or to default(TValue) if not.</param>
    /// <param name="slot">If provided, this will be set to a Slot that can be used to index this dictionary.</param>
    /// <returns>
    /// True when the lookup succeeded, false otherwise.
    /// </returns>
    public bool TryGetValue(ref string key, out TValue value, out Slot slot = _)
    {
        if (TryGetSlot(ref key, out slot)) {
            value = this[slot];
            return true;
        } else {
            value = default;
            return false;
        }
    }

    /// <summary>
    /// Extended implementation of IDictionary.TryAdd. Attempts to add a new key/value pair to the dictionary if the
    /// key does not already exist. Optionally retrieves the value stored there if the key is already present, and
    /// optionally returns a Slot which can be used to index into this dictionary.
    /// </summary>
    /// <param name="key">The key to look up. If passed as a ref, it will be normalized before return.</param>
    /// <param name="value">The value to add. If passed as a ref and the key is already present, it will be
    ///                     updated with the currently-stored value for the key.</param>
    /// <param name="slot">If provided, this will be set to a Slot that can be used to index this dictionary.</param>
    /// <returns>
    /// True when the provided value was inserted into the dictionary, false when the key was already present.
    /// </returns>
    public bool TryAdd(ref string key, ref TValue value, out Slot slot = _)
    {
        if (!TryGetSlot(ref key, out slot)) {
            SetFromSlot(ref slot, value);
            return true;
        } else {
            value = GetFromSlot(slot);
            return false;
        }
    }

    public bool TryGetSlot(ref string key, out Slot slot, ref int index = 0)
    {
        slot = GetSlot(ref key, ref index);
        return slot.HasValue;
    }

    protected Slot GetSlot(ref string key, ref int index = 0)
    {
        key = key.ToLowerInvariant();
        index = keys.IndexOf(key, index);
        return new Slot
        {
            Key = key,
            Index = index,
        };
    }
    protected TValue GetFromSlot(Slot slot)
    {
        return slot.HasValue ? values[slot.Index] : throw new KeyNotFoundException();
    }
    protected void SetFromSlot(ref Slot slot, in TValue value)
    {
        if (slot.HasValue) {
            values[slot.Index] = value;
        } else {
            slot = slot with { Index = Count };
            keys.Add(slot.Key);
            values.Add(value);
        }
    }
}

I'd suggest that when passing a constant literal or expression to a ref parameter, that the ref keyword be forbidden at the callsite, in order to avoid violating programmer expectations that using ref results in a change of caller-visible state. For example, a getter-only (and non-ref) property or a readonly field on an object could reasonably be used as a pass-by-value source - imagine passing an array's Length field to a Read method as an in/out parameter - it represents the size of the buffer on entry, and it is assigned the amount of data read prior to exit. If ref were allowed on a pass-by-value argument, then the following would be valid syntax:

reader.Read(buffer, ref buffer.Length);

That looks like the code is able and expected to modify the length of the array itself! Instead, I'd recommend using the in keyword to specify an argument that is passed by value to a ref parameter, since overloads can't distinguish between in and ref. That still isn't quite right, as a callsite in usually means that the value is being received in a readonly context, while this is being received as a live ref. You could, similarly use out for an optional ref parameter, and the calling method would be responsible for initializing it to the default value.

For optional out parameters, I'd suggest that the declaration syntax be an assignment of the discard _ identifier. That mirrors the syntax that you need to use today, in which you specify out _ as the argument at the callsite. Callsite syntax isn't affected for out parameters, as there is no concept of a "value" to pass in.

Critically, the invoked method cannot tell the difference between an argument passed by copied-value or by true reference; the semantics and codegen of the called method does not change with the addition of optional parameters. As far as the called function is concerned, it has been passed a reference.

This change would promote code reuse, as functions can be called in one of several ways according to need, and from a codegen and performance performance standpoint, it's effectively free - these types of parameters tend to be used for things the method was going to be allocating local storage anyway, so it's just a form of caller-mediated stack setup.

[CyrusNajmabadi]

In general, i'm in support of this.

[HaloFour]

This would be a chance to "clean up" some of the COM interop behavior that C# has quietly offered since C# 4.0 but never supported in our own APIs.

[TahirAhmadov]

Is it really that much work to write out _?
Also, the use case is a little weird, why not use IEqualityComparer<string> which does case-insensitive comparisons?

[dmchurch]

It's not so much trouble to write a single out _, no. But the point of this is that it becomes very easy to move a stack variable from inside a method to outside of it. Instead of a code example (which I'll agree is utterly contrived, but consider it from the perspective of a more complicated normalization function than ToLower) I'll give you a process example:

Oftentimes when I'm writing a class, especially when it encapsulates some kind of expensive behavior, I'll start out by writing the expensive algorithms in private/protected methods, and once they're working I'll write the public "API" methods using the implementation methods as black-box building blocks.

This works great until I realize that just one of the API methods needs some information from inside one of those black boxes - that is to say, some internal part of the implementation needs to get exposed. At this point I have a few options:

1. I could write a second version of the implementation method which just calculates and returns the internal value. This might be able to use shortcuts since it's not performing the full operation, and if so, this might be a worthwhile solution, but otherwise it'll involve a lot of code duplication and harm CPU cache locality.
2. I could extract the part of the implementation method into a helper method which is called by the new API, and the original implementation method calls the helper. This is usually the right choice, but it gets more complicated if the setup calculation extracted into the helper method sets up, say, ten internal variables, only one of which needs to be exposed to the API. Now, in order to satisfy both the new API and the implementation, the helper method has to have ten out parameters, and the API method needs nine out _ arguments, which doesn't look fantastic.
3. If the setup calculation is more involved than makes sense to extract, you could add the out parameter to the original method. Then, to avoid breaking code, you either have to add an overload with the original signature to avoid changing any of the other call sites, or just add an out _ argument to every call.

Allowing out parameters to have the discard token as a default means that you get both call signatures in one method. If you have three default-discard out parameters, they take the place of up to eight boilerplate overloads - and the programmer doesn't have to decide which overloads to actually add to the class, weighing the increase in API complexity and IL size against the likelihood that those overloads actually get used. And, again, changing a local variable to an out parameter is very nearly a free operation, as far as performance is concerned.

Personally, I think the reduction in boilerplate alone makes this worthwhile. I also think that something like the out Slot call signature oughta get added to the standard IDictionary interface, given the extremely common C# pattern of if (!dict.TryGetValue(key, out value)) dict[key] = value = GetValue();, and given that the lookup process is generally the most expensive part of any map-like data type.

[CyrusNajmabadi]

Reopening. I think this idea has merit and should be discussed.

[dmchurch]

Sure, I'm willing. Did you have any questions or thoughts on it?

[franchyd]

This feature would allow me to remove many passthrough overloads from my Geometry and LinearMath utilities. Is this one of those things that's stuck in the "lobby for a champion" state where it's generally liked but lacks an LDM driver because of higher priorities?

[jaredpar]

This proposal basically has two components:

1. Ability for ref / out to be optional parameters
2. Ability for non-location arguments to be passed to ref / out parameters

For (1) I could see a potential use case for this. There is prior art to look at with our COM interop story but that has a very low usage rate. I think the proposal though still lacks compelling use cases and without those I doubt this would ever get championed.

For (2) I think it is unlikely to be accepted by LDM even with a champion. Let's take this argument for (2) and examine it:

Critically, the invoked method cannot tell the difference between an argument passed by copied-value or by true reference; the semantics and codegen of the called method does not change with the addition of optional parameters. As far as the called function is concerned, it has been passed a reference.

This is correct from the perspective of the invoked method. Whether the ref / out parameter is a temp or named location is immaterial to a large degree. This is not correct from the perspective of how the method is used. Many APIs are designed to work with real named locations, not temporaries.

Consider the constructor ReadOnlySpan<T>(ref readonly T reference). This constructor originally used in parameters but this lead to problems consuming the API. Very subtle differences in how the argument was passed to the constructor had dramatic impact on the lifetime of the constructed value and it constantly tripped customers up. To the point that one of our big motivations for doing ref readonly parameters was to fix the use case problems with this constructor. This is why I doubt a request to allow out / ref / ref readonly accept non-location arguments will get championed or approved by LDM. The topic's already been visited and it was decided strongly that we should not be doing this.

[franchyd]

Yeah I really just was hoping for case one, where I have many overloads such as:

bool Intersects(Line a, Line b);
bool Intersects(Line a, Line b, out Vector2 intersection);
bool Intersects(Line a, Line b, out Vector2 intersection, out float ax, out float bx);

Where the minimal overload is the most used so better for usability but the most specified one is used occasionally so still needed. While minimal to just declare the overloads to passthrough to the more specified implementation with discard parameters, it would be nice to not need to.