[Proposal]: Nullable unconstrained generic types

[MrTA2]

Nullable unconstrained generic types

• Proposed
• Prototype: Not Started
• Implementation: Not Started
• Specification: Not Started

Summary

[summary]: Implement nullable unconstrained generic types, like class Foo<T> { T? Bar() { ... } }

Motivation

[motivation]: This would get rid of the CS8627, and practically, allow the creation of methods such as FirstOrNull on generic collections, among other things.

Detailed design

[design]:
A new type, UniversalNullable<T>, is added to the framework:

public struct UniversalNullable<T>
{
  public UniversalNullable(T value) { this._hasValue = true; this._value = value; }

  public bool HasValue=>this._hasValue;
  public T Value => this._hasValue ? this._value : throw new Exception("...");

  bool _hasValue; T _value;
}

Whenever the C# compiler encounters any usage of T?, and T isn't constrained with class, struct, or a base type, it substitutes the above UniversalNullable<T> type. All "get" operations are wrapped in a .Value property get (optimizer to inline the getter body); "set" operations are replaced with either new UniversalNullable<T>() or new UniversalNullable<T>(value), depending on whether the value is null. All this can be mostly copied from the Nullable<T> implementation for structs. Then we can write code like this:

class Foo<T> where T : notnull // notnull is needed, otherwise we'll create nullable of nullable which is pointless
{  
  public Foo(T t)=>this._t=t;

  readonly T _t;

  public T? Bar(T? input)
  {
    if(input != null) // compiles to "if(input._hasValue)" - use the field for performance
    {
      // input is not null here
      T res = this.Bar(input); // compiler implicitly resolves "T" by using "._value" (the field), to make it faster: T res = this.Bar(input._value);
      return res; // compiles to "return new UniversalNullable<T>(res);"
    }
    T res2 = this.Bar(input); // warning, possible null; still compile using ".Value" (the property), getter body inlined for performance
    string str = input.ToString(); // warning, possible null; still compile using ".Value.ToString()" - unlike Nullable<T>, never call UniversalNullable<T>.ToString
    string? str2 = input?.ToString(); // OK; if T has interface constraints which have ToString overloads, those are available here (same as above line - even if the above line generates a warning and crashes at runtime)
    return null; // compiles to "return new UniversalNullable<T>();"
  }
  public T Bar(T input) // overloading works
  {
    return input;
  }
  public async Task<T?> BarAsync()
  {
    await Task.Delay(100);
    if(DateTime.Now.Hour < 12) return this.Bar(null); else return this.Bar(this._t);
  }
}
class Extensions
{
  public static T? FirstOrNull<T>(this IEnumerable<T> This)
  {
    foreach(var x in This) return x; //  "return new UniversalNullable<T>(x);"
    return null; // "return new UniversalNullable<T>();"
  }
}
var foo = new Foo<int>(123);
int? x = foo.Bar(null); // resolves Bar(T?); in IL, convert the result from UniversalNullable<int> to Nullable<int> before assignment
int x2 = foo.Bar(456); // resolves Bar(T)
var foo2 = new Foo<string>("abc");
string? y = foo2.Bar(null); // resolves Bar(T?) aka Bar(UniversalNullable<string>); in IL, extract the result from UniversalNullable<string> to string before assignment
string y2 = foo2.Bar("def"); // resolves Bar(T) aka Bar(string)
var foo3 = new Foo<int?>(123); // error

Drawbacks

[drawbacks]: This obviously introduces a minor performance reduction for reference types, since it has the field referencing overhead. (For struct types, it's a given, similarly to Nullable.)

Alternatives

[alternatives]: Leave things as is, keep generating CS8627. When type safe FirstOrNull methods are needed, implement two versions, one for classes and one for structs.

Unresolved questions

Design meetings

[AntiPasha]

I guess, that it was already covered in #3297

[MrTA2]

I guess, that it was already covered in #3297

I saw it, it's a related discussion, but it doesn't solve the issue once and for all, which this proposal does.

[YairHalberstadt]

I think there's a lot of edge cases which need to be considered here.

For example:

public interface I
{
    string String { get; }
}

#nullable disable
public M1<T>(T t) where T : I => Console.WriteLine(t == null ? "" : t.String);
# nullable enable

public M2<T>(T? t) where T : notnull, I => M1(t); // What happens here? Does this call `M1<UniversalNullable<T>>`?

Personally I think there's way too much that can go wrong with these kind of tricks. It just becomes too risky to put into the language.

[YairHalberstadt]

I'd say yes, if nullable is disabled, it just does T, so your code creates a recursive endless loop.

Where's the loop? it's calling M1?

[MrTA2]

public M2<T>(T? t) where T : notnull, I => M1(t); // What happens here? Does this call `M1<UniversalNullable<T>>`?

(Sorry I misread your code, didn't notice M1 vs M2)
This would fail to compile, because UniversalNullable doesn't implement I.

[YairHalberstadt]

This would fail to compile, because UniversalNullable doesn't implement I.

What about cases where it's more subtle.

For example, if T implements IEquatable<T>, and you create a Dictionary<T?, string> the dictionary will use ReferenceEquality instead of IEquatable.Equals since UniversalNullable doesn't implement IEquatable

[MrTA2]

For example, if T implements IEquatable<T>, and you create a Dictionary<T?, string> the dictionary will use ReferenceEquality instead of IEquatable.Equals since UniversalNullable doesn't implement IEquatable

Yes, but arguably Dictionary<,> should have where TKey:notnull which would prevent this, and the programmer should know not to do this anyway; think about it, purposefully supplying a nullable key type to a dictionary is nonsensical.

[YairHalberstadt]

Yes, but arguably Dictionary<,> should have where TKey:notnull which would prevent this, and the programmer should know not to do this anyway; think about it, purposefully supplying a nullable key type to a dictionary is nonsensical.

As it happens dictionary throws on a null key, but I have implementations of IDictionary where this is not the case and it's perfectly sensible.

Besides IEquatable is just an example. There are many cases where code dynamically checks for an interface. What if T implements IDisposable?

[333fred]

Unconstrained T? already means something in C# 9. Changing that will be a breaking change. As much as I would like to change Nullable<T> and make it generally applicable to all types, we have 20 years of library design not taking this into account, and we heavily discussed doing something like this as part of nullable only to conclude that that history is too high a barrier.

[MrTA2]

As it happens dictionary throws on a null key, but I have implementations of IDictionary where this is not the case and it's perfectly sensible.

Actually I just checked and both Dictionary<,> and IDictionary<,> already do where TKey:notnull, so I'm not sure what you're talking about.

Besides IEquatable is just an example. There are many cases where code dynamically checks for an interface. What if T implements IDisposable?

The same problems occur with Nullable<T> and struct types. If you had SomeClass<int?> and that class were to do T x=...; if(x is IFormattable) { ... } it would fail the same way. If we accepted this drawback with Nullable<T>, there's no reason to also accept it with UniversalNullable<T>.
This is wrong as pointed out below (otherwise string.Format("{0:000}", (int?)123) wouldn't work)

[MrTA2]

Unconstrained T? already means something in C# 9. Changing that will be a breaking change. As much as I would like to change Nullable<T> and make it generally applicable to all types, we have 20 years of library design not taking this into account, and we heavily discussed doing something like this as part of nullable only to conclude that that history is too high a barrier.

I'm not proposing to change Nullable<T> - that would definitely be a non-starter. I'm specifically proposing creating a new type, UniversalNullable<T>, just for this solution.

I do understand that unconstrained T? was implemented in C# 9, but I think that was a mistake. I wish I could propose this sooner. Is it at all possible to take that implementation out of C# 9 for now and revisit in C#10?

[YairHalberstadt]

Nullable<T> does all sorts of runtime magic to implement the same interfaces as T implements. You would need runtime magic for this as well.

[MrTA2]

Nullable<T> does all sorts of runtime magic to implement the same interfaces as T implements. You would need runtime magic for this as well.

True, I retract what I said previously. However, I did point out in my proposal that a lot of these under-the-hood shenanigans can be copied and pasted from Nullable<T> or just expanded to accept both Nullable<T> and UniversalNullable<T>.

[333fred]

I wish I could propose this sooner.

Again, we considered this. In depth. And we went forward with the conscious decision that we were blocking this feature.

[YairHalberstadt]

If you're requiring runtime magic, why not just get this to work as is without introducing a new magic type? The runtime jits separate code for struct T and reference type T so these would probably be significantly simpler.

[MrTA2]

If you're requiring runtime magic, why not just get this to work as is without introducing a new magic type? The runtime jits separate code for struct T and reference type T so these would probably be significantly simpler.

The magic type is needed to: 1) house the hasValue and the value; 2) allow method binding and overload resolution cleanly.

[YairHalberstadt]

The magic type is needed to: 1) house the hasValue and the value; 2) allow method binding and overload resolution cleanly

Why?

Just generate the same IL as you would if T wasn't Nullable, and then the runtime will be responsible for handling the case where T is a struct.

[MrTA2]

The magic type is needed to: 1) house the hasValue and the value; 2) allow method binding and overload resolution cleanly

Why?

Just generate the same IL as you would if T wasn't Nullable, and then the runtime will be responsible for handling the case where T is a struct.

I thought about that before. The problem is that for nullability to work with struct types, you need a "hasValue". And once you accept that premise, you realize that you end up creating 2 versions of the code for each unconstrained nullable generic parameter. C# needs to support multiple generic parameters, so it becomes 2^N versions of a method; before starting any execution, a special IL needs to be output which checks all the generics and jumps to the correct version. All this is very inefficient, both space and performance wise. I understand the upside of being able to reuse Nullable just for structs in this case, but the downsides are much worse. Furthermore, when the execution leaves the generic-aware context, more difficulties arise - how do we explain to the callers of our generic methods that sometimes they have to unbox Nullable, and sometimes not? In short, just packaging the references into a UniversalNullable, while adding a minor overhead, is much simpler: no multiple versions of the code; standard logic for get/set operations; etc. I think this is the only viable approach.

[333fred]

The problem is that for nullability to work with struct types, you need a "hasValue".

So, leaving off the issue compat for a moment, this is only kinda true. In an ideal world, we'd have the runtime niche-fill this. Structs do need a bool for tracking, yes, but reference types do not. They can be represented by the null reference. And if you make the runtime aware of this, and specialize the implementation, then it's totally viable. This is how rust is able to represent Option<ref T> as just 0, and would really be the only version of this I'd be interested in. I'll convert this to a discussion for continued conversation, just remember we did consider this at length.

[TahirAhmadov]

I mean that I think that what C# 9 did with T? was a mistake for 2 reasons. 1, it adds an inconsistent behavior between references and values - null vs "zero", whatever that happens to be. 2, it makes it impossible to do a proper solution (like the one I described above) with the syntax which would intuitively fit it - T?.

Water under the bridge. Going forward, and I understand it's not a high priority item (especially given the rather poor IMHO solution of "defaultability" being denoted by T?), I would implore the language team to consider doing something like UniversalNullable<> with a different symbol or keyword.

[CyrusNajmabadi]

I would implore the language team to consider doing something like UniversalNullable<> with a different symbol or keyword.

I'm not sure such a concept can exist. This is literally not how the CLR works here.

[TahirAhmadov]

Why not? What part of my proposal exactly goes against how the CLR works?

[CyrusNajmabadi]

Can you link to your proposal and indicate what it would compile down to.

[TahirAhmadov]

Please see thread post. I apologize in advance if I missed anything or the formatting is not up to standard - I don't create these often.

[TahirAhmadov]

I swear, I knew this would happen. I ran into a real world problem specifically because of the lack of this feature. Consider the following:

class Base<T> where T : notnull
{
  public virtual T? Prop { get; set; } // I can't do this, because if T is struct, this becomes just T
  public virtual Null<T> Prop { get; set; } // so I had to write my own "UniversalNullable"
}
class ChildClass<T> : Base<T> where T : class
{
}
class ChildStruct<T> : Base<T> where T : struct
{
}
public struct Null<T> where T : notnull
{
  public static implicit operator Null<T>(T value) => new Null<T>(value);
  public static explicit operator T(Null<T> nullable) => nullable.Value;
  public static bool operator ==(Null<T> first, Null<T> second) => first.Equals(second);
  public static bool operator !=(Null<T> first, Null<T> second) => !first.Equals(second);
  //public static bool operator ==(Null<T> nullable, object? obj) => nullable.Equals(obj);
  //public static bool operator !=(Null<T> nullable, object? obj) => !nullable.Equals(obj);
  //public static bool operator ==(object? obj, Null<T> nullable) => nullable.Equals(obj);
  //public static bool operator !=(object? obj, Null<T> nullable) => !nullable.Equals(obj);
  //public static explicit operator Null<T>(object? obj) => default;

  public static Null<T> Unbox(object? obj) => obj != null ? new Null<T>((T)obj) : default;

  public Null(T value) { this._hasValue = true; this._value = value; }
  public bool HasValue => this._hasValue;
  public T Value { get { if (!this._hasValue) throw new InvalidOperationException("Nullable object must have a value."); return this._value; } }
  public object? Box() => this._hasValue ? this._value : null;
  public override bool Equals(object? obj) { ... }
  public bool Equals(Null<T> obj) { ... }
  public override int GetHashCode() { ... }
  public override string ToString() { if (!this._hasValue) return ""; return this._value.ToString(); }
  readonly bool _hasValue;
  readonly T _value;
}
public static class Null
{
  public static Null<T> Create<T>(T value) where T : notnull => new Null<T>(value);
  public static Null<T> CreateClass<T>(T? value) where T : class => value != null ? new Null<T>(value) : default;
  public static Null<T> Create<T>(T? nullable) where T : struct => nullable.HasValue ? new Null<T>(nullable.Value) : default;
}
public static class _Null_T_Extensions
{
  public static T? GetOrNull<T>(this Null<T> This) where T : class => This.HasValue ? This.Value : null;
  public static T? GetOrNullStruct<T>(this Null<T> This) where T : struct => This.HasValue ? This.Value : null;
  public static TProp? IfNotNull<TValue, TProp>(this Null<TValue> This, Func<TValue, TProp> getProp) where TValue : notnull where TProp : class
	{ if (This != default) return getProp(This.Value); return null; }
  public static TProp? IfNotNullStruct<TValue, TProp>(this Null<TValue> This, Func<TValue, TProp> getProp) where TValue : notnull where TProp : struct
  { if (This != default) return getProp(This.Value); return null; }
}

Basically, almost everything C# and .NET do today behind the scenes for Nullable<T>, in terms of boxing and unboxing, equality, and conversions, I had to write myself and call it explicitly. That's the first big problem - I have to remember how to code "the right way" to use my Null<T>. Second, the syntax is off; I couldn't make it work so that Null<T> x; if(x != null) { } and x = null; all work - the only consistent syntax was if(x == default) and x = default;; so in code using this, default means null. Third, there were issues because interfaces don't work with implicit operators, so I had to add static methods to handle those. Fourth, interface propagation and type lifting don't work because that would require CLR changes (thankfully, I don't need them yet). Fifth, operators like ?. and ?? don't work, so I had to create extension methods IfNotNull, etc.
Needless to say, none of these would be an issue if this feature was added to the language.