Inferencing of nullability for delegates

[sjb-sjb]

The inferred nullability of types for parameters are not always applied to occurrences of those types in Func delegates. Consider the following:

public class xTestNullabilityInference
{
    public void Create<TSource, T0, T1>(
        Action<T0, T1> action,
        Func<TSource, T0> expression0,
        Func<TSource, T1> expression1,
        T0 fallback0 = default(T0),
        T1 fallback1 = default(T1)
    ) where TSource : class, new()
    {
        action.Invoke(
            expression0.Invoke(new TSource()),
            expression1.Invoke(new TSource())
        );
    }

    public void Test()
    {
        this.Create(
            (Subject subject, int x) => { Debug.WriteLine($"subject.X is {subject.X}, and x is {x}"); },  // *** (a)
            (Subject subject) => subject,
            (Subject subject) => subject.X
        );
    }

    public void Test2()
    {
        this.Create(
            (subject, x) => { Debug.WriteLine($"subject.X is {subject.X}, and x is {x}"); },  // *** (b)
            (Subject subject) => subject,
            (Subject subject) => subject.X
        );
    }
}

Expected: no nullability warnings. The Subject parameter in the lambda expression should not be nullable, because TSource should be inferred as non-nullable in the second and third arguments to Create.

Actual: 'subject' in the 2nd and 3rd arguments to Create is inferred to be non-nullable in both the argument and the body of the lambda expression. However, 'subject' in the 1st argument to create is inferred to be nullable. On line (a), a warning CS8662 is generated on the entire delegate saying "Nullability of reference types in type of parameter 'subject' of 'lambda expression' doesn't match the target delegate 'Action<Subject?, int>'. On line (b), a warning CS8602 is generated on 'subject.X' saying that 'subject' may be null here and indicating that the type of 'subject' is Subject?.

Noting that I am working with NETStandard.Library 2.0.3 and Microsoft.NETCore.Platforms 3.1.2 due to the fact that I am targeting UWP. Would the attributes in NETCore 5.0 prevent the problem described above?

[canton7]

If you remove T0 fallback0 = default(T0), the warning goes away.

The issue is of course that T0 fallback0 = default(T0) tells the compiler that if T0 is a reference type then it can be null, which is why (presumably) it's inferring T0 as a Subject? rather than a Subject. Yes it's inferring TSource as a non-nullable Subject but that's not the source of the warnings.

[sjb-sjb]

Interesting. Seems like quite a pain in the neck, too. I have a variant of this with 8 parameters instead of just 2. I do want the fallback parameters to be optional. How can I do it? With 8 type parameters, writing out all the variations of class vs struct constraints for all of them would mean 256 signatures, which seems like about 255 too many. Any suggestions?

[canton7]

If you make fallback0 a T0?, then that shuts it up. Link.

This functionality is being introduced in C# 9. If you're using 8, then you can add where T0 : class if possible.

[sjb-sjb]

What I conclude is that nullability is not really working in 8, it does not have the tools to enforce non-nullability (if desired) or to defeat all the problems caused by the fact that nullable types are not actually types and the fact that nullability cannot be applied to unconstrained types. When I search on the topic I see so many posts that all seem to boil down to these issues.

How about having a real type system for nullable types, defining the type that "T?" represents when T is an unconstrained type, and just using attributes to tell the compiler the intended type mapping for code that was written in the old language? I feel that we have gone down the wrong path on this one ...

[sjb-sjb]

Going back to the original question, how can I work around this in C#8 ?

[HaloFour]

@sjb-sjb

This was all debated at length. NRTs had to fit into an ecosystem nearly 2 decades old, where 95%+ of the existing APIs already enforce that parameters are not null. Creating a new type system would break that entire ecosystem and require everything to be rewritten. Having a perfect type system that nobody can or will use is pointless. NRTs aren't perfect, but they don't have to be. They need to be good enough guardrails to keep most developers from accidentally tripping over common cases. This is the same approach that a lot of other languages take, like Kotlin, Swift and Typescript. This is the approach Jetbrains has used for R# and IntelliJ for C# and Java respectively.

[sjb-sjb]

True you would have to add a lot of [NotNull] attributes in the existing ecosystem if you wanted to explicitly state the non-nullness of values that was not expressible in the old language. However (a) doing so in any given case would be optional and (b) I think that you are already doing that anyway under the current approach (or doing something more difficult, actually).

I don't see why the introduction of non-nullable reference types would really be more disruptive than the introduction of nullable value types was in C# 2.0. Both cases are introducing a whole new set of types in addition to the old types. The introduction of nullable value types didn't break the existing ecosystem, neither should the introduction of non-nullable reference types.

[HaloFour]

@sjb-sjb

What you're describing would require orders of magnitude more work for everyone to adopt the language feature. As states, the vast majority of APIs that accept references expects them to be non-nullable already. So asking everyone to change their APIs would impact the potential adoption of the feature. And you'd definitely still need all of those attributes to port existing null-check helper code.

I don't see why the introduction of non-nullable reference types would really be more disruptive than the introduction of nullable value types was in C# 2.0.

Considerably more-so. Ignoring the fact that NVTs were introduced fairly early on, they also widened the potential values. It wasn't possible to express a value type that was nullable previous, so you had to adopt the new type to make null possible. With NRTs you'd trying to narrow the types that can be expressed.

In any case all of this was already discussed and debated over the last 6+ years. The team is well aware of the shortcomings of the current approach, but they were deemed much less severe than the problems with these other approaches.

[CyrusNajmabadi]

I guess what I don't understand is why creating a new type system would break the existing ecosystem

A core idea on nrt is that it comes with no runtime behavior change. It is a way of understanding existing code, what it means, and what issues it potentially has.

If you introduce a new system one of two things will be the case:

1. Existing code will not be using it, in which case it doesn't buy you anything in terms of understanding the bills of lines of code out there and what issues it may have.
2. Existing code now uses this new system. However, now the new code is using a new type system. Which means it has different meaning and behavior at runtime. This is risky and potentially introduces bugs, making people far less willing to incrementally adopt this.

Nrt is safe to adopt and can be gradually and gently brought into an existing codebase. This affect of it was arguably the most important part of the design. There are other designs, but they would violate this, and we felt that would greatly diminish the value and adoption of the feature.

The introduction of nullable value types didn't break the existing ecosystem,

There is a very big difference introducing something a couple of years into a language, certain introducing it 20 years in. They are not comparable situations.

[sjb-sjb]

@CyrusNajmabadi , @HaloFour , I understand your perspective. Yes, I am sure that these issues were thought about in detail, and no doubt by people such as you who have far more language and compiler experience than myself. I know there are a lot of practical considerations that go into these things.

At the same time I am struggling with problems such as the one described in this issue, which remains unsolved. I am still interested in any solutions that can be suggested (other than, I suppose, just ignoring the compiler warnings). The fact that this seems to be an unsolvable problem triggered by adopting NRT, gives me the feeling that NRT in C#8 is incomplete ... it didn't feel like an easy adoption when I hit this.

[sjb-sjb]

Continued: From some of the comments above it seems that maybe I was not being fully clear on what I was suggesting. Imagine for a moment that instead of adding "?" notations, we had instead added a set of types, call it T! for each reference type T, with T! meaning a non-nullable reference type. You could relabel these T? and T instead of T and T!.

Edit: I know this has been discussed, I've seen a few of the problems with it. Not speaking as an expert but I do kind of think the problems could be solved in most cases if you allow lazy initialization. In other words, basically non-nullable and nullable types would be the same except a non-nullable reference type would throw whenever you do anything with it (like, assign it to a variable/parameter/return value, or compare it for equality) instead of only throwing when you de-reference it to access a member. So you would get "UninitializedNonnullableException" with T! far earlier than you get "NullReferenceException" today. Also of course you would not be able to assign a null to a nonnullable reference type variable (i.e. no nulls other than due to the lazy initialization). There would be a default(T!) which would be null, it's just that you would not be able to use it for much except as a parameter default. You might want to allow uninitialized non-nullable references to be copied as part of copying a struct that contained them (but if accessed separately they would throw).

On the upside, you would have the advantages of type checking and not having to guess what the compiler can or cannot infer about nullability, and would not have to create workarounds when the compiler cannot infer something.

As for the existing code base, it would not be impacted since existing code would continue to be valid and have the same meaning as before (relabeling T as T? if that syntax change were adopted). If you want to improve the old code, just start adding "!" (or deleting "?") on the public signatures e.g. where you have already documented that the value should not be null. This would not have to break other old code that called into the improved version; the compiler could make new code callable from old code by adding null checks when calling from old code into 'new' public signatures that involve non-nullable reference types. Some trickery might be needed to skip these redundant null checks when calling from new code to new code.

@CyrusNajmabadi, while I think the goal of improving the existing code base is laudable, I believe it is equally valid to add great new features to the language without impacting the old code base at all.

@HaloFour, in the approach I describe there would be no change to existing APIs and nothing forcing people to go back and apply attributes to the old API either. Also, in most cases the code in the new language would be using non-nullable variables, and this would match the fact that most of the old APIs expect non-null values.

Again, any solutions for the problem described in the issue above?

[HaloFour]

@sjb-sjb

The solution for C# 8.0 is to ignore the warnings, or to disable them for at least that code. Yes, I know that isn't ideal, but NRTs are a work in progress. Changes will be released in C# 9.0 that should improve these scenarios and some others as well. If you find any other cases where either it feels that the NRT analysis is failing you or you can't figure out how to influence it in a manner you want I would suggest to post more issues here. The team is interested in improving the ergonomics of the feature and making it less painful to use.

The ! notation for non-nullable reference types was suggested. In fact it was the first flavor of the proposal going back some six years. The problem is that requiring an explicit opt-in for what is already expected of the vast, vast majority of existing code would only reduce adoption and force a lot of existing code to be modified for no reason. And it still wouldn't solve the issues with generics that you're experiencing.

[CyrusNajmabadi]

The fact that this seems to be an unsolvable problem triggered by adopting NRT, gives me the feeling that NRT in C#8 is incomplete

NRT is incomplete. And will likely be incomplete as long as we're attempting to type the existing .net/c# type system. It is intended to be a continued work in progress where new language versions improve things in ways we think will substantively close the gap between NRT's expressiveness, the runtime's true nature, and the complexity of understanding waht a program is doing.

[CyrusNajmabadi]

I am still interested in any solutions that can be suggested

For 8.0, suppress the warning.

(other than, I suppose, just ignoring the compiler warnings).

If you want a language solution to this, you'll have to wait for 9.0. That's the version that includes changes to help support this case.

[CyrusNajmabadi]

I believe it is equally valid to add great new features to the language without impacting the old code base at all.

We do this. But that's not what NRT is. :)

[sjb-sjb]

I agree T?/T is better than T/T!, and as it happens I don't mind the idea of changing the language syntax. What would be the downside of having a full type system? The approach I described would not break existing APIs although it would require living with a compiler switch (not a problem from my point of view). As for improving old code, that could happen by applying the input/output attributes to old code and having the compiler check the conditions they express, and/or by migrating the old code to the new syntax over time.

[sjb-sjb]

As for the workaround, what I decided to do is get rid of the default values of the parameters. This can be accomplished if you are willing to swear enough ... default(T0)!.

[CyrusNajmabadi]

What would be the downside of having a full type system?

You'd now have two type systems.

[theunrepentantgeek]

For an example of the level of pain that would be caused by two parallel type systems, have a look at the experience of the Python community - Python 3 was a breaking change released in 2008 and it took twelve years before they finally managed to sunset Python 2. Even so, many significant Python 2 projects have not moved forward.

The C# language design team are well aware of these kinds of issues and bifurcation of the .NET ecosystem is something they are keen to avoid.

[sjb-sjb]

Well, Python 3 was not backward compatible with Python 2, while what I’m describing would be 100% backward compatible. If you don’t think it would be 100% backward compatible then let me know what you think wouldn’t work. To Cyrus’s comment, what I meant was two types, T and T?, not two incompatible type systems.

What’s done is done, though ...

[CyrusNajmabadi]

To Cyrus’s comment, what I meant was two types, T and T?

How do you propose that would work?

[sjb-sjb]

@CyrusNajmabadi , I described it in my second post yesterday, I have edited for clarification.

Noting though that someone pointed out that introducing a type T! was one of the first possibilities considered. Whatever the reason was for not taking that approach, it may also apply to what I described.

[sjb-sjb]

Closing this as the answer seems to be that C#9 will solve the problem in this example.

[sjb-sjb]

@CyrusNajmabadi commented that the purpose of NRT is to enable improvement of past code, and that we don't want to break existing code. And @theunrepentantgeek pointed out the pain that was caused in Python by making a breaking change.

At the same time, where do we want to be in 5 or 10 years? Do we want to still be in a place where we do not really definitely know that a non-nullable variable actually is not null? Do we still to still be testing for null and throwing runtime exceptions? I believe the answer is we want to eventually get to a place where non-nullable reference variables are definitely always not null, so that runtime tests and exceptions are eliminated. That means eventually we want to get to a typed approach to nullability.

So ... what about eventually introducing "#nullable typed" which would compile as a typed-based system. It seems eminently do-able to figure out the interaction between code compiled this way and code compiled with, say, #nullable enable.