Nullable backing fields

[sjb-sjb]

Summary

Features are proposed to support nullable backing fields for non-nullable properties ("late initialized properties"). The keyword 'is' would be used in addition to 'get' and 'set' (and the proposed 'init' and 'req') to signal that the backing field should be nullable. There would be appropriate changes to the behavior of 'get' when 'is' is specified.

This proposal follows on discussion of 'req' in #3630 and would co-exist well with other Property-related proposals such as 'init' and the 'field' keyword (#140).

Motivation

The proposed 'req' and 'init' keywords address the need for flexibility in initializing non-nullable properties, especially non-nullable reference properties, during construction and related initialization stages. This provides a compile-time enablement of deferred initialization and required initialization. However, there are many scenarios where initialization cannot occur early in the object lifetime i.e. at or near construction time.

An example is an object whose meaning in the domain / model design involves required properties, but where the operational process for initializing those properties involves a UI / user interface design. The properties need to remain null until they are filled in on the UI. It would not be appropriate to make all properties nullable in the domain model simply to accommodate a UI process. On the other hand it would be too laborious to create a separate object with nullable properties for use in the UI and then have to copy the values into the 'real' object. What makes sense in such scenarios is for the properties to remain uninitialized at construction time and then be initialized (and then validated) during the UI interaction.

Nullable backing fields for non-nullable properties are an excellent way to deal with these "late initialization" scenarios. Here is a simple example of a property we can use for discussion:

public Wheel FrontLeft {
    get => this._wheel ?? throw new UninitializedPropertyException();
    set => this._wheel = value;
}
private Wheel? _wheel; 

So far so good. Once the property is initialized by assigning through 'set', the domain model is complete in the sense that the property can be used "normally" from then on. Prior to the late initialization, the result of 'get' should be undefined and correspondingly this design would throw an exception.

However, there is a significant gap in this design because there is no encapsulated way to test whether or not the property has been initialized. This is a real problem because you cannot call 'get' in order to test whether or not the property has been initialized -- you would get an exception. And you do need to test for initialization as part of validating the object. An incredibly ugly solution would be to add

public bool FrontLeftIsInitialized { get => this._wheel != null; } 

The purpose of the proposal is to enable better use of such properties before the late initialization has occurred, by adding an 'is' accessor that tests the nullable backing field for null.

Proposal

1. The 'is' keyword would be introduced in a syntactic position similar to the existing 'get' and 'set' keywords. (This would not conflict with other syntactic uses of the 'is' keyword. }

2. The body of 'is' would return a boolean value. In custom properties, this would be defined by code provided in a boolean-valued expression or block following the 'is' keyword -- for example this user-provided code could test whether a backing DependencyProperty is null.

3. In auto-implemented properties where 'is' is present, the backing store would be a nullable field defined as follows:

3a. If the property has a reference type, then the backing store would be the corresponding nullable reference type (in #nullable disable environment, therefore, the backing store would have the same type as the property while in #nullable enable it would have the same type but be marked as nullable).

3b. If the property has a value type T then the backing store type would be Nullable<T>,; except that if T is already Nullable<S> then the backing store would be just T itself. As with reference types, then, the backing store type is a nullable version of the property type, unless the property type is already nullable in which case the backing store type would be the same as the property type. While it would be legal to include 'is' on a property that is already nullable, there would be little reason to do so.

4. In auto-implemented properties where 'is' is present, the body of 'is' would return true if and only if the backing store is not null. In custom properties where 'is' is present, as noted above the boolean result would be defined by user code.

5. In auto-implemented properties where 'is' is present, the 'get' method would throw an UninitializedPropertyException if the backing field is null. Note this would apply both for reference types and for value types. In custom properties where 'is' is present, the compiler would check that the value returned by 'get' is definitely assigned, and if the property is a reference type or a nullable value type then run-time behavior of 'get' would be modified to throw an exception if the return value is null. The purpose here is to pretty much require the programmer of the custom property to throw an exception (or have it thrown for them) if the backing store is null.

6. A new expression would be introduced to execute the 'is' accessor. There are various ways that this could be done, at this time my suggestion would be (in the example above) "car.FrontLeft is null" to test whether the 'is' accessor returned false. Alternatives would be along the lines of either "car.FrontLeft.is" which is not very aesthetic, or "car.FrontLeft != null" which is convenient but introduces obvious problems, or even possibilities like "car.FrontLeft !null" which looks pretty funny at first but actually would be more convenient to write than !(car.FrontLeft is null).

7. In terms of nullability inferencing, use of the the 'is' accessor described in 6 would not result in the inference that the property is nullable.

8. If 'is' is specified then a LateInitializedAttribute would be applied to the property by the compiler.

Interaction With Other Features

A. As relates to 'req' and 'init', the 'is' operator could be specified in addition to these or without these. However there would be limited value to specifying both. If you specified both 'is' and 'init', the effect would be that the property could only be initialized in the construction phase; it would be an immutable property so after the construction phase the result of 'is' should not change and 'get' would either always return the immutable value of the property or would always throw. Adding the 'req' keyword would result in 'is' always returning true and 'get' always returning the immutable value.

B. As relates to the proposed 'field' keyword, there would be no special interaction with 'is' other than the fact that 'field' could be accessed within the body of 'is'.

C. Concerning the use case of UI initialization and the need for data validation, I believe the reference implementation of RequiredAttribute uses the 'get' accessor to test whether the value is null or empty. This would not work well with a late-initialized property because the 'get' accessor could throw an exception. The attribute could be modified to first test for the LateInitializedAttribute and if it is present then use the 'is' accessor instead of the 'get' accessor. If this is not deemed sufficiently backwards-compatible then an alternative would be to introduce a new InitializedAttribute derived from ValidationAttribute. However in this case one would also have to do something to guarantee that if both [Initialized] and [Required] are specified then [Initialized] is tested first.

Unresolved Points

• When both 'is' and 'init' are specified for a custom property it would be appropriate to somehow force the result of 'is' to be immutable. The discussion in (A) above only says that 'is' "should not change", not that it "will not change". An additional mechanism is needed, for example either restricting the values that can be accessed within 'is' or else freezing the value somehow at the end of the construction phase.

[theunrepentantgeek]

This seems like a niche concern that can easily be solved in existing code; I don't see how it reaches the required threshold for consideration as a language feature.

If implemented, I foresee that it would seldom be used because the required behaviour for partially initialized objects would vary from team to team, from project to project, perhaps even from form to form within a specific project. It's therefore highly likely that any specific implementation would only handle a small fraction of cases.

Why is a library solution inappropriate?

I had a quick play using SharpLab and was able to quickly knock together this example:

public class C {

    private Optional<int> _age;

    public int Age 
    {
        get => _age;
        set => _age = value;
    }
    
    public bool HasAge => _age.HasValue;
}

The definition of Optional<T> is straightforward, and accommodates most any kind of wrapped value:

public struct Optional<T>
{
    private bool _hasValue;
    private T _value;
    
    private Optional(T value) 
    {
        _hasValue = true;
        _value = value;
    }
    
    public bool HasValue => _hasValue;
    
    public static implicit operator T(Optional<T> optional) 
    {
        if (!optional._hasValue) 
        {
            throw new InvalidOperationException();
        }
        
        return optional._value;
    }
    
    public static implicit operator Optional<T>(T value)
    {
        return new Optional<T>(value);
    }
}

FWIW, wrappers like this are one of the very few places where implicit type casting is really useful and seldom causes grief.

[wanton7]

@sjb-sjb How often have you actually needed something like this for it to warrant a language feature? As a senior C# programmer having more stuff that will throw exceptions at runtime is ingredient for bad design. If I would see code like this in our company's codebase it would need a REALLY good reason blow up at runtime if it doesn't it will need to be refactored.

[sjb-sjb]

Well, I need it in most cases where the domain property is not nullable but it needs to be entered on the GUI. This is practically everything (FirstName, LastName) except where the domain design says that the value is optional (MiddleName). As I discussed above, the two obvious alternatives in the language currently for dealing with such a situation are both very unsatisfactory.

@theunrepentantgeek interesting idea. A comment though: Optional<T*gt; in your example is similar to Nullable<T>, in other words what you have there is a starting point very similar to the Wheel example that I gave but using a value type (int) instead of a reference type (Wheel). A conversion of a null Nullable<T> to a T would also throw an exception similarly as Optional<T> does. Looking at it, you are proposing a workaround by defining HasAge, similar to the FrontLeftIsInitialized workaround in my discussion. The point of the proposal is that we should not be forced into such workarounds.

The invention of non-nullable reference types has enabled us to express nullability in the language in a way that we were not able to before. We need to continue adding language support for using non-nullable types. Specifically in this proposal I am suggesting a way to express the relationship between a non-nullable property and a nullable backing field. It is a fundamental relationship that has hardly any language support at all currently. The proposed 'req' and 'init' keywords do provide some support for this relationship but they focus only on the relationship at construction time (initialization at construction). The 'is' keyword would address the relationship between non-nullable properties and nullable storage at run time (late initialization).

Also @wanton7 I know there are various philosophies on how to deal with incorrect calling code, whether to be resilient and respond as best possible or whether to blow up fast and loud so that the caller has to fix their code. Personally I'm in the latter camp. However this is a separate issue from what I am trying to address here. Potentially one could change the proposed language feature so that the "if uninitialized" action was configurable.

[wanton7]

@sjb-sjb by GUI do you mean what? UI in browser and something like ASP.NET Core? If it's something like that you should have separate classes that you model bind to and validate. It might feel like it laborious as you said but you should still do it. If you are creating an API then throwing an exceptions will cause 500 http status to be sent. It is not good to be doing that. You should be sending caller info what failed to validate.

Also by your explanation you are in the latter camp because you would throw exception later from uninitialized property but validation logic would actually take place earlier. So it would fail even faster.

[sjb-sjb]

Hi @wanton7, I do actually have separate view model classes that are used in binding properties to the UI. I won't go into details but your comment about the ordering of validation and initialization doesn't apply in my situation -- my philosophy about failing fast is actually what I said it is. Anyway, I agree that one possible approach is to make sure you have all the values off of the UI before construction of the model object. But there are many other approaches, and these should be supported rather than constrained by the language.

Specifically I believe that the language should have more support for late initialization of properties (uninitialized non-nullable properties). If you would like additional use cases, just consider all the places where people are assigning null! to initialize non-nullable properties and then having the value filled in by some other means, potentially after construction of the object. The language should have a way to better support that kind of situation. One can argue philosophically that this kind of thing shouldn't happen in the first place, but if we really believed that then we wouldn't allow "null!" to begin with (and actually I've argued for that kind of strict typing approach in the past). Be that as it may, the fact is that nulls are allowed into non-nullable properties, so, given that fact, we should provide more support for managing such situations.

[wanton7]

In our company's code bases we also separate models used in API and models used internally or just use API models properties are methods parameters. Actually this is one place #3630 would help by marking properties required if you add new property to class you use internally then compilation would fail if you forgot to copy it. In those API models we define only properties that are optional nullable. Because ASP.NET Core model binding already supports C# 8.0 nullable reference types we don't need to check them for null https://docs.microsoft.com/en-us/aspnet/core/mvc/models/validation?view=aspnetcore-3.1#required-attribute Non nullable reference types behave like they have [Required] attribute.

[wanton7]

@sjb-sjb And our codebases are mostly null! or default! free. We do use null! when creating EF Core contexts like this DbSet<UserEntity> Users { get; set; } = null!;. In some API request models we have some null! that would take inner models but ASP.NET Core model binding makes sure they are not null. If property is string we assign them to string FooBar { get; set; } = string.Empty instead using null!. But besides model binding and EF Core context our other code is completely null! free.

[sjb-sjb]

@wanton7 in EF Core you can avoid null! by using something like "public DbSet<Customer> Customers => this.Set<Customer>();" in your DbContext.

[canton7]

I'm surprised no one's mentioned it yet, but source generators seem like a good fit here? They can easily generate the property if you define the backing field, the getter can throw whatever donain-specific exception you need, and it can generate a suitable HasXXX property for you while you're at it.

There's an example if implementing INPC with a source generator in the cookbook: this would be very similar.

[sjb-sjb]

Hi @canton7 actually that's exactly what I'm doing, with T4. I just think I shouldn't have to write it all myself, the language should provide some support. Also I think that having a separate HasXxxxx property floating around in the class is just gross.

[JustNrik]

[MemberNotNullWhen("IsFrontLeftInitialized", true)]
public Wheel? FrontLeft { get; set; }
public bool IsFrontLeftInitialized { get; set; }

iirc this will be valid syntax which solves your use case, I think?

[333fred]

It's the inverse.

public Wheel? FrontLeft { get; set; }
[MemberNotNullWhen(true, nameof(FrontLeft))]
public bool IsFrontLeftInitialized { get; set; }

[sjb-sjb]

Again, in the example I want Wheel FrontLeft to be non-nullable, and I don't think we should have to have helper properties such as IsFrontLeftInitialized floating around.

[theunrepentantgeek]

I just think I shouldn't have to write it all myself, the language should provide some support.

The language does provide support - lots of it. You seem to be saying that you don't want to use it, and it needs to be something new. _If a problem can be solved by a library, or by tooling, then it should.

Your use case is very specific - there's a lot of very particular policy that you want, and that's the crux of the problem.
The language shouldn't be enforcing specific policy, because different teams take a different approach, so any specific policy will have limited applicability. C# isn't a language for @sjb-sjb alone, it's a language for everyone.

I don't think we should have to have helper properties such as IsFrontLeftInitialized floating around.

An alternative solution - one that I used with great effectiveness in Delphi - is to provide decorated properties. Harking back to my earlier example, you could write something like this:

public class C {

    private Property<int> _age;

    public Property<int> Age 
    {
        get => _age;
        set => _age = value;
    }
}

The struct Property<T> would be relatively similar to Optional<T>, but with slightly different consumption. If you wanted to test if the property has been set, you'd write c.Age.HasValue instead of c.HasAge.

[sjb-sjb]

@theunrepentantgeek Well, of course I don't think the language is just for me. Language is shared. But do you believe that the combination of non-nullable property and nullable backing store is specific and not very common? That would seem to be the crux of the matter. Because if it is very common then this is a piece of support for it that I believe would make sense.

Anything can be solved with a library. Your solution is a bit better than having the Has method outside the property. One would have to look at how it plays with EF. However generally I believe the combination of nullable backing store and non-nullable property is common enough to warrant language support in the form of an operator for testing whether the property has been initialized or not.

[YairHalberstadt]

The main thing I'm not understanding is what is so ugly about FrontLeftIsInitialized .

You want to be able to do the following things -

1. To be able to initialize wheel.
2. To only be able to access wheel once it has been initialized.
3. To check whether it's been initialized.

Therefore you need a:

1. setter for wheel
2. getter for wheel which throws if it's null
3. a property telling you if wheel is initialized.

That seems like really good clear code to me - it provides a clear, straightforward easily understood API which provides exactly the operations you need to do.

Now you could provide some language magic to allow you to do 3, and hide the necessary property from you, but why on earth would I prefer magic over clear code?

If your complaint is it's too much boilerplate to write, then I think source generators are a great solution to that.

[sjb-sjb]

It’s funny how something that seems like a no-brainer to one person seems opaque and excessive to others. Look at the effort being made, and rightly so, to eliminate the line that declares the backing field by introducing the ‘field’ keyword. To me, if we believe that nullable backing stores for non-nullable properties are a common pattern then this proposal is not very different in terms of the saving on boilerplate and increased encapsulation.

When you look at the long discussion about the challenges of having a full type system for non-nullables and the decision to use annotations instead of types, a fair bit of it revolves around the fact that if you have non-nullable reference types then a lot of problems are introduced by the fact that they cannot be allocated in arrays, properties and elsewhere because there is no default value for them. For people who want to use nullables as if they were types to the extent possible, there are only two solutions to such situations. One is you can organize all of your code around the principle of initializing everything at construction time. The proposals for ‘req’ and ‘init’ address this approach. The other is you can let the non-nullable be created with a null backing value and then make sure it is initialized before use. The ‘is’ proposal is meant to address that approach.

In other words, unless you want to say that the language should push everyone to initialize every non-nullable property at construction time, you have to accept that many people will want to initialize them after construction time. Then how are we to support that common situation? The ’is’ accessor does so.

[HaloFour]

unless you want to say that the language should push everyone to initialize every non-nullable property at construction time

Yes, the push is that every require property must be initialized at construction time. The compiler can't offer anything to help your scenario of lazy initialization with a totally unknown lifespan. There are no warnings that the compiler can provide because the compiler can never know whether or not all of the members are initialized at the point that you want to use them. And runtime enforcement of said rules goes against very explicit design decisions that have already been made around NRTs and are likely to also be made around required properties.

IMO, what you're asking for is rife with policy concerns and shouldn't be embedded into the language. It's exactly the kind of scenario that would be solvable with source generators as you can choose what it considered "initialized" and what to do with the property when it isn't. You can choose that a null non-nullable reference type will throw, when the language has already established the exact opposite behavior. You can choose that a "required property" is only satisfied when a specific value is assigned, which is not a part of that proposal. You can choose to throw at runtime, which the language very likely will not do due to the cost of wrapping every single accessor with validation.

[sjb-sjb]

Well, I agree that the construction-time-only approach is a coherent language philosophy. I can see a lot of favorable aspects of it.

In the GUI use case I am describing, without applying workarounds this language philosophy has very specific implications such as the target model entity cannot be constructed until all of view model properties have been supplied by the user. This means entity view models cannot be dependent on model entities (as is done for example in the Template10 validation / UI design) but must instead be free-standing independent objects. When I started my development that indeed was the approach I took but I quickly abandoned it in favor of having the entity view models hang off of entities and creating them lazily (and referencing them weakly, by the way) because it greatly simplifies the management of the objects. There are various pluses and minuses. At any rate, which of these approaches to take should be up to the system designers rather than being driven by the language.

To me, it feels like specific complex design approaches and implications in this case are being caused by a language philosophy. From that point of view I would rather have the language be a bit more agnostic as to run-time vs compile-time checking of initialization. At the same time, I see the appeal of having non-nullables always actually be non-null after construction time -- it brings us closer to a pure type system, which I initially argued in favor of. On the third hand, the fact that we have the null-forgiving operator reduces our ability to rely on non-nullables actually being non-null in the first place, so, being this deeply stepped ...

Commenting on the overhead of wrapping accessors with null-checking, this would only occur if the programmer requested it by using the 'is' keyword. So, not something that would interfere with existing uses or add overhead when it is not desired.