Feature request: An "undefined" value in addition to "null"

[vhe1]

As web backend developer, the thing that is hardest for me is to deal with the two ways javascript frontends have to do nulls, undefined (member not in json) and null (member in json and has null assigned).

I'm constantly in discussions with my client developers and this is no fun at all.

The typical usage is described in rfc 7386, it's obvious, sensible and there is no way at all for us in the backend to deal with it because we only have null.

Of course, we could deserialize the json into Dynamic or a set of nested Dictionary<String,Object> structures but that would lose all type safety and convenience. Then we might as well use Typescript (which has it) and node.js for backend programming.

This is the biggest gripe I currently have.

So, is there a way to allow us backend developers to correctly derive the client's intent from a json with nulls and missing members?

Maybe another method for ValueType would do it for the simple stuff? No idea about references though. On the other hand, C++ has so many pointers now, maybe c# can stand another reference type?

[MgSam]

This sounds like a serialization issue, not a language issue. As you yourself say, it's horrible that JavaScript has two ways of saying null, so we wouldn't want to propagate that mistake to other languages.

[Pzixel]

@vhe1 null itself is a billion-dollar mistake and you want to just square it. It's JS problems that it has two entities to describe the almost same thing - absence of something. In my practice no one cares if value is not setted at all or it's just an unusable value. If field is required - you are rising an error, if it isn't then both are valid. But you have a lot of problems with passing null or undefined guidelines or comparing with absence... It's a no-no feature, don't even think about it.

[vhe1]

I did not say that "it's horrible that JavaScript has two ways of saying null", I said it's horrible that we can't use the information contained in this in a typed backend.

Re "null is a billion dollar mistake", that's just wrong. The problem is not that there is null but that there is just one of it, forced to express two different intents. (Please remember the discussons in SQL. They focused precisely on the fact that it was not clear which null.) Having two different nulls solves that.

[svick]

@Pzixel

In my practice no one cares if value is not setted at all or it's just an unusable value. If field is required - you are rising an error, if it isn't then both are valid. But you have a lot of problems with passing null or undefined guidelines or comparing with absence... It's a no-no feature, don't even think about it.

@vhe1 links to an RFC for JSON Merge Patch, which is an actual specification that differentiates between undefined and null. So it may not be a choice to "not think about it", if you need to interact with a system that uses that format.

[HaloFour]

Optional<T>. Done.

[svick]

@vhe1 The problem with null in a statically typed language like C# is that you don't have a choice about using it, it's always a valid value. C# is trying to move away from that (see #36). Adding a second value that is valid for all reference types would be a move in the other direction and, in my opinion, a bad idea.

I think what you should do is to use the type system to your advantage: instead of a property of some type T, use a property of type, say, Value<T>, whose API looks something like this:

public struct Value<T>
{
    public static Value<T> Undefined { get; }
    public static Value<T> Null { get; }
    public static Value<T> Create(T value); // throws if value is null

    public bool IsUndefined { get; }
    public bool IsNull { get; }
    public T Value { get; } // throws exception when IsUndefined == true or IsNull == true
}

Then you teach your JSON serializer about Value<T>, and you're set. This has the advantages that it limits undefined to your serialization code and makes it very clear that undefined (and null) is a possibility.

There is no need to pollute all of C# with this.

[vhe1]

I've tried stuff along those lines but either it looks horrible in any swagger definition or it has a hole somewhere else.
Maybe it's also telling that there is no nuget package for this yet.

But I'll try some more then.

Thanks a lot for bothering!

[aluanhaddad]

Newtonsoft.Json.NullValueHandling.Ignore

[vhe1]

aluanhadad: How does this help me during deserialization?
The point is that I get json like this and want to extract the null/undef information from it.

[aluanhaddad]

@vhe1 1

I'm may have written something a bit wrong below, I am really tired 😪

So the simple scenario is C# producer, using Newtonsoft (the JSON one, not the DSON one), paired with a JavaScript consumer using JSON.parse and JSON.stringify.

To do this, we need to form a semantic bridge, across JSON, from C# types (CLR) to JavaScript values (ECMAScript) types. JSON has structural types only. There are no named constructs, it is nice and simple. It also happens to be a rough subset of ECMAScript literals which makes this mapping relatively straightforward.

The idea is basically to adopt the view that null in C# maps to undefined in JavaScript.¹
This means that null is not serialized into any of your responses and that it is not plucked from any of your requests.

We can enforce this as follows:
C#

class Person
{
    public string Name { get; set; }
    public int? Age { get; se; }
}

var person = new Person
{ 
    Name = "Anders"
}; // Age is null

JSON

{
  "name" : "Anders"
}

JavaScript

// literal from JSON.parse(response.data)
const person = { 
  name: 'Anders'
};

So we don't know the age of our person on either side and our REST interface allows it to be unspecified. So all is well.

Now when the JavaScript client wants to send me some values, he can do anything he wants

1. Nice JavaScript approach

const person = {
  name: 'Douglas'
};

request.data = JSON.stringify(person); // JSON => { "name": "Douglas" };

2. Ugly JavaScript approach (bad style)

const person = { 
  name: 'Douglas',
  age: null
};

request.data = JSON.stringify(person); // JSON => { "name": "Douglas", "age": null };

Thanks to Newtonsoft.Json.NullValueHandling.Ignore, the C# code on the server will simply ignore the difference between null and undefined (missing) that the client produces. This works partially because I made Age of type int? but you have to in order to map these correctly regardless of how you handle serialization.

C#

public HttpResponseMessage Post(Person person)
{
    var person = JsonConvert.DeserializeObject<Person>(Model);
    Console.WriteLine(person.Name); // => "Douglas"
    Console.WriteLine(person.Age); // => null
}

(1) This applies equally to TypeScript (obviously as TS is JS). Interestingly, while the community is somewhat divided, a fair number of ECMAScript experts such as Douglas Crockford, as well as the TypeScript team, have stated that null is unnecessary and have removed it from their code bases, include the TypeScript compiler.

[vhe1]

You are still trying to solve the wrong problem. The problem I want to solve is to READ a json object according to rfc 7386 as I wrote in my initial feature request posting. I'm not interested in creating such a json, I want to READ it and in the generated c# object distinguish between JSON null and JSON undefined, in order to properly deal with it according to that rfc. So, according to rfc 7386, {name: "XXX", age: null} and {name:"XXX"} have two distinct meanings. The first one updates the name and /deletes the age from whatever persistence layer there is/, the second one updates the name and /leaves the age alone/.

[Pzixel]

@vhe1 then you just should write your own deserializer and wrap every propery with Option<T>. C# shouldn't implement such feature because there is some language implementing it. It easily can be done if serializer distinguish between these two states. And probably JSON.Net is following it when using JObject.Parse. The problem is that JavaScript (and thus JSON specification) mix dictionary and ordinal object type. In C# you just cannot have undefined - all objects of class A always has all properties. On the other hand we have Dictionary<string, object> and you always know when value is not present in dictionary (ContainsKey -> false) or value is null (foo["bar"] == null).

[vhe1]

@Pzixel, I understand that, I was just replying to aluanhaddad.

[aluanhaddad]

@vhe1 I misunderstood your use case. You are dealing with partial updates. In that case, I think the solution is indeed what @HaloFour and @Pzixel are suggesting.

You can wrap your fields in an optional value. You can configure your serializer to understand it, which is not hard.

The Option<T> type itself is trivial to write.

[john-cd]

Sidebar: the R language has both NULL and NA. Missing values are represented by the symbol NA (not available).

Having NA is really useful in stats, data science, etc...

[PaulGarnerUk]

@jnm2

If you don't know ahead of time which JSON attributes will exist, I'd say it's more useful to inspect a JSON DOM than to deserialize to a POCO

I don't understand how checking a DOM will help? We still need to deserialize to a C# object, and the C# object needs a property with three possible states: value specified, explicit null value, or unspecified value.

@Pzixel - I at least feel you understand the issue, thanks. 👍 The Optional<T> in Roslyn is useful, but it needs a custom de/serializer amongst other workarounds. As I've said - this is the route I'm using to tackle it, but it hasn't been easy.

[YairHalberstadt]

How would you Serialise a C# object that had an undefined value for one of its properties?

You'd end up putting in a feature request to add undefined to Json. And then you'd need to add reallyReallyUndefined to C#. And the never ending cycle continues...

[HaloFour]

@PaulGarnerUk

IMO, the real problem is that the BCL doesn't have a proper stock Optional<T> type. If it did, all of the standard serializers could support it out of the box and you wouldn't have to do anything aside declaring your fields as that type.

[vdachev]

I've been writing C# code for the last 13 years and I've already mentioned several cases when an undefined would make life much easier. LINQ query syntax was not required, neither was the ? for INullable, using for IDisposable... but they make C# development a charm.
I'm not saying the solution is necessarily introducing an undefined keyword. It could be some language syntax extension around Optional, or a feature like a union (as @orthoxerox suggested).

@theunrepentantgeek , does ACM advise against the very existance of null values? Nope, and the C# team makes an effort in 8.0 to add safety checks, not to scap null altogether. JavaScript lacks those safety checks and that's the reason undefined is an issue. TypeScript has the language features to enable developers to avoid problems with both null and undefined inherited from JavaScript.

@CyrusNajmabadi , in SQL you can choose which values to include in an INSERT and UPDATE statements, and WHERE clauses. Entity Framework Core handles these scenarios with its own workaround - object tracking.

@YairHalberstadt ,

How would you Serialise a C# object that had an undefined value for one of its properties?

The property will not be present in JSON - that's what undefined is for, unassigned, not set...

[PaulGarnerUk]

How would you Serialise a C# object that had an undefined value for one of its properties?

Turtles all the way down! No, by not specifying the property/value in JSON, then by definition it has an undefined value (neither a tangible value, nor null). Undefined does not get serialized/deserialized, as opposed to null which, as you know, can be serialized.

[jnm2]

@PaulGarnerUk

We still need to deserialize to a C# object, and the C# object needs a property with three possible states: value specified, explicit null value, or unspecified value.

Why is this the only option? I can imagine taking code which uses the POCO and rewriting it to use the DOM. For something that has a very dynamic schema, this might even be preferable.

[CyrusNajmabadi]

It sounds like people are talking about an API they'd like, or the behavior of existing APIs. I still haven't seen a use case fo C# having an actual undefined value.

[CyrusNajmabadi]

but it needs a custom de/serializer amongst other workarounds

Wouldn't 'undefined' be the same way? All these tools and apis would have to be updated to handle this new intrinsic value. If they can be updated to handle 'undefined', why can't they be updated to handle 'Optional'?

[theunrepentantgeek]

The crux of the problem seems to be the desire to have a distinct value (a sentinel value) that means don't overwrite.

To illustrate, you can use null as the sentinel value that means keep the existing value:

public class Person
{
    public string FullName { get; private set; }
    public string KnownAs { get; private set; }
    public string FamilyName { get; private set; }

    public void Patch(Person other)
    {
        if (other.FullName is not null) FullName = other.FullName;
        if (other.KnownAs is not null) KnownAs = other.KnownAs;
        if (other.FamilyName is not null) FamilyName = other.FamilyName;
    }
}

But that doesn't let you patch existing properties to null, because you're using null to indicate don't overwrite.

So you introduce a new sentinel value, let's say undefined.

public class Person
{
    public string FullName { get; private set; }
    public string KnownAs { get; private set; }
    public string FamilyName { get; private set; }

    public void Patch(Person other)
    {
        if (other.FullName is not undefined) FullName = other.FullName;
        if (other.KnownAs is not undefined) KnownAs = other.KnownAs;
        if (other.FamilyName is not undefined) FamilyName = other.FamilyName;
    }
}

The problem is, now you can't patch existing properties to undefined, because that's the sentinel value that indicates don't overwrite.

All you've done is to move the problem down the road one step.

The problem is caused by the use of a sentinel value instead of a distinct flag.

What I did, when I needed to solve this exact problem, was to create a struct that captured whether I'd specified a value or not. With epic amounts of imagination, I called it Specifiable<T>.

[yaakov-h]

The problem is, now you can't patch existing properties to undefined, because that's the sentinel value that indicates don't overwrite.

And with NoSQL systems, this is actually a perfectly valid use-case.

[vdachev]

but it needs a custom de/serializer amongst other workarounds

Wouldn't 'undefined' be the same way? All these tools and apis would have to be updated to handle this new intrinsic value. If they can be updated to handle 'undefined', why can't they be updated to handle 'Optional'?

So were all the API that use Task, Nullable, etc.

[CyrusNajmabadi]

@vdachev Exactly. Task was added to the framework and the libraries updated to use it. No need to make a language change here. This should likely be moved to corefx with a request to make a suitable Optional/Some/None type for this purpose.

[vdachev]

@vdachev Exactly. Task was added to the framework and the libraries updated to use it. No need to make a language change here. This should likely be moved to corefx with a request to make a suitable Optional/Some/None type for this purpose.

The point in this discussion is that there are lots of scenarios when an undefined-like feature would be extremely helpful. Every library/solution addresses the problem in a different way and once you get need to "bridge" between a few technologies (e.g. between MVC and Entity Framework Core) you essentially have to "translate" between them. A standard solution would certainly reduce the effort of developing these bridges.

Or course, it all depends on the way such support is implemented:

• One way is to get a decent Option support in BCL (.NET Standard?) first and then get syntactic sugar around it in the language (as @HaloFour suggested).
• Implement it as a language feature over CLR (or DLR).
• Something else, I can't think of now.

[CyrusNajmabadi]

The point in this discussion is that there are lots of scenarios when an undefined-like feature would be extremely helpful. Every library/solution addresses the problem in a different way and once you get need to "bridge" between a few technologies (e.g. between MVC and Entity Framework Core) you essentially have to "translate" between them. A standard solution would certainly reduce the effort of developing these bridges.

As mentioned, the standard solution would be from Corefx, not from C#. C# is not the standard. It's one of many languages here. You want a corefx solution so that tehse libraries can actually adopt it. I recommend you open a bug there. If tehre is some library that is created, and it becomes popular enough, there can be consideration around if there should be a language feature.

Implement it as a language feature, if the CLR (or DLR).

I don't know what this means.

[jnm2]

DBNull.Value is crying and wondering why everyone is looking for a new sentinel value.