Unable to serialise null values while ignoring fields not specifically set

[TSlump]

Elastic.Clients.Elasticsearch version: 9.1.4

Elasticsearch version: 9.0.7

Description:
I'm currently migrating from the old Nest package to the new elasticsearch package, however the serialiser behaviour seems to have flipped... Before, it seemed that the Nest serialiser would include null values, unless they weren't specifically set in the model to null, then it would ignore them.
With the new elasticsearch package, the serialiser seems to be ignoring null values by default. This is causing issues when updating a record with a null value, as the update will be ignored...

I can flip the behaviour back in the client serialiser global settings:
options.DefaultIgnoreCondition = JsonIgnoreCondition.Never;
However, this is causing issues around the elastic security models such as: IndicesPrivileges as the model includes a query model with a large amount of fields. We don't set most of them, but as null isn't ignored the request is failing as some of those fields cannot be null. As this isn't our model we cannot add [JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] to these fields...

Alternatively, I could leave the serialiser as it is, and add [JsonIgnore(Condition = JsonIgnoreCondition.Never)] to almost every field in each model we have. I believe that this would work, however it feels quite drastic and not the intended method. I wanted to check if there is a simpler approach that I am missing, this worked well with the Nest client so I would be surprised if that previous behaviour had been scrapped.

Thanks !

[flobernd]

Hi @TSlump ,

this change in behavior is not explicitly driven by the Elasticsearch client, but it's definitely expected.

NEST uses the Utf8Json library for serialization while the new client uses the modern System.Text.JSON library.

I'm surprised that you experience issue with the IndicesPrivileges. The new client uses two different serializer instances: one for "Elasticsearch types" (like the mentioned one) and one for "source/document/user-defined types".

Changing the default ignore condition only for the SourceSerializer should work fine in your case.

Could you please show some code that allows me to reproduce the issue?

[TSlump]

Hi @flobernd , thank you for your response!

I didn't realise there were two serialisers, thats interesting... I change the global serialiser settings like so:

        var settings = new ElasticsearchClientSettings(new SingleNodePool(new Uri(configuration.Url)),
                sourceSerializer: (_, settings) => new DefaultSourceSerializer(settings, ElasticSearchClientConfigureOptions))
            .DisableDirectStreaming()
            .ServerCertificateValidationCallback(CertificateValidations.AllowAll);

         var elasticsearchClient = new ElasticsearchClient(settings);

    private static void ElasticSearchClientConfigureOptions(JsonSerializerOptions options)
    {
        options.DefaultIgnoreCondition = JsonIgnoreCondition.Never;
    }

And this seems to be affecting the serialising of the "Elasticsearch types". It could be how we instantiate them, I create the objects like this:

new PutRoleRequest(MapUserIdToRoleName(user.Id))
        {
            Applications = GetApplicationPrivileges(profileSettings),

            Cluster = GetClusterPrivileges(profileSettings),

            Indices = GetIndicesPrivileges(user, profileSettings)
        };

where the IndicesPrivelege model is added inside the GetIndicesPrivileges method:

indicesPrivileges.AddRange(IndexGroupConstants.Select(index => new IndicesPrivileges
                    {
                        Names = MapIndexAliasToIndexName(index),

                        Privileges = [RoleConstants.ReadIndexPrivilegeName, RoleConstants.ViewIndexMetaDataPrivilegeName],

                        Query = MapToUserQuery(user.Id, index)
                    }));

As I'm instantiating the model directly instead of fluently, could this be making the model go through the "source/document/user-defined types" serialiser ?

[flobernd]

Thanks for the additional details @TSlump , I'll have a look 🙂 The object initializer style and the fluent API are equivalent (the new fluent API internally wraps the corresponding objects) and should not affect this.

[flobernd]

@TSlump , I checked and this is indeed a bug / limitation that we currently have not addressed. Due to different factors, Query has the type object here (instead of e.g. Elastic.Clients.Elasticsearch.QueryDsl.Query). All object types indeed are serialized by the SourceSerializer.

I can however offer you a workaround for now:

Elasticsearch happily accepts a JSON formatted string for this field. This means, you could change your MapToUserQuery() method to return a string instead of a Query. The correct JSON string representation should be produced like described here:
https://www.elastic.co/docs/reference/elasticsearch/clients/dotnet/request-response-serialization#serialization

Please let me know, if that solves the immediate problem. I'll create a follow up issue to properly fix this.

[TSlump]

Hi @flobernd , ah that makes sense with the seen behaviour !

Thank you for the work around, I've implemented the query as a string instead of the 'Query' object and everything looks to be working ! I really appreciate your quick responses and help :)