Provide a helper for reading selected fields from search response

[DRKV333]

Is your feature request related to a problem? Please describe.

Consider a search like this, that uses DocvalueFields:

SearchResponse<Dialog> search = await elasticClient.SearchAsync<Dialog>(s => s
    .Indices(Indices.Index<Dialog>())
    .Query(q => q
        .Match(m => m
            .Field(x => x.Text)
            .Query("hello")
        )
    )
    .Size(1)
    .Source(false)
    .DocvalueFields(
        f => f.Field(x => x.Ver)
    )
);

To get the actual value of this field from the response, I have to do something like this:

int ver = ((JsonElement)search.Hits.Single().Fields!["ver"])[0].GetInt32();

Which is not intuitive and very inconvenient.

Describe the solution you'd like

It would be great to have some sort of extension method, that would let me do this without all the weird casting and conversion. Maybe something like this:

int ver = search.Hits.Single().GetFieldValues(x => x.Ver).Single();

Describe alternatives you've considered

I have a helper like this in my own code:

private static readonly JsonSerializerOptions fieldValuesOptions = new() { NumberHandling = JsonNumberHandling.AllowReadingFromString };

public static IEnumerable<TField> GetFieldValues<TDoc, TField>(this Hit<TDoc> hit, Inferrer inferrer, Expression<Func<TDoc, TField>> expression) =>
    ((JsonElement)hit.Fields![inferrer.Field(expression)]).EnumerateArray().Select(x => x.Deserialize<TField>(fieldValuesOptions)!);

But I sill feel like this should not be this difficult to do, and there should be a builtin helper.

Additional context

This applies to other ways of selecting specific fields in the search query, such as "stored_fields" or "fields". (Though not _source filtering.) The only info I found about this subject is this really old Stack Overflow question, where all the answers seem to be completely outdated: https://stackoverflow.com/questions/23688365/load-specific-fields-in-elasticsearch-nest-query
So it might also be nice to write something about this in the documentation.

[flobernd]

@DRKV333 Hi, thanks for reporting!

These properties are modeled as Dictionary<string, any> in the specification to indicate that we are dealing with an object like structure. This is not very convenient to consume in statically typed languages as you demonstrate.

A potentially (but breaking) solution would be to change the type to JsonElement. This would allow you to further deserialize the returned fields into a strongly typed POCO.

[DRKV333]

Yeah, having just a single JsonElement for the fields would make a lot more sense.

Deserializing it into a proper DTO would have have some challenges though. Every property would have to be a list, since every field is returned as an array. If multi-fields are involved there will be dots in the property names, so you'd need to use JsonPropertyNameAttributes everywhere or a custom naming policy. And a new class would be necessary for each query with different fields.

In my queries where I use Hit.Fields, I only need to read one or two fields, which is why I'm using the helper with the expression. I also originally expected to be able to do this out of the box, since everywhere else in the library it's possible to specify fields with expressions, which is very nice.

[flobernd]

Deserializing it into a proper DTO would have have some challenges though. Every property would have to be a list, since every field is returned as an array. If multi-fields are involved there will be dots in the property names, so you'd need to use JsonPropertyNameAttributes everywhere or a custom naming policy. And a new class would be necessary for each query with different fields.

Indeed. But these kinds of inconveniences are hard to avoid when retrieving the Lucene fields directly.

Is there any reason for not using source filtering? Using docvalue_fields in particular can even degrade performance due to increased memory allocations - depending on the concrete situation.

I have a helper like this in my own code:

Just a heads up regarding this code: x.Deserialize<T>(...) might not work correctly in all cases (e.g. if the SourceSerializer has been replaced / customized). If possible, you should invoke client.SourceSerializer.Deserialize<T>(...) instead.

[DRKV333]

I realize the fields look like that for a reason, I'm just saying it would still be relevant to have some sort of helper to retrieve them, because deserializing the whole fields JsonElement into a class would also be a bit annoying.

I haven't benchmarked my particular use case yet, so I don't know which is better. But I said, this is not limited to doc_values, fields and stored_fields work the same way. And these can be better than source filtering, if for instance the _source is huge.

client.SourceSerializer only takes Streams, so that doesn't really work here.

[flobernd]

I realize the fields look like that for a reason, I'm just saying it would still be relevant to have some sort of helper to retrieve them, because deserializing the whole fields JsonElement into a class would also be a bit annoying.

Deserialization into a POCO/DTO is a common pattern in the Elasticsearch .NET client and I personally don't think that we should introduce a different concept (helper) only for this particular case.

I haven't benchmarked my particular use case yet, so I don't know which is better. But I said, this is not limited to doc_values, fields and stored_fields work the same way. And these can be better than source filtering, if for instance the _source is huge.

There definitely are valid use-cases. I was just curious, because in 99% of the cases you are probably better suited using source filtering.

client.SourceSerializer only takes Streams, so that doesn't really work here.

It's only briefly documented for RequestResponseSerializer (see here), but the Elastic.Transport.Extensions namespace provides extension methods for Serializer that should cover most of the System.Text.Json Serialize/Deserialize overloads.