Enable a doc values sparse index on the timestamp field

[salvatore-campagna]

This PR introduces support for a sparse doc values index for the @timestamp field in DateFieldMapper when specific conditions are met:

• The index mode is set to LOGSDB.
• The field name is @timestamp and mapped as a date field.
• The field is included in the primary sort configuration.
• The field has doc values and indexing is not disabled explicitly (not index: false).

When all the conditions above hold true, we:

• We use the corresponding Sorted Set doc values format with indexing enabled (DocValuesSkipIndexType.RANGE is used in Lucene as the only available sparse index at the moment).
• Disable indexing of the @timestamp field, dropping the inverted index in favor of the sparse doc values index.

Some queries might experience slower performance as a result of using a doc values sparse index instead of an inverted index.

Disabling the inverted index on the @timestamp field while enabling the sparse doc values index is expected to:

• Reduce the storage footprint depending on the size of the inverted index relative to the sparse index.
• Improve indexing throughput by reducing the amount of data written during segment flushes.

[salvatore-campagna]

This does not work in case index: true explicitly because isSet is true but getValue() == getDefaultValue(). As in the other PR I think the implementation of isConfigured is not correct.

[salvatore-campagna]

In case of @timestamp for isConfigured we have two possibilities:

1. isConfigured() is true which means the user explicitly set the index value to true or false. In such case no matter the value of index the sparse doc values index should be disabled, and we would use the inverted index.
2. isConfigured() is false: this means the index parameter is not set by default which means, for LogsDB, under certain conditions, we can use the sparse index instead of the inverted index.

[martijnvg]

Maybe we should then just check for isSet() ?

[salvatore-campagna]

If we use isSet than the other test fails...for instance testFieldTypeWithSkipDocValues_LogsDBMode

[salvatore-campagna]

This is why I had to add the indexSettings

[martijnvg]

I think we should also allow when timestamp is secondary sort?

[salvatore-campagna]

Yeah I left it like that for the moment because that was not really an issue with the tests I have at the moment. Anyway I will replace that method with something like isSortedOnTimestamp.

[martijnvg]

Maybe we should then just check for isSet() ?

[felixbarny]

• The index mode is set to LOGSDB.

Can we extend this so that this is also enabled for TSDB?

• The field name is @timestamp and mapped as a date field.

Can we extend this to also work for when @timestamp is mapped to date_nanos?

[martijnvg]

Can we extends this so that this is also enabled for TSDB?
Can we extend this to also work for when @timestamp is mapped to date_nanos?

I think we can do this. The scope of this PR is enable sparse index in favor for indexed data structures, so that we can analyze the results in elastic/logs nightly benchmark. Note that everything is behind a feature flag. We can do this in a follow ups after we analyzed the nightly benchmark results. For tsdb, we like have a separate effort of checking how using sparse index effects querying (also on dimension fields).