ignore_above default to 8191 for logsdb

[salvatore-campagna]

In LogsDB we would like to use a default value of 8191 for the index-level setting
index.mapping.ignore_above. The value for ignore_above is the character count,
but Lucene counts bytes. Here we set the limit to 32766 / 4 = 8191 since UTF-8
characters may occupy at most 4 bytes.

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)

[dnhatn]

The changes look good, but I think 1024 is too small to be the default.

[salvatore-campagna]

@felixbarny @andrewkroh We would like to know if this value is good for our integrations.
Especially for fields that do not have a value set at mapping level, an index-level setting would result in applying a possibly unwanted limit. For fields which might include stack traces we might end-up missing the ability to search on them. This would result in a regression for fields not using ignore_above and expecting no limit.

My opinion is that we can still have a default...maybe changing the default to a larger value...2k, 4k...and that making sure that fields requiring larger-then-the-default value are explicit about it.

[felixbarny]

1024 is the default that is used in ECS. I'm not sure where this exact number was determined and why we haven't chosen a slightly larger default. The intention behind it is to avoid document rejections when running into the Lucene's byte-length limit of 32kb. I think we can choose a generic default for LogsDB that would prevent us running into that limit (with any unicode character) but that's bigger than the ECS limit of 1024.

While I think it makes sense to have a default value for ignore_above in LogsDB to avoid rejections, I think we should discuss whether there are better alternatives to ignore_above. For example, we could ignore values only if they would actually exceed the hard-limit in Lucene. Or instead of ignoring them entirely, we could truncate the values. There are some discussions in this issue that we may want to revise: #60329

What seems missing in your PR is to actually use the index setting in keyword field mappers and updating the docs to describe that there's a way to set an index-level default, similar to what we do with ignore_malformed.

[salvatore-campagna]

1024 is the default that is used in ECS. I'm not sure where this exact number was determined and why we haven't chosen a slightly larger default. The intention behind it is to avoid document rejections when running into the Lucene's byte-length limit of 32kb. I think we can choose a generic default for LogsDB that would prevent us running into that limit (with any unicode character) but that's bigger than the ECS limit of 1024.

While I think it makes sense to have a default value for ignore_above in LogsDB to avoid rejections, I think we should discuss whether there are better alternatives to ignore_above. For example, we could ignore values only if they would actually exceed the hard-limit in Lucene. Or instead of ignoring them entirely, we could truncate the values. There are some discussions in this issue that we may want to revise: #60329

What seems missing in your PR is to actually use the index setting in keyword field mappers and updating the docs to describe that there's a way to set an index-level default, similar to what we do with ignore_malformed.

The ignore_above index setting was already introduced by #113121.
In this PR we just change the default value from Inter.MAX_VALUE to 1024 (or any other suitable value we decide) for LogsDB.

[salvatore-campagna]

@felixbarny @andrewkroh we need feedback here, possibly before GA release. I would rather go for a high value like 8k to 12k rather than not having it.

[felixbarny]

+1 on having a default value for ignore_above for now that's potentially higher than 1024 but still guarantees that we're not hitting the hard limit of 32kb in Lucene.

What's the highest value of ignore_above that would guarantee us to be under the hard-limit in Lucene? How are we encoding chars and what's the highest number of bytes per char? Is there any other static overhead in the encoding? How are we handling unicode code points that consist of multiple characters?

[salvatore-campagna]

+1 on having a default value for ignore_above for now that's potentially higher than 1024 but still guarantees that we're not hitting the hard limit of 32kb in Lucene.

What's the highest value of ignore_above that would guarantee us to be under the hard-limit in Lucene? How are we encoding chars and what's the highest number of bytes per char? Is there any other static overhead in the encoding? How are we handling unicode code points that consist of multiple characters?

From our documentation:

NOTE: The value for `ignore_above` is the _character count_, but Lucene counts
bytes. If you use UTF-8 text with many non-ASCII characters, you may want to
set the limit to `32766 / 4 = 8191` since UTF-8 characters may occupy at most
4 bytes.

In Lucene the hard limit is 32766 bytes.

The highest value for ignore_above that guarantees being under the Lucene limit, accounting for the worst-case 4-byte encoding, is 8191.

Applying the value will depend on encoding and character count will match bytes count only if all characters are encoded using one byte. In UTF-8 encoding might result in 1 to 4 bytes per character if we consider also non-ascii characters but I guess for logging purposes we can safely assume that we are dealing with ASCII characters.

(AFAIK Elasticsearch uses UTF-8 encoding for strings).

About the static overhead I don't see any issue...keywords and text are normally converted to arrays of bytes.

[felixbarny]

The highest value for ignore_above that guarantees being under the Lucene limit, accounting for the worst-case 4-byte encoding, is 8191.

Let's set the default limit to 8191 then.

[salvatore-campagna]

@elasticmachine update branch

[elasticmachine]

merge conflict between base and head

[kkrik-es]

How can we override this for an index with logsdb mode?

[kkrik-es]

I see, this is the default value.. Let's add a comment above, or move it to a static helper function for clarity.

[salvatore-campagna]

The

settings -> { ... }

lambda just determines the default value if no explicit value is provided for the setting.
I will extract the lambda in a method with a descriptive name.

[salvatore-campagna]

@elasticmachine update branch

[elasticmachine]

merge conflict between base and head

[salvatore-campagna]

@elasticmachine update branch

[salvatore-campagna]

@martijnvg @felixbarny I need an approval if we ant to merge this

[martijnvg]

LGTM

[salvatore-campagna]

@elasticmachine update branch

[elasticsearchmachine]

💔 Backport failed

The backport operation could not be completed due to the following error:

An unexpected error occurred when attempting to backport this PR.

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 113442

[lkts]

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Questions ?

Please refer to the Backport tool documentation

[lkts]

💚 All backports created successfully

Status	Branch	Result
✅	8.16

Questions ?

Please refer to the Backport tool documentation