Do not respect synthetic_source_keep=arrays if type parses arrays #127795
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
elasticsearchmachine
added
needs:triage
parkertimmins
added
>enhancement
:StorageEngine/Mapping
Collaborator
|
Pinging @elastic/es-storage-engine (Team:StorageEngine) |
elasticsearchmachine
removed
the
needs:triage
Collaborator
|
Hi @parkertimmins, I've created a changelog YAML for you. |
Contributor
Author
|
Fixes #126155 |
…tic#127693) We replace usages of time sensitive `DataStream#getDefaultBackingIndexName` with the retrieval of the name via an API call. The problem with using the time sensitive method is that we can have test failures around midnight. Relates elastic#123376
Currently, we don't run the coordinating can_match to skip unmatched shards in field-caps. Most of the time, this is fine, but the current field-caps fails when the target shards are unavailable, even if they don't match the index filter. This change integrates the coordinating can_match into field-caps to prevent failures in such cases.
Now that security manager is gone, the policy files are no longer needed. This commit removes the server, test and plugin specific policy files
) Apache Lucene 10.2 exposes a new search strategy for executing filtered searches over HNSW graphs. This PR switches to utilizing that strategy by default as it generally provides a much better recall/latency pareto frontier than our regular hnsw fanout search. Additionally, a new tech-preview setting is provided to potentially revert to the old fanout behavior if issues arise.
With the SecurityManager gone, the PrivilegedOperations class is no longer needed, these operations can be called directly.
…sReference (elastic#127404) We have a couple of places in the codebase where we do the transition from the stream to the reference. We can save some code and make this a little less error-prone by having a conversion method with move-style semantics and enabling the use of try-with-resources. Also, this enables a couple of optimizations down the line and unlinking the list of pages and moving it to the reference instead of nulling it out is a bit nicer to the CPU caches also.
The SecureMockMaker is a way for mockito to be allowed SM permissions for proxying. Since the Security Manager is no longer used, secure mocks are no longer needed. This commit removes them.
…out to request (elastic#126805) * Fixing bug with listener and adding timeout * Update docs/changelog/126805.yaml * Fixing tests * Fixing writeTo
…ionAuthorizationIT testIndicesPrivilegesAreEnforcedForCcrRestoreSessionActions elastic#127782
* Avoid time-based expiry of channel stats or else `testHttpClientStats` will fail if running multiple iterations for more than 5m. * Assert all bytes received in `testHttpClientStats`.
Now that entitlements are always enabled, there is no need for duplicating PR jobs for entitlements enabled and not enabled. This commit removes the buildkite setup for the entitlements jobs.
In order to match the pattern for the other Cloud ESS image, provide a default entrypoint to the container that points to the configured docker entrypoint. This is described in https://github.com/elastic/elasticsearch/blob/1a1763c591c4c32bf66f0df3bce2040e8f19a1a2/distribution/docker/README.md?plain=1#L16-L19 and implemented in https://github.com/elastic/elasticsearch/blob/1a1763c591c4c32bf66f0df3bce2040e8f19a1a2/distribution/docker/src/docker/Dockerfile.ess#L37-L40
Since SecurityManager is no longer used, the custom subclass of SecurityManager, SecureSM, is no longer needed.
…astic#123426) Removed one unnecessary sort (the shards are already sorted in the same order) and ported two more to JDK's list sort which beats Lucene's timsort by quite a margin nowadays (the mutations are done straight on the array backing the list and that saves endless indirection). Also, removed a needless intermediary to and from set copy and an unnecessary conditional. This is all motivated by spacetime project benchmarking showing this stuff as one of the biggest contributors to search transport thread use.
Contributor
Author
|
Oof, not sure what I did when trying to rebase 🤦♂️ |
Contributor
Author
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
13 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Types that parse arrays directly should not need to store values in
_ignored_sourceifsynthetic_source_keep=arrays. Since they have custom handling of arrays, it provides no benefit to store in_ignored_sourcewhen there are multiple values of the type.For example, consider the document with
pointfield of typegeo_point:This set of points is not subject to
synthetic_source_keep=arrayssince thegeo_pointmapper does custom array parsing. Since this is the case, it does not make sense to require that the following equivalent document use_ignored_source:Currently, this second document does use
_ignored_source, which is a waste of space. This PR causes the second document to behave the same as the first, not using_ignored_source.