Introduce FallbackSyntheticSourceBlockLoader and apply it to keyword fields #119546
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lkts
changed the title
kkrik-es
reviewed
Jan 6, 2025
| * and we would fall back to (potentially synthetic) _source. However, in case of synthetic source, there is actually no need to | ||
| * construct the entire _source. We know that there is no doc_values and stored fields, and therefore we will be using fallback synthetic | ||
| * source. That is equivalent to just reading _ignored_source stored field directly and doing an in-place synthetic source just | ||
| * for this field. |
There was a problem hiding this comment.
I'm still trying to understand if this is necessary. #113827 introduced filtering in source construction. If we set SourceFilter in SourceLoader.Synthetic initialization to filter for the requested fields only, do we get a more generic solution that covers all uses of synthetic source? I assume the latter is used in SourceBlockLoader.
There was a problem hiding this comment.
I had a conversation with @martijnvg about this. There is a more general problem here. SourceProvider/SourceLoader infra we are currently using is designed for _search and get use cases and just doesn't fit into ESQL. See #117792. So this is not solely an optimization but also allows to remove workarounds like #117792.
lkts
force-pushed
the
synthetic_source_block_loader
branch
from
d048615 to
ada0166
Compare
lkts
force-pushed
the
synthetic_source_block_loader
branch
from
0a33390 to
4e2bff8
Compare
lkts
commented
Jan 9, 2025
| } | ||
|
|
||
| @Override | ||
| public boolean canReuse(int startingDocID) { |
There was a problem hiding this comment.
I don't know what to do here.
There was a problem hiding this comment.
Like BlockStoredFieldsReader this should return true, since this implementation doesn't keep references to data structures that can't be reused between threads.
martijnvg
reviewed
Jan 10, 2025
There was a problem hiding this comment.
I think this is in the right direction @lkts!
I have a comment about ignore_above, which I think currently isn't handled? And the new block loader implementation will be used when a keyword field isn't stored and doesn't have doc values or normalizer is configured?
| } | ||
|
|
||
| @Override | ||
| public boolean canReuse(int startingDocID) { |
There was a problem hiding this comment.
Like BlockStoredFieldsReader this should return true, since this implementation doesn't keep references to data structures that can't be reused between threads.
| var reader = new FallbackSyntheticSourceBlockLoader.Reader<BytesRef>() { | ||
| @Override | ||
| public void convertValue(Object value, List<BytesRef> accumulator) { | ||
| assert value instanceof BytesRef; |
There was a problem hiding this comment.
maybe remove assert? The cast below would always fail if value is something else than BytesRef.
server/src/main/java/org/elasticsearch/index/mapper/KeywordFieldMapper.java
Show resolved
Hide resolved
test/framework/src/main/java/org/elasticsearch/index/mapper/BlockLoaderTestCase.java
Show resolved
Hide resolved
lkts
added
>enhancement
:StorageEngine/Mapping
|
Hi @lkts, I've created a changelog YAML for you. |
|
Pinging @elastic/es-storage-engine (Team:StorageEngine) |
martijnvg
approved these changes
Feb 4, 2025
| }; | ||
| } | ||
|
|
||
| private String applyIgnoreAboveAndNormalizer(String value) { |
There was a problem hiding this comment.
This got me thinking, but I think is expected in the context of es|ql? You would never see un-normalized or ignored values. With synthetic source (and normal source), you just get what got stored.
There was a problem hiding this comment.
Yes, this is how reading from stored source works as well.
| } | ||
| } | ||
|
|
||
| // TODO figure out how to handle XContentDataHelper#voidValue() |
There was a problem hiding this comment.
iirc voidValue means to ignore ignored source, because we do have values in doc values for a field? I'm trying to remember in what cases we use void value.
There was a problem hiding this comment.
voidValue means "don't trust doc values for this field when doing synthetic source". It's to avoid copied values from showing up in synthetic source.
I think it's meaningless in this context because we need all values for this field regardless how they are represented in source. So if there are any other values for this field we'll decode them and if not then there was nothing copied to it. I'll remove this comment in a follow-up.
dnhatn
approved these changes
Feb 4, 2025
💔 Backport failed
You can use sqren/backport to manually backport by running |
💚 All backports created successfully
Questions ?Please refer to the Backport tool documentation |
lkts
added a commit
to lkts/elasticsearch
that referenced
this pull request
Feb 5, 2025
…fields (elastic#119546) (cherry picked from commit e885da1) # Conflicts: # server/src/test/java/org/elasticsearch/index/mapper/blockloader/KeywordFieldBlockLoaderTests.java # test/framework/src/main/java/org/elasticsearch/index/mapper/BlockLoaderTestCase.java
elasticsearchmachine
pushed a commit
that referenced
this pull request
Feb 5, 2025
…yword fields (#119546) (#121800) * Introduce FallbackSyntheticSourceBlockLoader and apply it to keyword fields (#119546) (cherry picked from commit e885da1) # Conflicts: # server/src/test/java/org/elasticsearch/index/mapper/blockloader/KeywordFieldBlockLoaderTests.java # test/framework/src/main/java/org/elasticsearch/index/mapper/BlockLoaderTestCase.java * Add non-backported piece
dnhatn
added a commit
that referenced
this pull request
Jun 6, 2025
dnhatn
added a commit
to dnhatn/elasticsearch
that referenced
this pull request
Jun 6, 2025
We miss appending null when ignored_source is not available. Our randomized tests already cover this case, but we do not check it when loading fields. I labelled this non-issue for an unreleased bug. Closes elastic#128959 Relates elastic#119546
ldematte
added a commit
to mosche/elasticsearch
that referenced
this pull request
Jun 10, 2025
commit e6b8a64 Author: Lorenzo Dematte <[email protected]> Date: Tue Jun 10 12:02:16 2025 +0200 PR comments commit ad0902e Author: Moritz Mack <[email protected]> Date: Fri Jun 6 13:33:37 2025 +0200 Update ReproduceInfoPrinter to correctly print a reproduction for lucene / BC upgrade tests. Relates to ES-12005 commit 78b4168 Author: Alexander Spies <[email protected]> Date: Fri Jun 6 11:37:53 2025 +0200 ESQL: Throw ISE instead of IAE for illegal block in page (elastic#128960) IAE gets reported as a 400 status code, but that's inappropriate when inconsistent pages are always bugs, and should be reported with a 500. Throw ISE instead. commit 29e68bd Author: Aurélien FOUCRET <[email protected]> Date: Fri Jun 6 11:24:23 2025 +0200 [ES|QL] Fix test releases for telemetry. (elastic#128971) commit 1a76bc2 Author: Bogdan Pintea <[email protected]> Date: Fri Jun 6 11:01:49 2025 +0200 ESQL: Workaround for RLike handling of empty lang pattern (elastic#128895) Lucene's `org.apache.lucene.util.automaton.Operations#getSingleton` fails with an Automaton for a `REGEXP_EMPTY` `RegExp`. This adds a workaround for that, to check the type of automaton before calling into that failing method. Closes elastic#128813 commit e24fd32 Author: Aurélien FOUCRET <[email protected]> Date: Fri Jun 6 10:25:28 2025 +0200 [ES|QL] Enable the completion command as a tech preview feature (elastic#128948) commit 3f03775 Author: Niels Bauman <[email protected]> Date: Fri Jun 6 09:00:24 2025 +0200 Remove non-test usages of `Metadata.Builder#putCustom` (elastic#128801) This removes all non-test usages of ``` Metadata.Builder.putCustom(String type, ProjectCustom custom) ``` And replaces it with appropriate calls to the equivalent method on `ProjectMetadata.Builder`. In most cases this _does not_ make the code project aware, but does reduce the number of deprecated methods in use. commit 330d127 Author: Niels Bauman <[email protected]> Date: Fri Jun 6 08:07:59 2025 +0200 Make utility methods in `IndexLifecycleTransition` project-aware (elastic#128930) Modifies the methods to work with a project scope rather than a cluster scope. This is part of an iterative process to make ILM project-aware. commit 1b5720d Author: Aurélien FOUCRET <[email protected]> Date: Fri Jun 6 07:50:52 2025 +0200 [ES|QL] Fix test releases for LookupJoinTypesIT. (elastic#128985) commit 40cf2d3 Author: Tim Vernum <[email protected]> Date: Fri Jun 6 13:09:31 2025 +1000 Add "extension" attribute validation to IdP SPs (elastic#128805) This extends the change from elastic#128176 to validate the "custom attributes" on a per Service Provider basis. Each Service Provider (whether registered or wildcard based) has a field "attributes.extensions" which is a list of attribute names that may be provided by the caller of "/_idp/saml/init". Service Providers that have not be configured with extension attributes will reject any custom attributes in SAML init. This necessitates a new field in the service provider index (but only if the new `extensions` attribute is set). The template has been updated, but there is no data migration because the `saml-service-provider` index does not exist in any of the environments into which we wish to deploy this change. commit 496fb2d Author: Jordan Powers <[email protected]> Date: Thu Jun 5 19:50:09 2025 -0700 Skip UTF8 to UTF16 conversion during document indexing (elastic#126492) When parsing documents, we receive the document as UTF-8 encoded data which we then parse and convert the fields to java-native UTF-16 encoded Strings. We then convert these strings back to UTF-8 for storage in lucene. This patch skips the redundant conversion, instead passing lucene a direct reference to the received UTF-8 bytes when possible. commit c34f8b6 Author: Tim Vernum <[email protected]> Date: Fri Jun 6 12:03:25 2025 +1000 Improve cache invalidation in IdP SP cache (elastic#128890) The Identity Provider's Service Provider cache had two issues: 1. It checked for identity based on sequence numbers, but didn't include the `seq_no_primary_term` parameter on searches, which means the sequence would always by `-2` 2. It didn't track whether the index was deleted, which means it could be caching values from an old version of the index This commit fixes both of these issues. In practice neither issue was a problem because there are no deployments that use index-based service providers, however the 2nd issue did cause some challenges for testing. commit 923f029 Author: Nhat Nguyen <[email protected]> Date: Thu Jun 5 18:09:58 2025 -0700 Fix block loader with missing ignored source (elastic#129006) We miss appending null when ignored_source is not available. Our randomized tests already cover this case, but we do not check it when loading fields. I labelled this non-issue for an unreleased bug. Closes elastic#128959 Relates elastic#119546 commit 0f8178a Author: Bogdan Pintea <[email protected]> Date: Fri Jun 6 02:02:24 2025 +0200 ESQL: Forward port 8.19 RegexMatch serialization change version (elastic#128979) Fwd port ESQL_REGEX_MATCH_WITH_CASE_INSENSITIVITY_8_19. Related: elastic#128919. commit ee716f1 Author: Simon Chase <[email protected]> Date: Thu Jun 5 15:20:08 2025 -0700 transport: edit TransportConnectionListener for close exceptions (elastic#129015) The TransportConnectionListener interface has previously included the Transport.Connection being closed and unregistered in its onNodeDisconnected callback. This is not in use, and can be removed as it is also available in the onConnectionClosed callback. It is being replaced with a Nullable exception that caused the close. This is being used in pending work (ES-11448) to differentiate network issues from node restarts. Closes ES-12007 commit aceaf23 Merge: f18f4ee 159c57f Author: elasticsearchmachine <[email protected]> Date: Thu Jun 5 19:27:54 2025 +0000 Merge patch/serverless-fix into main commit 159c57f Author: Rene Groeschke <[email protected]> Date: Tue Jun 3 11:56:27 2025 +0200 Prepare serverless patch including elastic#128784 elastic#128740 (elastic#128807) * Change default for vector.rescoring.directio to false (elastic#128784) On serverless (and potentially elsewhere), direct IO is not available, which can cause BBQ shards to fail to read with org.apache.lucene.CorruptIndexException when this setting is true. * Optimize sparse vector stats collection (elastic#128740) This change improves the performance of sparse vector statistics gathering by using the document count of terms directly, rather than relying on the field name field to compute stats. By avoiding per-term disk/network reads and instead leveraging statistics already loaded into leaf readers at index opening, we expect to significantly reduce overhead. Relates to elastic#128583 --------- Co-authored-by: Dave Pifke <[email protected]> Co-authored-by: Jim Ferenczi <[email protected]>
valeriy42
pushed a commit
to valeriy42/elasticsearch
that referenced
this pull request
Jun 12, 2025
We miss appending null when ignored_source is not available. Our randomized tests already cover this case, but we do not check it when loading fields. I labelled this non-issue for an unreleased bug. Closes elastic#128959 Relates elastic#119546
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR contains a prototype of
FallbackSyntheticSourceBlockLoader. This is an alternative implementation of block loading for fields that use fallback synthetic source. Currently the approach is to synthesize the entire_sourceand extract the relevant part of it. This implementation skips synthesizing and directly reads underlying_ignored_sourcefield and parses its value.This implementation should mostly remove cases when
_sourceis synthesized to be used in block loader. (Most) Block loaders already use doc_values or stored fields directly, meaning in most cases_sourceis not synthesized. There are some exceptions like geo fields that only ever use source and therefore synthetic source. However geo fields also use fallback synthetic source so will be covered by implementation similar to this PR.See #115394. Note that the issue mentions working with
ignore_aboveandignore_malformedbut (at least for keyword fields) the logic is actually the opposite -ignore_aboveand normalization identical to parsing are applied to values from source before being appended to block. So currently the "expected" result of block loader is to never have ignored values. This PR intends to follow the same approach but it of course can be discussed.Includes #119415 for now.