ESQL: loading unmapped fields on synthetic _source#144112
Merged
GalLalouche merged 15 commits intoelastic:mainfrom Mar 26, 2026
Merged
ESQL: loading unmapped fields on synthetic _source#144112GalLalouche merged 15 commits intoelastic:mainfrom
GalLalouche merged 15 commits intoelastic:mainfrom
Conversation
GalLalouche
force-pushed
the
fix/synthetic_load
branch
from
33bcf97 to
231f767
Compare
…3916) Add forUnmappedLoad flag to KeywordFieldType so only the unmapped-keyword path (created by ESQL for PotentiallyUnmappedKeywordEsField) converts decoded Object values to string in the fallback synthetic source reader; mapped keyword fields keep assuming BytesRef. Add synthetic-source test cases to unmapped-load and unmapped-nullify: - partial_mapping_synthetic_sample_data (unmapped_event_duration as long) - unmappedNumericFromK8sSyntheticTsIndex (TS k8s_unmapped, network.cost) Made-with: Cursor
GalLalouche
force-pushed
the
fix/synthetic_load
branch
from
0ee8e70 to
ce8e992
Compare
elasticsearchmachine
added
the
needs:triage
GalLalouche
changed the title
GalLalouche
added
>test
Collaborator
|
Pinging @elastic/es-analytical-engine (Team:Analytics) |
Collaborator
|
Hi @GalLalouche, I've created a changelog YAML for you. |
GalLalouche
commented
Mar 17, 2026
...lugin/esql/qa/testFixtures/src/main/resources/data/partial_mapping_synthetic_sample_data.csv
Outdated
Show resolved
Hide resolved
astefan
reviewed
Mar 23, 2026
| public static final TypeParser PARSER = createTypeParserWithLegacySupport(Builder::new); | ||
|
|
||
| public static final class KeywordFieldType extends TextFamilyFieldType { | ||
| public static class KeywordFieldType extends TextFamilyFieldType { |
Contributor
There was a problem hiding this comment.
I wouldn't change this class unless someone from the core ES team approves it. My advice is to seek approval for this from the relevant team.
...in/esql/src/main/java/org/elasticsearch/xpack/esql/planner/EsPhysicalOperationProviders.java
Outdated
Show resolved
Hide resolved
romseygeek
reviewed
Mar 24, 2026
| return new FallbackSyntheticSourceBlockLoader.SingleValueReader<BytesRef>(nullValueBytes) { | ||
| @Override | ||
| public void convertValue(Object value, List<BytesRef> accumulator) { | ||
| String stringValue = ((BytesRef) value).utf8ToString(); |
Contributor
There was a problem hiding this comment.
I think you're fine to change this line rather than making KeywordFieldType non-final. ie:
String stringValue = value instanceof BytesRef br ? br.utf8ToString() : value.toString();
Contributor
Author
There was a problem hiding this comment.
I wanted to be extra defensive and avoid changing behavior if the field is not potentially unmapped, but if you're fine with it then I'm fine with it :)
astefan
reviewed
Mar 24, 2026
| 172.21.2.113 |null |null |null | ||
| 172.21.2.162 |null |null |null | ||
| 172.21.3.15 |null |null |null | ||
| client_ip:ip | foo:null | bar:keyword | baz:null |
Contributor
There was a problem hiding this comment.
These changes were unnecessary.
Contributor
Author
There was a problem hiding this comment.
Of course not, but I used a script to fix the alignments for the tests I did add, and it just makes it easier to align the entire file rather than align a subset of it. I can revert it if you think it's a blocker.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix loading of unmapped numeric values when the
_sourceis synthetic. Also add a few more timeseries tests, since now we can actually run them!Resolves #143916.