Add ignore_malformed and null_values to test data generation #121983
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lkts
added
>test
Collaborator
|
Pinging @elastic/es-storage-engine (Team:StorageEngine) |
kkrik-es
reviewed
Feb 7, 2025
| } | ||
|
|
||
| private static boolean matchCountsEqualExact(List<String> actualNormalized, List<String> expectedNormalized) { | ||
| HashMap<String, Integer> counts = new HashMap<>(); |
Contributor
There was a problem hiding this comment.
Nit: let's use TreeMap for maps, it helps with debugging..
kkrik-es
reviewed
Feb 7, 2025
| } | ||
| } | ||
|
|
||
| return counts.isEmpty(); |
Contributor
There was a problem hiding this comment.
Don't we want to return something about the delta? Maybe return a string for counts, or null?
kkrik-es
reviewed
Feb 7, 2025
| * @return | ||
| */ | ||
| static Supplier<Object> defaults(Supplier<Object> rawValues, DataSource dataSource) { | ||
| var nulls = dataSource.get(new DataSourceRequest.NullWrapper()); |
Contributor
There was a problem hiding this comment.
I think I missed the null value inject, do we have existing logic that randomly injects nulls, or the null_value if one is provided?
Collaborator
💚 Backport successful
|
elasticsearchmachine
pushed a commit
that referenced
this pull request
Feb 7, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds generation of
null_valueandignore_malformed(including malformed values generation) mapping parameters to fields that support them (number fields). This is done in preparation to work on a fallback synthetic source block loader for number fields since the test infra is shared.