[9.0] Initial support for unmapped fields (Plus a few others) (#119886, #121261, #122607) #122804
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…uggestProfilesWithHint elastic#121116
…ohereEmbeddings {upgradedNodes=1} elastic#121362
…erank {upgradedNodes=1} elastic#121363
…ohereEmbeddings {upgradedNodes=2} elastic#121364
…Block {upgradedNodes=2} elastic#121365
…Block {upgradedNodes=3} elastic#121366
…Block {upgradedNodes=1} elastic#121367
…lastic#121109) (elastic#121386) (cherry picked from commit 1225b07) # Conflicts: # rest-api-spec/build.gradle Co-authored-by: Oleksandr Kolomiiets <[email protected]>
…temFeaturesMigration {cluster=UPGRADED} elastic#121115
…121204) (elastic#121339) * Refactor: separate package for entitlement records (elastic#121204) * Fix PolicyManagerTests after package move (elastic#121304) * Fix PolicyManagerTests after package move * Unmute
…lastic#121397) This commit adds the data dirs, config dir and temp dir into entitlement bootstrapping. It doesn't yet use them in entitlement policies, but makes them available to use within initialization.
(cherry picked from commit 198a187)
(cherry picked from commit 82b9b56)
…estCase org.elasticsearch.lucene.RollingUpgradeLuceneIndexCompatibilityTestCase elastic#121423
This commit adds an EntitlementTest annotation that can be used on classes containing test actions for entitlements. The annotation mirrors the parameters of CheckAction. Only file check actions are currently converted, the rest can be moved and annotated as followups. Note that the check action name is simply the method name, no fancy name manipulation is done.
…entParserTests (elastic#121075) (elastic#121328) * fix StatementParserTests.testNamedFunctionArgumentInMap, testNamedFunctionArgumentWithUnsupportedNamedParameterTypes * separate snapshot test from release test --------- Co-authored-by: Elastic Machine <[email protected]>
* Integrate watsonx reranking to inference api * Add api_version to the watsonx api call * Fix the return_doc option * Add top_n parameter to task_settings * Add truncate_input_tokens parameter to task_settings * Add test for IbmWatonxRankedResponseEntity * Add test for IbmWatonxRankedRequestEntity * Add test for IbmWatonxRankedRequest * [CI] Auto commit changes from spotless * Add changelog * Fix transport version * Add test for IbmWatsonxService * Remove canHandleStreamingResponses * Add requireNonNull for modelId and projectId * Remove maxInputToken method * Convert all optionals to required * [CI] Auto commit changes from spotless * Set minimal_supported version to be ML_INFERENCE_IBM_WATSONX_RERANK_ADDED * Remove extraction of unused fields from IbmWatsonxRerankServiceSettings * Add space * Add space --------- Co-authored-by: Saikat Sarkar <[email protected]> Co-authored-by: elasticsearchmachine <[email protected]>
…21103) (elastic#121401) * Adding fields and making some optional * Fixing tests
This commit forces the delegate for ES logging to always use the String version of LogManager.getLogger instead of the one taking a Class. The reason is that if a classloader is not in the hierarchy of the app classloader, the ES logging configuration will not be found. By using the String variant, the app classloader is always used. Co-authored-by: Ryan Ernst <[email protected]>
…#121426) * Refactor: remove duplicate canWrite methods. This serves as a good example of how Path and File handling could be specialized in the future, but as long as they are identical, the duplication causes more harm than good. * Refactor: just one neverEntitled. The original motivation was to avoid allocating a lambda object on each call, but since that's a highly optimized operation in the JVM, it's unlikely to make a difference in practice, and this smacks of premature optimization. We're pretty liberal about lambdas elsewhere, so let's not sweat it here until we have some evidence that it matters. * Remove dead code
…lastic#120852) (elastic#121449) * correct line and column numbers of missing named parameters
* Entitlement IT cases for reflection * EntitlementBootstrap selfTest using reflection * Remove errant logging setting * Lambdas instead of booleans * [CI] Auto commit changes from spotless * Refactor: Extract lambdas to method refs --------- Co-authored-by: elasticsearchmachine <[email protected]>
Under very unfortunate conditions tests that check xContent objects roundtrip parsing (like i.e. [SearchHitsTests testFromXContent](elastic#97716) can fail when we happen to randomly pick YAML xContent type and create random (realistic)Unicode character sequences that may contain the character U+0085 (133) from the [Latin1 code page](https://de.wikipedia.org/wiki/Unicodeblock_Lateinisch-1,_Erg%C3%A4nzung). That specific character doesn't get parsed back to its original form for YAML xContent, which can lead to [rare but hard to diagnose test failures](elastic#97716 (comment)). This change adds logic to AbstractXContentTestCase#test() which lies at the core of most of our xContent roundtrip tests that disallows test instances containing that particular character when using YAML xContent type. Closes elastic#97716
…ic#122605) (elastic#122725) While testing elastic#122591, I realized we need to grand read/write permission to the logs dir to server. This PR adds the `logsDir` to the bootstrap parameters, and uses it in the `server` policy.
…lastic#122733) This PR fixes SecuritySingleNodeTestCase and ProfileIntegTests tests. - The security single node test failures are solved by ensuring every test starts with security index created and available. This is in order to have consistent state for every test. With the changes introduce in the elastic#120323 PR, only the first test would execute with .security index being created async. Subsequent tests would execute without security index creation due to the fact that whole cluster is wiped after each test. This caused a flakiness only for the first test, because there was no mechanism in place to ensure that the .security index is active before test execution. - The profile integration tests are solved by introducing an anonymous role which don't have application privileges. The application privileges are resolved from the .security index and assigned to all users, including the es_test_root user which is used during cluster wiping. Due to asynchronous nature of cluster setup and .security index creation, this now causes flakiness. The main problem is that wiping is done asynchronously and uses es_test_root which had assigned anonymous rac_role which depends on .security index being available for search in order to resolve application privileges. The application privilege resolution is done in buildRoleFromDescriptors which currently does not wait for security index availability(can be improved - but still wouldn't fix internal cluster tests). This wasn't a problem before just because we simply return empty results when .security index does not exist. There is some complexity in making internal clusters wait for availability of security shards before the test, so I think this solution is acceptable given that it's not required for this tests to have anonymous role with application privileges. Resolves elastic#121022 Resolves elastic#121096 Resolves elastic#121101 Resolves elastic#120988 Resolves elastic#121108 Resolves elastic#120983 Resolves elastic#120987 Resolves elastic#121179 Resolves elastic#121183 Resolves elastic#121346 Resolves elastic#121151 Resolves elastic#120985 Resolves elastic#121039 Resolves elastic#121483 Resolves elastic#121116 Resolves elastic#121258 Resolves elastic#121486 (cherry picked from commit 369c641) # Conflicts: # muted-tests.yml
…2743) This module was removed in elastic#119384 so we can drop it from the `RestrictedBuildApiService` now.
…) (elastic#122747) Add missing file entitlements for server. closes elastic#122569, closes elastic#122568, closes elastic#122567, closes elastic#122566 (Note, some failures are due to a `NoSuchAlgorithmException` when fips is enabled. This issue is unrelated to entitlements) (cherry picked from commit cc96791) # Conflicts: # muted-tests.yml
) * Remove `@UpdateForV9` from `RepositoryConflictException` elastic#122730 added checks in 9.1.0 that `backwardCompatibleMessage` won't be serialized * [CI] Auto commit changes from spotless --------- Co-authored-by: elasticsearchmachine <[email protected]>
…arch.telemetry.apm.ApmAgentSettingsIT elastic#122546
This is no longer used since FilesEntitlement was added.
…2771) This commit switches to using records in place of classes for FileData impls. It moves them outside the interface so that they can be private. This does not change the fact the interface is sealed, nor the factory methods for creating them.
This test uncovered an issue in the suggest timeout handling logic. Addressed with elastic#122675 in 9.0. The test can be unmuted. It got muted due to failures happened right before the merge was merged.
) (elastic#122787) This commit adds instrumentation for File methods that require read permissions. see elastic#122109 for the write side
Co-authored-by: Alex Kattathra Johnson <[email protected]>
…fied` (elastic#122729) (elastic#122798) The size parameter hasn't been accepted since 8.0, end users should use max_docs and if the user specify it, we can defer to the standard error message produced by the parser.
- Fix lucene compatibility tests - failing due to buildscan setup - Update reference to luceneBwcTest commit
* ESQL: Support for _index metadata field in CsvTests * Extract INDEX constant to MetadataAttribute * Add comment on capability
This PR adds initial support for unmapped fields, using the INSIST clause. For starters, this unmapped fields without a cast. Note that the INSIST keyword is potentially a placeholder, as the method of defining an unmapped field might change in the future, e.g., use a special magic function. As this is currently under development, the actual syntax is INSIST_🐔. First stage of elastic#120072. Specifically, the following features are implemented in this PR: * Support for INSIST keyword without a cast. In particular, if the type being INSISTed upon is mapped to anything other than KEYWORD, it will result in an InvalidMappedField. There is no support for union type resolution on top of INSIST. Future PRs will handle these conflicts. There is support for multiple parameters, or INSIST on top of INSIST which is equivalent. * Enforcing that INSIST must always be on top of a FROM or another INSIST. While this may change in the future, e.g., handling cases like `FROM foo | EVAL x = 1 | INSIST bar` will not be done in this PR, as it makes handling INSIST too complicated.
GalLalouche
added
backport
Team:Analytics
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
58 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Manual backporting of #119886, #121261, #122607.