Updates Watsonx inference integration with rerank feature #123340
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…taFieldsRecoveryTests testSnapshotRecovery {p0=true p1=false} elastic#122550
…taFieldsRecoveryTests testSnapshotRecovery {p0=false p1=true} elastic#122551
This PR adds initial support for unmapped fields, using the INSIST clause. For starters, this unmapped fields without a cast. Note that the INSIST keyword is potentially a placeholder, as the method of defining an unmapped field might change in the future, e.g., use a special magic function. As this is currently under development, the actual syntax is INSIST_🐔. First stage of elastic#120072. Specifically, the following features are implemented in this PR: * Support for INSIST keyword without a cast. In particular, if the type being INSISTed upon is mapped to anything other than KEYWORD, it will result in an InvalidMappedField. There is no support for union type resolution on top of INSIST. Future PRs will handle these conflicts. There is support for multiple parameters, or INSIST on top of INSIST which is equivalent. * Enforcing that INSIST must always be on top of a FROM or another INSIST. While this may change in the future, e.g., handling cases like `FROM foo | EVAL x = 1 | INSIST bar` will not be done in this PR, as it makes handling INSIST too complicated.
This test was missed back in my "fix int4" PR. elastic#121437 closes: elastic#121412
…elastic#122535) When reindexing data stream indices, parts of the index metadata needs to be copied from the source index to destination index, so that ILM and data stream lifecycle function properly. This adds a new CopyLifecycleIndexMetadataTransportAction which copies the following metadata from a source index to a destination index: - creation date setting - rollover info - ILM custom metadata
Temp dir access is necessary all over Elasticsearch, and in general is not sensitive. With Security Manager all code in ES, including plugins, are given read/write access to the temp dir. This commit mimicks that behavior with entitlements.
Currently the SLM health indicator in health report turns YELLOW when snapshots fail for a number of times. However, the SLM health indicator stays GREEN if snapshot is not completed (no success or failure) for a long time. This change adds a new optional setting unhealthy_if_no_snapshot_within to SLM policy, that sets a time threshold. If the SLM policy has not had a successful snapshot for longer than the threshold, the SLM health indicator will turn YELLOW.
This moves from `isOperator(ThreadContext`) method from
org.elasticsearch.xpack.security.operator.OperatorPrivileges
to
org.elasticsearch.xpack.core.security.operator.OperatorPrivilegesUtil
so that it can be used by modules that depend on x-pack-core (without
needing to depend on x-pack-security)
…csearch.xpack.esql.qa.multi_node.EsqlSpecIT elastic#121411
…csearch.entitlement.qa.EntitlementsDeniedIT elastic#122566
…icsearch.entitlement.qa.EntitlementsAllowedIT elastic#122567
… org.elasticsearch.entitlement.qa.EntitlementsAllowedNonModularIT elastic#122568
…org.elasticsearch.entitlement.qa.EntitlementsDeniedNonModularIT elastic#122569
* Restrict routing on sort fields to enterprise license * sync * bypass checking for serverless * Node deprecation warning for indexes and component templates with source mode in mapping * Revert "Node deprecation warning for indexes and component templates with source mode in mapping" This reverts commit 0fd4ca7. * address comments
Obvious SO exception possibilitiy if we encounter exceptions back to back in the callback. Use promise style pattern instead of callback in the individual loops to limit stack-depth (this should be good enough for a fix, technically you could still run into very deep stacks if the search completes between the `isDone` check and adding the listener back-to-back a couple times but practically this should be good enough since all the instant-done situations are from the search fully failing outright. closes elastic#116175
We have at least this one test where the combination of running in the single CPU environment and a relatively large document count + dynamic mapping updates exceeds the 10s threshold in very rare cases. Lets try moving this to 30s to hopefully turn "rare" into practically impossible. I could see this being an issue in other tests or becoming one, so I think it's fine to raise the timeout for all tests. closes elastic#116126 closes elastic#115815
We recently fixed timeout handling in the suggest phase. A test failure on SearchTimeoutIT surfaced an issue with the current approach. In case partial results are allowed, it may happen that some shards time out while executing the suggest phase and some don't. SearchPhaseController assumes that if one shard has suggest results, all of the other shards will have suggest results too. We could address that assertion and check is the search timed out, instead this commit changes timeout handling in the suggest phase to return an empty suggestion instead of null. This seems appropriate in terms of providing some results and makes the assertion about non null suggestions in SearchPhaseController happy. Relates to elastic#122357 Closes elastic#122548
This is required to update to newer version of gradle enterprise and the gradle enterprise plugin.
…22589) Unable to reproduce however the coordinator node that is meant to route the write requests might've not been ready to do so in due time. This PR adds an ensureStableCluster in the test setup method. Fixes elastic#120605
…y for Files policies (elastic#122370)
…lastic#121392) Adds support of converting existing data stream to a system data stream as part of existing system_index_metadata_upgrade_service task
…=reference/snapshot-restore/restore-snapshot/line_408} elastic#123192
When running Elasticsearch in tests we occassionally have JDK crashes. It's important to capture the state of the JVM at the time of the crash. We currently capture the hs_err_pid file for the crash, but the resulting core file exists in a directory that is no captured. This commit adjusts the capture patterns to also get core files.
…estRestoredIndexManagedByLocalPolicySkipsIllegalActions elastic#123202
…toryIsWrittenWithFailure elastic#123203
…ed (elastic#123190) elastic#122887 set read-only on the source indices of a data stream reindex. To test behavior when the source index was already read-only, testDataStreamUpgrade sometimes set the old write index to read-only. But the rest end point for adding a read-only block always sets verifed-read-only. This caused the old write index to not need to be upgraded, resulting in an incorrect value for total_indices_requiring_upgrade
Speeds up the VALUES agg when collecting from many buckets.
Specifically, this speeds up the algorithm used to `finish` the
aggregation. Most specifically, this makes the algorithm more tollerant
to large numbers of groups being collected. The old algorithm was
`O(n^2)` with the number of groups. The new one is `O(n)`
```
(groups)
1 219.683 ± 1.069 -> 223.477 ± 1.990 ms/op
1000 426.323 ± 75.963 -> 463.670 ± 7.275 ms/op
100000 36690.871 ± 4656.350 -> 7800.332 ± 2775.869 ms/op
200000 89422.113 ± 2972.606 -> 21920.288 ± 3427.962 ms/op
400000 timed out at 10 minutes -> 40051.524 ± 2011.706 ms/op
```
The `1` group version was not changed at all. That's just noise in the
measurement. The small bump in the `1000` case is almost certainly worth
it and real. The huge drop in the `100000` case is quite real.
Creating (and more importantly eventually resizing) a fresh stream output makes up a large chunk of the runtime of this aggregation. Also, recursively calling an inline consumer makes this logic even more confusing and adds additional overhead as escape analysis will not be able to remove the allocation of the consumer. => just call a method recursively and reuse the output
Don't simultaneously restart multiple nodes in a cluster. It causes data races when multiple primaries are trying to mark the `[[.snapshot-blob-cache][0]]` shard as stale. ``` org.elasticsearch.cluster.action.shard.ShardStateAction$NoLongerPrimaryShardException: primary term [2] did not match current primary term [4] at org.elasticsearch.cluster.action.shard.ShardStateAction$ShardFailedClusterStateTaskExecutor.execute(ShardStateAction.java:355) ~[elasticsearch-8.18.0-SNAPSHOT.jar:8.18.0-SNAPSHOT] at org.elasticsearch.cluster.service.MasterService.innerExecuteTasks(MasterService.java:1075) ~[elasticsearch-8.18.0-SNAPSHOT.jar:8.18.0-SNAPSHOT] at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:1038) ~[elasticsearch-8.18.0-SNAPSHOT.jar:8.18.0-SNAPSHOT] at org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:245) ~[elasticsearch-8.18.0-SNAPSHOT.jar:8.18.0-SNAPSHOT] at org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.lambda$run$2(MasterService.java:1691) ~[elasticsearch-8.18.0-SNAPSHOT.jar:8.18.0-SNAPSHOT] at org.elasticsearch.action.ActionListener.run(ActionListener.java:452) ~[elasticsearch-8.18.0-SNAPSHOT.jar:8.18.0-SNAPSHOT] at org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.run(MasterService.java:1688) ~[elasticsearch-8.18.0-SNAPSHOT.jar:8.18.0-SNAPSHOT] at org.elasticsearch.cluster.service.MasterService$5.lambda$doRun$0(MasterService.java:1283) ~[elasticsearch-8.18.0-SNAPSHOT.jar:8.18.0-SNAPSHOT] at org.elasticsearch.action.ActionListener.run(ActionListener.java:452) ~[elasticsearch-8.18.0-SNAPSHOT.jar:8.18.0-SNAPSHOT] at org.elasticsearch.cluster.service.MasterService$5.doRun(MasterService.java:1262) ~[elasticsearch-8.18.0-SNAPSHOT.jar:8.18.0-SNAPSHOT] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044) ~[elasticsearch-8.18.0-SNAPSHOT.jar:8.18.0-SNAPSHOT] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) ~[elasticsearch-8.18.0-SNAPSHOT.jar:8.18.0-SNAPSHOT] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] at java.lang.Thread.run(Thread.java:1575) ~[?:?] ``` Resolve elastic#118374 Resolve elastic#120442
kosabogi
added
auto-backport
|
Documentation preview: |
|
Warning It looks like this PR modifies one or more |
elasticsearchmachine
added
needs:triage
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
68 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR updates the Watsonx AI inference documentation to include the new rerank feature.