Skip to content

Commit 73593d0

Browse files
przemekwitekprzemekwitek
committed
Squashed commit of the following:
commit a692cbd0b01314d6e149179b9f41ddc530f52762 Author: David Turner <[email protected]> Date: Mon Jul 21 08:16:16 2025 +0100 Log failure in `internalSend` (#131418) An exception here should be impossible, but we don't assert that, nor do we emit a log message to prove it didn't happen in a production environment. This commit adds the missing log and assert. commit 08cbdbce11cc3cd5f53756c8b7242d8df0a52f06 Author: elasticsearchmachine <[email protected]> Date: Mon Jul 21 08:44:22 2025 +0200 Mute org.elasticsearch.test.rest.yaml.RcsCcsCommonYamlTestSuiteIT test {p0=search/40_indices_boost/Indices boost with alias} #131598 commit b3715901c064c1b7a0d85d4c35387a335ed22140 Author: Nick Tindall <[email protected]> Date: Mon Jul 21 16:07:59 2025 +1000 Add shard write-load to cluster info (#131496) Relates: ES-12419, ES-12420 commit 47bb53569ab7c55deeb7d6aa27132df9f63cd1d1 Author: Yang Wang <[email protected]> Date: Mon Jul 21 14:49:51 2025 +1000 [Test] Wait for expected master on all nodes (#131399) This PR changes the test to simply wait expected master on every node instead of selectively waiting on one non-master and one master node. The later is problematic since it uses API that is not suitable when the cluster is changing master. Relates: #127213 commit 6dd4d675f854ee1b13d6b5351615348ae4113f4e Author: Yang Wang <[email protected]> Date: Mon Jul 21 12:25:36 2025 +1000 Executing shard recovery in project context (#130525) See title commit a07e7e91b7a3fb25ac08b265e1715c6a2dec0eec Author: elasticsearchmachine <[email protected]> Date: Sun Jul 20 17:40:03 2025 +0200 Mute org.elasticsearch.packaging.test.DockerTests test010Install #131376 commit 3c914e188f56405d2ce0ddcb5cfeab8a55f5ef78 Author: elasticsearchmachine <[email protected]> Date: Sun Jul 20 00:28:43 2025 +0200 Mute org.elasticsearch.xpack.esql.action.CrossClusterQueryWithPartialResultsIT testPartialResults #131481 commit 256a7fed46cd387d2d61c11ed1356998ecd15ea7 Author: Larisa Motova <[email protected]> Date: Fri Jul 18 19:08:52 2025 -1000 Skip downsample IT test with TS command when command not present (#131556) Fixes #131500 commit 74cd00654cfef2ffa12fdd1bc82f0bd38a2e4622 Author: Jeremy Dahlgren <[email protected]> Date: Fri Jul 18 19:15:37 2025 -0400 Remove redundant method for getting the remote cluster names (#130495) This change removes RemoteClusterService.getRemoteClusterNames() since getRegisteredRemoteClusterNames() provides the same functionality. The comment in getRegisteredRemoteClusterNames() was removed since it is no longer accurate after the change in PR #47891. commit 668acccabae4149f43c6b365dc79014f40ecc5fe Author: elasticsearchmachine <[email protected]> Date: Fri Jul 18 23:44:27 2025 +0200 Mute org.elasticsearch.action.admin.cluster.node.tasks.CancellableTasksIT testRemoveBanParentsOnDisconnect #131562 commit ac398e3b61f6a9d66a01f62d8887c44efd7721a2 Author: Stef Nestor <[email protected]> Date: Fri Jul 18 15:30:41 2025 -0600 New Slow log troubleshooting video (#131557) commit dc96c362f517ad0e9676285001ddcd7e03e36d5a Author: Ruben van Staden <[email protected]> Date: Fri Jul 18 17:16:58 2025 -0400 otel-data: enable failure store for OTEL datastreams (#131395) commit beb18a87c3279f7d9b9426cbd50df20f65c12d8d Author: Jan-Kazlouski-elastic <[email protected]> Date: Fri Jul 18 23:26:20 2025 +0300 Add Llama support to Inference Plugin (#130092) * Refactor Hugging Face service settings and completion request methods for consistency * Add Llama model support for embeddings and chat completions * Refactor Llama request classes to improve secret settings handling * Refactor DeltaParser in LlamaStreamingProcessor to improve argument handling * Enhance Llama streaming processing by adding support for nullable object arrays * [CI] Auto commit changes from spotless * Fix error messages in LlamaActionCreator * [CI] Auto commit changes from spotless * Add detailed Javadoc comments to Llama classes for improved documentation * Enhance LlamaChatCompletionResponseHandler to support mid-stream error handling and improve error response parsing * Add Javadoc comments to Llama classes for improved documentation and clarity * Fix checkstyle * Update LlamaEmbeddingsRequest to use mediaTypeWithoutParameters for content type header * Add unit tests for LlamaActionCreator and related models * Add unit tests for LlamaChatCompletionServiceSettings to validate configuration parsing and serialization * Add unit tests for LlamaEmbeddingsServiceSettings to validate configuration parsing and serialization * Add unit tests for LlamaEmbeddingsServiceSettings to validate various configuration scenarios * Add unit tests for LlamaChatCompletionResponseHandler to validate error response handling * Refactor Llama embedding and chat completion tests for consistency and clarity * Add unit tests for LlamaChatCompletionRequestEntity to validate message serialization * Add unit tests for LlamaEmbeddingsRequest to validate request creation and truncation behavior * Add unit tests for LlamaEmbeddingsRequestEntity to validate XContent serialization * Add unit tests for LlamaErrorResponse to validate error handling from HTTP responses * Add unit tests for LlamaChatCompletionServiceSettings to validate configuration parsing and serialization * Add tests for LlamaService request configuration validation and error handling * Fix error message formatting in LlamaServiceTests for better localization support * Refactor Llama model classes to implement accept method for action visitors * Hide Llama service from configuration API to enhance security and reduce exposure * Refactor Llama model classes to remove modelId and update embedding request handling * Refactor Llama request classes to use pattern matching for secret settings * Update embeddings handler to use HuggingFace response entity * Refactor Mistral model classes to remove modelId and update rate limit hashing * Refactor Mistral action classes to remove taskSettings parameter and streamline action creation * Refactor Llama and Mistral models to remove taskSettings parameter and simplify model instantiation * Refactor Llama service tests to use Model instead of CustomModel and update similarity measure to DOT_PRODUCT * Remove unused tests and imports from LlamaServiceTests * Add chunking settings support to Llama embeddings model tests * Add changelog * Add support for version checks in Llama settings and define new transport version * Refactor Llama model assertions and remove unused version support methods * Refactor Llama service constructors to include ClusterService and improve error message handling --------- Co-authored-by: elasticsearchmachine <[email protected]> commit f664cf5926345958978205907cc7d429f2403345 Author: Alexey Ivanov <[email protected]> Date: Fri Jul 18 19:04:17 2025 +0100 Add proper Diff support for ProjectStateRegistry (#130917) This change introduces proper diff support for ProjectStateRegistry instead of CompleteDiff and Entry to ProjectStateRegistry, to allow addition of more project state fields in the future commit 1dc6bc0ca385baecf77ab6ba46822417e66b5791 Author: Richard Dennehy <[email protected]> Date: Fri Jul 18 18:55:59 2025 +0100 Disable queryable built-in feature for smoke-test-plugins-ssl (#131523) (#131543) commit be2f024412a13995daeb8dc24734613513cf787e Author: Stanislav Malyshev <[email protected]> Date: Fri Jul 18 11:37:38 2025 -0600 Revert "Put shards failure under a cap flag (#131371)" (#131542) This reverts commit 04119405d141bcb390509f766ebbb7660b0ffe72. commit feafb3a2ef4ae9204b99becb797502b81cfa15e4 Author: Pat Whelan <[email protected]> Date: Fri Jul 18 12:35:35 2025 -0400 [ML] Track inference deployments (#131442) Record duration and errors when Inference Endpoints deploy Trained Models. The new metric is `es.inference.trained_model.deployment.time`. Refactored `InferenceStats` into server so it can be used in `InferenceServiceExtension` and passed to InferenceServices rather than remain at the Transport layer. commit a6d6d3ec8a3092bfaf4b1d0bfb34df66690a1716 Author: Ankita Kumar <[email protected]> Date: Fri Jul 18 12:34:38 2025 -0400 Resharding: Route requests to source shard while target is pre-handoff (#131306) This PR for the Autosharding project, routes DocWriteRequests (index/delete/update requests) that are meant for the target shards, to source shards, until the target shard is ready. commit 330debad664b087e4f419b77af8f4e1b61b0b443 Author: elasticsearchmachine <[email protected]> Date: Fri Jul 18 18:23:30 2025 +0200 Mute org.elasticsearch.xpack.esql.qa.multi_node.GenerativeIT test #131508 commit 0f00f69b137bf58f14c1c50d83cfa4d5cab8ed51 Author: Nhat Nguyen <[email protected]> Date: Fri Jul 18 08:40:30 2025 -0700 Optimized path for single segment in TS source (#131502) When only one segment remains for scanning, we can avoid moving the only segment iterator between the main queue and the sub-queue. This optimized path should also work with a forced-merge shard. commit 90699d3cc31f8a943b60b130c104a7bac920652c Author: Kathleen DeRusso <[email protected]> Date: Fri Jul 18 11:27:34 2025 -0400 Fix semantic highlighting bug on flat quantized fields (#131525) * Fix semantic highlighting bug on flat quantized fields * Update docs/changelog/131525.yaml commit 929f65b94c445bc1e3e64443ee20e61e76f556e5 Author: Pat Whelan <[email protected]> Date: Fri Jul 18 10:51:33 2025 -0400 [Persistent Tasks] Assign based on ProjectId (#130391) Pass the ProjectId from PersistentTaskClusterService through to all PersistentTasksExecutors when creating node assignments. These PersistentTasksExecutors require the ProjectId during node assignment: - OpenJobPersistentTasksExecutor - SnapshotUpgradeTaskExecutor - StartDatafeedPersistentTasksExecutor - TransformPersistentTasksExecutor commit a786c93a3b1882e51bd9b4dec21eaaff4eac14c7 Author: Michail Romaios <[email protected]> Date: Fri Jul 18 16:42:08 2025 +0200 refactor: enhance semantic_text inference error msg (#131519) commit efa71d7206a8291ee362f56a0b65e3e75f5758a8 Author: elasticsearchmachine <[email protected]> Date: Fri Jul 18 16:33:09 2025 +0200 Mute org.elasticsearch.xpack.downsample.DownsampleIT testAggMetricInEsqlTSAfterDownsampling #131500 commit e67e50b3f19d2326acf5e7b17211849d28aad210 Author: Julian Kiryakov <[email protected]> Date: Fri Jul 18 10:20:07 2025 -0400 [DOCS][ESQL] Fix release version in Docs for RLIKE LIST (#131465) RLIKE LIST did not manage to make it into 9.1. In this PR, we modify the documentation to make it clear that it will be available in 9.2, not 9.1 commit 6543e50cda593db90b8c215b116f2532bef597ff Author: Quentin Pradet <[email protected]> Date: Fri Jul 18 17:56:21 2025 +0400 Fix msearch rest-api-spec (#130627) * Fix msearch rest-api-spec * Add YAML tests for added parameters commit f5c6b35dab29b0dafa17101e236771d66fa4daf6 Author: elasticsearchmachine <[email protected]> Date: Fri Jul 18 15:33:41 2025 +0200 Mute org.elasticsearch.xpack.search.CrossClusterAsyncSearchIT testCancellationViaTimeoutWithAllowPartialResultsSetToFalse #131248 commit 06e39c0377e3769eae91487b6c2d22978f554b43 Author: Alexander Spies <[email protected]> Date: Fri Jul 18 15:30:57 2025 +0200 ESQL: Disallow remote enrich after lu join (#131426) Fix https://github.com/elastic/elasticsearch/issues/129372 Due to how remote ENRICH is [planned](https://github.com/elastic/elasticsearch/blob/32e50d0d94e27ee559d24bf9d5463ba6e64d1788/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/planner/mapper/Mapper.java#L93), it interacts in special ways with pipeline breakers, in particular LIMIT and TopN; when these are encountered upstream from a remote ENRICH, these nodes are copied and executed a second time after the remote ENRICH. We'd like to allow remote ENRICH after LOOKUP JOIN, but that forces the lookup to be remote as well; this has its own interactions with pipeline breakers: in particular, LIMITs and TopNs cannot just be duplicated after LOOKUP JOIN, as LOOKUP JOIN may add new rows. For now, let's just forbid any usage of remote ENRICH after LOOKUP JOINs; remote ENRICH is mostly relevant for CCS, and LOOKUP JOIN doesn't support that in 9.1/8.19, anyway. There is separate work that enables remote LOOKUP JOINs on remote clusters and adds the correct validations; we can later build support for remote ENRICH + LOOKUP JOIN on top of that. (C.f. my comment [here](https://github.com/elastic/elasticsearch/issues/129372#issuecomment-3083024230) and my draft https://github.com/elastic/elasticsearch/pull/131286 for enabling this.) commit 732bab022779e746a4817e6156975ac62600f059 Author: Kostas Krikellas <[email protected]> Date: Fri Jul 18 16:16:51 2025 +0300 Skip tests with TS command for non-snapshot builds (#131518) The `TS` source is guarded by a feature flag. Fixes #131500 commit 27a09d8529ee75cbac015111d05766ba7e3f5250 Author: Lorenzo Dematté <[email protected]> Date: Fri Jul 18 15:04:07 2025 +0200 Upgrade apm-agent to 1.55.0 (#131510) commit d70093b3ad4cbf5a4b671cb1c7f5aa496ff05238 Author: Tommaso Teofili <[email protected]> Date: Fri Jul 18 14:41:37 2025 +0200 ScoreTests capability check (#131516) commit 439b8e68bb7a33b22e1f6d7966247a0015ae893d Author: Nik Everett <[email protected]> Date: Fri Jul 18 08:37:14 2025 -0400 ESQL: Split large pages on load sometimes (#131053) This adds support for splitting `Page`s of large values when loading from single segment, non-descending hits. This is hottest code path as it's how we load data for aggregation. So! We had to make very very very sure this doesn't slow down the fast path of loading doc values. Caveat - this only defends against loading large values via the row-by-row load mechanism that we use for stored fields and _source. That covers the most common kinds of large values - mostly `text` and geo fields. If we need to split further on docs values, we'll have to invent something for them specifically. For now, just row-by-row. This works by flipping the order in which we load row-by-row and column-at-a-time values. Previously we loaded all column-at-a-time values first because that was simpler. Then we loaded all of the row-by-row values. Now we save the column-at-a-time values and instead load row-by-row until the `Page`'s estimated size is larger than a "jumbo" size which defaults to a megabyte. Once we load enough rows that we estimate the page is "jumbo", we then stop loading rows. The Page will look like this: ``` | txt1 | int | txt2 | long | double | |------|-----|------|------|--------| | XXXX | | XXXX | | | | XXXX | | XXXX | | | | XXXX | | XXXX | | | | XXXX | | XXXX | | | | XXXX | | XXXX | | | | XXXX | | XXXX | | | <-- after loading this row | | | | | | we crossed to "jumbo" size | | | | | | | | | | | | | | | | | | <-- these rows are entirely empty | | | | | | | | | | | | ``` Then we chop the page to the last row: ``` | txt1 | int | txt2 | long | double | |------|-----|------|------|--------| | XXXX | | XXXX | | | | XXXX | | XXXX | | | | XXXX | | XXXX | | | | XXXX | | XXXX | | | | XXXX | | XXXX | | | | XXXX | | XXXX | | | ``` Then fill in the column-at-a-time columns: ``` | txt1 | int | txt2 | long | double | |------|-----|------|------|--------| | XXXX | 1 | XXXX | 11 | 1.0 | | XXXX | 2 | XXXX | 22 | -2.0 | | XXXX | 3 | XXXX | 33 | 1e9 | | XXXX | 4 | XXXX | 44 | 913 | | XXXX | 5 | XXXX | 55 | 0.1234 | | XXXX | 6 | XXXX | 66 | 3.1415 | ``` And then we return *that* `Page`. On the next `Driver` iteration we start from where we left off. commit 40585a2199e454bb3028e09c8a6bf0162c3079b6 Author: Julian Kiryakov <[email protected]> Date: Fri Jul 18 08:33:58 2025 -0400 Add checks that optimizers do not modify the layout (#130855) Add verification that the optimizers do not modify the number of attributes and the attribute datatype. We add special handling for Lookup Join, by checking EsQueryExec esQueryExec && esQueryExec.indexMode() == LOOKUP and another special handling for ProjectAwayColumns.ALL_FIELDS_PROJECTED Closes #125576 commit c9d5076224a93c8a012083a258358adbbb446b02 Author: Luigi Dell'Aquila <[email protected]> Date: Fri Jul 18 13:26:12 2025 +0200 ES|QL: fix generative test (#131515) commit 8682c181c6585a0e70fa0670a8d7729cc8bd12aa Author: Iván Cea Fontenla <[email protected]> Date: Fri Jul 18 11:49:04 2025 +0200 [ESQL] Fix TopN grouping aggregates bug with non-qualifying intermediate states (#129633) Continuation of https://github.com/elastic/elasticsearch/pull/127148 When datanodes send the STATS intermediate states to the coordinator, it aggregates them. Now, however, the TopN groups sent by a datanode may not be acceptable in the coordinator (Because it has better values already), so it will discard such values. However, the engine wasn't handling intermediate groups with nulls (TopNBlockHash uses nulls to discard unused groups). See https://github.com/elastic/elasticsearch/blob/main/x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/aggregation/GroupingAggregator.java#L47 _This code isn't connected with the query yet, so there's no bug in production_ commit 9f22533c17d0eaf53402193243772997e3921603 Author: Joshua Adams <[email protected]> Date: Fri Jul 18 10:38:17 2025 +0100 [TEST] Remove YAML Tests (#131434) Removes YAML tests for the `/_cluster/allocation/explain` API. The tests passed in alternate values for the APIs. An example is passing "true" for fields expecting a boolean value. While this is explicitly supported by the API, this is not the correct place to be testing this behaviour, and resulted in the API specification failing validation. Relates #127028 commit fb6e0c0c571b9d5db0938a2e154587a8479065f9 Author: elasticsearchmachine <[email protected]> Date: Fri Jul 18 11:29:09 2025 +0200 Mute org.elasticsearch.xpack.downsample.DownsampleWithBasicRestIT test {p0=downsample-with-security/10_basic/Downsample index} #131513 commit db369bb7a64116fe69f2b37e4ce8da1f2f7d6a62 Author: Moritz Mack <[email protected]> Date: Fri Jul 18 11:15:13 2025 +0200 Disable entitlements for DirectIOIT, the suite requires delegation to work (#131505) Disable entitlements for DirectIOIT, the suite requires delegation to work. On main the suite is skipped (direct IO is disabled by default), but this blocks backports. commit 3c11ab29a356ea988829950366733aff541d1275 Author: elasticsearchmachine <[email protected]> Date: Fri Jul 18 10:22:00 2025 +0200 Mute org.elasticsearch.xpack.esql.qa.single_node.GenerativeIT test #131508 commit d637927e3d4db6d2f5f7bbf3c70ea36493c861b9 Author: Alexander Spies <[email protected]> Date: Fri Jul 18 09:21:09 2025 +0200 ESQL: Unmute some generative tests (#131446) The corresponding issue #116781 has already been fixed. commit a29d8ea0f4396e9881cb4298a29a87390e5f5ccd Author: elasticsearchmachine <[email protected]> Date: Fri Jul 18 09:14:35 2025 +0200 Mute org.elasticsearch.xpack.esql.action.EsqlActionBreakerIT testFromEvalStats #131503 commit 9edf9f65174dc808c99fe49e3c47660d0d814c14 Author: Nhat Nguyen <[email protected]> Date: Thu Jul 17 23:51:10 2025 -0700 Fix NPE in TimeSeriesExtractFieldOperator (#131497) We miss checking whether a field exists when populating the dimension attributes. This issue occurs when a field exists in some, but not all target indices. commit 654d1f4a7e466502bc8f83037e4b8d53d18f29d7 Author: Yang Wang <[email protected]> Date: Fri Jul 18 12:00:51 2025 +1000 Add missing query params to index recovery API spec (#131490) This PR adds the missing ignore_unavailable, allow_no_indices and expand_wildcards query parameters. commit d365412c12c45b863188ecc692e76c719d04f441 Author: elasticsearchmachine <[email protected]> Date: Fri Jul 18 01:25:27 2025 +0200 Mute org.elasticsearch.packaging.test.DockerTests test050BasicApiTests #120911 commit a528ebb0cbd22cf1411dbe8f63a23e5db3798dc7 Author: elasticsearchmachine <[email protected]> Date: Fri Jul 18 00:39:37 2025 +0200 Mute org.elasticsearch.test.rest.yaml.RcsCcsCommonYamlTestSuiteIT test {p0=vector-tile/20_aggregations/stats agg} #131484 commit 7e7093a24f7a11f9f6eac8b4994719f8b3c85c89 Author: elasticsearchmachine <[email protected]> Date: Fri Jul 18 00:16:53 2025 +0200 Mute org.elasticsearch.xpack.esql.heap_attack.HeapAttackIT testLookupExplosionNoFetch #128720 commit decce5659e02485e4aed5c7fefdbfb6306eb9fd0 Author: Nik Everett <[email protected]> Date: Thu Jul 17 17:24:50 2025 -0400 ESQL: Unmuted fixed tests (#131476) Fixed by #131370 or #130963. Closes #130505 Closes #130504 Closes #130501 Closes #131024 commit 560ffb973d71a1a9cd7b16942d2b6d2f87c86037 Author: Samiul Monir <[email protected]> Date: Thu Jul 17 16:57:15 2025 -0400 [Inference Timeout] Supply inference context to all third party services (#131251) * Refactoring inference services to accept context * fix linting issues * adding mock cluster service to fix IT test * refactoring to remove duplication in constructors * remove unnecessary blank line * refactor to have uniform constructor call * refactor to have uniform constructor call for sagemaker * fix linting issues * fix failed unit tests --------- Co-authored-by: Elastic Machine <[email protected]> commit 6ed50e1bae883b2f5628f96447b2322f364ce1b3 Author: Nik Everett <[email protected]> Date: Thu Jul 17 15:40:17 2025 -0400 Explain `ignore_above` better (#129284) This concept is complicated. Closes #128991 Co-authored-by: Larisa Motova <[email protected]> Co-authored-by: Liam Thompson <[email protected]> commit 56477d81a9ec03ac9fe304c166519fccabaab305 Author: Liam Thompson <[email protected]> Date: Thu Jul 17 21:21:53 2025 +0200 Split retrievers docs and redirect anchors (#131385) commit cf5d40fa1fc389bfb9e6158064389e20e62a7004 Author: Benjamin Trent <[email protected]> Date: Thu Jul 17 14:55:51 2025 -0400 Fix diskbbq flush logic (#131470) I accidentally broke recall on flush by allowing vectors to be double quantized. Additionally, we shouldn't use the first vector as a centroid, this can harm recall significantly when there is just one centroid. recall before this change: ``` index_name index_type num_docs index_time(ms) force_merge_time(ms) num_segments ------------------------------------- ---------- -------- -------------- -------------------- ------------ corpus-dbpedia-entity-E5-small-0.fvec ivf 1000000 25820 0 14 corpus-dbpedia-entity-E5-small-0.fvec ivf 1000000 0 41693 0 index_name index_type n_probe latency(ms) net_cpu_time(ms) avg_cpu_count QPS recall visited filter_selectivity ------------------------------------- ---------- ------- ----------- ---------------- ------------- ------ ------ --------- ------------------ corpus-dbpedia-entity-E5-small-0.fvec ivf 50 13.05 0.00 0.00 76.61 0.63 285267.44 1.00 corpus-dbpedia-entity-E5-small-0.fvec ivf 150 31.92 0.00 0.00 31.33 0.68 629033.22 1.00 corpus-dbpedia-entity-E5-small-0.fvec ivf 200 34.79 0.00 0.00 28.74 0.69 679699.13 1.00 corpus-dbpedia-entity-E5-small-0.fvec ivf 500 39.40 0.00 0.00 25.38 0.71 794375.05 1.00 corpus-dbpedia-entity-E5-small-0.fvec ivf 1000 45.99 0.00 0.00 21.74 0.72 940493.52 1.00 corpus-dbpedia-entity-E5-small-0.fvec ivf 50 1.52 0.00 0.00 655.74 0.74 24201.82 1.00 corpus-dbpedia-entity-E5-small-0.fvec ivf 150 2.94 0.00 0.00 340.43 0.85 67943.31 1.00 corpus-dbpedia-entity-E5-small-0.fvec ivf 200 3.81 0.00 0.00 262.81 0.87 89575.99 1.00 corpus-dbpedia-entity-E5-small-0.fvec ivf 500 7.67 0.00 0.00 130.38 0.93 213586.44 1.00 corpus-dbpedia-entity-E5-small-0.fvec ivf 1000 14.85 0.00 0.00 67.33 0.96 402628.11 1.00 ``` With this fix: ``` index_name index_type num_docs index_time(ms) force_merge_time(ms) num_segments ------------------------------------- ---------- -------- -------------- -------------------- ------------ corpus-dbpedia-entity-E5-small-0.fvec ivf 1000000 25304 0 15 corpus-dbpedia-entity-E5-small-0.fvec ivf 1000000 0 42110 0 index_name index_type n_probe latency(ms) net_cpu_time(ms) avg_cpu_count QPS recall visited filter_selectivity ------------------------------------- ---------- ------- ----------- ---------------- ------------- ------ ------ --------- ------------------ corpus-dbpedia-entity-E5-small-0.fvec ivf 50 12.63 0.00 0.00 79.18 0.89 285527.22 1.00 corpus-dbpedia-entity-E5-small-0.fvec ivf 150 32.49 0.00 0.00 30.77 0.94 619783.37 1.00 corpus-dbpedia-entity-E5-small-0.fvec ivf 200 35.46 0.00 0.00 28.20 0.95 667903.47 1.00 corpus-dbpedia-entity-E5-small-0.fvec ivf 500 40.38 0.00 0.00 24.76 0.97 781959.74 1.00 corpus-dbpedia-entity-E5-small-0.fvec ivf 1000 48.62 0.00 0.00 20.57 0.98 931017.40 1.00 corpus-dbpedia-entity-E5-small-0.fvec ivf 50 1.55 0.00 0.00 643.09 0.74 23595.57 1.00 corpus-dbpedia-entity-E5-small-0.fvec ivf 150 2.98 0.00 0.00 335.29 0.85 66299.43 1.00 corpus-dbpedia-entity-E5-small-0.fvec ivf 200 3.81 0.00 0.00 262.64 0.87 87416.15 1.00 corpus-dbpedia-entity-E5-small-0.fvec ivf 500 8.80 0.00 0.00 113.64 0.93 209061.37 1.00 corpus-dbpedia-entity-E5-small-0.fvec ivf 1000 16.18 0.00 0.00 61.81 0.96 394906.29 1.00 ``` commit 9fa06245a5187713f8175493f6548c9733bc2cfa Author: Larisa Motova <[email protected]> Date: Thu Jul 17 08:31:25 2025 -1000 [ES|QL] Support implicit casting of aggregate_metric_double (#129108) This commit adds support for implicit casting of aggregate_metric_double when present with other numerics for a limited set of aggregation functions: - Max / MaxOverTime - Min / MinOverTime - Sum / SumOverTime - Count / CountOverTime - Avg / AvgOverTime Attempting to use fields mapped to aggregate_metric_double in one index but some other numeric in another index in any other context will still require explicit casting with ToAggregateMetricDouble commit 221998d9fb68952165e0dad75719a05edc1be483 Author: eyalkoren <[email protected]> Date: Thu Jul 17 21:23:27 2025 +0300 Revert "Support Fields API in conditional ingest processors (#121914)" (#131452) This reverts commit a6f0f6fb4d1ddc38d6aba871ad8d196e0cd16406. commit 78f613362364998b96bde769c76f8e8b0df6ba47 Author: Nhat Nguyen <[email protected]> Date: Thu Jul 17 11:10:49 2025 -0700 Fix data race in InlineStats (#131370) Although blocks/vectors are immutable and safe to share between threads, their references are currently not thread-safe, which can lead to data races. Previously, blocks/vectors were exclusively owned by a single thread, but this is no longer always the case with InlineJoin. We should consider switching to AbstractRefCounted, which is thread-safe, and benchmark it with many-fields use cases to ensure there is no performance regression. As a temporary solution, this change clones the values block in InlineJoin until thread-safe blocks/vectors are available. commit 280793df0960ffe4ab1ccd5681458d948b27773b Author: Ruben van Staden <[email protected]> Date: Thu Jul 17 13:35:28 2025 -0400 apm-data: enable failure store for newly created APM datastreams (#131296) commit 7a01565a59e210e99152d247b3a5bf2e915d8f37 Author: Jordan Powers <[email protected]> Date: Thu Jul 17 09:42:19 2025 -0700 Fix match_only_text keyword multi-field bug (#131383) In #131314 we fixed match_only_text fields with ignore_above keyword multi-fields in the case that the keyword multi-field is stored. However, the issue is still present if the keyword field is not stored, but instead has doc values. This patch fixes that case. commit 6da8f9239246d74b53e24debd31f886fda059292 Author: Pat Whelan <[email protected]> Date: Thu Jul 17 10:57:41 2025 -0400 [ML] Block trained model updates from inference (#130940) When the Trained Model has been deployed through the Inference Endpoint API, it can only be updated using the Inference Endpoint API. When the Trained Model has been deployed and then attached to an Inference Endpoint, it can only be updated using the Trained Model API. Fix #129999 Co-authored-by: elasticsearchmachine <[email protected]> Co-authored-by: David Kyle <[email protected]> commit e2bb47c3bb5be3ed77ac5026db40d9297b458f36 Author: Samiul Monir <[email protected]> Date: Thu Jul 17 10:51:29 2025 -0400 Fix Semantic Query Rewrite Interception Drops Boosts (#129282) * fix boosting for knn * Fixing for match query * fixing for match subquery * fix for sparse vector query boost * fix linting issues * Update docs/changelog/129282.yaml * update changelog * Copy constructor with match query * util function to create sparseVectorBuilder for sparse query * util function for knn query to support boost * adding unit tests for all intercepted query terms * Adding yaml test for match,sparse, and knn * Adding queryname support for nested query * fix code styles * Fix failed yaml tests * Update docs/changelog/129282.yaml * update yaml tests to expand test scenarios * Updating knn to copy constructor * adding yaml tests for multiple indices * refactoring match query to adjust boost and queryname and move to copy constructor * refactoring sparse query to adjust boost and queryname and move to copy constructor * [CI] Auto commit changes from spotless * Refactor sparse vector to adjust boost and queryname in the top level * Refactor knn vector to adjust boost and queryname in the top level * fix knn combined query * fix unit tests * fix lint issues * remove unused code * Update inference feature name * Remove double boosting issue from match * Fix double boosting in match test yaml file * move to bool level for match semantic boost * fix double boosting for sparse vector * fix double boosting for sparse vector in yaml test * fix knn combined query * fix knn combined query * fix sparse combined query * fix knn yaml test for combined query * refactoring unit tests * linting * fix match query unit test * adding copy constructor for match query * refactor copy match builder to intercepter * [CI] Auto commit changes from spotless * fix unit tests * update yaml tests * fix match yaml test * fix yaml tests with 4 digits error margin * unit tests are now more randomized --------- Co-authored-by: Elastic Machine <[email protected]> Co-authored-by: elasticsearchmachine <[email protected]> commit 628828fd1d16c5a8d2845e0b1b859dbd07c5a3ab Author: Ignacio Vera <[email protected]> Date: Thu Jul 17 15:50:05 2025 +0100 [DiskBBQ] Write the raw centroid on the posting list file instead of the centroids file (#131421) commit f739673b48454a879d39a072a7f01bd95dfa497c Author: Ben Chaplin <[email protected]> Date: Thu Jul 17 10:48:19 2025 -0400 Fix bug in point in time response (#131391) Correct response which had swapped "skipped" and "failed" shard counts. commit df2fc49a1a036d8ed30389354182a94371638cfc Author: Moritz Mack <[email protected]> Date: Thu Jul 17 16:46:00 2025 +0200 Support entitlements in internal cluster tests (#130710) * To prevent an implicit grant-all if storing node homes inside the Java temp dir, the temporary folder of ESTestCase is configured separately from the Java temp dir in internalClusterTests (by means of the system property tempDir, see TestRuleTemporaryFilesCleanup) * Move ReloadingDatabasesWhilePerformingGeoLookupsIT from internalClusterTest to test, file permissions in internalClusterTest are stricter on the lucene tempDir commit 3b1523a3a8e9cf44fb38448fe876131cf7d5bf36 Author: Jonathan Buttner <[email protected]> Date: Thu Jul 17 10:27:38 2025 -0400 [ML] Refactoring streaming error handling (#131316) * Refactoring google gemini streaming error handling * Updating comments commit fd971e84c223295ec26dc34e7584cd74bdf663c0 Author: Yang Wang <[email protected]> Date: Fri Jul 18 00:15:25 2025 +1000 Update regions_by_endpoint for AWS sdk upgrade. (#131400) Also add test to ensure the file has at least one entry for each region so that it is easy to spot missing regions in future upgrades. Relates: #131050 Resolves: #131392 commit df985e60504735a0e401415c1094c3231a01b326 Author: Joshua Adams <[email protected]> Date: Thu Jul 17 14:30:50 2025 +0100 Add `includeDiskInfo` to `toString()` (#131358) Adds the `includeDiskInfo` parameter to the `cluster/allocation/explain` `toString()` method, and adds tests. commit d06b0c8c17429ff5eee853e193edc27b70cc28ad Author: Evgenii-Kazannik <[email protected]> Date: Thu Jul 17 15:24:02 2025 +0200 Add Azure AI Rerank support (#129848) * Add Azure AI Rerank support * address comments * address comments * refactor azure ai studio service * update rerank task settings test * add provider for rerank commit f9eee6c216ca7574c66a03b5dfe9ebc18cda051d Author: Ignacio Vera <[email protected]> Date: Thu Jul 17 13:14:46 2025 +0100 [DiskBBQ] Use PackedLongValues to hold offsets on heap while writing (#131411) commit 8fb9fb5cee69e9420572f8b87873be3e52d982dd Author: Quentin Pradet <[email protected]> Date: Thu Jul 17 15:51:45 2025 +0400 Remove 'index' from snapshot clear_cache query params (#131067) It's already part of the path parts, it's not useful to duplicate it in query parameters. commit 32e50d0d94e27ee559d24bf9d5463ba6e64d1788 Author: Lorenzo Dematté <[email protected]> Date: Thu Jul 17 11:37:09 2025 +0200 Migrate x-pack-autoscaling REST tests (#131365) This PR migrates legacy rest tests in the x-pack autoscaling module commit ec7f77becb29ad049327cc05770454e28029bb2c Author: Jan Kuipers <[email protected]> Date: Thu Jul 17 10:24:30 2025 +0200 ES|QL categorize options (#131104) * ES|QL categorize options * refactor options * fix serialization * polish * add verfications * better test coverage + polish code * better test coverage + polish code commit ac0c50820a540f64ad835bd52dff004dd4fc60af Author: kanoshiou <[email protected]> Date: Thu Jul 17 16:23:04 2025 +0800 ESQL: Fix inconsistent column order in MV_EXPAND (#129745) The new attribute generated by MV_EXPAND should remain in the original position. The projection added by ProjectAwayColumns does not respect the original order of attributes. Make ProjectAwayColumns respect the order of attributes to fix this. commit adcb2a57cecd5acbfb46282f27cab589aaac1c94 Author: elasticsearchmachine <[email protected]> Date: Thu Jul 17 09:43:49 2025 +0200 Mute org.elasticsearch.packaging.test.DockerTests test072RunEsAsDifferentUserAndGroup #131412 commit efd311016358e1f738aaf42627b1beb409190f27 Author: Nhat Nguyen <[email protected]> Date: Wed Jul 16 22:36:47 2025 -0700 Remove ordinal grouping path in aggregations (#131307) With the ordinal grouping operator removed in #131133, this PR removes the corresponding code path in the grouping aggregator function, as it is no longer needed. Relates #131133 commit f135998b118d4e58922300cf942506ae25cd7ab1 Author: Lisa Cawley <[email protected]> Date: Wed Jul 16 17:00:21 2025 -0700 [DOCS] Augment self-managed connector tutorials (#131127) commit 6832ca40b3986ceb1b0a7daefd8bec193f47701d Author: elasticsearchmachine <[email protected]> Date: Wed Jul 16 23:32:27 2025 +0200 Mute org.elasticsearch.xpack.downsample.DataStreamLifecycleDownsampleDisruptionIT testDataStreamLifecycleDownsampleRollingRestart #131394 commit 04119405d141bcb390509f766ebbb7660b0ffe72 Author: Stanislav Malyshev <[email protected]> Date: Wed Jul 16 13:09:41 2025 -0600 Put shards failure under a cap flag (#131371) * Put shards failure under a cap flag commit b57ee3bb3adcef8a36f68cef7c69ee0b26503d3d Author: Ignacio Vera <[email protected]> Date: Wed Jul 16 20:03:16 2025 +0100 [DiskBBQ] Remove posting lists offsets from meta file (#131379) this commit proposes to remove the offsets list from the meta file, instead it adds the offsets just after the centroids, together with the raw centroid. commit 9226a656d5aab86a9f1ec836731fad198940381b Author: O.K. <[email protected]> Date: Wed Jul 16 21:48:08 2025 +0300 [DOCS]: fix decimal digit filter reference (#129695) commit 38b7bfcaf39bb78b80da75adeb9acc5afa3aad9d Author: Sam Xiao <[email protected]> Date: Wed Jul 16 13:06:36 2025 -0400 Mark watcher NotMultiProjectCapable and replace deprecated multi-project methods (#131313) commit 9c6cf9045639da26c39916221fea955cdff63f60 Author: Dan Rubinstein <[email protected]> Date: Wed Jul 16 12:44:48 2025 -0400 Enable force inference endpoint deleting for invalid models and after stopping model deployment fails (#129090) * Enable force inference endpoint deleting for invalid models and after stopping model deployment fails * Update docs/changelog/129090.yaml --------- Co-authored-by: Elastic Machine <[email protected]> commit 037ddaa5c870d770d42de35741f3f53f90c6c0bb Author: Pat Whelan <[email protected]> Date: Wed Jul 16 12:00:38 2025 -0400 [ML] Remove SageMaker Elastic updates (#131301) Rather than silently drop the payload, throw a validation error when Users try to send task settings in the update payload for SageMaker inference with the Elastic API. commit 0cf275eca411a4311cf405bbbb4d0cb0ed931c3e Author: Ben Chaplin <[email protected]> Date: Wed Jul 16 11:50:33 2025 -0400 Refactor AsyncSearchErrorTraceIT to use assertBusy (#131328) commit 1da16920ded879ea561b62d416fdf6c4aa674d17 Author: elasticsearchmachine <[email protected]> Date: Wed Jul 16 17:36:24 2025 +0200 Mute org.elasticsearch.packaging.test.DockerTests test130JavaHasCorrectOwnership #131369 commit ae75a98d6c06a7d091e3613f121285a160a047fc Author: Lorenzo Dematté <[email protected]> Date: Wed Jul 16 17:31:05 2025 +0200 Add exception logging when interrupted (#131153) commit a04f07571cc80752d643c9ba161e2424813c31f9 Author: elasticsearchmachine <[email protected]> Date: Wed Jul 16 17:16:01 2025 +0200 Mute org.elasticsearch.packaging.test.DockerTests test140CgroupOsStatsAreAvailable #131372 commit 718d54a0a63025fd331ec5b1ab255e79898045ff Author: elasticsearchmachine <[email protected]> Date: Wed Jul 16 17:05:59 2025 +0200 Mute org.elasticsearch.packaging.test.DockerTests test070BindMountCustomPathConfAndJvmOptions #131366 commit 467fc7ad744f22a86b0e6daed0db51791935b7a5 Author: elasticsearchmachine <[email protected]> Date: Wed Jul 16 16:40:29 2025 +0200 Mute org.elasticsearch.xpack.test.rest.XPackRestIT test {p0=ml/delete_expired_data/Test delete expired data with body parameters} #131364 commit 04104bb6493c6339dfe9a1a52bf7b549d2b62e9e Author: elasticsearchmachine <[email protected]> Date: Wed Jul 16 16:39:57 2025 +0200 Mute org.elasticsearch.xpack.esql.vector.VectorSimilarityFunctionsIT testSimilarityBetweenConstantVectorAndField {functionName=v_cosine similarityFunction=COSINE} #131363 commit 11e8d18447b087c810c3b9f84a1a781ef3823c8e Author: elasticsearchmachine <[email protected]> Date: Wed Jul 16 16:39:50 2025 +0200 Mute org.elasticsearch.xpack.esql.vector.VectorSimilarityFunctionsIT testDifferentDimensions {functionName=v_cosine similarityFunction=COSINE} #131362 commit 32a4698c5eeab320995c948cd2f5e3783613c1c1 Author: elasticsearchmachine <[email protected]> Date: Wed Jul 16 16:39:42 2025 +0200 Mute org.elasticsearch.xpack.esql.vector.VectorSimilarityFunctionsIT testSimilarityBetweenConstantVectors {functionName=v_cosine similarityFunction=COSINE} #131361 commit ff1ad0c271d720179a3f3b42cdd65d2caef7466c Author: Tommaso Teofili <[email protected]> Date: Wed Jul 16 16:32:11 2025 +0200 Check SCORE_FUNCTION capability in VerifierTests (#131352) commit 1832248827c861293b66eb5fb3bce6477ca3991e Author: Ievgen Degtiarenko <[email protected]> Date: Wed Jul 16 15:39:11 2025 +0200 Replace deprecated routingTable table call in tests (#131005) commit 27bba64b84fe206b6fab915ac3582a083fab24c3 Author: Liam Thompson <[email protected]> Date: Wed Jul 16 15:37:59 2025 +0200 [DOCS] Remove misused applies_to tag (#131349) * [DOCS] Remove misused applies_to tag commit 44497b7b05720a5de2608b0a40c6678e50a2fc6f Author: Benjamin Trent <[email protected]> Date: Wed Jul 16 09:35:35 2025 -0400 Adj ivf postings list building (#130843) * Make postings list building more IO friendly * iter * iter * fixing assertion * [CI] Auto commit changes from spotless --------- Co-authored-by: elasticsearchmachine <[email protected]> commit ce481e5f8050f8f3aa6d118d08a4a0275eb8a997 Author: Pat Whelan <[email protected]> Date: Wed Jul 16 08:26:39 2025 -0400 [Transform] Read metadata from Project State (#131205) Add a ProjectState version of the TransformMetadata helper methods. commit 2af0e9f5e10366571bfebb0f297d6d928510dc29 Author: David Turner <[email protected]> Date: Wed Jul 16 11:36:27 2025 +0100 Add note on o11y to architecture guide (#131291) Adds brief overview that mostly just serves as a location for links to other more comprehensive design documentation elsewhere in the source tree. Closes ES-9874 commit ebdfa9c0f5b582469010f2d0335051058e10c23e Author: David Turner <[email protected]> Date: Wed Jul 16 10:48:02 2025 +0100 Upgrade AWS Java SDK to 2.31.78 (#131050) This picks up the fix for the locale bug reported at https://github.com/aws/aws-sdk-java-v2/issues/5968. This reverts the patching of the library added in commit 2697a3a87215c92efb3c9bd6c42b99ae8a63f2a7, except for the test enhancement. Fix confirmed in e.g. locale `ar-ER` with ./gradlew ":plugins:discovery-ec2:test" \ --tests "org.elasticsearch.discovery.ec2.Ec2DiscoveryTests.testFilterByTags" \ -Dtests.seed=596874EED28A2B92 -Dtests.locale=ar-ER -Dtests.timezone=NET \ -Druntime.java=24 commit a6f0f6fb4d1ddc38d6aba871ad8d196e0cd16406 Author: eyalkoren <[email protected]> Date: Wed Jul 16 11:46:17 2025 +0300 Support Fields API in conditional ingest processors (#121914) commit 6ffe27d030aef155d85dc1cc1eb939a5a9fe8fbb Author: Carlos Delgado <[email protected]> Date: Wed Jul 16 10:19:18 2025 +0200 ESQL - KNN function uses prefilters when pushed down to Lucene (#131004) commit 71466813223574d6a8b4b33dab8433e3b877664c Author: Luigi Dell'Aquila <[email protected]> Date: Wed Jul 16 10:15:08 2025 +0200 Add docs for ES|QL query logs (#131287) commit 01c09cba84998aa4c11824e44f18a44bfac0bfa8 Author: David Turner <[email protected]> Date: Wed Jul 16 08:12:48 2025 +0100 Simplify `expectedFinalRegisterValue` computation (#131274) In repository analysis we keep an `AtomicLong` counting the number of successfully-completed increments on the contended register, but we only check this value if all the increments succeed. We know how many increments we enqueue up-front so there's no need to count them as they complete. This commit removes the unnecessary counter. commit f8b89a378abd8bc990f9955eb2c68d7acf701424 Author: elasticsearchmachine <[email protected]> Date: Wed Jul 16 08:38:52 2025 +0200 Mute org.elasticsearch.test.rest.yaml.RcsCcsCommonYamlTestSuiteIT test {p0=search/110_field_collapsing/field collapsing, inner_hits and maxConcurrentGroupRequests} #131348 commit dc48b4b28b8b3bd0b6b1909e26da6374c374322a Author: Tim Vernum <[email protected]> Date: Wed Jul 16 13:44:09 2025 +1000 Add attribute count to SamlAttribute toString (#131173) Sometimes SAML IdPs send what _should_ be a list of values as a single comma-separated string. That is, we expect something using SAML's multi-valued attribute feature: <saml:Attribute NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:uri" Name="http://idp.example.org/attributes/groups" FriendlyName="groups"> <saml:AttributeValue>engineering</saml:AttributeValue> <saml:AttributeValue>elasticsearch-admins</saml:AttributeValue> <saml:AttributeValue>employees</saml:AttributeValue> </saml:Attribute> but we get <saml:Attribute NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:uri" Name="http://idp.example.org/attributes/groups" FriendlyName="groups"> <saml:AttributeValue>engineering,elasticsearch-admins,employees</saml:AttributeValue> </saml:Attribute> In order to help detect these cases, this commit changes the `toString()` on `SamlAttribute` to include the length (e.g. `(len=1)`) at the end Relates: #84379, #102769 commit 13aceaa78dba62881fc47670c6fd2f58780314db Author: Kuni Sen <[email protected]> Date: Wed Jul 16 12:24:44 2025 +0900 Update reference-docs-links.txt to include serverless doc reference link (#131175) * Update reference-docs-links.txt * Update reference-docs-links.txt * Update ReferenceDocs.java commit 393f70703f1f62969fad8c13598dddd887753f28 Author: Stanislav Malyshev <[email protected]> Date: Tue Jul 15 19:02:15 2025 -0600 Disable in release build due to pragma (#131340) commit 004af14a0a0a0679a9bd526b5548d80a92257c9a Author: Niels Bauman <[email protected]> Date: Tue Jul 15 20:45:03 2025 -0300 Remove `ClusterStateSerializationTests` (#130877) These tests do not seem valuable anymore, now that multi-project has been merged some time ago. Additionally, creating objects for a cluster state this way is tricky, as those objects might get new features/fields, which would break serialization. Closes #130872 commit f9b9e7cfa9d4b2ef25a479353d5e11de21c7722c Author: elasticsearchmachine <[email protected]> Date: Wed Jul 16 01:33:45 2025 +0200 Mute org.elasticsearch.packaging.test.DockerTests test171AdditionalCliOptionsAreForwarded #120925 commit dac6021d140e09054072bd5c6d10d0d8fd018a10 Author: elasticsearchmachine <[email protected]> Date: Wed Jul 16 00:55:21 2025 +0200 Mute org.elasticsearch.packaging.test.DockerTests test071BindMountCustomPathWithDifferentUID #120917 commit 71d04a4b411ea7ef4a6ab47652e40d95300700d1 Author: elasticsearchmachine <[email protected]> Date: Wed Jul 16 00:55:12 2025 +0200 Mute org.elasticsearch.packaging.test.DockerTests test022InstallPluginsFromLocalArchive #116866 commit 7331271f33da644e7bb8c9691a8ccd415012c1e2 Author: Pablo <[email protected]> Date: Tue Jul 15 15:37:46 2025 -0700 Tightening rate verification (#131114) commit 8f6f7633bcecefc4b61ed68bec6dbf4da694b0dc Author: Nhat Nguyen <[email protected]> Date: Tue Jul 15 15:31:31 2025 -0700 Fail request when all target shards fail in runtime (#131177) If all target shards, excluding skipped shards, fail, we should fail the entire query regardless of the partial_results configuration or skip_unavailable setting. This behavior does not fully align with the search API, where skip_unavailable ignores all failures from remote clusters and only fails the request when all shards in the local cluster fail. However, we believe the proposed behavior is more sensible than the existing behavior in the search API. Closes #128994 commit 000c71e60d8bc5bbf2a2c624180a73d12279e2e3 Author: elasticsearchmachine <[email protected]> Date: Wed Jul 16 00:30:46 2025 +0200 Mute org.elasticsearch.xpack.esql.analysis.VerifierTests testMatchInsideEval #131336 commit c12f382ae5f2d0012e5d209709fb172841fa59f7 Author: elasticsearchmachine <[email protected]> Date: Wed Jul 16 00:30:38 2025 +0200 Mute org.elasticsearch.xpack.esql.analysis.VerifierTests testFullTextFunctionsOnlyAllowedInWhere #131335 commit 42a271eaa4db6bcaaa31b1f538db315c1d2f5dab Author: elasticsearchmachine <[email protected]> Date: Wed Jul 16 00:30:27 2025 +0200 Mute org.elasticsearch.xpack.esql.expression.function.fulltext.ScoreTests testSerializationOfSimple {TestCase=<boolean>} #131334 commit 6f8be9c4ff502874d60a06fe227ab5f691ebde2a Author: Jordan Powers <[email protected]> Date: Tue Jul 15 15:19:28 2025 -0700 Fix match_only_text for keyword multi-fields with ignore_above (#131314) commit 7bbb211cb9b0abcc5bb5aae5674aac7b51a2f5fa Author: Quentin Pradet <[email protected]> Date: Wed Jul 16 00:42:40 2025 +0400 Add require_alias to reindex rest-api-spec (#130813) commit 6ccc4494c36b464defc993c9a57dca7614eb3f94 Author: Brendan Cully <[email protected]> Date: Tue Jul 15 12:51:13 2025 -0700 Log start of memory throttling at info (#131310) This is symmetric with logging when throttling stops, and will make it easier to debug throttling issues. commit fde4c0c596337c42c08492c75fbf0387a4c182d3 Author: Benjamin Trent <[email protected]> Date: Tue Jul 15 15:31:35 2025 -0400 Adds support for more than one input doc file for Knnindextester (#131308) commit 57598222772c3c4cb06d7639c3d1db50d8542c97 Author: Mark Tozzi <[email protected]> Date: Tue Jul 15 15:11:20 2025 -0400 Clean up non-parameterized tests in the wrong place (#131049) In the course of other work I found a few places where we were creating non-parameterized tests from within the parameterized test drivers for a few ESQL functions. This causes those tests to be run for every parameter combination, even though the tests themselves do not change anything, resulting in a lot of extra test overhead for no additional coverage. I cleaned up three classes: EndsWithTests ran 832 tests before this, and now runs 640 tests StartsWithTests ran 208 before this, and now runs 160 InTests ran 168 before this, and now runs 28 So overall, that's 369 redundant test runs removed (including the fact that the 11 tests I moved still run once in their new classes), and further savings if we later expand those parameterized tests. commit f0c30f272da3ff36f1a65524cc0e63a07389800a Author: Julian Kiryakov <[email protected]> Date: Tue Jul 15 14:39:11 2025 -0400 Add support for RLIKE (LIST) with pushdown (#129929) Adds support for RLIKE function alternative syntax with a list of patterns. Examples: ROW message = "foobar" | WHERE message RLIKE ("foo.*", "bar.") The new syntax is documented as part of the existing RLIKE function documentation. We will use the existing RLike java implementation for existing cases using the old syntax and one list argument case to improve mixed cluster compatibility. The RLikeList is pushed down as a single Automaton to improve performance. commit 03975a363ac36f16e4645b026f55816426d02b5e Author: Sam Xiao <[email protected]> Date: Tue Jul 15 14:19:15 2025 -0400 Make TransportDeleteDatabaseConfigurationAction project aware (#131300) commit bfab88c430a727de63188687a2c4655276a10212 Author: Stanislav Malyshev <[email protected]> Date: Tue Jul 15 11:51:57 2025 -0600 Fix testStopQueryLocal - always use the same client (same node) (#131253) commit b1c78ce2de591c014a727e9444438c80ff43db31 Author: Sam Xiao <[email protected]> Date: Tue Jul 15 13:49:21 2025 -0400 Make TransportGetDatabaseConfigurationAction project aware (#130065) commit fcd5afaa5d24c2647c40043d4b6906175e72467b Author: Sam Xiao <[email protected]> Date: Tue Jul 15 13:48:27 2025 -0400 Mark custom GeoIP database NotMultiProjectCapable (#130064) commit 366bc0068e6bfdfe055f4eb6e220ec891e2b9eb2 Author: Stanislav Malyshev <[email protected]> Date: Tue Jul 15 10:19:07 2025 -0600 Refactor and simplify missing index & unavailability handling (#131252) commit 516990c075b9f47f63dedd9c8f0e33e13ca4e915 Author: Nhat Nguyen <[email protected]> Date: Tue Jul 15 09:00:41 2025 -0700 Remove ordinal grouping operator (#131133) The ordinals grouping operator was introduced to speed up aggregation before ordinal blocks and related optimizations in block hashes were available. However, this operator has several issues: 1. It only supports single grouping with the `keyword` type and requires `doc_values`. 2. It needs a separate aggregation implementation, which currently lacks test coverage. We had performance issues with the `VALUES` aggregation using this operator (see #130576). 3. It can be slower and use more memory when the target documents have sparse cardinality (see #98963). 4. Ad-hoc planning, although this can now be addressed with local plans. Although the ordinals grouping operator is slightly faster than the hash operator with ordinal blocks, its complexity now outweighs the benefits. This PR proposes removing the operator. Below is the NYC_taxis benchmark. Closes #98963 commit 04ae5275bee342be105f15ae7728cca2be6259e7 Author: Fang Xing <[email protected]> Date: Tue Jul 15 11:55:58 2025 -0400 [ES|QL] Substitute date_trunc with round_to when the pre-calculated rounding points are available (#128639) * consolidate min/max in SearchStats and substitue date_trunc/bucket with round_to commit c2fa78fda121c966277e4728d1aa1836152ca938 Author: Nhat Nguyen <[email protected]> Date: Tue Jul 15 08:50:27 2025 -0700 Speed up reading multivalued keywords (#131061) This change speeds up reading multi-valued keyword fields by leveraging ordinals. Before: ``` Benchmark (layout) (name) Mode Cnt Score Error Units ValuesSourceReaderBenchmark.benchmark in_order keyword_mv avgt 7 318.332 ± 1.660 ns/op ``` After: ``` Benchmark (layout) (name) Mode Cnt Score Error Units ValuesSourceReaderBenchmark.benchmark in_order keyword_mv avgt 7 96.659 ± 0.932 ns/op ``` commit 1669e8d438dbd9b88630bfc501d6d84b71da8e41 Author: Pablo <[email protected]> Date: Tue Jul 15 08:19:23 2025 -0700 Redesigining randomized tests for TS functions (#129971) * Working on rate integration test * Sketch of new rate tests. Unsure of the source of variation >10pct * Tuning and improving auto-test for rate-based aggregations * fixup * addressing comments * fixup * improve failure rate * tuned deviation limits to reduce flakiness * fixup - wider error margin * wider margin * wider range for aggs commit 1541b146403f90bf7425b16c4f18e98e360463fa Author: Joshua Adams <[email protected]> Date: Tue Jul 15 16:03:40 2025 +0100 Add Query Parameters to Cluster Allocation Explain API (#129342) The cluster allocation explain API now accepts parameters via the request body, via path parameters passed in the URL, but not via both. Issue: https://github.com/elastic/elasticsearch/issues/127028 commit 9d4c3f84ff58d2310b1a9bb000b1e75fc82dc2a5 Author: Iraklis Psaroudakis <[email protected]> Date: Tue Jul 15 17:55:30 2025 +0300 Fix concurrent list in merge test (#131186) Fixes #131165 commit 86c778768b22eaf1fa4a23c9ad788ed12fe46bd5 Author: Sam Xiao <[email protected]> Date: Tue Jul 15 10:39:23 2025 -0400 Remove deprecated multi-project methods in Enterprise GeoIP downloader (#130062) commit 480615633cbf5c7e0b2f67a23f04d9e26f040592 Author: Slobodan Adamović <[email protected]> Date: Tue Jul 15 15:20:42 2025 +0200 Unmute docker tests, add more logging and increase startup timeout (#131203) Unmuting 33 DockerTests with added additional logging on failures. Also, bumping docker startup timeout to account for slow security auto-configuration, and additionally moving CA certificates inside the retry while loop. Resolves #129808 Resolves #123437 Resolves #120911 Resolves #120914 Resolves #120910 Resolves #120918 Resolves #120925 Resolves #124990 Resolves #116636 Resolves #125680 Resolves #116147 Resolves #126232 Resolves #126145 Resolves #116866 Resolves #126936 Resolves #127081 Resolves #127158 Resolves #127138 Resolves #128018 Resolves #128031 Resolves #128110 Resolves #128006 Resolves #128174 Resolves #128144 Resolves #128165 Resolves #128075 Resolves #128117 Resolves #128121 Resolves #128115 Resolves #128230 Resolves #128120 Resolves #128996 Resolves #128867 commit f1ddd4c31282155dbc5db955003abd2494ecf9b4 Author: Carlos Delgado <[email protected]> Date: Tue Jul 15 14:49:25 2025 +0200 ESQL: dense_vector cosine similarity function (#130641) commit 730308c6898a1ba94b59f790e0699e6fe021074d Author: Carlos Delgado <[email protected]> Date: Tue Jul 15 14:15:09 2025 +0200 Unmute #130640 - test failure was unrelated (#131285) commit 92ab85d26d33e0110723c65347fafa08872bd618 Author: Moritz Mack <[email protected]> Date: Tue Jul 15 13:21:54 2025 +0200 Log NotEntitledExceptions using logger with <package> suffix (#131031) Log NotEntitledExceptions using logger with `<component>.<module>.<package>` suffix (instead of `<component>.<module>`) for more fine-grained mutes, but remaining backwards compatible regarding existing mutes. Relates to ES-12231 commit bfb3aee8ea530c76419c89b2a4caa2b81fbf8833 Author: Tim Rühsen <[email protected]> Date: Tue Jul 15 13:11:00 2025 +0200 [Profiling] Manually downsample to 20k events if required (#131196) * [Profiling] Manually downsample to 20k events if required Since the random sampler aggregation is limited to p values of 0..0.5, add an explicit downsampling for p values of 0.5..1.0. With this, the latency of fetching stacktraces and stackframes can be further reduced. * Fix number of reported samples commit e4ff3f8bae28b78cfd10770fe076309296d8042c Author: Luigi Dell'Aquila <[email protected]> Date: Tue Jul 15 12:59:42 2025 +0200 ES|QL: fix generative tests (#131283) commit fa6e3e75aa9ec2dcc6e4316a1ae5ab1f71141a64 Author: Niels Bauman <[email protected]> Date: Tue Jul 15 07:39:53 2025 -0300 Remove non-test usages of `Metadata.Builder#indices` (#131240) Replaces the usages with non-deprecated versions. commit fd037bd846c5d63b11cce2012d1f161cbdc801db Author: Tommaso Teofili <[email protected]> Date: Tue Jul 15 11:55:57 2025 +0200 retain the scores of portions of an ES|QL query, via a score function (#127551) commit 7139aa1c5228bd649fc1cc3dd36c7df1160ec0de Author: Quentin Pradet <[email protected]> Date: Tue Jul 15 12:01:52 2025 +0400 Add register_operation_count to snapshot.repository_analyze rest-api-spec (#131082) * Add register_operation_count to snapshot.repository_analyze * Remove mention of realistic experiments for consistency Co-authored-by: Yang Wang <[email protected]> --------- Co-authored-by: Yang Wang <[email protected]> commit f67c1f03735e1b58d4f9d77ed7ff40f014308ea5 Author: Ievgen Degtiarenko <[email protected]> Date: Tue Jul 15 09:50:35 2025 +0200 Ensure single segment in test (#131077) commit b3b83cc7e91595dc1a03e401257a7691763300b3 Author: elasticsearchmachine <[email protected]> Date: Tue Jul 15 09:37:34 2025 +0200 Update docs for v9.0.4 release (#131144) commit 6f5579656c6a6ca573669fe733463ed2da35d9e2 Author: David Turner <[email protected]> Date: Tue Jul 15 08:32:49 2025 +0100 Improve lost-increment message in repo analysis (#131200) Today repository analysis may fail with a message like the following: [test-repo] register [test-register-contended-F_NNXHrSSDGveoeyj1skwg] should have value [10] but instead had value [OptionalBytesReference[00 00 00 00 00 00 00 09]] …
1 parent f073aa2 commit 73593d0

File tree

3 files changed

+20
-188
lines changed
  • x-pack/plugin/esql
    • compute/src
    • src/main/java/org/elasticsearch/xpack/esql/expression/function/grouping

3 files changed

+20
-188
lines changed

‎x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/aggregation/blockhash/BlockHash.java‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -143,6 +143,13 @@ public enum OutputFormat {
143143
/**
144144
* Configuration for a BlockHash group spec that is doing text categorization.
145145
*/
146+
public record CategorizeDef(String analyzer, OutputFormat outputFormat, int similarityThreshold) {
147+
public enum OutputFormat {
148+
REGEX,
149+
TOKENS
150+
}
151+
}
152+
146153
public record GroupSpec(
147154
int channel,
148155
ElementType elementType,

‎x-pack/plugin/esql/compute/src/test/java/org/elasticsearch/compute/OperatorTests.java‎

Lines changed: 0 additions & 188 deletions
Original file line numberDiff line numberDiff line change
@@ -152,194 +152,6 @@ public void testQueryOperator() throws IOException {
152152
}
153153
}
154154

155-
public void testGroupingWithOrdinals() throws Exception {
156-
DriverContext driverContext = driverContext();
157-
BlockFactory blockFactory = driverContext.blockFactory();
158-
159-
final String gField = "g";
160-
final int numDocs = 2856; // between(100, 10000);
161-
final Map<BytesRef, Long> expectedCounts = new HashMap<>();
162-
int keyLength = randomIntBetween(1, 10);
163-
try (BaseDirectoryWrapper dir = newDirectory(); RandomIndexWriter writer = new RandomIndexWriter(random(), dir)) {
164-
for (int i = 0; i < numDocs; i++) {
165-
Document doc = new Document();
166-
BytesRef key = new BytesRef(randomByteArrayOfLength(keyLength));
167-
SortedSetDocValuesField docValuesField = new SortedSetDocValuesField(gField, key);
168-
doc.add(docValuesField);
169-
writer.addDocument(doc);
170-
expectedCounts.compute(key, (k, v) -> v == null ? 1 : v + 1);
171-
}
172-
writer.commit();
173-
Map<BytesRef, Long> actualCounts = new HashMap<>();
174-
175-
try (DirectoryReader reader = writer.getReader()) {
176-
List<Operator> operators = new ArrayList<>();
177-
if (randomBoolean()) {
178-
operators.add(new ShuffleDocsOperator(blockFactory));
179-
}
180-
operators.add(new AbstractPageMappingOperator() {
181-
@Override
182-
protected Page process(Page page) {
183-
return page.appendBlock(driverContext.blockFactory().newConstantIntBlockWith(1, page.getPositionCount()));
184-
}
185-
186-
@Override
187-
public String toString() {
188-
return "Add(1)";
189-
}
190-
});
191-
operators.add(
192-
new OrdinalsGroupingOperator(
193-
shardIdx -> new KeywordFieldMapper.KeywordFieldType("g").blockLoader(mockBlContext()),
194-
List.of(new ValuesSourceReaderOperator.ShardContext(reader, () -> SourceLoader.FROM_STORED_SOURCE, 0.2)),
195-
ElementType.BYTES_REF,
196-
0,
197-
gField,
198-
List.of(CountAggregatorFunction.supplier().groupingAggregatorFactory(INITIAL, List.of(1))),
199-
randomPageSize(),
200-
driverContext
201-
)
202-
);
203-
List<BlockHash.GroupSpec> groups = List.of(new BlockHash.GroupSpec(0, ElementType.BYTES_REF));
204-
operators.add(
205-
new HashAggregationOperator(
206-
groups,
207-
List.of(CountAggregatorFunction.supplier().groupingAggregatorFactory(FINAL, List.of(1, 2))),
208-
() -> BlockHash.build(groups, driverContext.blockFactory(), randomPageSize(), false),
209-
driverContext
210-
)
211-
);
212-
Driver driver = TestDriverFactory.create(
213-
driverContext,
214-
luceneOperatorFactory(
215-
reader,
216-
List.of(new LuceneSliceQueue.QueryAndTags(new MatchAllDocsQuery(), List.of())),
217-
LuceneOperator.NO_LIMIT
218-
).get(driverContext),
219-
operators,
220-
new PageConsumerOperator(page -> {
221-
BytesRefBlock keys = page.getBlock(0);
222-
LongBlock counts = page.getBlock(1);
223-
for (int i = 0; i < keys.getPositionCount(); i++) {
224-
BytesRef spare = new BytesRef();
225-
keys.getBytesRef(i, spare);
226-
actualCounts.put(BytesRef.deepCopyOf(spare), counts.getLong(i));
227-
}
228-
page.releaseBlocks();
229-
})
230-
);
231-
OperatorTestCase.runDriver(driver);
232-
assertThat(actualCounts, equalTo(expectedCounts));
233-
assertDriverContext(driverContext);
234-
org.elasticsearch.common.util.MockBigArrays.ensureAllArraysAreReleased();
235-
}
236-
}
237-
assertThat(blockFactory.breaker().getUsed(), equalTo(0L));
238-
}
239-
240-
// TODO: Remove ordinals grouping operator or enable it GroupingAggregatorFunctionTestCase
241-
public void testValuesWithOrdinalGrouping() throws Exception {
242-
DriverContext driverContext = driverContext();
243-
BlockFactory blockFactory = driverContext.blockFactory();
244-
245-
final int numDocs = between(100, 1000);
246-
Map<BytesRef, Set<Long>> expectedValues = new HashMap<>();
247-
try (BaseDirectoryWrapper dir = newDirectory(); RandomIndexWriter writer = new RandomIndexWriter(random(), dir)) {
248-
String VAL_NAME = "val";
249-
String KEY_NAME = "key";
250-
for (int i = 0; i < numDocs; i++) {
251-
Document doc = new Document();
252-
BytesRef key = new BytesRef(Integer.toString(between(1, 100)));
253-
SortedSetDocValuesField keyField = new SortedSetDocValuesField(KEY_NAME, key);
254-
doc.add(keyField);
255-
if (randomBoolean()) {
256-
int numValues = between(0, 2);
257-
for (int v = 0; v < numValues; v++) {
258-
long val = between(1, 1000);
259-
var valuesField = new SortedNumericDocValuesField(VAL_NAME, val);
260-
doc.add(valuesField);
261-
expectedValues.computeIfAbsent(key, k -> new HashSet<>()).add(val);
262-
}
263-
}
264-
writer.addDocument(doc);
265-
}
266-
writer.commit();
267-
try (DirectoryReader reader = writer.getReader()) {
268-
List<Operator> operators = new ArrayList<>();
269-
if (randomBoolean()) {
270-
operators.add(new ShuffleDocsOperator(blockFactory));
271-
}
272-
operators.add(
273-
new ValuesSourceReaderOperator(
274-
blockFactory,
275-
List.of(
276-
new ValuesSourceReaderOperator.FieldInfo(
277-
VAL_NAME,
278-
ElementType.LONG,
279-
unused -> new BlockDocValuesReader.LongsBlockLoader(VAL_NAME)
280-
)
281-
),
282-
List.of(new ValuesSourceReaderOperator.ShardContext(reader, () -> {
283-
throw new UnsupportedOperationException();
284-
}, 0.2)),
285-
0
286-
)
287-
);
288-
operators.add(
289-
new OrdinalsGroupingOperator(
290-
shardIdx -> new KeywordFieldMapper.KeywordFieldType(KEY_NAME).blockLoader(mockBlContext()),
291-
List.of(new ValuesSourceReaderOperator.ShardContext(reader, () -> SourceLoader.FROM_STORED_SOURCE, 0.2)),
292-
ElementType.BYTES_REF,
293-
0,
294-
KEY_NAME,
295-
List.of(new ValuesLongAggregatorFunctionSupplier().groupingAggregatorFactory(INITIAL, List.of(1))),
296-
randomPageSize(),
297-
driverContext
298-
)
299-
);
300-
List<BlockHash.GroupSpec> groups = List.of(new BlockHash.GroupSpec(0, ElementType.BYTES_REF));
301-
operators.add(
302-
new HashAggregationOperator(
303-
groups,
304-
List.of(new ValuesLongAggregatorFunctionSupplier().groupingAggregatorFactory(FINAL, List.of(1))),
305-
() -> BlockHash.build(groups, driverContext.blockFactory(), randomPageSize(), false),
306-
driverContext
307-
)
308-
);
309-
Map<BytesRef, Set<Long>> actualValues = new HashMap<>();
310-
Driver driver = TestDriverFactory.create(
311-
driverContext,
312-
luceneOperatorFactory(
313-
reader,
314-
List.of(new LuceneSliceQueue.QueryAndTags(new MatchAllDocsQuery(), List.of())),
315-
LuceneOperator.NO_LIMIT
316-
).get(driverContext),
317-
operators,
318-
new PageConsumerOperator(page -> {
319-
BytesRefBlock keyBlock = page.getBlock(0);
320-
LongBlock valueBlock = page.getBlock(1);
321-
BytesRef spare = new BytesRef();
322-
for (int p = 0; p < page.getPositionCount(); p++) {
323-
var key = keyBlock.getBytesRef(p, spare);
324-
int valueCount = valueBlock.getValueCount(p);
325-
for (int i = 0; i < valueCount; i++) {
326-
long val = valueBlock.getLong(valueBlock.getFirstValueIndex(p) + i);
327-
boolean added = actualValues.computeIfAbsent(BytesRef.deepCopyOf(key), k -> new HashSet<>()).add(val);
328-
assertTrue(actualValues.toString(), added);
329-
}
330-
}
331-
page.releaseBlocks();
332-
})
333-
);
334-
OperatorTestCase.runDriver(driver);
335-
assertDriverContext(driverContext);
336-
assertThat(actualValues, equalTo(expectedValues));
337-
org.elasticsearch.common.util.MockBigArrays.ensureAllArraysAreReleased();
338-
}
339-
}
340-
assertThat(blockFactory.breaker().getUsed(), equalTo(0L));
341-
}
342-
343155
public void testPushRoundToToQuery() throws IOException {
344156
long firstGroupMax = randomLong();
345157
long secondGroupMax = randomLong();

‎x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/grouping/Bucket.java‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -649,4 +649,17 @@ private static double round(double value, int n) {
649649
return new BigDecimal(value).setScale(n, RoundingMode.HALF_UP).doubleValue();
650650
}
651651
}
652+
653+
@Override
654+
public Expression surrogate(SearchStats searchStats) {
655+
// LocalSubstituteSurrogateExpressions should make sure this doesn't happen
656+
assert searchStats != null : "SearchStats cannot be null";
657+
return maybeSubstituteWithRoundTo(
658+
source(),
659+
field(),
660+
buckets(),
661+
searchStats,
662+
(interval, minValue, maxValue) -> getDateRounding(FoldContext.small(), minValue, maxValue)
663+
);
664+
}
652665
}

0 commit comments

Comments
 (0)