Skip to content

Add "emitEmptyBuckets" parameter to the "Bucket" function. #130008

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 4 commits into from
Closed

Add "emitEmptyBuckets" parameter to the "Bucket" function. #130008

wants to merge 4 commits into from

Conversation

@przemekwitek
Copy link
Contributor

@przemekwitek przemekwitek commented Jun 25, 2025 •
edited
Loading

-- DRAFT --

This PR extends the BUCKET function by adding the ability to emit empty buckets (i.e. buckets with no data).

Currently, the BUCKET function can work in two modes and can have 2 or 4 arguments (see docs).
With this PR, the 5th parameter (emitEmptyBuckets, name to be finalized) is introduced. The emitEmptyBuckets parameter is of boolean type and has the following semantics:

  • false: empty buckets will not be emitted (this is the current behavior)
  • true: empty buckets will be emitted. In this case, from and to parameters are required, even when the desired bucket size is provided explicitly.
    The default value for emitEmptyBuckets is false for BWC.

@przemekwitek przemekwitek force-pushed the empty_bucket branch 2 times, most recently from 5fed6ed to dd15df3 Compare June 27, 2025 08:42
@github-actions
Copy link
Contributor

github-actions bot commented Jul 10, 2025 •
edited
Loading

🔍 Preview links for changed docs

More links …

@przemekwitek
Squashed commit of the following:
73593d0 
commit a692cbd0b01314d6e149179b9f41ddc530f52762
Author: David Turner <[email protected]>
Date:   Mon Jul 21 08:16:16 2025 +0100

    Log failure in `internalSend` (#131418)

    An exception here should be impossible, but we don't assert that, nor do
    we emit a log message to prove it didn't happen in a production
    environment. This commit adds the missing log and assert.

commit 08cbdbce11cc3cd5f53756c8b7242d8df0a52f06
Author: elasticsearchmachine <[email protected]>
Date:   Mon Jul 21 08:44:22 2025 +0200

    Mute org.elasticsearch.test.rest.yaml.RcsCcsCommonYamlTestSuiteIT test {p0=search/40_indices_boost/Indices boost with alias} #131598

commit b3715901c064c1b7a0d85d4c35387a335ed22140
Author: Nick Tindall <[email protected]>
Date:   Mon Jul 21 16:07:59 2025 +1000

    Add shard write-load to cluster info (#131496)

    Relates: ES-12419, ES-12420

commit 47bb53569ab7c55deeb7d6aa27132df9f63cd1d1
Author: Yang Wang <[email protected]>
Date:   Mon Jul 21 14:49:51 2025 +1000

    [Test] Wait for expected master on all nodes (#131399)

    This PR changes the test to simply wait expected master on every node
    instead of selectively waiting on one non-master and one master node.
    The later is problematic since it uses API that is not suitable when the
    cluster is changing master.

    Relates: #127213

commit 6dd4d675f854ee1b13d6b5351615348ae4113f4e
Author: Yang Wang <[email protected]>
Date:   Mon Jul 21 12:25:36 2025 +1000

    Executing shard recovery in project context (#130525)

    See title

commit a07e7e91b7a3fb25ac08b265e1715c6a2dec0eec
Author: elasticsearchmachine <[email protected]>
Date:   Sun Jul 20 17:40:03 2025 +0200

    Mute org.elasticsearch.packaging.test.DockerTests test010Install #131376

commit 3c914e188f56405d2ce0ddcb5cfeab8a55f5ef78
Author: elasticsearchmachine <[email protected]>
Date:   Sun Jul 20 00:28:43 2025 +0200

    Mute org.elasticsearch.xpack.esql.action.CrossClusterQueryWithPartialResultsIT testPartialResults #131481

commit 256a7fed46cd387d2d61c11ed1356998ecd15ea7
Author: Larisa Motova <[email protected]>
Date:   Fri Jul 18 19:08:52 2025 -1000

    Skip downsample IT test with TS command when command not present (#131556)

    Fixes #131500

commit 74cd00654cfef2ffa12fdd1bc82f0bd38a2e4622
Author: Jeremy Dahlgren <[email protected]>
Date:   Fri Jul 18 19:15:37 2025 -0400

    Remove redundant method for getting the remote cluster names (#130495)

    This change removes RemoteClusterService.getRemoteClusterNames() since
    getRegisteredRemoteClusterNames() provides the same functionality.
    The comment in getRegisteredRemoteClusterNames() was removed since
    it is no longer accurate after the change in PR #47891.

commit 668acccabae4149f43c6b365dc79014f40ecc5fe
Author: elasticsearchmachine <[email protected]>
Date:   Fri Jul 18 23:44:27 2025 +0200

    Mute org.elasticsearch.action.admin.cluster.node.tasks.CancellableTasksIT testRemoveBanParentsOnDisconnect #131562

commit ac398e3b61f6a9d66a01f62d8887c44efd7721a2
Author: Stef Nestor <[email protected]>
Date:   Fri Jul 18 15:30:41 2025 -0600

    New Slow log troubleshooting video (#131557)

commit dc96c362f517ad0e9676285001ddcd7e03e36d5a
Author: Ruben van Staden <[email protected]>
Date:   Fri Jul 18 17:16:58 2025 -0400

    otel-data: enable failure store for OTEL datastreams (#131395)

commit beb18a87c3279f7d9b9426cbd50df20f65c12d8d
Author: Jan-Kazlouski-elastic <[email protected]>
Date:   Fri Jul 18 23:26:20 2025 +0300

    Add Llama support to Inference Plugin (#130092)

    * Refactor Hugging Face service settings and completion request methods for consistency

    * Add Llama model support for embeddings and chat completions

    * Refactor Llama request classes to improve secret settings handling

    * Refactor DeltaParser in LlamaStreamingProcessor to improve argument handling

    * Enhance Llama streaming processing by adding support for nullable object arrays

    * [CI] Auto commit changes from spotless

    * Fix error messages in LlamaActionCreator

    * [CI] Auto commit changes from spotless

    * Add detailed Javadoc comments to Llama classes for improved documentation

    * Enhance LlamaChatCompletionResponseHandler to support mid-stream error handling and improve error response parsing

    * Add Javadoc comments to Llama classes for improved documentation and clarity

    * Fix checkstyle

    * Update LlamaEmbeddingsRequest to use mediaTypeWithoutParameters for content type header

    * Add unit tests for LlamaActionCreator and related models

    * Add unit tests for LlamaChatCompletionServiceSettings to validate configuration parsing and serialization

    * Add unit tests for LlamaEmbeddingsServiceSettings to validate configuration parsing and serialization

    * Add unit tests for LlamaEmbeddingsServiceSettings to validate various configuration scenarios

    * Add unit tests for LlamaChatCompletionResponseHandler to validate error response handling

    * Refactor Llama embedding and chat completion tests for consistency and clarity

    * Add unit tests for LlamaChatCompletionRequestEntity to validate message serialization

    * Add unit tests for LlamaEmbeddingsRequest to validate request creation and truncation behavior

    * Add unit tests for LlamaEmbeddingsRequestEntity to validate XContent serialization

    * Add unit tests for LlamaErrorResponse to validate error handling from HTTP responses

    * Add unit tests for LlamaChatCompletionServiceSettings to validate configuration parsing and serialization

    * Add tests for LlamaService request configuration validation and error handling

    * Fix error message formatting in LlamaServiceTests for better localization support

    * Refactor Llama model classes to implement accept method for action visitors

    * Hide Llama service from configuration API to enhance security and reduce exposure

    * Refactor Llama model classes to remove modelId and update embedding request handling

    * Refactor Llama request classes to use pattern matching for secret settings

    * Update embeddings handler to use HuggingFace response entity

    * Refactor Mistral model classes to remove modelId and update rate limit hashing

    * Refactor Mistral action classes to remove taskSettings parameter and streamline action creation

    * Refactor Llama and Mistral models to remove taskSettings parameter and simplify model instantiation

    * Refactor Llama service tests to use Model instead of CustomModel and update similarity measure to DOT_PRODUCT

    * Remove unused tests and imports from LlamaServiceTests

    * Add chunking settings support to Llama embeddings model tests

    * Add changelog

    * Add support for version checks in Llama settings and define new transport version

    * Refactor Llama model assertions and remove unused version support methods

    * Refactor Llama service constructors to include ClusterService and improve error message handling

    ---------

    Co-authored-by: elasticsearchmachine <[email protected]>

commit f664cf5926345958978205907cc7d429f2403345
Author: Alexey Ivanov <[email protected]>
Date:   Fri Jul 18 19:04:17 2025 +0100

    Add proper Diff support for ProjectStateRegistry (#130917)

    This change introduces proper diff support for ProjectStateRegistry instead of CompleteDiff and Entry to ProjectStateRegistry, to allow addition of more project state fields in the future

commit 1dc6bc0ca385baecf77ab6ba46822417e66b5791
Author: Richard Dennehy <[email protected]>
Date:   Fri Jul 18 18:55:59 2025 +0100

    Disable queryable built-in feature for smoke-test-plugins-ssl (#131523) (#131543)

commit be2f024412a13995daeb8dc24734613513cf787e
Author: Stanislav Malyshev <[email protected]>
Date:   Fri Jul 18 11:37:38 2025 -0600

    Revert "Put shards failure under a cap flag (#131371)" (#131542)

    This reverts commit 04119405d141bcb390509f766ebbb7660b0ffe72.

commit feafb3a2ef4ae9204b99becb797502b81cfa15e4
Author: Pat Whelan <[email protected]>
Date:   Fri Jul 18 12:35:35 2025 -0400

    [ML] Track inference deployments (#131442)

    Record duration and errors when Inference Endpoints deploy Trained
    Models. The new metric is `es.inference.trained_model.deployment.time`.

    Refactored `InferenceStats` into server so it can be used in
    `InferenceServiceExtension` and passed to InferenceServices rather than
    remain at the Transport layer.

commit a6d6d3ec8a3092bfaf4b1d0bfb34df66690a1716
Author: Ankita Kumar <[email protected]>
Date:   Fri Jul 18 12:34:38 2025 -0400

    Resharding: Route requests to source shard while target is pre-handoff (#131306)

    This PR for the Autosharding project, routes DocWriteRequests (index/delete/update requests) that are meant for the target shards, to source shards, until the target shard is ready.

commit 330debad664b087e4f419b77af8f4e1b61b0b443
Author: elasticsearchmachine <[email protected]>
Date:   Fri Jul 18 18:23:30 2025 +0200

    Mute org.elasticsearch.xpack.esql.qa.multi_node.GenerativeIT test #131508

commit 0f00f69b137bf58f14c1c50d83cfa4d5cab8ed51
Author: Nhat Nguyen <[email protected]>
Date:   Fri Jul 18 08:40:30 2025 -0700

    Optimized path for single segment in TS source (#131502)

    When only one segment remains for scanning, we can avoid moving the only
    segment iterator between the main queue and the sub-queue. This
    optimized path should also work with a forced-merge shard.

commit 90699d3cc31f8a943b60b130c104a7bac920652c
Author: Kathleen DeRusso <[email protected]>
Date:   Fri Jul 18 11:27:34 2025 -0400

    Fix semantic highlighting bug on flat quantized fields (#131525)

    * Fix semantic highlighting bug on flat quantized fields

    * Update docs/changelog/131525.yaml

commit 929f65b94c445bc1e3e64443ee20e61e76f556e5
Author: Pat Whelan <[email protected]>
Date:   Fri Jul 18 10:51:33 2025 -0400

    [Persistent Tasks] Assign based on ProjectId (#130391)

    Pass the ProjectId from PersistentTaskClusterService through to all
    PersistentTasksExecutors when creating node assignments.

    These PersistentTasksExecutors require the ProjectId during node
    assignment:
    - OpenJobPersistentTasksExecutor
    - SnapshotUpgradeTaskExecutor
    - StartDatafeedPersistentTasksExecutor
    - TransformPersistentTasksExecutor

commit a786c93a3b1882e51bd9b4dec21eaaff4eac14c7
Author: Michail Romaios <[email protected]>
Date:   Fri Jul 18 16:42:08 2025 +0200

    refactor: enhance semantic_text inference error msg (#131519)

commit efa71d7206a8291ee362f56a0b65e3e75f5758a8
Author: elasticsearchmachine <[email protected]>
Date:   Fri Jul 18 16:33:09 2025 +0200

    Mute org.elasticsearch.xpack.downsample.DownsampleIT testAggMetricInEsqlTSAfterDownsampling #131500

commit e67e50b3f19d2326acf5e7b17211849d28aad210
Author: Julian Kiryakov <[email protected]>
Date:   Fri Jul 18 10:20:07 2025 -0400

    [DOCS][ESQL] Fix release version in Docs for RLIKE LIST (#131465)

    RLIKE LIST did not manage to make it into 9.1.
    In this PR, we modify the documentation to make it clear that it will be available in 9.2, not 9.1

commit 6543e50cda593db90b8c215b116f2532bef597ff
Author: Quentin Pradet <[email protected]>
Date:   Fri Jul 18 17:56:21 2025 +0400

    Fix msearch rest-api-spec (#130627)

    * Fix msearch rest-api-spec

    * Add YAML tests for added parameters

commit f5c6b35dab29b0dafa17101e236771d66fa4daf6
Author: elasticsearchmachine <[email protected]>
Date:   Fri Jul 18 15:33:41 2025 +0200

    Mute org.elasticsearch.xpack.search.CrossClusterAsyncSearchIT testCancellationViaTimeoutWithAllowPartialResultsSetToFalse #131248

commit 06e39c0377e3769eae91487b6c2d22978f554b43
Author: Alexander Spies <[email protected]>
Date:   Fri Jul 18 15:30:57 2025 +0200

    ESQL: Disallow remote enrich after lu join (#131426)

    Fix https://github.com/elastic/elasticsearch/issues/129372

    Due to how remote ENRICH is
    [planned](https://github.com/elastic/elasticsearch/blob/32e50d0d94e27ee559d24bf9d5463ba6e64d1788/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/planner/mapper/Mapper.java#L93),
    it interacts in special ways with pipeline breakers, in particular LIMIT
    and TopN; when these are encountered upstream from a remote ENRICH,
    these nodes are copied and executed a second time after the remote
    ENRICH.

    We'd like to allow remote ENRICH after LOOKUP JOIN, but that forces the
    lookup to be remote as well; this has its own interactions with pipeline
    breakers: in particular, LIMITs and TopNs cannot just be duplicated
    after LOOKUP JOIN, as LOOKUP JOIN may add new rows.

    For now, let's just forbid any usage of remote ENRICH after LOOKUP
    JOINs; remote ENRICH is mostly relevant for CCS, and LOOKUP JOIN doesn't
    support that in 9.1/8.19, anyway.

    There is separate work that enables remote LOOKUP JOINs on remote
    clusters and adds the correct validations; we can later build support
    for remote ENRICH + LOOKUP JOIN on top of that. (C.f. my comment
    [here](https://github.com/elastic/elasticsearch/issues/129372#issuecomment-3083024230)
    and my draft https://github.com/elastic/elasticsearch/pull/131286 for
    enabling this.)

commit 732bab022779e746a4817e6156975ac62600f059
Author: Kostas Krikellas <[email protected]>
Date:   Fri Jul 18 16:16:51 2025 +0300

    Skip tests with TS command for non-snapshot builds (#131518)

    The `TS` source is guarded by a feature flag.

    Fixes #131500

commit 27a09d8529ee75cbac015111d05766ba7e3f5250
Author: Lorenzo Dematté <[email protected]>
Date:   Fri Jul 18 15:04:07 2025 +0200

    Upgrade apm-agent to 1.55.0 (#131510)

commit d70093b3ad4cbf5a4b671cb1c7f5aa496ff05238
Author: Tommaso Teofili <[email protected]>
Date:   Fri Jul 18 14:41:37 2025 +0200

    ScoreTests capability check (#131516)

commit 439b8e68bb7a33b22e1f6d7966247a0015ae893d
Author: Nik Everett <[email protected]>
Date:   Fri Jul 18 08:37:14 2025 -0400

    ESQL: Split large pages on load sometimes (#131053)

    This adds support for splitting `Page`s of large values when loading
    from single segment, non-descending hits. This is hottest code path as
    it's how we load data for aggregation. So! We had to make very very very
    sure this doesn't slow down the fast path of loading doc values.

    Caveat - this only defends against loading large values via the
    row-by-row load mechanism that we use for stored fields and _source.
    That covers the most common kinds of large values - mostly `text` and
    geo fields. If we need to split further on docs values, we'll have to
    invent something for them specifically. For now, just row-by-row.

    This works by flipping the order in which we load row-by-row and
    column-at-a-time values. Previously we loaded all column-at-a-time
    values first because that was simpler. Then we loaded all of the
    row-by-row values. Now we save the column-at-a-time values and instead
    load row-by-row until the `Page`'s estimated size is larger than a "jumbo"
    size which defaults to a megabyte.

    Once we load enough rows that we estimate the page is "jumbo", we then
    stop loading rows. The Page will look like this:

    ```
    | txt1 | int | txt2 | long | double |
    |------|-----|------|------|--------|
    | XXXX |     | XXXX |      |        |
    | XXXX |     | XXXX |      |        |
    | XXXX |     | XXXX |      |        |
    | XXXX |     | XXXX |      |        |
    | XXXX |     | XXXX |      |        |
    | XXXX |     | XXXX |      |        | <-- after loading this row
    |      |     |      |      |        |     we crossed to "jumbo" size
    |      |     |      |      |        |
    |      |     |      |      |        |
    |      |     |      |      |        | <-- these rows are entirely empty
    |      |     |      |      |        |
    |      |     |      |      |        |
    ```

    Then we chop the page to the last row:
    ```
    | txt1 | int | txt2 | long | double |
    |------|-----|------|------|--------|
    | XXXX |     | XXXX |      |        |
    | XXXX |     | XXXX |      |        |
    | XXXX |     | XXXX |      |        |
    | XXXX |     | XXXX |      |        |
    | XXXX |     | XXXX |      |        |
    | XXXX |     | XXXX |      |        |
    ```

    Then fill in the column-at-a-time columns:
    ```
    | txt1 | int | txt2 | long | double |
    |------|-----|------|------|--------|
    | XXXX |   1 | XXXX |   11 |    1.0 |
    | XXXX |   2 | XXXX |   22 |   -2.0 |
    | XXXX |   3 | XXXX |   33 |    1e9 |
    | XXXX |   4 | XXXX |   44 |    913 |
    | XXXX |   5 | XXXX |   55 | 0.1234 |
    | XXXX |   6 | XXXX |   66 | 3.1415 |
    ```

    And then we return *that* `Page`. On the next `Driver` iteration we
    start from where we left off.

commit 40585a2199e454bb3028e09c8a6bf0162c3079b6
Author: Julian Kiryakov <[email protected]>
Date:   Fri Jul 18 08:33:58 2025 -0400

    Add checks that optimizers do not modify the layout (#130855)

    Add verification that the optimizers do not modify the number of attributes and the attribute datatype.
    We add special handling for Lookup Join, by checking EsQueryExec esQueryExec && esQueryExec.indexMode() == LOOKUP and another special handling for ProjectAwayColumns.ALL_FIELDS_PROJECTED

    Closes #125576

commit c9d5076224a93c8a012083a258358adbbb446b02
Author: Luigi Dell'Aquila <[email protected]>
Date:   Fri Jul 18 13:26:12 2025 +0200

    ES|QL: fix generative test (#131515)

commit 8682c181c6585a0e70fa0670a8d7729cc8bd12aa
Author: Iván Cea Fontenla <[email protected]>
Date:   Fri Jul 18 11:49:04 2025 +0200

    [ESQL] Fix TopN grouping aggregates bug with non-qualifying intermediate states (#129633)

    Continuation of https://github.com/elastic/elasticsearch/pull/127148

    When datanodes send the STATS intermediate states to the coordinator, it aggregates them.
    Now, however, the TopN groups sent by a datanode may not be acceptable in the coordinator (Because it has better values already), so it will discard such values.

    However, the engine wasn't handling intermediate groups with nulls (TopNBlockHash uses nulls to discard unused groups).
    See https://github.com/elastic/elasticsearch/blob/main/x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/aggregation/GroupingAggregator.java#L47

    _This code isn't connected with the query yet, so there's no bug in production_

commit 9f22533c17d0eaf53402193243772997e3921603
Author: Joshua Adams <[email protected]>
Date:   Fri Jul 18 10:38:17 2025 +0100

    [TEST] Remove YAML Tests (#131434)

    Removes YAML tests for the `/_cluster/allocation/explain` API. The tests
     passed in alternate values for the APIs. An example is passing "true"
     for fields expecting a boolean value. While this is explicitly
     supported by the API, this is not the correct place to be testing this
     behaviour, and resulted in the API specification failing
     validation.

    Relates #127028

commit fb6e0c0c571b9d5db0938a2e154587a8479065f9
Author: elasticsearchmachine <[email protected]>
Date:   Fri Jul 18 11:29:09 2025 +0200

    Mute org.elasticsearch.xpack.downsample.DownsampleWithBasicRestIT test {p0=downsample-with-security/10_basic/Downsample index} #131513

commit db369bb7a64116fe69f2b37e4ce8da1f2f7d6a62
Author: Moritz Mack <[email protected]>
Date:   Fri Jul 18 11:15:13 2025 +0200

    Disable entitlements for DirectIOIT, the suite requires delegation to work (#131505)

    Disable entitlements for DirectIOIT, the suite requires delegation to
    work.

    On main the suite is skipped (direct IO is disabled by default), but
    this blocks backports.

commit 3c11ab29a356ea988829950366733aff541d1275
Author: elasticsearchmachine <[email protected]>
Date:   Fri Jul 18 10:22:00 2025 +0200

    Mute org.elasticsearch.xpack.esql.qa.single_node.GenerativeIT test #131508

commit d637927e3d4db6d2f5f7bbf3c70ea36493c861b9
Author: Alexander Spies <[email protected]>
Date:   Fri Jul 18 09:21:09 2025 +0200

    ESQL: Unmute some generative tests (#131446)

    The corresponding issue #116781 has already been fixed.

commit a29d8ea0f4396e9881cb4298a29a87390e5f5ccd
Author: elasticsearchmachine <[email protected]>
Date:   Fri Jul 18 09:14:35 2025 +0200

    Mute org.elasticsearch.xpack.esql.action.EsqlActionBreakerIT testFromEvalStats #131503

commit 9edf9f65174dc808c99fe49e3c47660d0d814c14
Author: Nhat Nguyen <[email protected]>
Date:   Thu Jul 17 23:51:10 2025 -0700

    Fix NPE in TimeSeriesExtractFieldOperator (#131497)

    We miss checking whether a field exists when populating the dimension
    attributes. This issue occurs when a field exists in some, but not all
    target indices.

commit 654d1f4a7e466502bc8f83037e4b8d53d18f29d7
Author: Yang Wang <[email protected]>
Date:   Fri Jul 18 12:00:51 2025 +1000

    Add missing query params to index recovery API spec (#131490)

    This PR adds the missing ignore_unavailable, allow_no_indices and
    expand_wildcards query parameters.

commit d365412c12c45b863188ecc692e76c719d04f441
Author: elasticsearchmachine <[email protected]>
Date:   Fri Jul 18 01:25:27 2025 +0200

    Mute org.elasticsearch.packaging.test.DockerTests test050BasicApiTests #120911

commit a528ebb0cbd22cf1411dbe8f63a23e5db3798dc7
Author: elasticsearchmachine <[email protected]>
Date:   Fri Jul 18 00:39:37 2025 +0200

    Mute org.elasticsearch.test.rest.yaml.RcsCcsCommonYamlTestSuiteIT test {p0=vector-tile/20_aggregations/stats agg} #131484

commit 7e7093a24f7a11f9f6eac8b4994719f8b3c85c89
Author: elasticsearchmachine <[email protected]>
Date:   Fri Jul 18 00:16:53 2025 +0200

    Mute org.elasticsearch.xpack.esql.heap_attack.HeapAttackIT testLookupExplosionNoFetch #128720

commit decce5659e02485e4aed5c7fefdbfb6306eb9fd0
Author: Nik Everett <[email protected]>
Date:   Thu Jul 17 17:24:50 2025 -0400

    ESQL: Unmuted fixed tests (#131476)

    Fixed by #131370 or #130963.

    Closes #130505 Closes #130504 Closes #130501 Closes #131024

commit 560ffb973d71a1a9cd7b16942d2b6d2f87c86037
Author: Samiul Monir <[email protected]>
Date:   Thu Jul 17 16:57:15 2025 -0400

    [Inference Timeout] Supply inference context to all third party services (#131251)

    * Refactoring inference services to accept context

    * fix linting issues

    * adding mock cluster service to fix IT test

    * refactoring to remove duplication in constructors

    * remove unnecessary blank line

    * refactor to have uniform constructor call

    * refactor to have uniform constructor call for sagemaker

    * fix linting issues

    * fix failed unit tests

    ---------

    Co-authored-by: Elastic Machine <[email protected]>

commit 6ed50e1bae883b2f5628f96447b2322f364ce1b3
Author: Nik Everett <[email protected]>
Date:   Thu Jul 17 15:40:17 2025 -0400

    Explain `ignore_above` better (#129284)

    This concept is complicated.

    Closes #128991

    Co-authored-by: Larisa Motova <[email protected]>
    Co-authored-by: Liam Thompson <[email protected]>

commit 56477d81a9ec03ac9fe304c166519fccabaab305
Author: Liam Thompson <[email protected]>
Date:   Thu Jul 17 21:21:53 2025 +0200

    Split retrievers docs and redirect anchors  (#131385)

commit cf5d40fa1fc389bfb9e6158064389e20e62a7004
Author: Benjamin Trent <[email protected]>
Date:   Thu Jul 17 14:55:51 2025 -0400

    Fix diskbbq flush logic (#131470)

    I accidentally broke recall on flush by allowing vectors to be double
    quantized. Additionally, we shouldn't use the first vector as a
    centroid, this can harm recall significantly when there is just one
    centroid.

    recall before this change:

    ```
    index_name                             index_type  num_docs  index_time(ms)  force_merge_time(ms)  num_segments
    -------------------------------------  ----------  --------  --------------  --------------------  ------------
    corpus-dbpedia-entity-E5-small-0.fvec         ivf   1000000           25820                     0            14
    corpus-dbpedia-entity-E5-small-0.fvec         ivf   1000000               0                 41693             0

    index_name                             index_type  n_probe  latency(ms)  net_cpu_time(ms)  avg_cpu_count     QPS  recall    visited  filter_selectivity
    -------------------------------------  ----------  -------  -----------  ----------------  -------------  ------  ------  ---------  ------------------
    corpus-dbpedia-entity-E5-small-0.fvec         ivf       50        13.05              0.00           0.00   76.61    0.63  285267.44                1.00
    corpus-dbpedia-entity-E5-small-0.fvec         ivf      150        31.92              0.00           0.00   31.33    0.68  629033.22                1.00
    corpus-dbpedia-entity-E5-small-0.fvec         ivf      200        34.79              0.00           0.00   28.74    0.69  679699.13                1.00
    corpus-dbpedia-entity-E5-small-0.fvec         ivf      500        39.40              0.00           0.00   25.38    0.71  794375.05                1.00
    corpus-dbpedia-entity-E5-small-0.fvec         ivf     1000        45.99              0.00           0.00   21.74    0.72  940493.52                1.00
    corpus-dbpedia-entity-E5-small-0.fvec         ivf       50         1.52              0.00           0.00  655.74    0.74   24201.82                1.00
    corpus-dbpedia-entity-E5-small-0.fvec         ivf      150         2.94              0.00           0.00  340.43    0.85   67943.31                1.00
    corpus-dbpedia-entity-E5-small-0.fvec         ivf      200         3.81              0.00           0.00  262.81    0.87   89575.99                1.00
    corpus-dbpedia-entity-E5-small-0.fvec         ivf      500         7.67              0.00           0.00  130.38    0.93  213586.44                1.00
    corpus-dbpedia-entity-E5-small-0.fvec         ivf     1000        14.85              0.00           0.00   67.33    0.96  402628.11                1.00
    ```

    With this fix:

    ```
    index_name                             index_type  num_docs  index_time(ms)  force_merge_time(ms)  num_segments
    -------------------------------------  ----------  --------  --------------  --------------------  ------------
    corpus-dbpedia-entity-E5-small-0.fvec         ivf   1000000           25304                     0            15
    corpus-dbpedia-entity-E5-small-0.fvec         ivf   1000000               0                 42110             0

    index_name                             index_type  n_probe  latency(ms)  net_cpu_time(ms)  avg_cpu_count     QPS  recall    visited  filter_selectivity
    -------------------------------------  ----------  -------  -----------  ----------------  -------------  ------  ------  ---------  ------------------
    corpus-dbpedia-entity-E5-small-0.fvec         ivf       50        12.63              0.00           0.00   79.18    0.89  285527.22                1.00
    corpus-dbpedia-entity-E5-small-0.fvec         ivf      150        32.49              0.00           0.00   30.77    0.94  619783.37                1.00
    corpus-dbpedia-entity-E5-small-0.fvec         ivf      200        35.46              0.00           0.00   28.20    0.95  667903.47                1.00
    corpus-dbpedia-entity-E5-small-0.fvec         ivf      500        40.38              0.00           0.00   24.76    0.97  781959.74                1.00
    corpus-dbpedia-entity-E5-small-0.fvec         ivf     1000        48.62              0.00           0.00   20.57    0.98  931017.40                1.00
    corpus-dbpedia-entity-E5-small-0.fvec         ivf       50         1.55              0.00           0.00  643.09    0.74   23595.57                1.00
    corpus-dbpedia-entity-E5-small-0.fvec         ivf      150         2.98              0.00           0.00  335.29    0.85   66299.43                1.00
    corpus-dbpedia-entity-E5-small-0.fvec         ivf      200         3.81              0.00           0.00  262.64    0.87   87416.15                1.00
    corpus-dbpedia-entity-E5-small-0.fvec         ivf      500         8.80              0.00           0.00  113.64    0.93  209061.37                1.00
    corpus-dbpedia-entity-E5-small-0.fvec         ivf     1000        16.18              0.00           0.00   61.81    0.96  394906.29                1.00
    ```

commit 9fa06245a5187713f8175493f6548c9733bc2cfa
Author: Larisa Motova <[email protected]>
Date:   Thu Jul 17 08:31:25 2025 -1000

    [ES|QL] Support implicit casting of aggregate_metric_double (#129108)

    This commit adds support for implicit casting of aggregate_metric_double
    when present with other numerics for a limited set of aggregation
    functions:
    - Max / MaxOverTime
    - Min / MinOverTime
    - Sum / SumOverTime
    - Count / CountOverTime
    - Avg / AvgOverTime

    Attempting to use fields mapped to aggregate_metric_double in one index
    but some other numeric in another index in any other context will still
    require explicit casting with ToAggregateMetricDouble

commit 221998d9fb68952165e0dad75719a05edc1be483
Author: eyalkoren <[email protected]>
Date:   Thu Jul 17 21:23:27 2025 +0300

    Revert "Support Fields API in conditional ingest processors (#121914)" (#131452)

    This reverts commit a6f0f6fb4d1ddc38d6aba871ad8d196e0cd16406.

commit 78f613362364998b96bde769c76f8e8b0df6ba47
Author: Nhat Nguyen <[email protected]>
Date:   Thu Jul 17 11:10:49 2025 -0700

    Fix data race in InlineStats (#131370)

    Although blocks/vectors are immutable and safe to share between threads,
    their references are currently not thread-safe, which can lead to data
    races. Previously, blocks/vectors were exclusively owned by a single
    thread, but this is no longer always the case with InlineJoin. We should
    consider switching to AbstractRefCounted, which is thread-safe, and
    benchmark it with many-fields use cases to ensure there is no
    performance regression. As a temporary solution, this change clones the
    values block in InlineJoin until thread-safe blocks/vectors are
    available.

commit 280793df0960ffe4ab1ccd5681458d948b27773b
Author: Ruben van Staden <[email protected]>
Date:   Thu Jul 17 13:35:28 2025 -0400

    apm-data: enable failure store for newly created APM datastreams (#131296)

commit 7a01565a59e210e99152d247b3a5bf2e915d8f37
Author: Jordan Powers <[email protected]>
Date:   Thu Jul 17 09:42:19 2025 -0700

    Fix match_only_text keyword multi-field bug (#131383)

    In #131314 we fixed match_only_text fields with ignore_above keyword
    multi-fields in the case that the keyword multi-field is stored. However,
    the issue is still present if the keyword field is not stored, but instead
    has doc values.

    This patch fixes that case.

commit 6da8f9239246d74b53e24debd31f886fda059292
Author: Pat Whelan <[email protected]>
Date:   Thu Jul 17 10:57:41 2025 -0400

    [ML] Block trained model updates from inference (#130940)

    When the Trained Model has been deployed through the Inference Endpoint
    API, it can only be updated using the Inference Endpoint API.

    When the Trained Model has been deployed and then attached to an
    Inference Endpoint, it can only be updated using the Trained Model API.

    Fix #129999

    Co-authored-by: elasticsearchmachine <[email protected]>
    Co-authored-by: David Kyle <[email protected]>

commit e2bb47c3bb5be3ed77ac5026db40d9297b458f36
Author: Samiul Monir <[email protected]>
Date:   Thu Jul 17 10:51:29 2025 -0400

    Fix Semantic Query Rewrite Interception Drops Boosts (#129282)

    * fix boosting for knn

    * Fixing for match query

    * fixing for match subquery

    * fix for sparse vector query boost

    * fix linting issues

    * Update docs/changelog/129282.yaml

    * update changelog

    * Copy constructor with match query

    * util function to create sparseVectorBuilder for sparse query

    * util function for knn query to support boost

    * adding unit tests for all intercepted query terms

    * Adding yaml test for match,sparse, and knn

    * Adding queryname support for nested query

    * fix code styles

    * Fix failed yaml tests

    * Update docs/changelog/129282.yaml

    * update yaml tests to expand test scenarios

    * Updating knn to copy constructor

    * adding yaml tests for multiple indices

    * refactoring match query to adjust boost and queryname and move to copy constructor

    * refactoring sparse query to adjust boost and queryname and move to copy constructor

    * [CI] Auto commit changes from spotless

    * Refactor sparse vector to adjust boost and queryname in the top level

    * Refactor knn vector to adjust boost and queryname in the top level

    * fix knn combined query

    * fix unit tests

    * fix lint issues

    * remove unused code

    * Update inference feature name

    * Remove double boosting issue from match

    * Fix double boosting in match test yaml file

    * move to bool level for match semantic boost

    * fix double boosting for sparse vector

    * fix double boosting for sparse vector in yaml test

    * fix knn combined query

    * fix knn combined query

    * fix sparse combined query

    * fix knn yaml test for combined query

    * refactoring unit tests

    * linting

    * fix match query unit test

    * adding copy constructor for match query

    * refactor copy match builder to intercepter

    * [CI] Auto commit changes from spotless

    * fix unit tests

    * update yaml tests

    * fix match yaml test

    * fix yaml tests with 4 digits error margin

    * unit tests are now more randomized

    ---------

    Co-authored-by: Elastic Machine <[email protected]>
    Co-authored-by: elasticsearchmachine <[email protected]>

commit 628828fd1d16c5a8d2845e0b1b859dbd07c5a3ab
Author: Ignacio Vera <[email protected]>
Date:   Thu Jul 17 15:50:05 2025 +0100

    [DiskBBQ] Write the raw centroid on the posting list file instead of the centroids file (#131421)

commit f739673b48454a879d39a072a7f01bd95dfa497c
Author: Ben Chaplin <[email protected]>
Date:   Thu Jul 17 10:48:19 2025 -0400

    Fix bug in point in time response (#131391)

    Correct response which had swapped "skipped" and "failed" shard counts.

commit df2fc49a1a036d8ed30389354182a94371638cfc
Author: Moritz Mack <[email protected]>
Date:   Thu Jul 17 16:46:00 2025 +0200

    Support entitlements in internal cluster tests (#130710)

    * To prevent an implicit grant-all if storing node homes inside the Java temp dir, the temporary folder of ESTestCase is configured separately from the Java temp dir in internalClusterTests (by means of the system property tempDir, see TestRuleTemporaryFilesCleanup)

    * Move ReloadingDatabasesWhilePerformingGeoLookupsIT from internalClusterTest to test, file permissions in internalClusterTest are stricter on the lucene tempDir

commit 3b1523a3a8e9cf44fb38448fe876131cf7d5bf36
Author: Jonathan Buttner <[email protected]>
Date:   Thu Jul 17 10:27:38 2025 -0400

    [ML] Refactoring streaming error handling (#131316)

    * Refactoring google gemini streaming error handling

    * Updating comments

commit fd971e84c223295ec26dc34e7584cd74bdf663c0
Author: Yang Wang <[email protected]>
Date:   Fri Jul 18 00:15:25 2025 +1000

    Update regions_by_endpoint for AWS sdk upgrade. (#131400)

    Also add test to ensure the file has at least one entry for each region
    so that it is easy to spot missing regions in future upgrades.

    Relates: #131050 Resolves: #131392

commit df985e60504735a0e401415c1094c3231a01b326
Author: Joshua Adams <[email protected]>
Date:   Thu Jul 17 14:30:50 2025 +0100

    Add `includeDiskInfo` to `toString()` (#131358)

    Adds the `includeDiskInfo` parameter to the `cluster/allocation/explain`
     `toString()` method, and adds tests.

commit d06b0c8c17429ff5eee853e193edc27b70cc28ad
Author: Evgenii-Kazannik <[email protected]>
Date:   Thu Jul 17 15:24:02 2025 +0200

    Add Azure AI Rerank support (#129848)

    * Add Azure AI Rerank support

    * address comments

    * address comments

    * refactor azure ai studio service

    * update rerank task settings test

    * add provider for rerank

commit f9eee6c216ca7574c66a03b5dfe9ebc18cda051d
Author: Ignacio Vera <[email protected]>
Date:   Thu Jul 17 13:14:46 2025 +0100

    [DiskBBQ] Use PackedLongValues to hold offsets on heap while writing (#131411)

commit 8fb9fb5cee69e9420572f8b87873be3e52d982dd
Author: Quentin Pradet <[email protected]>
Date:   Thu Jul 17 15:51:45 2025 +0400

    Remove 'index' from snapshot clear_cache query params (#131067)

    It's already part of the path parts, it's not useful to duplicate it in query
    parameters.

commit 32e50d0d94e27ee559d24bf9d5463ba6e64d1788
Author: Lorenzo Dematté <[email protected]>
Date:   Thu Jul 17 11:37:09 2025 +0200

    Migrate x-pack-autoscaling REST tests (#131365)

    This PR migrates legacy rest tests in the x-pack autoscaling module

commit ec7f77becb29ad049327cc05770454e28029bb2c
Author: Jan Kuipers <[email protected]>
Date:   Thu Jul 17 10:24:30 2025 +0200

    ES|QL categorize options (#131104)

    * ES|QL categorize options

    * refactor options

    * fix serialization

    * polish

    * add verfications

    * better test coverage + polish code

    * better test coverage + polish code

commit ac0c50820a540f64ad835bd52dff004dd4fc60af
Author: kanoshiou <[email protected]>
Date:   Thu Jul 17 16:23:04 2025 +0800

    ESQL: Fix inconsistent column order in MV_EXPAND (#129745)

    The new attribute generated by MV_EXPAND should remain in the original position. The projection added by ProjectAwayColumns does not respect the original order of attributes.

    Make ProjectAwayColumns respect the order of attributes to fix this.

commit adcb2a57cecd5acbfb46282f27cab589aaac1c94
Author: elasticsearchmachine <[email protected]>
Date:   Thu Jul 17 09:43:49 2025 +0200

    Mute org.elasticsearch.packaging.test.DockerTests test072RunEsAsDifferentUserAndGroup #131412

commit efd311016358e1f738aaf42627b1beb409190f27
Author: Nhat Nguyen <[email protected]>
Date:   Wed Jul 16 22:36:47 2025 -0700

    Remove ordinal grouping path in aggregations (#131307)

    With the ordinal grouping operator removed in #131133, this PR removes
    the corresponding code path in the grouping aggregator function, as it
    is no longer needed.

    Relates #131133

commit f135998b118d4e58922300cf942506ae25cd7ab1
Author: Lisa Cawley <[email protected]>
Date:   Wed Jul 16 17:00:21 2025 -0700

    [DOCS] Augment self-managed connector tutorials (#131127)

commit 6832ca40b3986ceb1b0a7daefd8bec193f47701d
Author: elasticsearchmachine <[email protected]>
Date:   Wed Jul 16 23:32:27 2025 +0200

    Mute org.elasticsearch.xpack.downsample.DataStreamLifecycleDownsampleDisruptionIT testDataStreamLifecycleDownsampleRollingRestart #131394

commit 04119405d141bcb390509f766ebbb7660b0ffe72
Author: Stanislav Malyshev <[email protected]>
Date:   Wed Jul 16 13:09:41 2025 -0600

    Put shards failure under a cap flag (#131371)

    * Put shards failure under a cap flag

commit b57ee3bb3adcef8a36f68cef7c69ee0b26503d3d
Author: Ignacio Vera <[email protected]>
Date:   Wed Jul 16 20:03:16 2025 +0100

    [DiskBBQ] Remove posting lists offsets from meta file (#131379)

    this commit proposes to remove the offsets list from the meta file, instead it adds the offsets just after the centroids, together with the raw centroid.

commit 9226a656d5aab86a9f1ec836731fad198940381b
Author: O.K. <[email protected]>
Date:   Wed Jul 16 21:48:08 2025 +0300

    [DOCS]: fix decimal digit filter reference (#129695)

commit 38b7bfcaf39bb78b80da75adeb9acc5afa3aad9d
Author: Sam Xiao <[email protected]>
Date:   Wed Jul 16 13:06:36 2025 -0400

    Mark watcher NotMultiProjectCapable and replace deprecated multi-project methods (#131313)

commit 9c6cf9045639da26c39916221fea955cdff63f60
Author: Dan Rubinstein <[email protected]>
Date:   Wed Jul 16 12:44:48 2025 -0400

    Enable force inference endpoint deleting for invalid models and after stopping model deployment fails (#129090)

    * Enable force inference endpoint deleting for invalid models and after stopping model deployment fails

    * Update docs/changelog/129090.yaml

    ---------

    Co-authored-by: Elastic Machine <[email protected]>

commit 037ddaa5c870d770d42de35741f3f53f90c6c0bb
Author: Pat Whelan <[email protected]>
Date:   Wed Jul 16 12:00:38 2025 -0400

    [ML] Remove SageMaker Elastic updates (#131301)

    Rather than silently drop the payload,
    throw a validation error when Users try to send task settings in the
    update payload for SageMaker inference with the Elastic API.

commit 0cf275eca411a4311cf405bbbb4d0cb0ed931c3e
Author: Ben Chaplin <[email protected]>
Date:   Wed Jul 16 11:50:33 2025 -0400

    Refactor AsyncSearchErrorTraceIT to use assertBusy (#131328)

commit 1da16920ded879ea561b62d416fdf6c4aa674d17
Author: elasticsearchmachine <[email protected]>
Date:   Wed Jul 16 17:36:24 2025 +0200

    Mute org.elasticsearch.packaging.test.DockerTests test130JavaHasCorrectOwnership #131369

commit ae75a98d6c06a7d091e3613f121285a160a047fc
Author: Lorenzo Dematté <[email protected]>
Date:   Wed Jul 16 17:31:05 2025 +0200

    Add exception logging when interrupted (#131153)

commit a04f07571cc80752d643c9ba161e2424813c31f9
Author: elasticsearchmachine <[email protected]>
Date:   Wed Jul 16 17:16:01 2025 +0200

    Mute org.elasticsearch.packaging.test.DockerTests test140CgroupOsStatsAreAvailable #131372

commit 718d54a0a63025fd331ec5b1ab255e79898045ff
Author: elasticsearchmachine <[email protected]>
Date:   Wed Jul 16 17:05:59 2025 +0200

    Mute org.elasticsearch.packaging.test.DockerTests test070BindMountCustomPathConfAndJvmOptions #131366

commit 467fc7ad744f22a86b0e6daed0db51791935b7a5
Author: elasticsearchmachine <[email protected]>
Date:   Wed Jul 16 16:40:29 2025 +0200

    Mute org.elasticsearch.xpack.test.rest.XPackRestIT test {p0=ml/delete_expired_data/Test delete expired data with body parameters} #131364

commit 04104bb6493c6339dfe9a1a52bf7b549d2b62e9e
Author: elasticsearchmachine <[email protected]>
Date:   Wed Jul 16 16:39:57 2025 +0200

    Mute org.elasticsearch.xpack.esql.vector.VectorSimilarityFunctionsIT testSimilarityBetweenConstantVectorAndField {functionName=v_cosine similarityFunction=COSINE} #131363

commit 11e8d18447b087c810c3b9f84a1a781ef3823c8e
Author: elasticsearchmachine <[email protected]>
Date:   Wed Jul 16 16:39:50 2025 +0200

    Mute org.elasticsearch.xpack.esql.vector.VectorSimilarityFunctionsIT testDifferentDimensions {functionName=v_cosine similarityFunction=COSINE} #131362

commit 32a4698c5eeab320995c948cd2f5e3783613c1c1
Author: elasticsearchmachine <[email protected]>
Date:   Wed Jul 16 16:39:42 2025 +0200

    Mute org.elasticsearch.xpack.esql.vector.VectorSimilarityFunctionsIT testSimilarityBetweenConstantVectors {functionName=v_cosine similarityFunction=COSINE} #131361

commit ff1ad0c271d720179a3f3b42cdd65d2caef7466c
Author: Tommaso Teofili <[email protected]>
Date:   Wed Jul 16 16:32:11 2025 +0200

    Check SCORE_FUNCTION capability in VerifierTests (#131352)

commit 1832248827c861293b66eb5fb3bce6477ca3991e
Author: Ievgen Degtiarenko <[email protected]>
Date:   Wed Jul 16 15:39:11 2025 +0200

    Replace deprecated routingTable table call in tests (#131005)

commit 27bba64b84fe206b6fab915ac3582a083fab24c3
Author: Liam Thompson <[email protected]>
Date:   Wed Jul 16 15:37:59 2025 +0200

    [DOCS] Remove misused applies_to tag (#131349)

    * [DOCS] Remove misused applies_to tag

commit 44497b7b05720a5de2608b0a40c6678e50a2fc6f
Author: Benjamin Trent <[email protected]>
Date:   Wed Jul 16 09:35:35 2025 -0400

    Adj ivf postings list building (#130843)

    * Make postings list building more IO friendly

    * iter

    * iter

    * fixing assertion

    * [CI] Auto commit changes from spotless

    ---------

    Co-authored-by: elasticsearchmachine <[email protected]>

commit ce481e5f8050f8f3aa6d118d08a4a0275eb8a997
Author: Pat Whelan <[email protected]>
Date:   Wed Jul 16 08:26:39 2025 -0400

    [Transform] Read metadata from Project State (#131205)

    Add a ProjectState version of the TransformMetadata helper methods.

commit 2af0e9f5e10366571bfebb0f297d6d928510dc29
Author: David Turner <[email protected]>
Date:   Wed Jul 16 11:36:27 2025 +0100

    Add note on o11y to architecture guide (#131291)

    Adds brief overview that mostly just serves as a location for links to
    other more comprehensive design documentation elsewhere in the source
    tree.

    Closes ES-9874

commit ebdfa9c0f5b582469010f2d0335051058e10c23e
Author: David Turner <[email protected]>
Date:   Wed Jul 16 10:48:02 2025 +0100

    Upgrade AWS Java SDK to 2.31.78 (#131050)

    This picks up the fix for the locale bug reported at
    https://github.com/aws/aws-sdk-java-v2/issues/5968.

    This reverts the patching of the library added in commit
    2697a3a87215c92efb3c9bd6c42b99ae8a63f2a7, except for the test
    enhancement.

    Fix confirmed in e.g. locale `ar-ER` with

        ./gradlew ":plugins:discovery-ec2:test" \
            --tests "org.elasticsearch.discovery.ec2.Ec2DiscoveryTests.testFilterByTags" \
            -Dtests.seed=596874EED28A2B92 -Dtests.locale=ar-ER -Dtests.timezone=NET \
            -Druntime.java=24

commit a6f0f6fb4d1ddc38d6aba871ad8d196e0cd16406
Author: eyalkoren <[email protected]>
Date:   Wed Jul 16 11:46:17 2025 +0300

    Support Fields API in conditional ingest processors (#121914)

commit 6ffe27d030aef155d85dc1cc1eb939a5a9fe8fbb
Author: Carlos Delgado <[email protected]>
Date:   Wed Jul 16 10:19:18 2025 +0200

    ESQL - KNN function uses prefilters when pushed down to Lucene (#131004)

commit 71466813223574d6a8b4b33dab8433e3b877664c
Author: Luigi Dell'Aquila <[email protected]>
Date:   Wed Jul 16 10:15:08 2025 +0200

    Add docs for ES|QL query logs (#131287)

commit 01c09cba84998aa4c11824e44f18a44bfac0bfa8
Author: David Turner <[email protected]>
Date:   Wed Jul 16 08:12:48 2025 +0100

    Simplify `expectedFinalRegisterValue` computation (#131274)

    In repository analysis we keep an `AtomicLong` counting the number of
    successfully-completed increments on the contended register, but we only
    check this value if all the increments succeed. We know how many
    increments we enqueue up-front so there's no need to count them as they
    complete. This commit removes the unnecessary counter.

commit f8b89a378abd8bc990f9955eb2c68d7acf701424
Author: elasticsearchmachine <[email protected]>
Date:   Wed Jul 16 08:38:52 2025 +0200

    Mute org.elasticsearch.test.rest.yaml.RcsCcsCommonYamlTestSuiteIT test {p0=search/110_field_collapsing/field collapsing, inner_hits and maxConcurrentGroupRequests} #131348

commit dc48b4b28b8b3bd0b6b1909e26da6374c374322a
Author: Tim Vernum <[email protected]>
Date:   Wed Jul 16 13:44:09 2025 +1000

    Add attribute count to SamlAttribute toString (#131173)

    Sometimes SAML IdPs send what _should_ be a list of values as a single
    comma-separated string.

    That is, we expect something using SAML's multi-valued attribute
    feature:

        <saml:Attribute NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:uri"
           Name="http://idp.example.org/attributes/groups" FriendlyName="groups">
           <saml:AttributeValue>engineering</saml:AttributeValue>
           <saml:AttributeValue>elasticsearch-admins</saml:AttributeValue>
           <saml:AttributeValue>employees</saml:AttributeValue>
        </saml:Attribute>

    but we get

        <saml:Attribute NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:uri"
           Name="http://idp.example.org/attributes/groups" FriendlyName="groups">
           <saml:AttributeValue>engineering,elasticsearch-admins,employees</saml:AttributeValue>
        </saml:Attribute>

    In order to help detect these cases, this commit changes the
    `toString()` on `SamlAttribute` to include the length (e.g. `(len=1)`)
    at the end

    Relates: #84379, #102769

commit 13aceaa78dba62881fc47670c6fd2f58780314db
Author: Kuni Sen <[email protected]>
Date:   Wed Jul 16 12:24:44 2025 +0900

    Update reference-docs-links.txt to include serverless doc reference link  (#131175)

    * Update reference-docs-links.txt

    * Update reference-docs-links.txt

    * Update ReferenceDocs.java

commit 393f70703f1f62969fad8c13598dddd887753f28
Author: Stanislav Malyshev <[email protected]>
Date:   Tue Jul 15 19:02:15 2025 -0600

    Disable in release build due to pragma (#131340)

commit 004af14a0a0a0679a9bd526b5548d80a92257c9a
Author: Niels Bauman <[email protected]>
Date:   Tue Jul 15 20:45:03 2025 -0300

    Remove `ClusterStateSerializationTests` (#130877)

    These tests do not seem valuable anymore, now that multi-project has
    been merged some time ago. Additionally, creating objects for a cluster
    state this way is tricky, as those objects might get new
    features/fields, which would break serialization.

    Closes #130872

commit f9b9e7cfa9d4b2ef25a479353d5e11de21c7722c
Author: elasticsearchmachine <[email protected]>
Date:   Wed Jul 16 01:33:45 2025 +0200

    Mute org.elasticsearch.packaging.test.DockerTests test171AdditionalCliOptionsAreForwarded #120925

commit dac6021d140e09054072bd5c6d10d0d8fd018a10
Author: elasticsearchmachine <[email protected]>
Date:   Wed Jul 16 00:55:21 2025 +0200

    Mute org.elasticsearch.packaging.test.DockerTests test071BindMountCustomPathWithDifferentUID #120917

commit 71d04a4b411ea7ef4a6ab47652e40d95300700d1
Author: elasticsearchmachine <[email protected]>
Date:   Wed Jul 16 00:55:12 2025 +0200

    Mute org.elasticsearch.packaging.test.DockerTests test022InstallPluginsFromLocalArchive #116866

commit 7331271f33da644e7bb8c9691a8ccd415012c1e2
Author: Pablo <[email protected]>
Date:   Tue Jul 15 15:37:46 2025 -0700

    Tightening rate verification (#131114)

commit 8f6f7633bcecefc4b61ed68bec6dbf4da694b0dc
Author: Nhat Nguyen <[email protected]>
Date:   Tue Jul 15 15:31:31 2025 -0700

    Fail request when all target shards fail in runtime (#131177)

    If all target shards, excluding skipped shards, fail, we should fail the
    entire query regardless of the partial_results configuration or
    skip_unavailable setting. This behavior does not fully align with the
    search API, where skip_unavailable ignores all failures from remote
    clusters and only fails the request when all shards in the local cluster
    fail. However, we believe the proposed behavior is more sensible than
    the existing behavior in the search API.

    Closes #128994

commit 000c71e60d8bc5bbf2a2c624180a73d12279e2e3
Author: elasticsearchmachine <[email protected]>
Date:   Wed Jul 16 00:30:46 2025 +0200

    Mute org.elasticsearch.xpack.esql.analysis.VerifierTests testMatchInsideEval #131336

commit c12f382ae5f2d0012e5d209709fb172841fa59f7
Author: elasticsearchmachine <[email protected]>
Date:   Wed Jul 16 00:30:38 2025 +0200

    Mute org.elasticsearch.xpack.esql.analysis.VerifierTests testFullTextFunctionsOnlyAllowedInWhere #131335

commit 42a271eaa4db6bcaaa31b1f538db315c1d2f5dab
Author: elasticsearchmachine <[email protected]>
Date:   Wed Jul 16 00:30:27 2025 +0200

    Mute org.elasticsearch.xpack.esql.expression.function.fulltext.ScoreTests testSerializationOfSimple {TestCase=<boolean>} #131334

commit 6f8be9c4ff502874d60a06fe227ab5f691ebde2a
Author: Jordan Powers <[email protected]>
Date:   Tue Jul 15 15:19:28 2025 -0700

    Fix match_only_text for keyword multi-fields with ignore_above (#131314)

commit 7bbb211cb9b0abcc5bb5aae5674aac7b51a2f5fa
Author: Quentin Pradet <[email protected]>
Date:   Wed Jul 16 00:42:40 2025 +0400

    Add require_alias to reindex rest-api-spec (#130813)

commit 6ccc4494c36b464defc993c9a57dca7614eb3f94
Author: Brendan Cully <[email protected]>
Date:   Tue Jul 15 12:51:13 2025 -0700

    Log start of memory throttling at info (#131310)

    This is symmetric with logging when throttling stops, and
    will make it easier to debug throttling issues.

commit fde4c0c596337c42c08492c75fbf0387a4c182d3
Author: Benjamin Trent <[email protected]>
Date:   Tue Jul 15 15:31:35 2025 -0400

    Adds support for more than one input doc file for Knnindextester (#131308)

commit 57598222772c3c4cb06d7639c3d1db50d8542c97
Author: Mark Tozzi <[email protected]>
Date:   Tue Jul 15 15:11:20 2025 -0400

    Clean up non-parameterized tests in the wrong place (#131049)

    In the course of other work I found a few places where we were creating non-parameterized tests from within the parameterized test drivers for a few ESQL functions. This causes those tests to be run for every parameter combination, even though the tests themselves do not change anything, resulting in a lot of extra test overhead for no additional coverage.

    I cleaned up three classes:

        EndsWithTests ran 832 tests before this, and now runs 640 tests
        StartsWithTests ran 208 before this, and now runs 160
        InTests ran 168 before this, and now runs 28

    So overall, that's 369 redundant test runs removed (including the fact that the 11 tests I moved still run once in their new classes), and further savings if we later expand those parameterized tests.

commit f0c30f272da3ff36f1a65524cc0e63a07389800a
Author: Julian Kiryakov <[email protected]>
Date:   Tue Jul 15 14:39:11 2025 -0400

    Add support for RLIKE (LIST) with pushdown (#129929)

    Adds support for RLIKE function alternative syntax with a list of patterns.
    Examples:

    ROW message = "foobar"
    | WHERE message RLIKE ("foo.*", "bar.")
    The new syntax is documented as part of the existing RLIKE function documentation. We will use the existing RLike java implementation for existing cases using the old syntax and one list argument case to improve mixed cluster compatibility.
    The RLikeList is pushed down as a single Automaton to improve performance.

commit 03975a363ac36f16e4645b026f55816426d02b5e
Author: Sam Xiao <[email protected]>
Date:   Tue Jul 15 14:19:15 2025 -0400

    Make TransportDeleteDatabaseConfigurationAction project aware (#131300)

commit bfab88c430a727de63188687a2c4655276a10212
Author: Stanislav Malyshev <[email protected]>
Date:   Tue Jul 15 11:51:57 2025 -0600

    Fix testStopQueryLocal - always use the same client (same node) (#131253)

commit b1c78ce2de591c014a727e9444438c80ff43db31
Author: Sam Xiao <[email protected]>
Date:   Tue Jul 15 13:49:21 2025 -0400

    Make TransportGetDatabaseConfigurationAction project aware (#130065)

commit fcd5afaa5d24c2647c40043d4b6906175e72467b
Author: Sam Xiao <[email protected]>
Date:   Tue Jul 15 13:48:27 2025 -0400

    Mark custom GeoIP database NotMultiProjectCapable (#130064)

commit 366bc0068e6bfdfe055f4eb6e220ec891e2b9eb2
Author: Stanislav Malyshev <[email protected]>
Date:   Tue Jul 15 10:19:07 2025 -0600

    Refactor and simplify missing index & unavailability handling (#131252)

commit 516990c075b9f47f63dedd9c8f0e33e13ca4e915
Author: Nhat Nguyen <[email protected]>
Date:   Tue Jul 15 09:00:41 2025 -0700

    Remove ordinal grouping operator (#131133)

    The ordinals grouping operator was introduced to speed up aggregation
    before ordinal blocks and related optimizations in block hashes were
    available. However, this operator has several issues:

    1. It only supports single grouping with the `keyword` type and requires
    `doc_values`.

    2. It needs a separate aggregation implementation, which currently lacks
    test coverage. We had performance issues with the `VALUES` aggregation
    using this operator (see #130576).

    3. It can be slower and use more memory when the target documents have
    sparse cardinality (see #98963).

    4. Ad-hoc planning, although this can now be addressed with local plans.

    Although the ordinals grouping operator is slightly faster than the hash
    operator with ordinal blocks, its complexity now outweighs the benefits.
    This PR proposes removing the operator. Below is the NYC_taxis
    benchmark.

    Closes #98963

commit 04ae5275bee342be105f15ae7728cca2be6259e7
Author: Fang Xing <[email protected]>
Date:   Tue Jul 15 11:55:58 2025 -0400

    [ES|QL] Substitute date_trunc with round_to when the pre-calculated rounding points are available (#128639)

    * consolidate min/max in SearchStats and substitue date_trunc/bucket with round_to

commit c2fa78fda121c966277e4728d1aa1836152ca938
Author: Nhat Nguyen <[email protected]>
Date:   Tue Jul 15 08:50:27 2025 -0700

    Speed up reading multivalued keywords (#131061)

    This change speeds up reading multi-valued keyword fields by leveraging ordinals.

    Before:
    ```
    Benchmark                              (layout)          (name)  Mode  Cnt     Score    Error  Units
    ValuesSourceReaderBenchmark.benchmark  in_order      keyword_mv  avgt    7   318.332 ±  1.660  ns/op
    ```

    After:
    ```
    Benchmark                              (layout)          (name)  Mode  Cnt     Score    Error  Units
    ValuesSourceReaderBenchmark.benchmark  in_order      keyword_mv  avgt    7    96.659 ±  0.932  ns/op
    ```

commit 1669e8d438dbd9b88630bfc501d6d84b71da8e41
Author: Pablo <[email protected]>
Date:   Tue Jul 15 08:19:23 2025 -0700

    Redesigining randomized tests for TS functions (#129971)

    * Working on rate integration test

    * Sketch of new rate tests. Unsure of the source of variation >10pct

    * Tuning and improving auto-test for rate-based aggregations

    * fixup

    * addressing comments

    * fixup

    * improve failure rate

    * tuned deviation limits to reduce flakiness

    * fixup - wider error margin

    * wider margin

    * wider range for aggs

commit 1541b146403f90bf7425b16c4f18e98e360463fa
Author: Joshua Adams <[email protected]>
Date:   Tue Jul 15 16:03:40 2025 +0100

    Add Query Parameters to Cluster Allocation Explain API (#129342)

    The cluster allocation explain API now accepts parameters via the
    request body, via path parameters passed in the URL, but not via both.

    Issue: https://github.com/elastic/elasticsearch/issues/127028

commit 9d4c3f84ff58d2310b1a9bb000b1e75fc82dc2a5
Author: Iraklis Psaroudakis <[email protected]>
Date:   Tue Jul 15 17:55:30 2025 +0300

    Fix concurrent list in merge test (#131186)

    Fixes #131165

commit 86c778768b22eaf1fa4a23c9ad788ed12fe46bd5
Author: Sam Xiao <[email protected]>
Date:   Tue Jul 15 10:39:23 2025 -0400

    Remove deprecated multi-project methods in Enterprise GeoIP downloader (#130062)

commit 480615633cbf5c7e0b2f67a23f04d9e26f040592
Author: Slobodan Adamović <[email protected]>
Date:   Tue Jul 15 15:20:42 2025 +0200

    Unmute docker tests, add more logging and increase startup timeout (#131203)

    Unmuting 33 DockerTests with added additional logging on failures.
    Also, bumping docker startup timeout to account for slow security
    auto-configuration, and additionally moving CA certificates inside
    the retry while loop.

    Resolves #129808
    Resolves #123437
    Resolves #120911
    Resolves #120914
    Resolves #120910
    Resolves #120918
    Resolves #120925
    Resolves #124990
    Resolves #116636
    Resolves #125680
    Resolves #116147
    Resolves #126232
    Resolves #126145
    Resolves #116866
    Resolves #126936
    Resolves #127081
    Resolves #127158
    Resolves #127138
    Resolves #128018
    Resolves #128031
    Resolves #128110
    Resolves #128006
    Resolves #128174
    Resolves #128144
    Resolves #128165
    Resolves #128075
    Resolves #128117
    Resolves #128121
    Resolves #128115
    Resolves #128230
    Resolves #128120
    Resolves #128996
    Resolves #128867

commit f1ddd4c31282155dbc5db955003abd2494ecf9b4
Author: Carlos Delgado <[email protected]>
Date:   Tue Jul 15 14:49:25 2025 +0200

    ESQL: dense_vector cosine similarity function (#130641)

commit 730308c6898a1ba94b59f790e0699e6fe021074d
Author: Carlos Delgado <[email protected]>
Date:   Tue Jul 15 14:15:09 2025 +0200

    Unmute #130640 - test failure was unrelated (#131285)

commit 92ab85d26d33e0110723c65347fafa08872bd618
Author: Moritz Mack <[email protected]>
Date:   Tue Jul 15 13:21:54 2025 +0200

    Log NotEntitledExceptions using logger with <package> suffix (#131031)

    Log NotEntitledExceptions using logger with `<component>.<module>.<package>` suffix (instead of `<component>.<module>`) for more fine-grained mutes, but remaining backwards compatible regarding existing mutes.

    Relates to ES-12231

commit bfb3aee8ea530c76419c89b2a4caa2b81fbf8833
Author: Tim Rühsen <[email protected]>
Date:   Tue Jul 15 13:11:00 2025 +0200

    [Profiling] Manually downsample to 20k events if required (#131196)

    * [Profiling] Manually downsample to 20k events if required

    Since the random sampler aggregation is limited to p values of 0..0.5,
    add an explicit downsampling for p values of 0.5..1.0.

    With this, the latency of fetching stacktraces and stackframes can be
    further reduced.

    * Fix number of reported samples

commit e4ff3f8bae28b78cfd10770fe076309296d8042c
Author: Luigi Dell'Aquila <[email protected]>
Date:   Tue Jul 15 12:59:42 2025 +0200

    ES|QL: fix generative tests (#131283)

commit fa6e3e75aa9ec2dcc6e4316a1ae5ab1f71141a64
Author: Niels Bauman <[email protected]>
Date:   Tue Jul 15 07:39:53 2025 -0300

    Remove non-test usages of `Metadata.Builder#indices` (#131240)

    Replaces the usages with non-deprecated versions.

commit fd037bd846c5d63b11cce2012d1f161cbdc801db
Author: Tommaso Teofili <[email protected]>
Date:   Tue Jul 15 11:55:57 2025 +0200

    retain the scores of portions of an ES|QL query, via a score function (#127551)

commit 7139aa1c5228bd649fc1cc3dd36c7df1160ec0de
Author: Quentin Pradet <[email protected]>
Date:   Tue Jul 15 12:01:52 2025 +0400

    Add register_operation_count to snapshot.repository_analyze rest-api-spec (#131082)

    * Add register_operation_count to snapshot.repository_analyze

    * Remove mention of realistic experiments for consistency

    Co-authored-by: Yang Wang <[email protected]>

    ---------

    Co-authored-by: Yang Wang <[email protected]>

commit f67c1f03735e1b58d4f9d77ed7ff40f014308ea5
Author: Ievgen Degtiarenko <[email protected]>
Date:   Tue Jul 15 09:50:35 2025 +0200

    Ensure single segment in test (#131077)

commit b3b83cc7e91595dc1a03e401257a7691763300b3
Author: elasticsearchmachine <[email protected]>
Date:   Tue Jul 15 09:37:34 2025 +0200

    Update docs for v9.0.4 release (#131144)

commit 6f5579656c6a6ca573669fe733463ed2da35d9e2
Author: David Turner <[email protected]>
Date:   Tue Jul 15 08:32:49 2025 +0100

    Improve lost-increment message in repo analysis (#131200)

    Today repository analysis may fail with a message like the following:

        [test-repo] register [test-register-contended-F_NNXHrSSDGveoeyj1skwg]
        should have value [10] but instead had value
        [OptionalBytesReference[00 00 00 00 00 00 00 09]]

    …
@przemekwitek przemekwitek deleted the empty_bucket branch November 7, 2025 09:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

v9.3.0 WIP

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@przemekwitek @elasticsearchmachine