Merge branch '9.0' into backport/9.0/pr-128823

mosche · web-flow · commit 852a6de08ee8 · 2025-07-14T10:17:27.000+02:00
diff --git a/README.asciidoc b/README.asciidoc
@@ -265,7 +265,7 @@ For the complete Elasticsearch documentation visit
 https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html[elastic.co].
 
 For information about our documentation processes, see the
-xref:docs/README.asciidoc[docs README].
+xref:https://github.com/elastic/elasticsearch/blob/main/docs/README.md[docs README].
 
 [[examples]]
 == Examples and guides
diff --git a/docs/changelog/130924.yaml b/docs/changelog/130924.yaml
@@ -0,0 +1,6 @@
+pr: 130924
+summary: Check field data type before casting when applying geo distance sort
+area: Search
+type: bug
+issues:
+ - 129500
diff --git a/docs/changelog/131032.yaml b/docs/changelog/131032.yaml
@@ -0,0 +1,5 @@
+pr: 131032
+summary: "Fix: `GET _synonyms` returns synonyms with empty rules"
+area: Relevance
+type: bug
+issues: []
diff --git a/docs/changelog/131081.yaml b/docs/changelog/131081.yaml
@@ -0,0 +1,6 @@
+pr: 131081
+summary: Fix knn search error when dimensions are not set
+area: Vector Search
+type: bug
+issues:
+ - 129550
diff --git a/docs/reference/elasticsearch/index.md b/docs/reference/elasticsearch/index.md
@@ -1 +1,15 @@
-# Elasticsearch and index management
+# Elasticsearch
+
+This section contains reference information for {{es}} and index management features.
+
+To learn more about {{es}} features and how to get started, refer to the [{{es}}](docs-content://solutions/search.md) documentation.
+
+For more details about query and scripting languages, check these sections:
+
+* [Query languages](../query-languages/index.md)
+* [Scripting languages](../scripting-languages/index.md)
+
+{{es}} also provides the following REST APIs:
+
+* [{{es}} API](https://www.elastic.co/docs/api/doc/elasticsearch)
+* [{{es}} Serverless API](https://www.elastic.co/docs/api/doc/elasticsearch-serverless)
diff --git a/docs/reference/elasticsearch/rest-apis/index.md b/docs/reference/elasticsearch/rest-apis/index.md
diff --git a/docs/reference/elasticsearch/rest-apis/retrievers.md b/docs/reference/elasticsearch/rest-apis/retrievers.md
@@ -291,7 +291,10 @@ See also [this hybrid search example](docs-content://solutions/search/retrievers
 `rank_window_size`
 :   (Optional, integer)
 
-    This value determines the size of the individual result sets per query. A higher value will improve result relevance at the cost of performance. The final ranked result set is pruned down to the search request’s [size](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-search#search-size-param). `rank_window_size` must be greater than or equal to `size` and greater than or equal to `1`. Defaults to the `size` parameter.
+    This value determines the size of the individual result sets per query.
+    A higher value will improve result relevance at the cost of performance.
+    The final ranked result set is pruned down to the search request’s [size](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-search#search-size-param). `rank_window_size` must be greater than or equal to `size` and greater than or equal to `1`.
+    Defaults to 10.
 
 
 `filter`
@@ -323,7 +326,10 @@ An [RRF](/reference/elasticsearch/rest-apis/reciprocal-rank-fusion.md) retriever
 `rank_window_size`
 :   (Optional, integer)
 
-    This value determines the size of the individual result sets per query. A higher value will improve result relevance at the cost of performance. The final ranked result set is pruned down to the search request’s [size](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-search#search-size-param). `rank_window_size` must be greater than or equal to `size` and greater than or equal to `1`. Defaults to the `size` parameter.
+    This value determines the size of the individual result sets per query.
+    A higher value will improve result relevance at the cost of performance.
+    The final ranked result set is pruned down to the search request’s [size](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-search#search-size-param). `rank_window_size` must be greater than or equal to `size` and greater than or equal to `1`.
+    Defaults to 10.
 
 
 `filter`
diff --git a/docs/reference/query-languages/esql/limitations.md b/docs/reference/query-languages/esql/limitations.md
@@ -254,3 +254,6 @@ Work around this limitation by converting the field to single value with one of
 * CSV export from Discover shows no more than 10,000 rows. This limit only applies to the number of rows that are retrieved by the query and displayed in Discover. Queries and aggregations run on the full data set.
 * Querying many indices at once without any filters can cause an error in kibana which looks like `[esql] > Unexpected error from Elasticsearch: The content length (536885793) is bigger than the maximum allowed string (536870888)`. The response from {{esql}} is too long. Use [`DROP`](/reference/query-languages/esql/commands/processing-commands.md#esql-drop) or [`KEEP`](/reference/query-languages/esql/commands/processing-commands.md#esql-keep) to limit the number of fields returned.
 
+## Known issues [esql-known-issues]
+
+Refer to [Known issues](/release-notes/known-issues.md) for a list of known issues for {{esql}}.
diff --git a/docs/release-notes/known-issues.md b/docs/release-notes/known-issues.md
@@ -34,3 +34,12 @@ This issue will be fixed in a future patch release (see [PR #126990](https://git
      DELETE _index_template/.watches
      POST /_watcher/_start
      ```
+
+* A bug in the ES|QL STATS command may yield incorrect results. The bug only happens in very specific cases that follow this pattern: `STATS ... BY keyword1, keyword2`, i.e. the command must have exactly two grouping fields, both keywords, where the first field has high cardinality (more than 65k distinct values).
+
+  The bug is described in detail in [this issue](https://github.com/elastic/elasticsearch/issues/130644).
+  The problem was introduced in 8.16.0 and [fixed](https://github.com/elastic/elasticsearch/pull/130705) in 8.17.9, 8.18.7, 9.0.4.
+
+  Possible workarounds include:
+    * switching the order of the grouping keys (eg. `STATS ... BY keyword2, keyword1`, if the `keyword2` has a lower cardinality)
+    * reducing the grouping key cardinality, by filtering out values before STATS
diff --git a/muted-tests.yml b/muted-tests.yml
@@ -282,6 +282,9 @@ tests:
 - class: org.elasticsearch.index.engine.ThreadPoolMergeExecutorServiceDiskSpaceTests
   method: testUnavailableBudgetBlocksNewMergeTasksFromStartingExecution
   issue: https://github.com/elastic/elasticsearch/issues/130205
+- class: org.elasticsearch.packaging.test.DockerTests
+  method: test080ConfigurePasswordThroughEnvironmentVariableFile
+  issue: https://github.com/elastic/elasticsearch/issues/128075
 
 # Examples:
 #
diff --git a/qa/mixed-cluster/build.gradle b/qa/mixed-cluster/build.gradle
@@ -89,6 +89,7 @@ buildParams.bwcVersions.withWireCompatible { bwcVersion, baseName ->
         setting 'health.master_history.no_master_transitions_threshold', '10'
       }
       requiresFeature 'es.index_mode_feature_flag_registered', Version.fromString("8.0.0")
+      requiresFeature 'sub_objects_auto', Version.fromString("8.16.0")
       if (bwcVersion.before(Version.fromString("8.18.0"))) {
         jvmArgs '-da:org.elasticsearch.index.mapper.DocumentMapper'
         jvmArgs '-da:org.elasticsearch.index.mapper.MapperService'
diff --git a/rest-api-spec/src/main/resources/rest-api-spec/api/search_mvt.json b/rest-api-spec/src/main/resources/rest-api-spec/api/search_mvt.json
@@ -73,6 +73,15 @@
         "description":"Determines the geometry type for features in the aggs layer.",
         "default":"grid"
       },
+      "grid_agg":{
+        "type":"enum",
+        "options":[
+          "geotile",
+          "geohex"
+        ],
+        "description":"Aggregation used to create a grid for `field`.",
+        "default":"geotile"
+      },
       "size":{
         "type":"int",
         "description":"Maximum number of features to return in the hits layer. Accepts 0-10000.",
diff --git a/rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/search.vectors/40_knn_search.yml b/rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/search.vectors/40_knn_search.yml
@@ -630,3 +630,36 @@ setup:
           properties:
             embedding:
               type: dense_vector
+
+
+---
+"Searching with no data dimensions specified":
+  - requires:
+      cluster_features: "search.vectors.no_dimensions_bugfix"
+      reason: "Search with no dimensions bugfix"
+
+  - do:
+      indices.create:
+        index: empty-test
+        body:
+          mappings:
+            properties:
+              vector:
+                type: dense_vector
+                index: true
+
+  - do:
+      search:
+        index: empty-test
+        body:
+          fields: [ "name" ]
+          knn:
+            field: vector
+            query_vector: [ -0.5, 90.0, -10, 14.8, -156.0 ]
+            k: 3
+            num_candidates: 3
+            rescore_vector:
+              oversample: 1.5
+            similarity: 0.1
+
+  - match: { hits.total.value: 0 }
diff --git a/rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/synonyms/40_synonyms_sets_get.yml b/rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/synonyms/40_synonyms_sets_get.yml
@@ -163,3 +163,33 @@ setup:
 
   - match:
       count: 12
+
+---
+"Return empty rule set":
+  - requires:
+      cluster_features: [ synonyms_set.get.return_empty_synonym_sets ]
+      reason: "synonyms_set get api return empty synonym sets"
+
+  - do:
+      synonyms.put_synonym:
+        id: empty-synonyms
+        body:
+          synonyms_set: []
+
+  - do:
+      synonyms.get_synonyms_sets: {}
+
+  - match:
+      count: 4
+
+  - match:
+      results:
+        - synonyms_set: "empty-synonyms"
+          count: 0
+        - synonyms_set: "test-synonyms-1"
+          count: 3
+        - synonyms_set: "test-synonyms-2"
+          count: 1
+        - synonyms_set: "test-synonyms-3"
+          count: 2
+
diff --git a/server/src/main/java/module-info.java b/server/src/main/java/module-info.java
@@ -430,6 +430,7 @@
             org.elasticsearch.index.mapper.MapperFeatures,
             org.elasticsearch.index.IndexFeatures,
             org.elasticsearch.search.SearchFeatures,
+            org.elasticsearch.synonyms.SynonymFeatures,
             org.elasticsearch.script.ScriptFeatures,
             org.elasticsearch.search.retriever.RetrieversFeatures,
             org.elasticsearch.action.admin.cluster.stats.ClusterStatsFeatures;
diff --git a/server/src/main/java/org/elasticsearch/index/mapper/vectors/DenseVectorFieldMapper.java b/server/src/main/java/org/elasticsearch/index/mapper/vectors/DenseVectorFieldMapper.java
@@ -31,6 +31,7 @@
 import org.apache.lucene.index.VectorEncoding;
 import org.apache.lucene.index.VectorSimilarityFunction;
 import org.apache.lucene.search.FieldExistsQuery;
+import org.apache.lucene.search.MatchNoDocsQuery;
 import org.apache.lucene.search.Query;
 import org.apache.lucene.search.join.BitSetProducer;
 import org.apache.lucene.util.BitUtil;
@@ -2026,6 +2027,9 @@ public Query createKnnQuery(
                     "to perform knn search on field [" + name() + "], its mapping must have [index] set to [true]"
                 );
             }
+            if (dims == null) {
+                return new MatchNoDocsQuery("No data has been indexed for field [" + name() + "]");
+            }
             return switch (getElementType()) {
                 case BYTE -> createKnnByteQuery(queryVector.asByteVector(), k, numCands, filter, similarityThreshold, parentFilter);
                 case FLOAT -> createKnnFloatQuery(
diff --git a/server/src/main/java/org/elasticsearch/index/shard/IndexShard.java b/server/src/main/java/org/elasticsearch/index/shard/IndexShard.java
@@ -1344,11 +1344,12 @@ public RefreshStats refreshStats() {
     }
 
     public FlushStats flushStats() {
+        final Engine engine = getEngineOrNull();
         return new FlushStats(
             flushMetric.count(),
             periodicFlushMetric.count(),
             TimeUnit.NANOSECONDS.toMillis(flushMetric.sum()),
-            getEngineOrNull() != null ? getEngineOrNull().getTotalFlushTimeExcludingWaitingOnLockInMillis() : 0L
+            engine != null ? engine.getTotalFlushTimeExcludingWaitingOnLockInMillis() : 0L
         );
     }
 
diff --git a/server/src/main/java/org/elasticsearch/search/SearchFeatures.java b/server/src/main/java/org/elasticsearch/search/SearchFeatures.java
@@ -25,9 +25,10 @@ public Set<NodeFeature> getFeatures() {
     }
 
     public static final NodeFeature RETRIEVER_RESCORER_ENABLED = new NodeFeature("search.retriever.rescorer.enabled");
+    public static final NodeFeature SEARCH_WITH_NO_DIMENSIONS_BUGFIX = new NodeFeature("search.vectors.no_dimensions_bugfix");
 
     @Override
     public Set<NodeFeature> getTestFeatures() {
-        return Set.of(RETRIEVER_RESCORER_ENABLED);
+        return Set.of(RETRIEVER_RESCORER_ENABLED, SEARCH_WITH_NO_DIMENSIONS_BUGFIX);
     }
 }
diff --git a/server/src/main/java/org/elasticsearch/search/sort/GeoDistanceSortBuilder.java b/server/src/main/java/org/elasticsearch/search/sort/GeoDistanceSortBuilder.java
@@ -599,7 +599,13 @@ private IndexGeoPointFieldData fieldData(SearchExecutionContext context) {
                 throw new IllegalArgumentException("failed to find mapper for [" + fieldName + "] for geo distance based sort");
             }
         }
-        return context.getForField(fieldType, MappedFieldType.FielddataOperation.SEARCH);
+        IndexFieldData<?> indexFieldData = context.getForField(fieldType, MappedFieldType.FielddataOperation.SEARCH);
+        if (indexFieldData instanceof IndexGeoPointFieldData) {
+            return (IndexGeoPointFieldData) indexFieldData;
+        }
+        throw new IllegalArgumentException(
+            "unable to apply geo distance sort to field [" + fieldName + "] of type [" + fieldType.typeName() + "]"
+        );
     }
 
     private Nested nested(SearchExecutionContext context) throws IOException {
diff --git a/server/src/main/java/org/elasticsearch/synonyms/SynonymFeatures.java b/server/src/main/java/org/elasticsearch/synonyms/SynonymFeatures.java
@@ -0,0 +1,24 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the "Elastic License
+ * 2.0", the "GNU Affero General Public License v3.0 only", and the "Server Side
+ * Public License v 1"; you may not use this file except in compliance with, at
+ * your election, the "Elastic License 2.0", the "GNU Affero General Public
+ * License v3.0 only", or the "Server Side Public License, v 1".
+ */
+
+package org.elasticsearch.synonyms;
+
+import org.elasticsearch.features.FeatureSpecification;
+import org.elasticsearch.features.NodeFeature;
+
+import java.util.Set;
+
+public class SynonymFeatures implements FeatureSpecification {
+    private static final NodeFeature RETURN_EMPTY_SYNONYM_SETS = new NodeFeature("synonyms_set.get.return_empty_synonym_sets");
+
+    @Override
+    public Set<NodeFeature> getTestFeatures() {
+        return Set.of(RETURN_EMPTY_SYNONYM_SETS);
+    }
+}
diff --git a/server/src/main/java/org/elasticsearch/synonyms/SynonymsManagementAPIService.java b/server/src/main/java/org/elasticsearch/synonyms/SynonymsManagementAPIService.java
@@ -45,6 +45,9 @@
 import org.elasticsearch.indices.SystemIndexDescriptor;
 import org.elasticsearch.rest.RestStatus;
 import org.elasticsearch.search.aggregations.BucketOrder;
+import org.elasticsearch.search.aggregations.bucket.filter.Filters;
+import org.elasticsearch.search.aggregations.bucket.filter.FiltersAggregationBuilder;
+import org.elasticsearch.search.aggregations.bucket.filter.FiltersAggregator;
 import org.elasticsearch.search.aggregations.bucket.terms.Terms;
 import org.elasticsearch.search.aggregations.bucket.terms.TermsAggregationBuilder;
 import org.elasticsearch.search.builder.SearchSourceBuilder;
@@ -89,6 +92,8 @@ public class SynonymsManagementAPIService {
     private static final int MAX_SYNONYMS_SETS = 10_000;
     private static final String SYNONYM_RULE_ID_FIELD = SynonymRule.ID_FIELD.getPreferredName();
     private static final String SYNONYM_SETS_AGG_NAME = "synonym_sets_aggr";
+    private static final String RULE_COUNT_AGG_NAME = "rule_count";
+    private static final String RULE_COUNT_FILTER_KEY = "synonym_rules";
     private static final int SYNONYMS_INDEX_MAPPINGS_VERSION = 1;
     private final int maxSynonymsSets;
 
@@ -179,27 +184,45 @@ private static XContentBuilder mappings() {
         }
     }
 
+    /**
+     * Returns all synonym sets with their rule counts, including empty synonym sets.
+     * @param from The index of the first synonym set to return
+     * @param size The number of synonym sets to return
+     * @param listener The listener to return the synonym sets to
+     */
     public void getSynonymsSets(int from, int size, ActionListener<PagedResult<SynonymSetSummary>> listener) {
+        BoolQueryBuilder synonymSetQuery = QueryBuilders.boolQuery()
+            .should(QueryBuilders.termQuery(OBJECT_TYPE_FIELD, SYNONYM_SET_OBJECT_TYPE))
+            .should(QueryBuilders.termQuery(OBJECT_TYPE_FIELD, SYNONYM_RULE_OBJECT_TYPE))
+            .minimumShouldMatch(1);
+
+        // Aggregation query to count only synonym rules (excluding synonym set objects)
+        FiltersAggregationBuilder ruleCountAggregation = new FiltersAggregationBuilder(
+            RULE_COUNT_AGG_NAME,
+            new FiltersAggregator.KeyedFilter(RULE_COUNT_FILTER_KEY, QueryBuilders.termQuery(OBJECT_TYPE_FIELD, SYNONYM_RULE_OBJECT_TYPE))
+        );
+
         client.prepareSearch(SYNONYMS_ALIAS_NAME)
             .setSize(0)
             // Retrieves aggregated synonym rules for each synonym set, excluding the synonym set object type
-            .setQuery(QueryBuilders.termQuery(OBJECT_TYPE_FIELD, SYNONYM_RULE_OBJECT_TYPE))
+            .setQuery(synonymSetQuery)
             .addAggregation(
                 new TermsAggregationBuilder(SYNONYM_SETS_AGG_NAME).field(SYNONYMS_SET_FIELD)
                     .order(BucketOrder.key(true))
                     .size(maxSynonymsSets)
+                    .subAggregation(ruleCountAggregation)
             )
             .setPreference(Preference.LOCAL.type())
             .execute(new ActionListener<>() {
                 @Override
                 public void onResponse(SearchResponse searchResponse) {
                     Terms termsAggregation = searchResponse.getAggregations().get(SYNONYM_SETS_AGG_NAME);
                     List<? extends Terms.Bucket> buckets = termsAggregation.getBuckets();
-                    SynonymSetSummary[] synonymSetSummaries = buckets.stream()
-                        .skip(from)
-                        .limit(size)
-                        .map(bucket -> new SynonymSetSummary(bucket.getDocCount(), bucket.getKeyAsString()))
-                        .toArray(SynonymSetSummary[]::new);
+                    SynonymSetSummary[] synonymSetSummaries = buckets.stream().skip(from).limit(size).map(bucket -> {
+                        Filters ruleCountFilters = bucket.getAggregations().get(RULE_COUNT_AGG_NAME);
+                        Filters.Bucket ruleCountBucket = ruleCountFilters.getBucketByKey(RULE_COUNT_FILTER_KEY);
+                        return new SynonymSetSummary(ruleCountBucket.getDocCount(), bucket.getKeyAsString());
+                    }).toArray(SynonymSetSummary[]::new);
 
                     listener.onResponse(new PagedResult<>(buckets.size(), synonymSetSummaries));
                 }
diff --git a/server/src/main/resources/META-INF/services/org.elasticsearch.features.FeatureSpecification b/server/src/main/resources/META-INF/services/org.elasticsearch.features.FeatureSpecification
@@ -13,6 +13,7 @@ org.elasticsearch.rest.action.admin.cluster.ClusterRerouteFeatures
 org.elasticsearch.index.IndexFeatures
 org.elasticsearch.index.mapper.MapperFeatures
 org.elasticsearch.search.SearchFeatures
+org.elasticsearch.synonyms.SynonymFeatures
 org.elasticsearch.search.retriever.RetrieversFeatures
 org.elasticsearch.script.ScriptFeatures
 org.elasticsearch.cluster.routing.RoutingFeatures
diff --git a/server/src/test/java/org/elasticsearch/search/sort/GeoDistanceSortBuilderTests.java b/server/src/test/java/org/elasticsearch/search/sort/GeoDistanceSortBuilderTests.java
@@ -23,6 +23,7 @@
 import org.elasticsearch.index.mapper.GeoPointFieldMapper;
 import org.elasticsearch.index.mapper.MappedFieldType;
 import org.elasticsearch.index.mapper.NestedPathFieldMapper;
+import org.elasticsearch.index.mapper.NumberFieldMapper;
 import org.elasticsearch.index.query.GeoValidationMethod;
 import org.elasticsearch.index.query.MatchAllQueryBuilder;
 import org.elasticsearch.index.query.MatchNoneQueryBuilder;
@@ -99,6 +100,9 @@ public static GeoDistanceSortBuilder randomGeoDistanceSortBuilder() {
 
     @Override
     protected MappedFieldType provideMappedFieldType(String name) {
+        if (name.equals("double")) {
+            return new NumberFieldMapper.NumberFieldType(name, NumberFieldMapper.NumberType.DOUBLE);
+        }
         return new GeoPointFieldMapper.GeoPointFieldType(name);
     }
 
@@ -531,6 +535,12 @@ public void testBuildInvalidPoints() throws IOException {
             );
             assertEquals("illegal longitude value [-360.0] for [GeoDistanceSort] for field [fieldName].", ex.getMessage());
         }
+        {
+            GeoDistanceSortBuilder sortBuilder = new GeoDistanceSortBuilder("double", 0.0, 180.0);
+            sortBuilder.validation(GeoValidationMethod.STRICT);
+            IllegalArgumentException ex = expectThrows(IllegalArgumentException.class, () -> sortBuilder.build(searchExecutionContext));
+            assertEquals("unable to apply geo distance sort to field [double] of type [double]", ex.getMessage());
+        }
     }
 
     /**
diff --git a/x-pack/plugin/async-search/build.gradle b/x-pack/plugin/async-search/build.gradle
diff --git a/x-pack/test/idp-fixture/src/main/resources/oidc/Dockerfile b/x-pack/test/idp-fixture/src/main/resources/oidc/Dockerfile

Original file line number	Diff line number	Diff line change
`@@ -282,6 +282,9 @@ tests:`
`282`	`282`	`- class: org.elasticsearch.index.engine.ThreadPoolMergeExecutorServiceDiskSpaceTests`
`283`	`283`	`method: testUnavailableBudgetBlocksNewMergeTasksFromStartingExecution`
`284`	`284`	`issue: https://github.com/elastic/elasticsearch/issues/130205`
	`285`	`+- class: org.elasticsearch.packaging.test.DockerTests`
	`286`	`+ method: test080ConfigurePasswordThroughEnvironmentVariableFile`
	`287`	`+ issue: https://github.com/elastic/elasticsearch/issues/128075`
`285`	`288`
`286`	`289`	`# Examples:`
`287`	`290`	`#`