Merge branch 'main' into share-histo-serialization

Author: JonasKunz

.claude/settings.local.json

@@ -1,4 +1,7 @@
 
+# claude
+.claude
+
 # intellij files
 .idea/
 *.iml

.gitignore

@@ -0,0 +1,5 @@
+pr: 129693
+summary: Add top level normalizer for linear retriever
+area: Search
+type: enhancement
+issues: []

bin/chezmoi

@@ -31,9 +31,16 @@ Combining `query` and `retrievers` is not supported.
 `normalizer` {applies_to}`stack: ga 9.1`
 :   (Optional, String)
 
-    The normalizer to use when using the [multi-field query format](../retrievers.md#multi-field-query-format).
+    The top-level normalizer to use when combining results.
     See [normalizers](#linear-retriever-normalizers) for supported values.
     Required when `query` is specified.
+    
+    When used with the [multi-field query format](../retrievers.md#multi-field-query-format) (`query` parameter), normalizes scores per [field grouping](../retrievers.md#multi-field-field-grouping).
+    Otherwise serves as the default normalizer for any sub-retriever that doesn't specify its own normalizer. Per-retriever normalizers always take precedence over the top-level normalizer.
+
+    :::{note}
+    **Top-level normalizer support for sub-retrievers**: The ability to use a top-level normalizer as a default for sub-retrievers was introduced in Elasticsearch 9.2+. In earlier versions, only per-retriever normalizers are supported.
+    :::
 
     ::::{warning}
     Avoid using `none` as that will disable normalization and may bias the result set towards lexical matches.
@@ -74,9 +81,10 @@ Each entry in the `retrievers` array specifies the following parameters:
 `normalizer`
 :   (Optional, String)
 
-    Specifies how the retriever’s score will be normalized before applying the specified `weight`.
+    Specifies how the retriever's score will be normalized before applying the specified `weight`.
     See [normalizers](#linear-retriever-normalizers) for supported values.
-    Defaults to `none`.
+    If not specified, uses the top-level `normalizer` or defaults to `none` if no top-level normalizer is set.
+    {applies_to}`stack: ga 9.2`
 
 See also [this hybrid search example](retrievers-examples.md#retrievers-examples-linear-retriever) using a linear retriever on how to independently configure and apply normalizers to retrievers.
 
@@ -94,7 +102,7 @@ The `linear` retriever supports the following normalizers:
 
 ## Example
 
-This example of a hybrid search weights KNN results five times more heavily than BM25 results in the final ranking.
+This example of a hybrid search weights KNN results five times more heavily than BM25 results in the final ranking, with a top-level normalizer applied to all retrievers.
 
 ```console
 GET my_index/_search
@@ -105,23 +113,33 @@ GET my_index/_search
         {
           "retriever": {
             "knn": {
-              ...
+              "field": "title_vector",
+              "query_vector": [0.1, 0.2, 0.3],
+              "k": 10,
+              "num_candidates": 100
             }
           },
           "weight": 5 # KNN query weighted 5x
         },
         {
           "retriever": {
             "standard": {
-              ...
+              "query": {
+                "match": {
+                  "title": "elasticsearch"
+                }
+              }
             }
           },
           "weight": 1.5 # BM25 query weighted 1.5x
         }
-      ]
+      ],
+      "normalizer": "minmax"
     }
   }
 }
 ```
 
+In this example, the `minmax` normalizer is applied to both the kNN retriever and the standard retriever. The top-level normalizer serves as a default that can be overridden by individual sub-retrievers. When using the multi-field query format, the top-level normalizer is applied to all generated inner retrievers.
+
 See also [this hybrid search example](retrievers-examples.md#retrievers-examples-linear-retriever).

docs/changelog/129693.yaml

@@ -16,16 +16,21 @@ $$$query-rules$$$
 * Personalized metadata about users (e.g. country, language, etc)
 * A particular topic
 * A referring site
-* etc.
+
 
 Query rules define a metadata key that will be used to match the metadata provided in the [rule retriever](/reference/elasticsearch/rest-apis/retrievers/rule-retriever.md) with the criteria specified in the rule.
 
 When a query rule matches the rule metadata according to its defined criteria, the query rule action is applied to the underlying `organic` query.
 
 For example, a query rule could be defined to match a user-entered query string of `pugs` and a country `us` and promote adoptable shelter dogs if the rule query met both criteria.
 
-Rules are defined using the [query rules API](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-query_rules) and searched using the [rule retriever](/reference/elasticsearch/rest-apis/retrievers/rule-retriever.md) or the [rule query](/reference/query-languages/query-dsl/query-dsl-rule-query.md).
+You can create and manage query rules using either:
+- [Query rules API]({{es-apis}}v9/group/endpoint-query_rules)
+- [Query Rules UI](docs-content://solutions/search/query-rules-ui.md)
 
+You can search with query rules using either:
+- [Retrievers syntax](/reference/elasticsearch/rest-apis/retrievers/rule-retriever.md)
+- [Query DSL syntax](/reference/query-languages/query-dsl/query-dsl-rule-query.md)
 
 ## Rule definition [query-rule-definition]
 
@@ -68,7 +73,7 @@ The actions to take when the rule matches a query:
 Use `ids` when searching over a single index, and `docs` when searching over multiple indices. `ids` and `docs` cannot be combined in the same query.
 
 
-## Add query rules [add-query-rules]
+## Manage query rules [manage-query-rules]
 
 You can add query rules using the [Create or update query ruleset](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-query-rules-put-ruleset) call. This adds a ruleset containing one or more query rules that will be applied to queries that match their specified criteria.
 
@@ -145,6 +150,8 @@ There is a limit of 100 rules per ruleset. This can be increased up to 1000 usin
 
 You can use the [Get query ruleset](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-query-rules-get-ruleset) call to retrieve the ruleset you just created, the [List query rulesets](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-query-rules-list-rulesets) call to retrieve a summary of all query rulesets, and the [Delete query ruleset](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-query-rules-delete-ruleset) call to delete a query ruleset.
 
+To manage rules using the Query Rules UI, refer to [Manage query rules](https://www.elastic.co/docs/solutions/search/query-rules-ui#manage-existing-rules).
+
 
 ## Search using query rules [rule-query-search]
 

docs/reference/elasticsearch/rest-apis/retrievers/linear-retriever.md

@@ -108,7 +108,8 @@ public static FieldType tryParse(String name) {
             case "wildcard" -> FieldType.WILDCARD;
             case "passthrough" -> FieldType.PASSTHROUGH;
             case "match_only_text" -> FieldType.MATCH_ONLY_TEXT;
-            default -> throw new IllegalArgumentException("Unknown field type: " + name);
+            // Custom types will fail to parse and will return null
+            default -> null;
         };
     }
 

docs/reference/elasticsearch/rest-apis/searching-with-query-rules.md

@@ -17,6 +17,7 @@
 import org.elasticsearch.plugins.Plugin;
 import org.elasticsearch.xcontent.XContentBuilder;
 import org.junit.AssumptionViolatedException;
+import org.junit.Before;
 
 import java.io.IOException;
 import java.util.ArrayList;
@@ -39,6 +40,14 @@
 
 public class ExponentialHistogramFieldMapperTests extends MapperTestCase {
 
+    @Before
+    public void setup() {
+        assumeTrue(
+            "Only when exponential_histogram feature flag is enabled",
+            ExponentialHistogramFieldMapper.EXPONENTIAL_HISTOGRAM_FEATURE.isEnabled()
+        );
+    }
+
     protected Collection<? extends Plugin> getPlugins() {
         return Collections.singletonList(new ExponentialHistogramMapperPlugin());
     }

test/framework/src/main/java/org/elasticsearch/datageneration/FieldType.java

@@ -10,9 +10,11 @@
 import com.carrotsearch.randomizedtesting.annotations.Name;
 import com.carrotsearch.randomizedtesting.annotations.ParametersFactory;
 
+import org.elasticsearch.Build;
 import org.elasticsearch.test.cluster.ElasticsearchCluster;
 import org.elasticsearch.test.rest.yaml.ClientYamlTestCandidate;
 import org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase;
+import org.junit.Before;
 import org.junit.ClassRule;
 
 public class ExponentialHistogramYamlTestSuiteIT extends ESClientYamlSuiteTestCase {
@@ -21,6 +23,12 @@ public ExponentialHistogramYamlTestSuiteIT(@Name("yaml") ClientYamlTestCandidate
         super(testCandidate);
     }
 
+    @Before
+    public void setup() {
+        // TODO: remove when FeatureFlag is removed and add minimum required version to yaml spec
+        assumeTrue("Only when exponential_histogram feature flag is enabled", Build.current().isSnapshot());
+    }
+
     @ParametersFactory
     public static Iterable<Object[]> parameters() throws Exception {
         return ESClientYamlSuiteTestCase.createParameters();

x-pack/plugin/mapper-exponential-histogram/src/test/java/org/elasticsearch/xpack/exponentialhistogram/ExponentialHistogramFieldMapperTests.java

@@ -835,4 +835,37 @@ public XContentBuilder toXContent(XContentBuilder builder, Params params) throws
         );
         assertThat(numAsyncCalls.get(), equalTo(4));
     }
+
+    public void testMixedNormalizerInheritance() throws IOException {
+        client().prepareIndex(INDEX).setId("1").setSource("field1", "elasticsearch only", "field2", "no technology here").get();
+        client().prepareIndex(INDEX).setId("2").setSource("field1", "no elasticsearch", "field2", "technology only").get();
+        client().prepareIndex(INDEX).setId("3").setSource("field1", "search term", "field2", "no technology").get();
+        refresh(INDEX);
+
+        LinearRetrieverBuilder linearRetriever = new LinearRetrieverBuilder(
+            List.of(
+                CompoundRetrieverBuilder.RetrieverSource.from(
+                    new StandardRetrieverBuilder(QueryBuilders.matchQuery("field1", "elasticsearch"))
+                ),
+                CompoundRetrieverBuilder.RetrieverSource.from(
+                    new StandardRetrieverBuilder(QueryBuilders.matchQuery("field2", "technology"))
+                ),
+                CompoundRetrieverBuilder.RetrieverSource.from(new StandardRetrieverBuilder(QueryBuilders.matchQuery("field1", "search")))
+            ),
+            null,
+            null,
+            MinMaxScoreNormalizer.INSTANCE,
+            10,
+            new float[] { 1.0f, 1.0f, 1.0f },
+            new ScoreNormalizer[] { null, L2ScoreNormalizer.INSTANCE, null }
+        );
+
+        assertThat(linearRetriever.getNormalizers()[0], equalTo(MinMaxScoreNormalizer.INSTANCE));
+        assertThat(linearRetriever.getNormalizers()[1], equalTo(L2ScoreNormalizer.INSTANCE));
+        assertThat(linearRetriever.getNormalizers()[2], equalTo(MinMaxScoreNormalizer.INSTANCE));
+
+        assertResponse(client().prepareSearch(INDEX).setSource(new SearchSourceBuilder().retriever(linearRetriever)), searchResponse -> {
+            assertThat(searchResponse.getHits().getTotalHits().value(), equalTo(3L));
+        });
+    }
 }