Merge branch 'main' into bugfix/address-source-confirmed-npe

Author: benwtrent

BUILDING.md

@@ -144,7 +144,7 @@ To wire this registered cluster into a `TestClusterAware` task (e.g. `RestIntegT
 Additional integration tests for a certain Elasticsearch modules that are specific to certain cluster configuration can be declared in a separate so called `qa` subproject of your module.
 
 The benefit of a dedicated project for these tests are:
-- `qa` projects are dedicated two specific use-cases and easier to maintain
+- `qa` projects are dedicated to specific use-cases and easier to maintain
 - It keeps the specific test logic separated from the common test logic.
 - You can run those tests in parallel to other projects of the build.
 

docs/changelog/125922.yaml

@@ -0,0 +1,5 @@
+pr: 125922 
+summary: Fix text structure NPE when fields in list have null value
+area: Machine Learning
+type: bug
+issues: []

docs/changelog/127229.yaml

@@ -0,0 +1,6 @@
+pr: 127229
+summary: Return BAD_REQUEST when a field scorer references a missing field
+area: Ranking
+type: bug
+issues:
+ - 127162

docs/reference/enrich-processor/date-processor.md

@@ -6,7 +6,6 @@ mapped_pages:
 
 # Date processor [date-processor]
 
-
 Parses dates from fields, and then uses the date or timestamp as the timestamp for the document. By default, the date processor adds the parsed date as a new field called `@timestamp`. You can specify a different field by setting the `target_field` configuration parameter. Multiple date formats are supported as part of the same date processor definition. They will be used sequentially to attempt parsing the date field, in the same order they were defined as part of the processor definition.
 
 $$$date-options$$$
@@ -16,7 +15,7 @@ $$$date-options$$$
 | `field` | yes | - | The field to get the date from. |
 | `target_field` | no | @timestamp | The field that will hold the parsed date. |
 | `formats` | yes | - | An array of the expected date formats. Can be a [java time pattern](/reference/elasticsearch/mapping-reference/mapping-date-format.md) or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N. |
-| `timezone` | no | UTC | The timezone to use when parsing the date. Supports [template snippets](docs-content://manage-data/ingest/transform-enrich/ingest-pipelines.md#template-snippets). |
+| `timezone` | no | UTC | The default [timezone](#date-processor-timezones) used by the processor. Supports [template snippets](docs-content://manage-data/ingest/transform-enrich/ingest-pipelines.md#template-snippets). |
 | `locale` | no | ENGLISH | The locale to use when parsing the date, relevant when parsing month names or week days. Supports [template snippets](docs-content://manage-data/ingest/transform-enrich/ingest-pipelines.md#template-snippets). |
 | `output_format` | no | `yyyy-MM-dd'T'HH:mm:ss.SSSXXX` | The format to use when writing the date to `target_field`. Must be a valid [java time pattern](/reference/elasticsearch/mapping-reference/mapping-date-format.md). |
 | `description` | no | - | Description of the processor. Useful for describing the purpose of the processor or its configuration. |
@@ -25,6 +24,20 @@ $$$date-options$$$
 | `on_failure` | no | - | Handle failures for the processor. See [Handling pipeline failures](docs-content://manage-data/ingest/transform-enrich/ingest-pipelines.md#handling-pipeline-failures). |
 | `tag` | no | - | Identifier for the processor. Useful for debugging and metrics. |
 
+## Timezones [date-processor-timezones]
+
+The `timezone` option may have two effects on the behavior of the processor:
+ - If the string being parsed matches a format representing a local date-time, such as `yyyy-MM-dd HH:mm:ss`, it will be assumed to be in the timezone specified by this option. This is not applicable if the string matches a format representing a zoned date-time, such as `yyyy-MM-dd HH:mm:ss zzz`: in that case, the timezone parsed from the string will be used. It is also not applicable if the string matches an absolute time format, such as `epoch_millis`.
+ - The date-time will be converted into the timezone given by this option before it is formatted and written into the target field. This is not applicable if the `output_format` is an absolute time format such as `epoch_millis`.
+
+::::{warning}
+We recommend avoiding the use of short abbreviations for timezone names, since they can be ambiguous. For example, one JDK might interpret `PST` as `America/Tijuana`, i.e. Pacific (Standard) Time, while another JDK might interpret it as `Asia/Manila`, i.e. Philippine Standard Time. If your input data contains such abbreviations, you should convert them into either standard full names or UTC offsets before parsing them, using your own knowledge of what each abbreviation means in your data. See [below](#date-processor-short-timezone-example) for an example. (This does not apply to `UTC`, which is safe.)
+::::
+
+## Examples [date-processor-examples]
+
+### Simple example [date-processor-simple-example]
+
 Here is an example that adds the parsed date to the `timestamp` field based on the `initial_date` field:
 
 ```js
@@ -43,6 +56,8 @@ Here is an example that adds the parsed date to the `timestamp` field based on t
 }
 ```
 
+### Example using templated parameters [date-processor-templated-example]
+
 The `timezone` and `locale` processor parameters are templated. This means that their values can be extracted from fields within documents. The example below shows how to extract the locale/timezone details from existing fields, `my_timezone` and `my_locale`, in the ingested document that contain the timezone and locale values.
 
 ```js
@@ -62,3 +77,49 @@ The `timezone` and `locale` processor parameters are templated. This means that
 }
 ```
 
+### Example dealing with short timezone abbreviations safely [date-processor-short-timezone-example]
+
+In the example below, the `message` field in the input is expected to be a string formed of a local date-time in `yyyyMMddHHmmss` format, a timezone abbreviated to one of `PST`, `CET`, or `JST` representing Pacific, Central European, or Japan time, and a payload. This field is split up using a `grok` processor, then the timezones are converted into full names using a `script` processor, then the date-time is parsed using a `date` processor, and finally the unwanted fields are discarded using a `drop` processor.
+
+```js
+{
+  "description" : "...",
+  "processors": [
+    {
+      "grok": {
+        "field": "message",
+        "patterns": ["%{DATESTAMP_EVENTLOG:local_date_time} %{TZ:short_tz} %{GREEDYDATA:payload}"],
+        "pattern_definitions": {
+          "TZ": "[A-Z]{3}"
+        }
+      }
+    },
+    {
+      "script": {
+        "source": "ctx['full_tz'] = params['tz_map'][ctx['short_tz']]",
+        "params": {
+          "tz_map": {
+            "PST": "America/Los_Angeles",
+            "CET": "Europe/Amsterdam",
+            "JST": "Asia/Tokyo"
+          }
+        }
+      }
+    },
+    {
+      "date": {
+        "field": "local_date_time",
+        "formats": ["yyyyMMddHHmmss"],
+        "timezone": "{{{full_tz}}}"
+      }
+    },
+    {
+      "remove": {
+        "field": ["message", "local_date_time", "short_tz", "full_tz"]
+      }
+    }
+  ]
+}
+```
+
+With this pipeline, a `message` field with the value `20250102123456 PST Hello world` will result in a `@timestamp` field with the value `2025-01-02T12:34:56.000-08:00` and a `payload` field with the value `Hello world`. (Note: A `@timestamp` field will normally be mapped to a `date` type, and therefore it will be indexed as an integer representing milliseconds since the epoch, although the original format and timezone may be preserved in the `_source`.)

modules/data-streams/src/internalClusterTest/java/org/elasticsearch/datastreams/DataStreamIT.java

@@ -202,8 +202,8 @@ public void testBasicScenario() throws Exception {
         int numDocsFoo = randomIntBetween(2, 16);
         indexDocs("metrics-foo", numDocsFoo);
 
-        verifyDocs("metrics-bar", numDocsBar, 1, 1);
-        verifyDocs("metrics-foo", numDocsFoo, 1, 1);
+        verifyDocs("metrics-bar", numDocsBar);
+        verifyDocs("metrics-foo", numDocsFoo);
 
         RolloverResponse fooRolloverResponse = indicesAdmin().rolloverIndex(new RolloverRequest("metrics-foo", null)).get();
         assertThat(fooRolloverResponse.getNewIndex(), backingIndexEqualTo("metrics-foo", 2));
@@ -234,8 +234,8 @@ public void testBasicScenario() throws Exception {
         int numDocsFoo2 = randomIntBetween(2, 16);
         indexDocs("metrics-foo", numDocsFoo2);
 
-        verifyDocs("metrics-bar", numDocsBar + numDocsBar2, 1, 2);
-        verifyDocs("metrics-foo", numDocsFoo + numDocsFoo2, 1, 2);
+        verifyDocs("metrics-bar", numDocsBar + numDocsBar2);
+        verifyDocs("metrics-foo", numDocsFoo + numDocsFoo2);
 
         DeleteDataStreamAction.Request deleteDataStreamRequest = new DeleteDataStreamAction.Request(TEST_REQUEST_TIMEOUT, "metrics-*");
         client().execute(DeleteDataStreamAction.INSTANCE, deleteDataStreamRequest).actionGet();
@@ -469,7 +469,7 @@ public void testComposableTemplateOnlyMatchingWithDataStreamName() throws Except
 
         int numDocs = randomIntBetween(2, 16);
         indexDocs(dataStreamName, numDocs);
-        verifyDocs(dataStreamName, numDocs, 1, 1);
+        verifyDocs(dataStreamName, numDocs);
 
         GetDataStreamAction.Request getDataStreamRequest = new GetDataStreamAction.Request(TEST_REQUEST_TIMEOUT, new String[] { "*" });
         GetDataStreamAction.Response getDataStreamResponse = client().execute(GetDataStreamAction.INSTANCE, getDataStreamRequest)
@@ -504,7 +504,7 @@ public void testComposableTemplateOnlyMatchingWithDataStreamName() throws Except
 
         int numDocs2 = randomIntBetween(2, 16);
         indexDocs(dataStreamName, numDocs2);
-        verifyDocs(dataStreamName, numDocs + numDocs2, 1, 2);
+        verifyDocs(dataStreamName, numDocs + numDocs2);
 
         getDataStreamRequest = new GetDataStreamAction.Request(TEST_REQUEST_TIMEOUT, new String[] { "*" });
         getDataStreamResponse = client().execute(GetDataStreamAction.INSTANCE, getDataStreamRequest).actionGet();
@@ -959,7 +959,7 @@ public void testAliasActionsFailOnDataStreamBackingIndices() throws Exception {
         );
         client().execute(CreateDataStreamAction.INSTANCE, createDataStreamRequest).get();
 
-        String backingIndex = DataStream.getDefaultBackingIndexName(dataStreamName, 1);
+        String backingIndex = getDataStreamBackingIndexNames(dataStreamName).getFirst();
         AliasActions addAction = new AliasActions(AliasActions.Type.ADD).index(backingIndex).aliases("first_gen");
         IndicesAliasesRequest aliasesAddRequest = new IndicesAliasesRequest(TEST_REQUEST_TIMEOUT, TEST_REQUEST_TIMEOUT);
         aliasesAddRequest.addAliasAction(addAction);
@@ -2050,11 +2050,8 @@ static void verifyDocs(String dataStream, long expectedNumHits, List<String> exp
         });
     }
 
-    static void verifyDocs(String dataStream, long expectedNumHits, long minGeneration, long maxGeneration) {
-        List<String> expectedIndices = new ArrayList<>();
-        for (long k = minGeneration; k <= maxGeneration; k++) {
-            expectedIndices.add(DataStream.getDefaultBackingIndexName(dataStream, k));
-        }
+    static void verifyDocs(String dataStream, long expectedNumHits) {
+        List<String> expectedIndices = getDataStreamBackingIndexNames(dataStream);
         verifyDocs(dataStream, expectedNumHits, expectedIndices);
     }
 

modules/data-streams/src/internalClusterTest/java/org/elasticsearch/datastreams/DataStreamsSnapshotsIT.java

@@ -35,8 +35,8 @@
 import org.elasticsearch.action.support.IndicesOptions;
 import org.elasticsearch.action.support.master.AcknowledgedResponse;
 import org.elasticsearch.client.internal.Client;
-import org.elasticsearch.cluster.metadata.DataStream;
 import org.elasticsearch.cluster.metadata.DataStreamAlias;
+import org.elasticsearch.cluster.metadata.DataStreamTestHelper;
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.common.unit.ByteSizeUnit;
 import org.elasticsearch.index.Index;
@@ -301,8 +301,8 @@ public void testSnapshotAndRestoreInPlace() {
         RolloverRequest rolloverRequest = new RolloverRequest("ds", null);
         RolloverResponse rolloverResponse = client.admin().indices().rolloverIndex(rolloverRequest).actionGet();
         assertThat(rolloverResponse.isRolledOver(), is(true));
-        String backingIndexAfterSnapshot = DataStream.getDefaultBackingIndexName("ds", 2);
-        assertThat(rolloverResponse.getNewIndex(), equalTo(backingIndexAfterSnapshot));
+        String backingIndexAfterSnapshot = getDataStreamBackingIndexNames("ds").getFirst();
+        assertThat(rolloverResponse.getNewIndex(), DataStreamTestHelper.backingIndexEqualTo("ds", 2));
 
         // Close all backing indices of ds data stream:
         CloseIndexRequest closeIndexRequest = new CloseIndexRequest(".ds-ds-*");
@@ -337,7 +337,7 @@ public void testSnapshotAndRestoreInPlace() {
         rolloverRequest = new RolloverRequest("ds", null);
         rolloverResponse = client.admin().indices().rolloverIndex(rolloverRequest).actionGet();
         assertThat(rolloverResponse.isRolledOver(), is(true));
-        assertThat(rolloverResponse.getNewIndex(), equalTo(DataStream.getDefaultBackingIndexName("ds", 3)));
+        assertThat(rolloverResponse.getNewIndex(), DataStreamTestHelper.backingIndexEqualTo("ds", 3));
     }
 
     public void testFailureStoreSnapshotAndRestore() {
@@ -915,7 +915,7 @@ public void testDataStreamAndBackingIndicesAreRenamedUsingRegex() {
         List<GetDataStreamAction.Response.DataStreamInfo> dataStreamInfos = getDataStreamInfo("test-ds");
         assertThat(
             dataStreamInfos.get(0).getDataStream().getIndices().get(0).getName(),
-            is(DataStream.getDefaultBackingIndexName("test-ds", 1L))
+            DataStreamTestHelper.backingIndexEqualTo("test-ds", 1)
         );
 
         // data stream "ds" should still exist in the system

plugins/examples/rescore/src/yamlRestTest/resources/rest-api-spec/test/example-rescore/30_factor_field.yml

@@ -48,6 +48,24 @@ setup:
   - match: { hits.hits.1._score: 20 }
   - match: { hits.hits.2._score: 10 }
 
+---
+"referencing a missing field returns bad request":
+  - requires:
+      cluster_features: [ "search.rescorer.missing.field.bad.request" ]
+      reason: "Testing the behaviour change with this feature"
+  - do:
+      catch: bad_request
+      search:
+        index: test
+        body:
+          rescore:
+            example:
+              factor: 1
+              factor_field: missing
+  - match: { status: 400 }
+  - match: { error.root_cause.0.type: "illegal_argument_exception" }
+  - match: { error.root_cause.0.reason: "Missing value for field [missing]" }
+
 ---
 "sorted based on a numeric field and rescored based on a factor field using a window size":
   - do:

qa/smoke-test-http/src/internalClusterTest/java/org/elasticsearch/http/SearchErrorTraceIT.java

@@ -127,33 +127,7 @@ public void testSearchFailingQueryErrorTraceFalse() throws IOException {
         assertFalse(hasStackTrace.getAsBoolean());
     }
 
-    public void testDataNodeDoesNotLogStackTraceWhenErrorTraceTrue() throws IOException {
-        setupIndexWithDocs();
-
-        Request searchRequest = new Request("POST", "/_search");
-        searchRequest.setJsonEntity("""
-            {
-                "query": {
-                    "simple_query_string" : {
-                        "query": "foo",
-                        "fields": ["field"]
-                    }
-                }
-            }
-            """);
-
-        String errorTriggeringIndex = "test2";
-        int numShards = getNumShards(errorTriggeringIndex).numPrimaries;
-        try (var mockLog = MockLog.capture(SearchService.class)) {
-            ErrorTraceHelper.addUnseenLoggingExpectations(numShards, mockLog, errorTriggeringIndex);
-
-            searchRequest.addParameter("error_trace", "true");
-            getRestClient().performRequest(searchRequest);
-            mockLog.assertAllExpectationsMatched();
-        }
-    }
-
-    public void testDataNodeLogsStackTraceWhenErrorTraceFalseOrEmpty() throws IOException {
+    public void testDataNodeLogsStackTrace() throws IOException {
         setupIndexWithDocs();
 
         Request searchRequest = new Request("POST", "/_search");
@@ -173,10 +147,14 @@ public void testDataNodeLogsStackTraceWhenErrorTraceFalseOrEmpty() throws IOExce
         try (var mockLog = MockLog.capture(SearchService.class)) {
             ErrorTraceHelper.addSeenLoggingExpectations(numShards, mockLog, errorTriggeringIndex);
 
-            // error_trace defaults to false so we can test both cases with some randomization
-            if (randomBoolean()) {
+            // No matter the value of error_trace (empty, true, or false) we should see stack traces logged
+            int errorTraceValue = randomIntBetween(0, 2);
+            if (errorTraceValue == 0) {
+                searchRequest.addParameter("error_trace", "true");
+            } else if (errorTraceValue == 1) {
                 searchRequest.addParameter("error_trace", "false");
-            }
+            } // else empty
+
             getRestClient().performRequest(searchRequest);
             mockLog.assertAllExpectationsMatched();
         }
@@ -233,32 +211,7 @@ public void testMultiSearchFailingQueryErrorTraceFalse() throws IOException {
         assertFalse(hasStackTrace.getAsBoolean());
     }
 
-    public void testDataNodeDoesNotLogStackTraceWhenErrorTraceTrueMultiSearch() throws IOException {
-        setupIndexWithDocs();
-
-        XContentType contentType = XContentType.JSON;
-        MultiSearchRequest multiSearchRequest = new MultiSearchRequest().add(
-            new SearchRequest("test*").source(new SearchSourceBuilder().query(simpleQueryStringQuery("foo").field("field")))
-        );
-        Request searchRequest = new Request("POST", "/_msearch");
-        byte[] requestBody = MultiSearchRequest.writeMultiLineFormat(multiSearchRequest, contentType.xContent());
-        searchRequest.setEntity(
-            new NByteArrayEntity(requestBody, ContentType.create(contentType.mediaTypeWithoutParameters(), (Charset) null))
-        );
-
-        searchRequest.addParameter("error_trace", "true");
-
-        String errorTriggeringIndex = "test2";
-        int numShards = getNumShards(errorTriggeringIndex).numPrimaries;
-        try (var mockLog = MockLog.capture(SearchService.class)) {
-            ErrorTraceHelper.addUnseenLoggingExpectations(numShards, mockLog, errorTriggeringIndex);
-
-            getRestClient().performRequest(searchRequest);
-            mockLog.assertAllExpectationsMatched();
-        }
-    }
-
-    public void testDataNodeLogsStackTraceWhenErrorTraceFalseOrEmptyMultiSearch() throws IOException {
+    public void testDataNodeLogsStackTraceMultiSearch() throws IOException {
         setupIndexWithDocs();
 
         XContentType contentType = XContentType.JSON;
@@ -271,16 +224,19 @@ public void testDataNodeLogsStackTraceWhenErrorTraceFalseOrEmptyMultiSearch() th
             new NByteArrayEntity(requestBody, ContentType.create(contentType.mediaTypeWithoutParameters(), (Charset) null))
         );
 
-        // error_trace defaults to false so we can test both cases with some randomization
-        if (randomBoolean()) {
-            searchRequest.addParameter("error_trace", "false");
-        }
-
         String errorTriggeringIndex = "test2";
         int numShards = getNumShards(errorTriggeringIndex).numPrimaries;
         try (var mockLog = MockLog.capture(SearchService.class)) {
             ErrorTraceHelper.addSeenLoggingExpectations(numShards, mockLog, errorTriggeringIndex);
 
+            // No matter the value of error_trace (empty, true, or false) we should see stack traces logged
+            int errorTraceValue = randomIntBetween(0, 2);
+            if (errorTraceValue == 0) {
+                searchRequest.addParameter("error_trace", "true");
+            } else if (errorTraceValue == 1) {
+                searchRequest.addParameter("error_trace", "false");
+            } // else empty
+
             getRestClient().performRequest(searchRequest);
             mockLog.assertAllExpectationsMatched();
         }