Merge branch 'main' into lucene_snapshot

Author: thecoop

TESTING.asciidoc

@@ -681,12 +681,15 @@ There are multiple base classes for tests:
   directly by unit tests.
 * **`ESSingleNodeTestCase`**: This test case sets up a cluster that has a
   single node.
-* **`ESIntegTestCase`**: An integration test case that creates a cluster that
-  might have multiple nodes.
-* **`ESRestTestCase`**: An integration tests that interacts with an external
-  cluster via the REST API. This is used for Java based REST tests.
-* **`ESClientYamlSuiteTestCase` **: A subclass of `ESRestTestCase` used to run
-  YAML based REST tests.
+* **`ESIntegTestCase`**: An internal integration test that starts nodes within the same JVM as the test.
+    These allow you to test functionality that is not exposed via the REST API, or for verifying a certain internal state.
+    Additionally, you can easily simulate tricky distributed setups that are difficult to do in REST tests.
+    If you only need to start one node, use `ESSingleNodeTestCase` instead, which is a much lighter test setup.
+* **`ESRestTestCase`**: An integration test that interacts with an external
+  cluster via the REST API. This is used for Java based REST tests. This should
+  be the first choice for writing integration tests as these tests run in a much more
+  realistic setup.
+* **`ESClientYamlSuiteTestCase` **: A subclass of `ESRestTestCase` used to run YAML based REST tests.
 
 === Good practices
 

docs/changelog/133166.yaml

@@ -0,0 +1,6 @@
+pr: 133166
+summary: Improve Expanding Lookup Join performance by pushing a filter to the right
+  side of the lookup join
+area: ES|QL
+type: enhancement
+issues: []

docs/changelog/133369.yaml

@@ -0,0 +1,5 @@
+pr: 133369
+summary: Enable `date` `date_nanos` implicit casting
+area: ES|QL
+type: enhancement
+issues: []

docs/changelog/133793.yaml

@@ -0,0 +1,6 @@
+pr: 133793
+summary: "[Sentinel One] Add `manage`, `create_index`, `read`, `index`, `write`, `delete`, permission for third party agent indices `kibana_system`"
+area: Authorization
+type: enhancement
+issues:
+  - 133703

docs/changelog/133897.yaml

@@ -0,0 +1,6 @@
+pr: 133897
+summary: "Runtime fields: pass down runtime field name as source filter when source\
+  \ mode is synthetic"
+area: "Mapping"
+type: enhancement
+issues: []

docs/reference/query-languages/esql/_snippets/functions/description/knn.md

@@ -135,6 +135,62 @@ FROM events_*
 | 2023-10-23T12:27:28.948Z | 172.21.2.113 | 2764889 | Connected to 10.1.0.2 |
 | 2023-10-23T12:15:03.360Z | 172.21.2.162 | 3450233 | Connected to 10.1.0.3 |
 
+### Date and date_nanos union type [esql-multi-index-date-date-nanos-union]
+```{applies_to}
+stack: ga 9.2.0
+```
+When the type of an {{esql}} field is a *union* of `date` and `date_nanos` across different indices, {{esql}} automatically casts all values to the `date_nanos` type during query execution. This implicit casting ensures that all values are handled with nanosecond precision, regardless of their original type. As a result, users can write queries against such fields without needing to perform explicit type conversions, and the query engine will seamlessly align the types for consistent and precise results.
+
+`date_nanos` fields offer higher precision but have a narrower range of valid values compared to `date` fields. This limits their representable dates roughly from 1970 to 2262. This is because dates are stored as a `long` representing nanoseconds since the epoch. When a field is mapped as both `date` and `date_nanos` across different indices, {{esql}} defaults to the more precise `date_nanos` type. This behavior ensures that no precision is lost when querying multiple indices with differing date field types. For dates that fall outside the valid range of `date_nanos` in fields that are mapped to both `date` and `date_nanos` across different indices, {{esql}} returns null by default. However, users can explicitly cast these fields to the `date` type to obtain a valid value, with precision limited to milliseconds.
+
+For example, if the `@timestamp` field is mapped as `date` in one index and `date_nanos` in another, {{esql}} will automatically treat all `@timestamp` values as `date_nanos` during query execution. This allows users to write queries that utilize the `@timestamp` field without encountering type mismatch errors, ensuring accurate time-based operations and comparisons across the combined dataset.
+
+**index: events_date**
+
+```
+{
+  "mappings": {
+    "properties": {
+      "@timestamp":     { "type": "date" },
+      "client_ip":      { "type": "ip" },
+      "event_duration": { "type": "long" },
+      "message":        { "type": "keyword" }
+    }
+  }
+}
+```
+
+**index: events_date_nanos**
+
+```
+{
+  "mappings": {
+    "properties": {
+      "@timestamp":     { "type": "date_nanos" },
+      "client_ip":      { "type": "ip" },
+      "event_duration": { "type": "long" },
+      "message":        { "type": "keyword" }
+    }
+  }
+}
+```
+
+```esql
+FROM events_date*
+| EVAL date = @timestamp::date
+| KEEP @timestamp, date, client_ip, event_duration, message
+| SORT date
+```
+
+| @timestamp:date_nanos | date:date   | client_ip:ip | event_duration:long | message:keyword |
+|--------------------------| --- |--------------|---------| --- |
+| null                          |1969-10-23T13:33:34.937Z| 172.21.0.5   | 1232382 |Disconnected|
+| 2023-10-23T12:15:03.360Z      |2023-10-23T12:15:03.360Z| 172.21.2.162 | 3450233 |Connected to 10.1.0.3|
+| 2023-10-23T12:15:03.360103847Z|2023-10-23T12:15:03.360Z| 172.22.2.162 | 3450233 |Connected to 10.1.0.3|
+| 2023-10-23T12:27:28.948Z      |2023-10-23T12:27:28.948Z| 172.22.2.113 | 2764889 |Connected to 10.1.0.2|
+| 2023-10-23T12:27:28.948Z      |2023-10-23T12:27:28.948Z| 172.21.2.113 | 2764889 |Connected to 10.1.0.2|
+| 2023-10-23T13:33:34.937193Z   |2023-10-23T13:33:34.937Z| 172.22.0.5   | 1232382 |Disconnected|
+| null                          |2263-10-23T13:51:54.732Z| 172.21.3.15  | 725448  |Connection error|
 
 ## Index metadata [esql-multi-index-index-metadata]