Merge branch 'main' into pem-unwanted-test-tsrateit

Author: pabloem

TESTING.asciidoc

@@ -681,12 +681,15 @@ There are multiple base classes for tests:
   directly by unit tests.
 * **`ESSingleNodeTestCase`**: This test case sets up a cluster that has a
   single node.
-* **`ESIntegTestCase`**: An integration test case that creates a cluster that
-  might have multiple nodes.
-* **`ESRestTestCase`**: An integration tests that interacts with an external
-  cluster via the REST API. This is used for Java based REST tests.
-* **`ESClientYamlSuiteTestCase` **: A subclass of `ESRestTestCase` used to run
-  YAML based REST tests.
+* **`ESIntegTestCase`**: An internal integration test that starts nodes within the same JVM as the test.
+    These allow you to test functionality that is not exposed via the REST API, or for verifying a certain internal state.
+    Additionally, you can easily simulate tricky distributed setups that are difficult to do in REST tests.
+    If you only need to start one node, use `ESSingleNodeTestCase` instead, which is a much lighter test setup.
+* **`ESRestTestCase`**: An integration test that interacts with an external
+  cluster via the REST API. This is used for Java based REST tests. This should
+  be the first choice for writing integration tests as these tests run in a much more
+  realistic setup.
+* **`ESClientYamlSuiteTestCase` **: A subclass of `ESRestTestCase` used to run YAML based REST tests.
 
 === Good practices
 

docs/changelog/133369.yaml

@@ -0,0 +1,5 @@
+pr: 133369
+summary: Enable `date` `date_nanos` implicit casting
+area: ES|QL
+type: enhancement
+issues: []

docs/reference/query-languages/esql/esql-multi-index.md

@@ -135,6 +135,62 @@ FROM events_*
 | 2023-10-23T12:27:28.948Z | 172.21.2.113 | 2764889 | Connected to 10.1.0.2 |
 | 2023-10-23T12:15:03.360Z | 172.21.2.162 | 3450233 | Connected to 10.1.0.3 |
 
+### Date and date_nanos union type [esql-multi-index-date-date-nanos-union]
+```{applies_to}
+stack: ga 9.2.0
+```
+When the type of an {{esql}} field is a *union* of `date` and `date_nanos` across different indices, {{esql}} automatically casts all values to the `date_nanos` type during query execution. This implicit casting ensures that all values are handled with nanosecond precision, regardless of their original type. As a result, users can write queries against such fields without needing to perform explicit type conversions, and the query engine will seamlessly align the types for consistent and precise results.
+
+`date_nanos` fields offer higher precision but have a narrower range of valid values compared to `date` fields. This limits their representable dates roughly from 1970 to 2262. This is because dates are stored as a `long` representing nanoseconds since the epoch. When a field is mapped as both `date` and `date_nanos` across different indices, {{esql}} defaults to the more precise `date_nanos` type. This behavior ensures that no precision is lost when querying multiple indices with differing date field types. For dates that fall outside the valid range of `date_nanos` in fields that are mapped to both `date` and `date_nanos` across different indices, {{esql}} returns null by default. However, users can explicitly cast these fields to the `date` type to obtain a valid value, with precision limited to milliseconds.
+
+For example, if the `@timestamp` field is mapped as `date` in one index and `date_nanos` in another, {{esql}} will automatically treat all `@timestamp` values as `date_nanos` during query execution. This allows users to write queries that utilize the `@timestamp` field without encountering type mismatch errors, ensuring accurate time-based operations and comparisons across the combined dataset.
+
+**index: events_date**
+
+```
+{
+  "mappings": {
+    "properties": {
+      "@timestamp":     { "type": "date" },
+      "client_ip":      { "type": "ip" },
+      "event_duration": { "type": "long" },
+      "message":        { "type": "keyword" }
+    }
+  }
+}
+```
+
+**index: events_date_nanos**
+
+```
+{
+  "mappings": {
+    "properties": {
+      "@timestamp":     { "type": "date_nanos" },
+      "client_ip":      { "type": "ip" },
+      "event_duration": { "type": "long" },
+      "message":        { "type": "keyword" }
+    }
+  }
+}
+```
+
+```esql
+FROM events_date*
+| EVAL date = @timestamp::date
+| KEEP @timestamp, date, client_ip, event_duration, message
+| SORT date
+```
+
+| @timestamp:date_nanos | date:date   | client_ip:ip | event_duration:long | message:keyword |
+|--------------------------| --- |--------------|---------| --- |
+| null                          |1969-10-23T13:33:34.937Z| 172.21.0.5   | 1232382 |Disconnected|
+| 2023-10-23T12:15:03.360Z      |2023-10-23T12:15:03.360Z| 172.21.2.162 | 3450233 |Connected to 10.1.0.3|
+| 2023-10-23T12:15:03.360103847Z|2023-10-23T12:15:03.360Z| 172.22.2.162 | 3450233 |Connected to 10.1.0.3|
+| 2023-10-23T12:27:28.948Z      |2023-10-23T12:27:28.948Z| 172.22.2.113 | 2764889 |Connected to 10.1.0.2|
+| 2023-10-23T12:27:28.948Z      |2023-10-23T12:27:28.948Z| 172.21.2.113 | 2764889 |Connected to 10.1.0.2|
+| 2023-10-23T13:33:34.937193Z   |2023-10-23T13:33:34.937Z| 172.22.0.5   | 1232382 |Disconnected|
+| null                          |2263-10-23T13:51:54.732Z| 172.21.3.15  | 725448  |Connection error|
 
 ## Index metadata [esql-multi-index-index-metadata]
 

muted-tests.yml

@@ -513,12 +513,6 @@ tests:
 - class: org.elasticsearch.xpack.ml.integration.InferenceIT
   method: testInferClassificationModel
   issue: https://github.com/elastic/elasticsearch/issues/133448
-- class: org.elasticsearch.xpack.logsdb.LogsIndexingIT
-  method: testRouteOnSortFields
-  issue: https://github.com/elastic/elasticsearch/issues/133993
-- class: org.elasticsearch.xpack.logsdb.LogsIndexingIT
-  method: testShrink
-  issue: https://github.com/elastic/elasticsearch/issues/133875
 - class: org.elasticsearch.xpack.esql.action.CrossClusterQueryWithPartialResultsIT
   method: testPartialResults
   issue: https://github.com/elastic/elasticsearch/issues/131481
@@ -534,6 +528,9 @@ tests:
 - class: org.elasticsearch.xpack.cluster.routing.allocation.DataTierAllocationDeciderIT
   method: testShardsAreKeptInPreferredTierUntilTheNextTierIsInItsFinalState
   issue: https://github.com/elastic/elasticsearch/issues/134050
+- class: org.elasticsearch.cluster.ClusterInfoServiceIT
+  method: testMaxQueueLatenciesInClusterInfo
+  issue: https://github.com/elastic/elasticsearch/issues/134088
 
 # Examples:
 #

server/src/main/java/org/elasticsearch/TransportVersions.java

@@ -358,6 +358,7 @@ static TransportVersion def(int id) {
     public static final TransportVersion RESOLVE_INDEX_MODE_FILTER = def(9_149_0_00);
     public static final TransportVersion SEMANTIC_QUERY_MULTIPLE_INFERENCE_IDS = def(9_150_0_00);
     public static final TransportVersion ESQL_LOOKUP_JOIN_PRE_JOIN_FILTER = def(9_151_0_00);
+    public static final TransportVersion INFERENCE_API_DISABLE_EIS_RATE_LIMITING = def(9_152_0_00);
 
     /*
      * STOP! READ THIS FIRST! No, really,

server/src/main/java/org/elasticsearch/index/codec/PerFieldFormatSupplier.java

@@ -27,15 +27,37 @@
 import org.elasticsearch.index.mapper.Mapper;
 import org.elasticsearch.index.mapper.MapperService;
 import org.elasticsearch.index.mapper.SeqNoFieldMapper;
+import org.elasticsearch.index.mapper.TimeSeriesIdFieldMapper;
+import org.elasticsearch.index.mapper.TimeSeriesRoutingHashFieldMapper;
 import org.elasticsearch.index.mapper.vectors.DenseVectorFieldMapper;
 
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Set;
+
 /**
  * Class that encapsulates the logic of figuring out the most appropriate file format for a given field, across postings, doc values and
  * vectors.
  */
 public class PerFieldFormatSupplier {
 
-    private static final FeatureFlag SEQNO_FIELD_USE_TSDB_DOC_VALUES_FORMAT = new FeatureFlag("seqno_field_use_tsdb_doc_values_format");
+    static final FeatureFlag SEQNO_FIELD_USE_TSDB_DOC_VALUES_FORMAT = new FeatureFlag("seqno_field_use_tsdb_doc_values_format");
+    private static final Set<String> INCLUDE_META_FIELDS;
+
+    static {
+        // TODO: should we just allow all fields to use tsdb doc values codec?
+        // Avoid using tsdb codec for fields like _seq_no, _primary_term.
+        // But _tsid and _ts_routing_hash should always use the tsdb codec.
+        Set<String> includeMetaField = new HashSet<>(3);
+        includeMetaField.add(TimeSeriesIdFieldMapper.NAME);
+        includeMetaField.add(TimeSeriesRoutingHashFieldMapper.NAME);
+        if (SEQNO_FIELD_USE_TSDB_DOC_VALUES_FORMAT.isEnabled()) {
+            includeMetaField.add(SeqNoFieldMapper.NAME);
+        }
+        // Don't the include _recovery_source_size and _recovery_source fields, since their values can be trimmed away in
+        // RecoverySourcePruneMergePolicy, which leads to inconsistencies between merge stats and actual values.
+        INCLUDE_META_FIELDS = Collections.unmodifiableSet(includeMetaField);
+    }
 
     private static final DocValuesFormat docValuesFormat = new Lucene90DocValuesFormat();
     private static final KnnVectorsFormat knnVectorsFormat = new Lucene99HnswVectorsFormat();
@@ -126,13 +148,7 @@ boolean useTSDBDocValuesFormat(final String field) {
     }
 
     private boolean excludeFields(String fieldName) {
-        // TODO: should we just allow all fields to use tsdb doc values codec?
-        // Avoid using tsdb codec for fields like _seq_no, _primary_term.
-        // But _tsid and _ts_routing_hash should always use the tsdb codec.
-        return fieldName.startsWith("_")
-            && fieldName.equals("_tsid") == false
-            && fieldName.equals("_ts_routing_hash") == false
-            && (SEQNO_FIELD_USE_TSDB_DOC_VALUES_FORMAT.isEnabled() && fieldName.equals(SeqNoFieldMapper.NAME) == false);
+        return fieldName.startsWith("_") && INCLUDE_META_FIELDS.contains(fieldName) == false;
     }
 
     private boolean isTimeSeriesModeIndex() {

server/src/test/java/org/elasticsearch/index/codec/PerFieldMapperCodecTests.java

@@ -21,6 +21,10 @@
 import org.elasticsearch.index.codec.bloomfilter.ES87BloomFilterPostingsFormat;
 import org.elasticsearch.index.codec.postings.ES812PostingsFormat;
 import org.elasticsearch.index.mapper.MapperService;
+import org.elasticsearch.index.mapper.SeqNoFieldMapper;
+import org.elasticsearch.index.mapper.SourceFieldMapper;
+import org.elasticsearch.index.mapper.TimeSeriesIdFieldMapper;
+import org.elasticsearch.index.mapper.TimeSeriesRoutingHashFieldMapper;
 import org.elasticsearch.test.ESTestCase;
 
 import java.io.IOException;
@@ -201,6 +205,33 @@ public void testLogsIndexMode() throws IOException {
         assertThat((perFieldMapperCodec.useTSDBDocValuesFormat("response_size")), is(true));
     }
 
+    public void testMetaFields() throws IOException {
+        PerFieldFormatSupplier perFieldMapperCodec = createFormatSupplier(true, IndexMode.LOGSDB, MAPPING_3);
+        assertThat((perFieldMapperCodec.useTSDBDocValuesFormat(TimeSeriesIdFieldMapper.NAME)), is(true));
+        assertThat((perFieldMapperCodec.useTSDBDocValuesFormat(TimeSeriesRoutingHashFieldMapper.NAME)), is(true));
+        // See: PerFieldFormatSupplier why these fields shouldn't use tsdb codec
+        assertThat((perFieldMapperCodec.useTSDBDocValuesFormat(SourceFieldMapper.RECOVERY_SOURCE_NAME)), is(false));
+        assertThat((perFieldMapperCodec.useTSDBDocValuesFormat(SourceFieldMapper.RECOVERY_SOURCE_SIZE_NAME)), is(false));
+    }
+
+    public void testSeqnoField() throws IOException {
+        assumeTrue(
+            "seqno_field_use_tsdb_doc_values_format should be enabled",
+            PerFieldFormatSupplier.SEQNO_FIELD_USE_TSDB_DOC_VALUES_FORMAT.isEnabled()
+        );
+        PerFieldFormatSupplier perFieldMapperCodec = createFormatSupplier(true, IndexMode.LOGSDB, MAPPING_3);
+        assertThat((perFieldMapperCodec.useTSDBDocValuesFormat(SeqNoFieldMapper.NAME)), is(true));
+    }
+
+    public void testSeqnoFieldFeatureFlagDisabled() throws IOException {
+        assumeTrue(
+            "seqno_field_use_tsdb_doc_values_format should be disabled",
+            PerFieldFormatSupplier.SEQNO_FIELD_USE_TSDB_DOC_VALUES_FORMAT.isEnabled() == false
+        );
+        PerFieldFormatSupplier perFieldMapperCodec = createFormatSupplier(true, IndexMode.LOGSDB, MAPPING_3);
+        assertThat((perFieldMapperCodec.useTSDBDocValuesFormat(SeqNoFieldMapper.NAME)), is(false));
+    }
+
     private PerFieldFormatSupplier createFormatSupplier(boolean enableES87TSDBCodec, IndexMode mode, String mapping) throws IOException {
         Settings.Builder settings = Settings.builder();
         settings.put(IndexSettings.MODE.getKey(), mode);

x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/lucene/LuceneOperator.java

@@ -33,7 +33,6 @@
 
 import java.io.IOException;
 import java.io.UncheckedIOException;
-import java.util.Collection;
 import java.util.Collections;
 import java.util.HashSet;
 import java.util.List;
@@ -284,20 +283,14 @@ List<Object> tags() {
     public String toString() {
         StringBuilder sb = new StringBuilder();
         sb.append(this.getClass().getSimpleName()).append("[");
-        sb.append("shards = ").append(sortedUnion(processedShards, sliceQueue.remainingShardsIdentifiers()));
+        sb.append("shards = ")
+            .append(sliceQueue.partitioningStrategies().keySet().stream().sorted().collect(Collectors.joining(",", "[", "]")));
         sb.append(", maxPageSize = ").append(maxPageSize);
         describe(sb);
         sb.append("]");
         return sb.toString();
     }
 
-    private static Set<String> sortedUnion(Collection<String> a, Collection<String> b) {
-        var result = new TreeSet<String>();
-        result.addAll(a);
-        result.addAll(b);
-        return result;
-    }
-
     protected abstract void describe(StringBuilder sb);
 
     @Override

x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/lucene/LuceneSliceQueue.java

@@ -23,7 +23,6 @@
 import java.io.UncheckedIOException;
 import java.util.ArrayList;
 import java.util.Arrays;
-import java.util.Collection;
 import java.util.Collections;
 import java.util.Comparator;
 import java.util.HashMap;
@@ -178,17 +177,6 @@ public Map<String, PartitioningStrategy> partitioningStrategies() {
         return partitioningStrategies;
     }
 
-    public Collection<String> remainingShardsIdentifiers() {
-        List<String> remaining = new ArrayList<>(slices.length());
-        for (int i = 0; i < slices.length(); i++) {
-            LuceneSlice slice = slices.get(i);
-            if (slice != null) {
-                remaining.add(slice.shardContext().shardIdentifier());
-            }
-        }
-        return remaining;
-    }
-
     public static LuceneSliceQueue create(
         List<? extends ShardContext> contexts,
         Function<ShardContext, List<QueryAndTags>> queryFunction,

x-pack/plugin/esql/qa/server/single-node/src/javaRestTest/java/org/elasticsearch/xpack/esql/qa/single_node/RestEsqlIT.java

@@ -58,7 +58,6 @@
 import static org.elasticsearch.test.ListMatcher.matchesList;
 import static org.elasticsearch.test.MapMatcher.assertMap;
 import static org.elasticsearch.test.MapMatcher.matchesMap;
-import static org.elasticsearch.xpack.esql.action.EsqlCapabilities.Cap.IMPLICIT_CASTING_DATE_AND_DATE_NANOS;
 import static org.elasticsearch.xpack.esql.core.type.DataType.isMillisOrNanos;
 import static org.elasticsearch.xpack.esql.qa.rest.RestEsqlTestCase.Mode.SYNC;
 import static org.elasticsearch.xpack.esql.tools.ProfileParser.parseProfile;
@@ -715,9 +714,7 @@ public void testSuggestedCast() throws IOException {
                 Map<String, Object> results = entityAsMap(resp);
                 List<?> columns = (List<?>) results.get("columns");
                 DataType suggestedCast = DataType.suggestedCast(Set.of(listOfTypes.get(i), listOfTypes.get(j)));
-                if (IMPLICIT_CASTING_DATE_AND_DATE_NANOS.isEnabled()
-                    && isMillisOrNanos(listOfTypes.get(i))
-                    && isMillisOrNanos(listOfTypes.get(j))) {
+                if (isMillisOrNanos(listOfTypes.get(i)) && isMillisOrNanos(listOfTypes.get(j))) {
                     // datetime and date_nanos are casted to date_nanos implicitly
                     assertThat(columns, equalTo(List.of(Map.ofEntries(Map.entry("name", "my_field"), Map.entry("type", "date_nanos")))));
                 } else {