Merge branch 'main' into date-nanos-implicit-casting-behind-snapshot

Author: fang-xing-esql

docs/changelog/128866.yaml

@@ -0,0 +1,6 @@
+pr: 128866
+summary: Add `age_in_millis` to ILM Explain Response
+area: ILM+SLM
+type: enhancement
+issues:
+ - 103659

docs/reference/query-languages/esql/README.md

@@ -105,16 +105,95 @@ To help differentiate between the static and generated content, the generated co
 % This is generated by ESQL's AbstractFunctionTestCase. Do no edit it. See ../README.md for how to regenerate it.
 ```
 
+## Version differentiation in Docs V3
+
+> [!IMPORTANT]
+> Starting with 9.0, we no longer publish separate documentation branches for every minor release (`9.0`, `9.1`, `9.2`, etc.). 
+> This means there won't be a different page for `9.1`, `9.2`, and so on. Instead, all changes landing in subsequent minor releases **will appear on the same page**.
+
+Because we now publish just one docs set off of the `main` branch, we use the [`applies_to` metadata](https://elastic.github.io/docs-builder/syntax/applies/) to differentiate features and their availability across different versions. This is a [cumulative approach](https://elastic.github.io/docs-builder/contribute/#cumulative-docs): instead of creating separate pages for each product and release, we update a **single page** with product- and version-specific details over time.
+
+`applies_to` allows us to clearly communicate when features are introduced, when they transition from preview to GA, and which versions support specific functionality.
+
+This metadata accepts a lifecycle and an optional version.
+
+### Functions and operators
+
+Use the `@FunctionAppliesTo` annotation within the `@FunctionInfo` annotation on function and operator classes to specify the lifecycle and version for functions and operators.
+
+For example, to indicate that a function is in technical preview and applies to version 9.0.0, you would use:
+
+```java
+@FunctionInfo(
+    returnType = "boolean",
+    appliesTo = {
+        @FunctionAppliesTo(lifeCycle = FunctionAppliesToLifecycle.PREVIEW, version = "9.0.0")
+    },
+    ...
+)
+```
+
+When a feature evolves from preview in `9.0` to GA in `9.2`, add a new entry alongside the existing preview entry and remove the `preview = true` boolean:
+
+```java
+@FunctionInfo(
+    returnType = "boolean",
+    preview = false, //  the preview boolean can be removed (or flipped to false) when the function becomes GA
+    appliesTo = {
+        @FunctionAppliesTo(lifeCycle = FunctionAppliesToLifecycle.PREVIEW, version = "9.0.0"),
+        @FunctionAppliesTo(lifeCycle = FunctionAppliesToLifecycle.GA, version = "9.2.0")
+    },
+    ...
+)
+```
+
+We updated [`DocsV3Support.java`](https://github.com/elastic/elasticsearch/blob/main/x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/expression/function/DocsV3Support.java) to generate the `applies_to` metadata correctly for functions and operators.
+
+### Inline `applies_to` metadata
+
+Use [inline annotations](https://elastic.github.io/docs-builder/syntax/applies/#inline-annotations) to specify `applies_to` metadata in descriptions, parameter lists, etc.
+
+For example, the second item in this list is in technical preview as of version 9.2:
+
+```markdown
+- Item 1
+- Item 2 {applies_to}`stack: preview 9.2.`
+```
+
+### Key rules
+
+1. **Use the `preview = true` boolean** for any tech preview feature - this is required for the Kibana inline docs
+   - **Remove `preview = true`** only when the feature becomes GA on serverless and is _definitely_ going GA in the next minor release
+2. **Never delete `appliesTo` entries** - only add new ones as features evolve from preview to GA
+3. **Use specific versions** (`9.0.0`, `9.1.0`) when known, or just `PREVIEW` without a version if timing is uncertain
+4. **Add `applies_to` to examples** where necessary
+
+> [!IMPORTANT]
+> We don't use `applies_to` in the legacy asciidoc system for 8.x and earlier versions.
+
+### Supported lifecycles
+
+- `PREVIEW` - Feature is in technical preview
+- `GA` - Feature is generally available  
+- `DEPRECATED` - Feature is deprecated and will be removed in a future release
+- `UNAVAILABLE` - Feature is not available in the current version, but may be available in future releases
+
+> [!NOTE] 
+> Unreleased version information is automatically sanitized in the docs build output. For example, say you specify `preview 9.3.0`: 
+> - Before `9.3.0` is released, the live documentation will display "Planned for a future release" instead of the specific version number. 
+>  - This will be updated automatically when the version is released.
+
 ## Tutorials
 
 ### Adding a new command
 
 When adding a new command, for example adding the `CHANGE_POINT` command, do the following:
 1. Create a new file in the `_snippets/commands/layout` directory with the name of the command, for example `change_point.md`.
-2. Add the content for the command to the file. See other files in this directory for examples.
-3. Add the command to the list in `_snippets/lists/processing-commands.md`.
-4. Add an include directive to the `commands/processing-commands.md` file to include the new command.
-5. Add tested examples to the `_snippets/commands/examples` directory. See below for details.
+2. Ensure to specify what versions the command applies to. See [Version differentiation in Docs V3](#version-differentiation-in-docs-v3) for details. [Example PR](https://github.com/elastic/elasticsearch/pull/130314/files#diff-0ab90b6202c5d9eeea75dc95a7cb71dc4d720230342718bff887816771a5a803R3-R6).
+3. Add the content for the command to the file. See other files in this directory for examples.
+4. Add the command to the list in `_snippets/lists/processing-commands.md`.
+5. Add an include directive to the `commands/processing-commands.md` file to include the new command.
+6. Add tested examples to the `_snippets/commands/examples` directory. See below for details.
 
 ### Adding examples to commands
 

modules/data-streams/src/yamlRestTest/resources/rest-api-spec/test/data_stream/240_data_stream_settings.yml

@@ -327,8 +327,6 @@ setup:
         index: "my-data-stream-1"
         wait_for_status: green
 
-
-
   - do:
       indices.get_data_stream:
         name: my-data-stream-1
@@ -398,3 +396,96 @@ setup:
   - match: { .$idx0name.settings.index.number_of_shards: "1" }
   - match: { .$idx0name.settings.index.lifecycle.name: "my-policy" }
   - match: { .$idx0name.settings.index.lifecycle.prefer_ilm: "true" }
+
+---
+"Test null out settings component templates only":
+  - requires:
+      cluster_features: [ "logs_stream" ]
+      reason: requires setting 'logs_stream' to get or set data stream settings
+
+  - do:
+      cluster.put_component_template:
+        name: settings-template
+        body:
+          template:
+            settings:
+              lifecycle.name: my-policy
+
+  - do:
+      allowed_warnings:
+        - "index template [my-component-only-template] has index patterns [my-component-only-data-stream-*] matching patterns from existing older templates [global] with patterns (global => [*]); this template [my-component-only-template] will take precedence during new index creation"
+      indices.put_index_template:
+        name: my-component-only-template
+        body:
+          index_patterns: [ my-component-only-data-stream-* ]
+          data_stream: { }
+          composed_of:
+            - settings-template
+
+  - do:
+      indices.create_data_stream:
+        name: my-component-only-data-stream-1
+
+  - do:
+      cluster.health:
+        index: "my-component-only-data-stream-1"
+        wait_for_status: green
+
+  - do:
+      indices.get_data_stream:
+        name: my-component-only-data-stream-1
+  - match: { data_streams.0.name: my-component-only-data-stream-1 }
+  - match: { data_streams.0.settings: {} }
+  - match: { data_streams.0.effective_settings: null }
+
+  - do:
+      indices.put_data_stream_settings:
+        name: my-component-only-data-stream-1
+        body:
+          index:
+            lifecycle:
+              name: my-new-policy
+              prefer_ilm: true
+  - match: { data_streams.0.name: my-component-only-data-stream-1 }
+  - match: { data_streams.0.applied_to_data_stream: true }
+  - length: { data_streams.0.index_settings_results.applied_to_data_stream_and_backing_indices: 2  }
+  - match: { data_streams.0.settings.index.lifecycle.name: "my-new-policy" }
+  - match: { data_streams.0.settings.index.lifecycle.prefer_ilm: "true" }
+  - match: { data_streams.0.effective_settings.index.lifecycle.name: "my-new-policy" }
+  - match: { data_streams.0.effective_settings.index.lifecycle.prefer_ilm: "true" }
+
+  - do:
+      indices.put_data_stream_settings:
+        name: my-component-only-data-stream-1
+        body:
+          index:
+            lifecycle:
+              name: null
+              prefer_ilm: null
+  - match: { data_streams.0.name: my-component-only-data-stream-1 }
+  - match: { data_streams.0.applied_to_data_stream: true }
+  - length: { data_streams.0.index_settings_results.applied_to_data_stream_and_backing_indices: 2  }
+  - match: { data_streams.0.settings.index.lifecycle.name: null }
+  - match: { data_streams.0.settings.index.lifecycle.prefer_ilm: null }
+  - match: { data_streams.0.effective_settings.index.lifecycle.name: "my-policy" }
+  - match: { data_streams.0.effective_settings.index.lifecycle.prefer_ilm: null }
+
+  - do:
+      indices.get_data_stream_settings:
+        name: my-component-only-data-stream-1
+  - match: { data_streams.0.name: my-component-only-data-stream-1 }
+  - match: { data_streams.0.settings.index.lifecycle.name: null }
+  - match: { data_streams.0.settings.index.lifecycle.prefer_ilm: null }
+  - match: { data_streams.0.effective_settings.index.lifecycle.name: "my-policy" }
+  - match: { data_streams.0.effective_settings.index.lifecycle.prefer_ilm: null }
+
+  - do:
+      indices.get_data_stream:
+        name: my-component-only-data-stream-1
+  - set: { data_streams.0.indices.0.index_name: idx0name }
+
+  - do:
+      indices.get_settings:
+        index: my-component-only-data-stream-1
+  - match: { .$idx0name.settings.index.lifecycle.name: "my-policy" }
+  - match: { .$idx0name.settings.index.lifecycle.prefer_ilm: null }

server/src/main/java/org/elasticsearch/cluster/metadata/MetadataDataStreamsService.java

@@ -480,10 +480,9 @@ private DataStream createDataStreamForUpdatedDataStreamSettings(
         ProjectMetadata projectMetadata = clusterState.metadata().getProject(projectId);
         Map<String, DataStream> dataStreamMap = projectMetadata.dataStreams();
         DataStream dataStream = dataStreamMap.get(dataStreamName);
-        Settings existingSettings = dataStream.getSettings();
+        Settings existingDataStreamSettings = dataStream.getSettings();
 
-        Template.Builder templateBuilder = Template.builder();
-        Settings.Builder mergedSettingsBuilder = Settings.builder().put(existingSettings).put(settingsOverrides);
+        Settings.Builder mergedSettingsBuilder = Settings.builder().put(existingDataStreamSettings).put(settingsOverrides);
         /*
          * A null value for a setting override means that we remove it from the data stream, and let the value from the template (if any)
          * be used.
@@ -493,18 +492,18 @@ private DataStream createDataStreamForUpdatedDataStreamSettings(
                 mergedSettingsBuilder.remove(key);
             }
         });
-        Settings mergedSettings = mergedSettingsBuilder.build();
+        Settings mergedDataStreamSettings = mergedSettingsBuilder.build();
 
         final ComposableIndexTemplate template = lookupTemplateForDataStream(dataStreamName, projectMetadata);
-        ComposableIndexTemplate mergedTemplate = template.mergeSettings(mergedSettings);
+        Settings templateSettings = MetadataIndexTemplateService.resolveSettings(template, projectMetadata.componentTemplates());
+        Settings mergedEffectiveSettings = templateSettings.merge(mergedDataStreamSettings);
         MetadataIndexTemplateService.validateTemplate(
-            mergedTemplate.template().settings(),
+            mergedEffectiveSettings,
             dataStream.getEffectiveMappings(projectMetadata),
             indicesService
         );
 
-        templateBuilder.settings(mergedSettingsBuilder);
-        return dataStream.copy().setSettings(mergedSettings).build();
+        return dataStream.copy().setSettings(mergedDataStreamSettings).build();
     }
 
     private DataStream createDataStreamForUpdatedDataStreamMappings(

server/src/test/java/org/elasticsearch/index/mapper/flattened/FlattenedFieldMapperTests.java

@@ -931,6 +931,35 @@ public void testSyntheticSourceWithEmptyObject() throws IOException {
             {"field":{"key1":"foo"}}"""));
     }
 
+    public void testSyntheticSourceWithMatchesInNestedPath() throws IOException {
+        DocumentMapper mapper = createSytheticSourceMapperService(
+            mapping(b -> { b.startObject("field").field("type", "flattened").endObject(); })
+        ).documentMapper();
+
+        // This test covers a scenario that previously had a bug.
+        // Since a.b.c and b.b.d have a matching middle key `b`, and b.b.d starts with a `b`,
+        // startObject was not called for the first `b` in b.b.d.
+        // For a full explanation see this comment: https://github.com/elastic/elasticsearch/pull/129600#issuecomment-3024476134
+        var syntheticSource = syntheticSource(mapper, b -> {
+            b.startObject("field");
+            {
+                b.startObject("a");
+                {
+                    b.startObject("b").field("c", "1").endObject();
+                }
+                b.endObject();
+                b.startObject("b");
+                {
+                    b.startObject("b").field("d", "2").endObject();
+                }
+                b.endObject();
+            }
+            b.endObject();
+        });
+        assertThat(syntheticSource, equalTo("""
+            {"field":{"a":{"b":{"c":"1"}},"b":{"b":{"d":"2"}}}}"""));
+    }
+
     @Override
     protected boolean supportsCopyTo() {
         return false;

x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/ilm/IndexLifecycleExplainResponse.java

@@ -51,6 +51,7 @@ public class IndexLifecycleExplainResponse implements ToXContentObject, Writeabl
     private static final ParseField PREVIOUS_STEP_INFO_FIELD = new ParseField("previous_step_info");
     private static final ParseField PHASE_EXECUTION_INFO = new ParseField("phase_execution");
     private static final ParseField AGE_FIELD = new ParseField("age");
+    private static final ParseField AGE_IN_MILLIS_FIELD = new ParseField("age_in_millis");
     private static final ParseField TIME_SINCE_INDEX_CREATION_FIELD = new ParseField("time_since_index_creation");
     private static final ParseField REPOSITORY_NAME = new ParseField("repository_name");
     private static final ParseField SHRINK_INDEX_NAME = new ParseField("shrink_index_name");
@@ -83,6 +84,7 @@ public class IndexLifecycleExplainResponse implements ToXContentObject, Writeabl
             Objects.requireNonNullElse((Boolean) a[22], false)
             // a[13] == "age"
             // a[20] == "time_since_index_creation"
+            // a[23] = "age_in_millis"
         )
     );
     static {
@@ -121,6 +123,7 @@ public class IndexLifecycleExplainResponse implements ToXContentObject, Writeabl
             return BytesReference.bytes(builder);
         }, PREVIOUS_STEP_INFO_FIELD);
         PARSER.declareBoolean(ConstructingObjectParser.optionalConstructorArg(), SKIP_NAME);
+        PARSER.declareLong(ConstructingObjectParser.optionalConstructorArg(), AGE_IN_MILLIS_FIELD);
     }
 
     private final String index;
@@ -529,7 +532,10 @@ public XContentBuilder toXContent(XContentBuilder builder, Params params) throws
                     LIFECYCLE_DATE_FIELD.getPreferredName(),
                     lifecycleDate
                 );
-                builder.field(AGE_FIELD.getPreferredName(), getAge(nowSupplier).toHumanReadableString(2));
+
+                final TimeValue ageNow = getAge(nowSupplier);
+                builder.field(AGE_FIELD.getPreferredName(), ageNow.toHumanReadableString(2));
+                builder.field(AGE_IN_MILLIS_FIELD.getPreferredName(), ageNow.getMillis());
             }
             if (phase != null) {
                 builder.field(PHASE_FIELD.getPreferredName(), phase);

x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/xcontent/XContentUtils.java

@@ -112,9 +112,7 @@ public static void addAuthorizationInfo(final XContentBuilder builder, final Map
     private static void addSubjectInfo(XContentBuilder builder, Subject subject) throws IOException {
         switch (subject.getType()) {
             case USER -> builder.array(User.Fields.ROLES.getPreferredName(), subject.getUser().roles());
-            case API_KEY -> {
-                addApiKeyInfo(builder, subject);
-            }
+            case API_KEY -> addApiKeyInfo(builder, subject);
             case SERVICE_ACCOUNT -> builder.field("service_account", subject.getUser().principal());
             case CROSS_CLUSTER_ACCESS -> {
                 builder.startObject("cross_cluster_access");
@@ -129,7 +127,16 @@ private static void addSubjectInfo(XContentBuilder builder, Subject subject) thr
                 builder.endObject();
             }
             case CLOUD_API_KEY -> {
-                // TODO Add cloud API key information here
+                builder.startObject("cloud_api_key");
+                Map<String, Object> metadata = subject.getUser().metadata();
+                builder.field("id", subject.getUser().principal());
+                Object name = metadata.get(AuthenticationField.API_KEY_NAME_KEY);
+                if (name instanceof String) {
+                    builder.field("name", name);
+                }
+                builder.field("internal", metadata.get(AuthenticationField.API_KEY_INTERNAL_KEY));
+                builder.array(User.Fields.ROLES.getPreferredName(), subject.getUser().roles());
+                builder.endObject();
             }
         }
     }