Merge branch 'main' into mp/1

Author: prwhelan

build-conventions/src/main/java/org/elasticsearch/gradle/internal/conventions/PublishPlugin.java

@@ -175,6 +175,8 @@ private void addNameAndDescriptionToPom(Project project, NamedDomainObjectSet<Ma
     private static void configureWithShadowPlugin(Project project, MavenPublication publication) {
         var shadow = project.getExtensions().getByType(ShadowExtension.class);
         shadow.component(publication);
+        publication.artifact(project.getTasks().named("javadocJar"));
+        publication.artifact(project.getTasks().named("sourcesJar"));
     }
 
     private static void addScmInfo(XmlProvider xml, GitInfo gitInfo) {

docs/changelog/128866.yaml

@@ -0,0 +1,6 @@
+pr: 128866
+summary: Add `age_in_millis` to ILM Explain Response
+area: ILM+SLM
+type: enhancement
+issues:
+ - 103659

docs/changelog/129725.yaml

@@ -0,0 +1,5 @@
+pr: 129725
+summary: Throw a 400 when sorting for all types of range fields
+area: Search
+type: bug
+issues: []

docs/reference/elasticsearch/rest-apis/update-document.md

@@ -0,0 +1,264 @@
+---
+mapped_pages:
+  - https://www.elastic.co/guide/en/elasticsearch/reference/8.18/docs-update.html
+applies_to:
+  stack: all
+navigation_title: Update a document
+---
+
+# Update a document [update-document]
+
+The [Update API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-update) enables you to script document updates, allowing you to update, delete, or skip modifying a document. 
+
+The following examples show common use cases, such as incrementing a counter and adding or removing elements from a list, as well as how to:
+
+- [Update part of a document](#update-part-document)
+- [Detect noop updates](#detect-noop-updates)
+- [Insert or update documents with upsert](#upsert)
+- [Simplify upsert with doc_as_upsert](#doc-as-upsert)
+
+First, let's index a simple doc:
+
+```console
+PUT test/_doc/1
+{
+  "counter" : 1,
+  "tags" : ["red"]
+}
+```
+% TESTSETUP
+
+To increment the counter, you can submit an update request with the
+following script:
+
+```console
+POST test/_update/1
+{
+  "script" : {
+    "source": "ctx._source.counter += params.count",
+    "lang": "painless",
+    "params" : {
+      "count" : 4
+    }
+  }
+}
+```
+
+Similarly, you could use and update script to add a tag to the list of tags
+(this is just a list, so the tag is added even it exists):
+
+```console
+POST test/_update/1
+{
+  "script": {
+    "source": "ctx._source.tags.add(params.tag)",
+    "lang": "painless",
+    "params": {
+      "tag": "blue"
+    }
+  }
+}
+```
+
+You could also remove a tag from the list of tags. The Painless
+function to `remove` a tag takes the array index of the element
+you want to remove. To avoid a possible runtime error, you first need to
+make sure the tag exists. If the list contains duplicates of the tag, this
+script just removes one occurrence.
+
+```console
+POST test/_update/1
+{
+  "script": {
+    "source": "if (ctx._source.tags.contains(params.tag)) { ctx._source.tags.remove(ctx._source.tags.indexOf(params.tag)) }",
+    "lang": "painless",
+    "params": {
+      "tag": "blue"
+    }
+  }
+}
+```
+
+You can also add and remove fields from a document. For example, this script
+adds the field `new_field`:
+
+```console
+POST test/_update/1
+{
+  "script" : "ctx._source.new_field = 'value_of_new_field'"
+}
+```
+
+Conversely, this script removes the field `new_field`:
+
+```console
+POST test/_update/1
+{
+  "script" : "ctx._source.remove('new_field')"
+}
+```
+% TEST[continued]
+
+The following script removes a subfield from an object field:
+
+```console
+PUT test/_doc/1?refresh
+{
+  "my-object": {
+    "my-subfield": true
+  }
+}
+```
+
+```console
+POST test/_update/1
+{
+  "script": "ctx._source['my-object'].remove('my-subfield')"
+}
+```
+% TEST[continued]
+
+Instead of updating the document, you can also change the operation that is
+executed from within the script. For example, this request deletes the doc if
+the `tags` field contains `green`, otherwise it does nothing (`noop`):
+
+```console
+POST test/_update/1
+{
+  "script": {
+    "source": "if (ctx._source.tags.contains(params.tag)) { ctx.op = 'delete' } else { ctx.op = 'noop' }",
+    "lang": "painless",
+    "params": {
+      "tag": "green"
+    }
+  }
+}
+```
+
+## Update part of a document [update-part-document]
+
+The following partial update adds a new field to the
+existing document:
+
+```console
+POST test/_update/1
+{
+  "doc": {
+    "name": "new_name"
+  }
+}
+```
+
+If both `doc` and `script` are specified, then `doc` is ignored. If you
+specify a scripted update, include the fields you want to update in the script.
+
+
+## Detect noop updates [detect-noop-updates]
+
+By default updates that don't change anything detect that they don't change
+anything and return `"result": "noop"`:
+
+```console
+POST test/_update/1
+{
+  "doc": {
+    "name": "new_name"
+  }
+}
+```
+% TEST[continued]
+
+If the value of `name` is already `new_name`, the update
+request is ignored and the `result` element in the response returns `noop`:
+
+```console
+
+{
+   "_shards": {
+        "total": 0,
+        "successful": 0,
+        "failed": 0
+   },
+   "_index": "test",
+   "_id": "1",
+   "_version": 2,
+   "_primary_term": 1,
+   "_seq_no": 1,
+   "result": "noop"
+}
+```
+
+You can disable this behavior by setting `"detect_noop": false`:
+
+```console
+POST test/_update/1
+{
+  "doc": {
+    "name": "new_name"
+  },
+  "detect_noop": false
+}
+```
+
+## Insert or update documents with upsert [upsert]
+
+An upsert operation lets you update an existing document or insert a new one if it doesn't exist, in a single request.
+
+In this example, if the product with ID `1` exists, its price will be updated to `100`. If the product does not exist, a new document with ID `1` and a price of `50` will be inserted.
+
+```console
+POST /test/_update/1
+{
+  "doc": {
+    "product_price": 100
+  },
+  "upsert": {
+    "product_price": 50
+  }
+}
+```
+
+## Run a scripted upsert [scripted-upsert]
+
+To run the script whether or not the document exists, set `scripted_upsert` to
+`true`:
+
+```console
+POST test/_update/1
+{
+  "scripted_upsert": true,
+  "script": {
+    "source": """
+      if ( ctx.op == 'create' ) {
+        ctx._source.counter = params.count
+      } else {
+        ctx._source.counter += params.count
+      }
+    """,
+    "params": {
+      "count": 4
+    }
+  },
+  "upsert": {}
+}
+```
+
+## Simplify upsert with doc_as_upsert [doc-as-upsert]
+
+Instead of sending a partial `doc` plus an `upsert` doc, you can set
+`doc_as_upsert` to `true` to use the contents of `doc` as the `upsert`
+value:
+
+```console
+POST test/_update/1
+{
+  "doc": {
+    "name": "new_name"
+  },
+  "doc_as_upsert": true
+}
+```
+
+::::{note}
+Using [ingest pipelines](https://www.elastic.co/guide/en/elasticsearch/reference/8.18/ingest.html) with `doc_as_upsert` is not supported.
+::::

docs/reference/elasticsearch/toc.yml

@@ -105,6 +105,7 @@ toc:
             - file: rest-apis/searching-with-query-rules.md
             - file: rest-apis/shard-request-cache.md
             - file: rest-apis/term-vectors-examples.md
+            - file: rest-apis/update-document.md
             - file: rest-apis/update-cc-api-key-examples.md
             - file: rest-apis/vector-tile-search.md
   - file: mapping-reference/index.md

docs/reference/query-languages/esql/README.md

@@ -105,16 +105,95 @@ To help differentiate between the static and generated content, the generated co
 % This is generated by ESQL's AbstractFunctionTestCase. Do no edit it. See ../README.md for how to regenerate it.
 ```
 
+## Version differentiation in Docs V3
+
+> [!IMPORTANT]
+> Starting with 9.0, we no longer publish separate documentation branches for every minor release (`9.0`, `9.1`, `9.2`, etc.). 
+> This means there won't be a different page for `9.1`, `9.2`, and so on. Instead, all changes landing in subsequent minor releases **will appear on the same page**.
+
+Because we now publish just one docs set off of the `main` branch, we use the [`applies_to` metadata](https://elastic.github.io/docs-builder/syntax/applies/) to differentiate features and their availability across different versions. This is a [cumulative approach](https://elastic.github.io/docs-builder/contribute/#cumulative-docs): instead of creating separate pages for each product and release, we update a **single page** with product- and version-specific details over time.
+
+`applies_to` allows us to clearly communicate when features are introduced, when they transition from preview to GA, and which versions support specific functionality.
+
+This metadata accepts a lifecycle and an optional version.
+
+### Functions and operators
+
+Use the `@FunctionAppliesTo` annotation within the `@FunctionInfo` annotation on function and operator classes to specify the lifecycle and version for functions and operators.
+
+For example, to indicate that a function is in technical preview and applies to version 9.0.0, you would use:
+
+```java
+@FunctionInfo(
+    returnType = "boolean",
+    appliesTo = {
+        @FunctionAppliesTo(lifeCycle = FunctionAppliesToLifecycle.PREVIEW, version = "9.0.0")
+    },
+    ...
+)
+```
+
+When a feature evolves from preview in `9.0` to GA in `9.2`, add a new entry alongside the existing preview entry and remove the `preview = true` boolean:
+
+```java
+@FunctionInfo(
+    returnType = "boolean",
+    preview = false, //  the preview boolean can be removed (or flipped to false) when the function becomes GA
+    appliesTo = {
+        @FunctionAppliesTo(lifeCycle = FunctionAppliesToLifecycle.PREVIEW, version = "9.0.0"),
+        @FunctionAppliesTo(lifeCycle = FunctionAppliesToLifecycle.GA, version = "9.2.0")
+    },
+    ...
+)
+```
+
+We updated [`DocsV3Support.java`](https://github.com/elastic/elasticsearch/blob/main/x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/expression/function/DocsV3Support.java) to generate the `applies_to` metadata correctly for functions and operators.
+
+### Inline `applies_to` metadata
+
+Use [inline annotations](https://elastic.github.io/docs-builder/syntax/applies/#inline-annotations) to specify `applies_to` metadata in descriptions, parameter lists, etc.
+
+For example, the second item in this list is in technical preview as of version 9.2:
+
+```markdown
+- Item 1
+- Item 2 {applies_to}`stack: preview 9.2.`
+```
+
+### Key rules
+
+1. **Use the `preview = true` boolean** for any tech preview feature - this is required for the Kibana inline docs
+   - **Remove `preview = true`** only when the feature becomes GA on serverless and is _definitely_ going GA in the next minor release
+2. **Never delete `appliesTo` entries** - only add new ones as features evolve from preview to GA
+3. **Use specific versions** (`9.0.0`, `9.1.0`) when known, or just `PREVIEW` without a version if timing is uncertain
+4. **Add `applies_to` to examples** where necessary
+
+> [!IMPORTANT]
+> We don't use `applies_to` in the legacy asciidoc system for 8.x and earlier versions.
+
+### Supported lifecycles
+
+- `PREVIEW` - Feature is in technical preview
+- `GA` - Feature is generally available  
+- `DEPRECATED` - Feature is deprecated and will be removed in a future release
+- `UNAVAILABLE` - Feature is not available in the current version, but may be available in future releases
+
+> [!NOTE] 
+> Unreleased version information is automatically sanitized in the docs build output. For example, say you specify `preview 9.3.0`: 
+> - Before `9.3.0` is released, the live documentation will display "Planned for a future release" instead of the specific version number. 
+>  - This will be updated automatically when the version is released.
+
 ## Tutorials
 
 ### Adding a new command
 
 When adding a new command, for example adding the `CHANGE_POINT` command, do the following:
 1. Create a new file in the `_snippets/commands/layout` directory with the name of the command, for example `change_point.md`.
-2. Add the content for the command to the file. See other files in this directory for examples.
-3. Add the command to the list in `_snippets/lists/processing-commands.md`.
-4. Add an include directive to the `commands/processing-commands.md` file to include the new command.
-5. Add tested examples to the `_snippets/commands/examples` directory. See below for details.
+2. Ensure to specify what versions the command applies to. See [Version differentiation in Docs V3](#version-differentiation-in-docs-v3) for details. [Example PR](https://github.com/elastic/elasticsearch/pull/130314/files#diff-0ab90b6202c5d9eeea75dc95a7cb71dc4d720230342718bff887816771a5a803R3-R6).
+3. Add the content for the command to the file. See other files in this directory for examples.
+4. Add the command to the list in `_snippets/lists/processing-commands.md`.
+5. Add an include directive to the `commands/processing-commands.md` file to include the new command.
+6. Add tested examples to the `_snippets/commands/examples` directory. See below for details.
 
 ### Adding examples to commands