Merge branch 'main' into svilen-mihaylov/112822

Author: svilen-mihaylov-elastic

build.gradle

@@ -365,7 +365,7 @@ tasks.register("verifyBwcTestsEnabled") {
 
 tasks.register("branchConsistency") {
   description = 'Ensures this branch is internally consistent. For example, that versions constants match released versions.'
-  group 'Verification'
+  group = 'Verification'
   dependsOn ":verifyVersions", ":verifyBwcTestsEnabled"
 }
 

docs/changelog/117519.yaml

@@ -0,0 +1,20 @@
+pr: 117519
+summary: Remove `data_frame_transforms` roles
+area: Transform
+type: breaking
+issues: []
+breaking:
+  title: Remove `data_frame_transforms` roles
+  area: Transform
+  details: >-
+    `data_frame_transforms_admin` and `data_frame_transforms_user` were deprecated in
+    Elasticsearch 7 and are being removed in Elasticsearch 9.
+    `data_frame_transforms_admin` is now `transform_admin`.
+    `data_frame_transforms_user` is now `transform_user`.
+    Users must call the `_update` API to replace the permissions on the Transform before the
+    Transform can be started.
+  impact: >-
+    Transforms created with either the `data_frame_transforms_admin` or the
+    `data_frame_transforms_user` role will fail to start. The Transform will remain
+    in a `stopped` state, and its health will be red while displaying permission failures.
+  notable: false

docs/changelog/117949.yaml

@@ -0,0 +1,5 @@
+pr: 117949
+summary: Move `SlowLogFieldProvider` instantiation to node construction
+area: Infra/Logging
+type: bug
+issues: []

docs/changelog/118804.yaml

@@ -0,0 +1,15 @@
+pr: 118804
+summary: Add new experimental `rank_vectors` mapping for late-interaction second order
+  ranking
+area: Vector Search
+type: feature
+issues: []
+highlight:
+  title: Add new experimental `rank_vectors` mapping for late-interaction second order
+    ranking
+  body:
+      Late-interaction models are powerful rerankers. While their size and overall
+      cost doesn't lend itself for HNSW indexing, utilizing them as second order reranking
+      can provide excellent boosts in relevance. The new `rank_vectors` mapping allows for rescoring
+      over new and novel multi-vector late-interaction models like ColBERT or ColPali.
+  notable: true

docs/plugins/discovery-ec2.asciidoc

@@ -241,7 +241,7 @@ The `discovery-ec2` plugin can automatically set the `aws_availability_zone`
 node attribute to the availability zone of each node. This node attribute
 allows you to ensure that each shard has copies allocated redundantly across
 multiple availability zones by using the
-{ref}/modules-cluster.html#shard-allocation-awareness[Allocation Awareness]
+{ref}/shard-allocation-awareness.html#[Allocation Awareness]
 feature.
 
 In order to enable the automatic definition of the `aws_availability_zone`
@@ -333,7 +333,7 @@ labelled as `Moderate` or `Low`.
 
 * It is a good idea to distribute your nodes across multiple
 https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html[availability
-zones] and use {ref}/modules-cluster.html#shard-allocation-awareness[shard
+zones] and use {ref}/shard-allocation-awareness.html[shard
 allocation awareness] to ensure that each shard has copies in more than one
 availability zone.
 

docs/reference/analysis.asciidoc

@@ -9,8 +9,7 @@
 --
 
 _Text analysis_ is the process of converting unstructured text, like
-the body of an email or a product description, into a structured format that's
-optimized for search.
+the body of an email or a product description, into a structured format that's <<full-text-search,optimized for search>>.
 
 [discrete]
 [[when-to-configure-analysis]]

docs/reference/analysis/tokenizers.asciidoc

@@ -1,6 +1,14 @@
 [[analysis-tokenizers]]
 == Tokenizer reference
 
+.Difference between {es} tokenization and neural tokenization
+[NOTE]
+====
+{es}'s tokenization process produces linguistic tokens, optimized for search and retrieval.
+This differs from neural tokenization in the context of machine learning and natural language processing. Neural tokenizers translate strings into smaller, subword tokens, which are encoded into vectors for consumptions by neural networks.
+{es} does not have built-in neural tokenizers.
+====
+
 A _tokenizer_ receives a stream of characters, breaks it up into individual
 _tokens_ (usually individual words), and outputs a stream of _tokens_. For
 instance, a <<analysis-whitespace-tokenizer,`whitespace`>> tokenizer breaks

docs/reference/cat/nodeattrs.asciidoc

@@ -17,7 +17,7 @@ console. They are _not_ intended for use by applications. For application
 consumption, use the <<cluster-nodes-info,nodes info API>>.
 ====
 
-Returns information about <<shard-allocation-filtering,custom node attributes>>.
+Returns information about <<custom-node-attributes,custom node attributes>>.
 
 [[cat-nodeattrs-api-request]]
 ==== {api-request-title}

docs/reference/cluster.asciidoc

@@ -35,7 +35,7 @@ one of the following:
   master-eligible nodes, all data nodes, all ingest nodes, all voting-only
   nodes, all machine learning nodes, and all coordinating-only nodes.
 * a pair of patterns, using `*` wildcards, of the form `attrname:attrvalue`,
-  which adds to the subset all nodes with a custom node attribute whose name
+  which adds to the subset all nodes with a <<custom-node-attributes,custom node attribute>> whose name
   and value match the respective patterns. Custom node attributes are
   configured by setting properties in the configuration file of the form
   `node.attr.attrname: attrvalue`.

docs/reference/commands/node-tool.asciidoc

@@ -23,8 +23,8 @@ bin/elasticsearch-node repurpose|unsafe-bootstrap|detach-cluster|override-versio
 This tool has a number of modes:
 
 * `elasticsearch-node repurpose` can be used to delete unwanted data from a
-  node if it used to be a <<data-node,data node>> or a
-  <<master-node,master-eligible node>> but has been repurposed not to have one
+  node if it used to be a <<data-node-role,data node>> or a
+  <<master-node-role,master-eligible node>> but has been repurposed not to have one
   or other of these roles.
 
 * `elasticsearch-node remove-settings` can be used to remove persistent settings