Merge branch 'main' into version-sets

Author: thecoop

.github/workflows/comment-on-asciidoc-changes.yml

@@ -0,0 +1,21 @@
+---
+name: Comment on PR for .asciidoc changes
+
+on:
+  # We need to use pull_request_target to be able to comment on PRs from forks
+  pull_request_target:
+    types:
+      - synchronize
+      - opened
+      - reopened
+    branches:
+      - main
+      - master
+      - "9.0"
+
+jobs:
+  comment-on-asciidoc-change:
+    permissions:
+      contents: read
+      pull-requests: write
+    uses: elastic/docs-builder/.github/workflows/comment-on-asciidoc-changes.yml@main

branches.json

@@ -10,6 +10,9 @@
     {
       "branch": "9.0"
     },
+    {
+      "branch": "8.18"
+    },
     {
       "branch": "8.17"
     },

build-tools-internal/version.properties

@@ -1,5 +1,5 @@
 elasticsearch     = 9.0.0
-lucene            = 10.0.0
+lucene            = 10.1.0
 
 bundled_jdk_vendor = openjdk
 bundled_jdk = 23+37@3c5b90190c68498b986a97f276efd28a

docs/Versions.asciidoc

@@ -1,8 +1,8 @@
 
 include::{docs-root}/shared/versions/stack/{source_branch}.asciidoc[]
 
-:lucene_version:        10.0.0
-:lucene_version_path:   10_0_0
+:lucene_version:        10.1.0
+:lucene_version_path:   10_1_0
 :jdk:                   11.0.2
 :jdk_major:             11
 :build_type:            tar

docs/changelog/119308.yaml

@@ -0,0 +1,5 @@
+pr: 119308
+summary: Upgrade to Lucene 10.1.0
+area: Search
+type: upgrade
+issues: []

docs/changelog/121193.yaml

@@ -0,0 +1,18 @@
+pr: 121193
+summary: Enable LOOKUP JOIN in non-snapshot builds
+area: ES|QL
+type: enhancement
+issues:
+ - 121185
+highlight:
+  title: Enable LOOKUP JOIN in non-snapshot builds
+  body: |-
+    This effectively releases LOOKUP JOIN into tech preview. Docs will
+    follow in a separate PR.
+
+    - Enable the lexing/grammar for LOOKUP JOIN in non-snapshot builds.
+    - Remove the grammar for the unsupported `| JOIN ...` command (without `LOOKUP` as first keyword). The way the lexer modes work, otherwise we'd also have to enable `| JOIN ...` syntax on non-snapshot builds and would have to add additional validation to provide appropriate error messages.
+    - Remove grammar for `LOOKUP JOIN index AS ...` because qualifiers are not yet supported. Otherwise we'd have to put in additional validation as well to prevent such queries.
+
+    Also fix https://github.com/elastic/elasticsearch/issues/121185
+  notable: true

docs/reference/search/profile.asciidoc

@@ -176,7 +176,7 @@ The API returns the following result:
                 "time_in_nanos": 775274,
                 "children" : [
                   {
-                    "name": "SimpleTopScoreDocCollector",
+                    "name": "TopScoreDocCollector",
                     "reason": "search_top_hits",
                     "time_in_nanos": 775274
                   }
@@ -537,7 +537,7 @@ Looking at the previous example:
     "time_in_nanos": 775274,
     "children" : [
       {
-        "name": "SimpleTopScoreDocCollector",
+        "name": "TopScoreDocCollector",
         "reason": "search_top_hits",
         "time_in_nanos": 775274
       }
@@ -551,7 +551,7 @@ Looking at the previous example:
 
 
 We see a top-level collector named `QueryPhaseCollector` which holds a child
-`SimpleTopScoreDocCollector`. `SimpleTopScoreDocCollector` is the  default
+`TopScoreDocCollector`. `TopScoreDocCollector` is the  default
 "scoring and sorting" `Collector` used by {es}. The `reason` field attempts
 to give a plain English description of the class name. The `time_in_nanos`
 is similar to the time in the Query tree: a wall-clock time inclusive of all
@@ -751,7 +751,7 @@ The API returns the following result:
                 "time_in_nanos": 1945072,
                 "children": [
                   {
-                    "name": "SimpleTopScoreDocCollector",
+                    "name": "TopScoreDocCollector",
                     "reason": "search_top_hits",
                     "time_in_nanos": 22577
                   },
@@ -788,7 +788,7 @@ major portions of the query are represented:
 2. The second `TermQuery` (message:search) represents the `post_filter` query.
 
 The Collector tree is fairly straightforward, showing how a single
-QueryPhaseCollector that holds the normal scoring SimpleTopScoreDocCollector
+QueryPhaseCollector that holds the normal scoring TopScoreDocCollector
 used to collect top hits, as well as BucketCollectorWrapper to run all scoped
 aggregations.
 
@@ -1332,7 +1332,7 @@ One of the `dfs.knn` sections for a shard looks like the following:
         "rewrite_time" : 1275732,
         "collector" : [
             {
-                "name" : "SimpleTopScoreDocCollector",
+                "name" : "TopScoreDocCollector",
                 "reason" : "search_top_hits",
                 "time_in_nanos" : 17163
             }

docs/reference/search/search-your-data/semantic-text-hybrid-search

@@ -113,6 +113,7 @@ POST _tasks/<task_id>/_cancel
 ==== Perform hybrid search
 
 After reindexing the data into the `semantic-embeddings` index, you can perform hybrid search by using <<rrf,reciprocal rank fusion (RRF)>>. RRF is a technique that merges the rankings from both semantic and lexical queries, giving more weight to results that rank high in either search. This ensures that the final results are balanced and relevant.
+To extract the most relevant fragments from the original text and query, you can use the <<highlighting,highlight parameter>>:
 
 [source,console]
 ------------------------------------------------------------
@@ -142,6 +143,13 @@ GET semantic-embeddings/_search
         }
       ]
     }
+  },
+  "highlight": {
+    "fields": {
+        "semantic_text": {
+            "number_of_fragments": 2  <5>
+        }
+    }
   }
 }
 ------------------------------------------------------------
@@ -150,7 +158,7 @@ GET semantic-embeddings/_search
 <2> Lexical search is performed on the `content` field using the specified phrase.
 <3> The second `standard` retriever refers to the semantic search.
 <4> The `semantic_text` field is used to perform the semantic search.
-
+<5> Specifies the maximum number of fragments to return. See <<semantic-text-highlighting, semantic text highlighting>> for a more complete example.
 
 After performing the hybrid search, the query will return the top 10 documents that match both semantic and lexical search criteria. The results include detailed information about each document:
 
@@ -178,36 +186,14 @@ After performing the hybrid search, the query will return the top 10 documents t
         "_score": 0.032786883,
         "_rank": 1,
         "_source": {
-          "semantic_text": {
-            "inference": {
-              "inference_id": "my-elser-endpoint",
-              "model_settings": {
-                "task_type": "sparse_embedding"
-              },
-              "chunks": [
-                {
-                  "text": "What so many out there do not realize is the importance of what you do after you work out. You may have done the majority of the work, but how you treat your body in the minutes and hours after you exercise has a direct effect on muscle soreness, muscle strength and growth, and staying hydrated. Cool Down. After your last exercise, your workout is not over. The first thing you need to do is cool down. Even if running was all that you did, you still should do light cardio for a few minutes. This brings your heart rate down at a slow and steady pace, which helps you avoid feeling sick after a workout.",
-                  "embeddings": {
-                    "exercise": 1.571044,
-                    "after": 1.3603843,
-                    "sick": 1.3281639,
-                    "cool": 1.3227621,
-                    "muscle": 1.2645415,
-                    "sore": 1.2561599,
-                    "cooling": 1.2335974,
-                    "running": 1.1750668,
-                    "hours": 1.1104802,
-                    "out": 1.0991782,
-                    "##io": 1.0794281,
-                    "last": 1.0474665,
-                   (...) 
-                  }
-                }
-              ]
-            }
-          },
           "id": 8408852,
           "content": "What so many out there do not realize is the importance of (...)"
+        },
+        "highlight" : {
+            "semantic_text" : [
+                "... fragment_1 ...",
+                "... fragment_2 ..."
+            ]
         }
       }
     ]

docs/reference/troubleshooting/common-issues/disk-usage-exceeded.asciidoc

@@ -127,12 +127,14 @@ its {cloud}/ec-api-console.html[Elasticsearch API Console] to later
 with this resolution flow on {ess}, kindly reach out to
 https://support.elastic.co[Elastic Support] for assistance.
 
-== Prevent watermark errors
+[discrete]
+[[fix-watermark-errors-prevent]]
+=== Prevent watermark errors
 
-To avoid watermark errors in future, , perform one of the following actions:
+To avoid watermark errors in future, perform one of the following actions:
 
 * If you're using {ess}, {ece}, or {eck}: Enable <<xpack-autoscaling,autoscaling>>.
 
 * Set up {kibana-ref}/kibana-alerts.html[stack monitoring alerts] on top of
 <<monitor-elasticsearch-cluster,{es} monitoring>> to be notified before
-the flood-stage watermark is reached.
+the flood-stage watermark is reached.