Merge branch 'main' into increase_sparse_pruning_test_coverage

Author: mromaios

distribution/tools/server-cli/src/main/java/org/elasticsearch/server/cli/APMJvmOptions.java

@@ -329,7 +329,7 @@ static Path findAgentJar(String installDir) throws IOException, UserException {
 
         try (var apmStream = Files.list(apmModule)) {
             final List<Path> paths = apmStream.filter(
-                path -> path.getFileName().toString().matches("elastic-apm-agent-\\d+\\.\\d+\\.\\d+\\.jar")
+                path -> path.getFileName().toString().matches("elastic-apm-agent-java8-\\d+\\.\\d+\\.\\d+\\.jar")
             ).toList();
 
             if (paths.size() > 1) {

distribution/tools/server-cli/src/test/java/org/elasticsearch/server/cli/APMJvmOptionsTests.java

@@ -141,7 +141,7 @@ private Path makeFakeAgentJar() throws IOException {
         Path tempFile = createTempFile();
         Path apmPathDir = tempFile.getParent().resolve("modules").resolve("apm");
         Files.createDirectories(apmPathDir);
-        Path apmAgentFile = apmPathDir.resolve("elastic-apm-agent-0.0.0.jar");
+        Path apmAgentFile = apmPathDir.resolve("elastic-apm-agent-java8-0.0.0.jar");
         Files.move(tempFile, apmAgentFile);
 
         return tempFile.getParent();

docs/changelog/131723.yaml

@@ -0,0 +1,5 @@
+pr: 131723
+summary: Tests for FORK's evaluation of field names used in `field_caps` resolve calls
+area: Search
+type: bug
+issues: []

docs/changelog/132131.yaml

@@ -0,0 +1,7 @@
+pr: 132131
+summary: Updating `TransportSimulateIndexTemplateAction.resolveTemplate()` to account
+  for data stream overrides
+area: Indices APIs
+type: bug
+issues:
+ - 131425

docs/changelog/132210.yaml

@@ -0,0 +1,6 @@
+pr: 132210
+summary: Adding a `merge_type` parameter to the ingest simulate API
+area: Ingest Node
+type: enhancement
+issues:
+ - 131608

docs/changelog/132546.yaml

@@ -0,0 +1,5 @@
+pr: 132546
+summary: Improve EIS auth call logs and fix revocation bug
+area: Machine Learning
+type: bug
+issues: []

docs/changelog/132622.yaml

@@ -0,0 +1,5 @@
+pr: 132622
+summary: Push down compute engine value loading of long based singleton numeric doc value to the es819 tsdb doc values codec.
+area: "Codec"
+type: enhancement
+issues: []

docs/changelog/132646.yaml

@@ -0,0 +1,5 @@
+pr: 132646
+summary: Update EIS sparse and dense embedding max batch size to 16
+area: Machine Learning
+type: bug
+issues: []

docs/changelog/132651.yaml

@@ -0,0 +1,5 @@
+pr: 132651
+summary: Use java8 variant of apm-agent
+area: Infra/Core
+type: upgrade
+issues: []

docs/release-notes/known-issues.md

@@ -1,17 +1,43 @@
 ---
 navigation_title: "Known issues"
 mapped_pages:
-  - https://www.elastic.co/guide/en/elasticsearch/reference/current/es-connectors-known-issues.html
+  - https://www.elastic.co/guide/en/elasticsearch/reference/current/es-release-notes.html
 ---
 
 # Elasticsearch known issues [elasticsearch-known-issues]
+
 Known issues are significant defects or limitations that may impact your implementation. These issues are actively being worked on and will be addressed in a future release. Review the Elasticsearch known issues to help you make informed decisions, such as upgrading to a new version.
 
+## 9.1.1 [elasticsearch-9.1.1-known-issues]
+
+* An [optimization](https://github.com/elastic/elasticsearch/pull/125403) introduced in 9.1.0 contains a [bug](https://github.com/elastic/elasticsearch/pull/132597) that causes merges to fail for shrunk TSDB and LogsDB indices.
+  
+  Possible *temporary* workarounds include:
+  * Configure the ILM policy to not perform force merges after shrinking TSDB or LogsDB indices.
+  * Add `-Dorg.elasticsearch.index.codec.tsdb.es819.ES819TSDBDocValuesConsumer.enableOptimizedMerge=false` as a Java system property to all data nodes in the cluster and perform a rolling restart.
+    * *Important:* Remove this property when upgrading to the fixed version to re-enable merge optimization. Otherwise, merges will be slower.  
+
+The bug is addressed in version 9.1.2.  
+
 ## 9.1.0 [elasticsearch-9.1.0-known-issues]
-* An error in the configuration of vector indices with type `bbq_hnsw` may lead to significant search performance degradation on 9.1.0. To mitigate this, set the `-Dvector.rescoring.directio=false` JVM option on all search nodes, then restart the nodes.
-  This option can be removed in 9.1.1.
+
+* An [optimization](https://github.com/elastic/elasticsearch/pull/125403) introduced in 9.1.0 contains a [bug](https://github.com/elastic/elasticsearch/pull/132597) that causes merges to fail for shrunk TSDB and LogsDB indices.
+  
+  Possible *temporary* workarounds include:
+  * Configure the ILM policy to not perform force merges after shrinking TSDB or LogsDB indices.
+  * Add `-Dorg.elasticsearch.index.codec.tsdb.es819.ES819TSDBDocValuesConsumer.enableOptimizedMerge=false` as a Java system property to all data nodes in the cluster and perform a rolling restart.
+    * *Important:* Remove this property when upgrading to the fixed version to re-enable merge optimization. Otherwise, merges will be slower.  
+
+The bug is addressed in version 9.1.2.  
+
+* The `-Dvector.rescoring.directio` JVM option is enabled (set to `true`) by default. When used with `bbq_hnsw` type vector indices, this can cause significant search performance degradation; particularly when enough memory is available to hold all vector data. In some cases, kNN search latency can increase by as much as 10x. To mitigate this, set the JVM option `-Dvector.rescoring.directio=false` on all search nodes and restart them. This option can be removed in 9.1.1.
+
+**How do I know if my index vector type is `bbq_hnsw`?**
+
+* Prior to 9.1, the vector type had to be explicitly set to `bbq_hnsw`. Starting with 9.1, `bbq_hnsw` is the default vector type for dense vectors with more than 384 dimensions in new indices, unless another type is specified.
 
 ## 9.0.3 [elasticsearch-9.0.3-known-issues]
+
 * A bug in the merge scheduler in Elasticsearch 9.0.3 may prevent shards from closing when there isn’t enough disk space to complete a merge. As a result, operations such as closing or relocating an index may hang until sufficient disk space becomes available.
 To mitigate this issue, the disk space checker is disabled by default in 9.0.3 by setting `indices.merge.disk.check_interval` to `0` seconds. Manually enabling this setting is not recommended.