Merge branch 'main' into svilen/121525

Author: svilen-mihaylov-elastic

docs/changelog/126385.yaml

@@ -0,0 +1,6 @@
+pr: 126385
+summary: Filter out empty top docs results before merging
+area: Search
+type: bug
+issues:
+ - 126118

docs/changelog/126550.yaml

@@ -0,0 +1,6 @@
+pr: 126550
+summary: Add leniency to missing array values in mustache
+area: Infra/Scripting
+type: bug
+issues:
+ - 55200

docs/reference/search-connectors/es-connectors-known-issues.md

@@ -0,0 +1,124 @@
+---
+navigation_title: "Known issues"
+---
+
+# Connector known issues [es-connectors-known-issues]
+
+:::{important}
+Enterprise Search is not available in {{stack}} 9.0+.
+:::
+
+## Connector service [es-connectors-known-issues-connector-service]
+
+The connector service has the following known issues:
+
+* **OOM errors when syncing large database tables**
+
+    Syncs after the initial sync can cause out-of-memory (OOM) errors when syncing large database tables. This occurs because database connectors load and store IDs in memory. For tables with millions of records, this can lead to memory exhaustion if the connector service has insufficient RAM.
+
+    To mitigate this issue, you can:
+
+    * **Increase RAM allocation**:
+
+        * **Self-managed**: Increase RAM allocation for the machine/container running the connector service.
+
+            ::::{dropdown} RAM sizing guidelines
+            The following table shows the estimated RAM usage for loading IDs into memory.
+
+            |     |     |
+            | --- | --- |
+            | **Number of IDs** | **Memory Usage in MB (2X buffer)** |
+            | 1,000,000 | ≈ 45.78 MB |
+            | 10,000,000 | ≈ 457.76 MB |
+            | 50,000,000 | ≈ 2288.82 MB (≈ 2.29 GB) |
+            | 100,000,000 | ≈ 4577.64 MB (≈ 4.58 GB) |
+
+            ::::
+
+    * **Optimize** [**sync rules**](/reference/search-connectors/es-sync-rules.md):
+
+        * Review and optimize sync rules to filter and reduce data retrieved from the source before syncing.
+
+%    * **Use a self-managed connector** instead of a managed connector:
+
+%        * Because self-managed connectors run on your infrastructure, they are not subject to the same RAM limitations of the Enterprise Search node.
+
+* **Upgrades from deployments running on versions earlier than 8.9.0 can cause sync job failures**
+
+    Due to a bug, the `job_type` field mapping will be missing after upgrading from deployments running on versions earlier than 8.9.0. Sync jobs won’t be displayed in the Kibana UI (job history) and the connector service won’t be able to start new sync jobs. **This will only occur if you have previously scheduled sync jobs.**
+
+    To resolve this issue, you can manually add the missing field with the following command and trigger a sync job:
+
+    ```console
+    PUT .elastic-connectors-sync-jobs-v1/_mapping
+    {
+      "properties": {
+        "job_type": {
+          "type": "keyword"
+        }
+      }
+    }
+    ```
+
+* **The connector service will fail to sync when the connector tries to fetch more more than 2,147,483,647 (*2^31-1*) documents from a data source**
+
+    A workaround is to manually partition the data to be synced using multiple search indices.
+
+* **Custom scheduling might break when upgrading from version 8.6 or earlier.**
+
+    If you encounter the error `'custom_schedule_triggered': undefined method 'each' for nil:NilClass (NoMethodError)`, it means the custom scheduling feature migration failed. You can use the following manual workaround:
+
+    ```console
+    POST /.elastic-connectors/_update/connector-id
+    {
+      "doc": {
+        "custom_scheduling": {}
+      }
+    }
+    ```
+
+    This error can appear on Connectors or Crawlers that aren’t the cause of the issue. If the error continues, try running the above command for every document in the `.elastic-connectors` index.
+
+* **Connectors upgrading from 8.7 or earlier can be missing configuration fields**
+
+    A connector that was created prior to 8.8 can sometimes be missing configuration fields. This is a known issue for the MySQL connector but could also affect other connectors.
+
+    If the self-managed connector raises the error `Connector for <connector_id> has missing configuration fields: <field_a>, <field_b>...`, you can resolve the error by manually adding the missing configuration fields via the Dev Tools. Only the following two field properties are required, as the rest will be autopopulated by the self-managed connector:
+
+    * `type`: one of `str`, `int`, `bool`, or `list`
+    * `value`: any value, as long as it is of the correct `type` (`list` type values should be saved as comma-separated strings)
+
+        ```console
+        POST /.elastic-connectors/_update/connector_id
+        {
+          "doc" : {
+            "configuration": {
+              "field_a": {
+                "type": "str",
+                "value": ""
+              },
+              "field_b": {
+                "type": "bool",
+                "value": false
+              },
+              "field_c": {
+                "type": "int",
+                "value": 1
+              },
+              "field_d": {
+                "type": "list",
+                "value": "a,b"
+              }
+            }
+          }
+        }
+        ```
+
+* **Python connectors that upgraded from 8.7.1 will report document volumes in gigabytes (GB) instead of megabytes (MB)**
+
+    As a result, true document volume will be under-reported by a factor of 1024.
+
+
+## Individual connector known issues [es-connectors-known-issues-specific]
+
+Individual connectors may have additional known issues. Refer to [each connector’s reference documentation](/reference/search-connectors/index.md) for connector-specific known issues.

docs/reference/search-connectors/toc.yml

@@ -62,4 +62,5 @@ toc:
     - file: es-connectors-logs.md
 - file: use-cases.md
   children:
-  - file: es-connectors-overview-architecture.md
+  - file: es-connectors-overview-architecture.md
+- file: es-connectors-known-issues.md

docs/release-notes/index.md

@@ -25,29 +25,25 @@ To check for security updates, go to [Security announcements for the Elastic sta
 
 ### Highlights [elasticsearch-900-highlights]
 
-::::{dropdown} Add new experimental `rank_vectors` mapping for late-interaction second order ranking
-Late-interaction models are powerful rerankers. While their size and overall cost doesn't lend itself for HNSW indexing, utilizing them as second order reranking can provide excellent boosts in relevance. The new `rank_vectors` mapping allows for rescoring over new and novel multi-vector late-interaction models like ColBERT or ColPali.
+::::{dropdown} rank_vectors field type is now available for late-interaction ranking
 
-For more information, check [PR #118804](https://github.com/elastic/elasticsearch/pull/118804).
-::::
+[`rank_vectors`](../reference/elasticsearch/mapping-reference/rank-vectors.md) is a new field type released as an experimental feature in Elasticsearch 9.0. It is designed to be used with dense vectors and allows for late-interaction second order ranking.
 
-::::{dropdown} Enable LOOKUP JOIN in non-snapshot builds
-This effectively releases LOOKUP JOIN into tech preview. Docs will
-follow in a separate PR.
+Late-interaction models are powerful rerankers. While their size and overall cost doesn’t lend itself for HNSW indexing, utilizing them as second order reranking can provide excellent boosts in relevance. The new `rank_vectors` mapping allows for rescoring over new and novel multi-vector late-interaction models like ColBERT or ColPali.
+::::
 
-- Enable the lexing/grammar for LOOKUP JOIN in non-snapshot builds.
-- Remove the grammar for the unsupported `| JOIN ...` command (without `LOOKUP` as first keyword). The way the lexer modes work, otherwise we'd also have to enable `| JOIN ...` syntax on non-snapshot builds and would have to add additional validation to provide appropriate error messages.
-- Remove grammar for `LOOKUP JOIN index AS ...` because qualifiers are not yet supported. Otherwise we'd have to put in additional validation as well to prevent such queries.
+::::{dropdown} ES|QL LOOKUP JOIN is now available in technical preview
 
-Also fix https://github.com/elastic/elasticsearch/issues/121185
+[LOOKUP JOIN](../reference/query-languages/esql/esql-commands.md) is now available in technical preview. LOOKUP JOIN combines data from your ES|QL queries with matching records from a lookup index, enabling you to:
 
-For more information, check [PR #121193](https://github.com/elastic/elasticsearch/pull/121193).
+- Enrich your search results with reference data
+- Speed up root-cause analysis and security investigations
+- Join data across indices without complex queries
+- Reduce operational overhead when correlating events
 ::::
 
-::::{dropdown} Release semantic_text as a GA feature
-semantic_text is now an official GA (generally available) feature! This field type allows you to easily set up and perform semantic search with minimal ramp up time.
-
-For more information, check [PR #124669](https://github.com/elastic/elasticsearch/pull/124669).
+::::{dropdown} The semantic_text field type is now GA
+[`semantic_text`](../reference/elasticsearch/mapping-reference/semantic-text.md) is now an official GA (generally available) feature! This field type allows you to easily set up and perform semantic search with minimal ramp up time.
 ::::
 
 ### Features and enhancements [elasticsearch-900-features-enhancements]

modules/lang-mustache/src/main/java/org/elasticsearch/script/mustache/CustomReflectionObjectHandler.java

@@ -111,11 +111,11 @@ public Object get(Object key) {
             if ("size".equals(key)) {
                 return size();
             } else if (key instanceof Number number) {
-                return Array.get(array, number.intValue());
+                return number.intValue() >= 0 && number.intValue() < length ? Array.get(array, number.intValue()) : null;
             }
             try {
                 int index = Integer.parseInt(key.toString());
-                return Array.get(array, index);
+                return index >= 0 && index < length ? Array.get(array, index) : null;
             } catch (NumberFormatException nfe) {
                 // if it's not a number it is as if the key doesn't exist
                 return null;
@@ -169,11 +169,11 @@ public Object get(Object key) {
             if ("size".equals(key)) {
                 return col.size();
             } else if (key instanceof Number number) {
-                return Iterables.get(col, number.intValue());
+                return number.intValue() >= 0 && number.intValue() < col.size() ? Iterables.get(col, number.intValue()) : null;
             }
             try {
                 int index = Integer.parseInt(key.toString());
-                return Iterables.get(col, index);
+                return index >= 0 && index < col.size() ? Iterables.get(col, index) : null;
             } catch (NumberFormatException nfe) {
                 // if it's not a number it is as if the key doesn't exist
                 return null;

modules/lang-mustache/src/test/java/org/elasticsearch/script/mustache/MustacheTests.java

@@ -141,6 +141,26 @@ public void testMapInArrayAccess() throws Exception {
         assertThat(output, both(containsString("foo")).and(containsString("bar")));
     }
 
+    public void testMapInArrayBadAccess() throws Exception {
+        String template = "{{data.0.key}} {{data.2.key}}";
+        TemplateScript.Factory factory = engine.compile(null, template, TemplateScript.CONTEXT, Collections.emptyMap());
+        Map<String, Object> vars = new HashMap<>();
+        vars.put("data", new Object[] { singletonMap("key", "foo"), singletonMap("key", "bar") });
+        // assertThat(factory.newInstance(vars).execute(), equalTo("foo "));
+
+        vars.put("data", Arrays.asList(singletonMap("key", "foo"), singletonMap("key", "bar")));
+        factory.newInstance(vars);
+        assertThat(factory.newInstance(vars).execute(), equalTo("foo "));
+
+        // HashSet iteration order isn't fixed
+        Set<Object> setData = new HashSet<>();
+        setData.add(singletonMap("key", "foo"));
+        setData.add(singletonMap("key", "bar"));
+        vars.put("data", setData);
+        String output = factory.newInstance(vars).execute();
+        assertThat(output, both(containsString("foo")).and(not(containsString("bar"))));
+    }
+
     public void testSizeAccessForCollectionsAndArrays() throws Exception {
         String[] randomArrayValues = generateRandomStringArray(10, 20, false);
         List<String> randomList = Arrays.asList(generateRandomStringArray(10, 20, false));

muted-tests.yml

@@ -423,6 +423,18 @@ tests:
 - class: org.elasticsearch.smoketest.MlWithSecurityIT
   method: test {yaml=ml/get_trained_model_stats/Test get stats given trained models}
   issue: https://github.com/elastic/elasticsearch/issues/126510
+- class: org.elasticsearch.xpack.test.rest.XPackRestIT
+  method: test {p0=transform/transforms_stats/Test get multiple transform stats}
+  issue: https://github.com/elastic/elasticsearch/issues/126567
+- class: org.elasticsearch.xpack.test.rest.XPackRestIT
+  method: test {p0=transform/transforms_stats/Test get single transform stats when it does not have a task}
+  issue: https://github.com/elastic/elasticsearch/issues/126568
+- class: org.elasticsearch.repositories.blobstore.testkit.rest.SnapshotRepoTestKitClientYamlTestSuiteIT
+  method: test {p0=/10_analyze/Analysis without details}
+  issue: https://github.com/elastic/elasticsearch/issues/126569
+- class: org.elasticsearch.repositories.blobstore.testkit.analyze.S3RepositoryAnalysisRestIT
+  method: testRepositoryAnalysis
+  issue: https://github.com/elastic/elasticsearch/issues/126576
 
 # Examples:
 #

server/src/internalClusterTest/java/org/elasticsearch/index/shard/IndexShardIT.java

@@ -676,7 +676,7 @@ public void testInvalidateIndicesRequestCacheWhenRollbackEngine() throws Excepti
         final CountDownLatch engineResetLatch = new CountDownLatch(1);
         shard.acquireAllPrimaryOperationsPermits(ActionListener.wrap(r -> {
             try {
-                shard.resetEngineToGlobalCheckpoint();
+                shard.rollbackEngineToGlobalCheckpoint();
             } finally {
                 r.close();
                 engineResetLatch.countDown();

server/src/internalClusterTest/java/org/elasticsearch/indices/IndexingMemoryControllerIT.java

@@ -84,7 +84,8 @@ EngineConfig engineConfigWithLargerIndexingMemory(EngineConfig config) {
                 config.getRelativeTimeInNanosSupplier(),
                 config.getIndexCommitListener(),
                 config.isPromotableToPrimary(),
-                config.getMapperService()
+                config.getMapperService(),
+                config.getEngineResetLock()
             );
         }