Merge branch 'main' into sa-include-key-usage-in-http-certs

Author: slobodanadamovic

client/client-benchmark-noop-api-plugin/src/main/java/org/elasticsearch/plugin/noop/NoopPlugin.java

@@ -8,8 +8,6 @@
  */
 package org.elasticsearch.plugin.noop;
 
-import org.elasticsearch.action.ActionRequest;
-import org.elasticsearch.action.ActionResponse;
 import org.elasticsearch.action.ActionType;
 import org.elasticsearch.action.bulk.BulkResponse;
 import org.elasticsearch.action.search.SearchResponse;
@@ -41,10 +39,10 @@ public class NoopPlugin extends Plugin implements ActionPlugin {
     public static final ActionType<BulkResponse> NOOP_BULK_ACTION = new ActionType<>("mock:data/write/bulk");
 
     @Override
-    public List<ActionHandler<? extends ActionRequest, ? extends ActionResponse>> getActions() {
+    public List<ActionHandler> getActions() {
         return Arrays.asList(
-            new ActionHandler<>(NOOP_BULK_ACTION, TransportNoopBulkAction.class),
-            new ActionHandler<>(NOOP_SEARCH_ACTION, TransportNoopSearchAction.class)
+            new ActionHandler(NOOP_BULK_ACTION, TransportNoopBulkAction.class),
+            new ActionHandler(NOOP_SEARCH_ACTION, TransportNoopSearchAction.class)
         );
     }
 

docs/changelog/126264.yaml

@@ -0,0 +1,5 @@
+pr: 126264
+summary: '`FileWatchingService` shoudld not throw for missing file'
+area: Infra/Settings
+type: enhancement
+issues: []

docs/changelog/126417.yaml

@@ -0,0 +1,5 @@
+pr: 126417
+summary: Correctly handle nulls in nested paths in the remove processor
+area: Ingest Node
+type: bug
+issues: []

docs/release-notes/known-issues.md

@@ -4,154 +4,6 @@ mapped_pages:
   - https://www.elastic.co/guide/en/elasticsearch/reference/current/es-connectors-known-issues.html
 ---
 
-# Elasticsearch known issues
-
-## Connector known issues [es-connectors-known-issues]
-
-:::{important}
-Enterprise Search is not available in {{stack}} 9.0+.
-:::
-
-### Enterprise Search service: self-managed connectors [es-connectors-known-issues-enterprie-search-service]
-
-As of **8.10.0** self-managed connectors no longer require the Enterprise Search service to be running on your Elastic deployment. However, if you are upgrading connectors from versions *earlier than 8.9*, you’ll need to run Enterprise Search once to migrate your connectors to the new format.
-
-Some points to note about this migration:
-
-* This involves updating system indices that store configuration and sync history for your connectors.
-* This is an in-place operation, meaning no temporary or backup indices will be created.
-* Therefore, it is important to take a snapshot of the Elasticsearch cluster before upgrading— in the unlikely event of an index migration failure.
-
-If you have trouble with this migration, please contact support.
-
-::::{warning}
-To run self-managed connectors your self-deployed connector service version must match your Elasticsearch version. For example, if you’re running Elasticsearch 8.10.1, your connector service should be version 8.10.1.x. Elastic does not support deployments running mismatched versions (except during upgrades).
-
-::::
-
-
-
-### Connector service [es-connectors-known-issues-connector-service]
-
-The connector service has the following known issues:
-
-* **OOM errors when syncing large database tables**
-
-    Syncs after the initial sync can cause out-of-memory (OOM) errors when syncing large database tables. This occurs because database connectors load and store IDs in memory. For tables with millions of records, this can lead to memory exhaustion if the connector service has insufficient RAM.
-
-    To mitigate this issue, you can:
-
-    * **Increase RAM allocation**:
-
-        * **Elastic Cloud**: Upgrade the Enterprise Search instance to a larger size. Note that for Elastic managed connectors running on Elastic Cloud, the connector service runs on the Enterprise Search node. It only has access to up to 40% of the node’s RAM allocation.
-        * **Self-managed**: Increase RAM allocation for the machine/container running the connector service.
-
-            ::::{dropdown} RAM **sizing guidelines**
-            The following table shows the estimated RAM usage for loading IDs into memory.
-
-            |     |     |
-            | --- | --- |
-            | **Number of IDs** | **Memory Usage in MB (2X buffer)** |
-            | 1,000,000 | ≈ 45.78 MB |
-            | 10,000,000 | ≈ 457.76 MB |
-            | 50,000,000 | ≈ 2288.82 MB (≈ 2.29 GB) |
-            | 100,000,000 | ≈ 4577.64 MB (≈ 4.58 GB) |
-
-            ::::
-
-    * **Optimize** [**sync rules**](/reference/search-connectors/es-sync-rules.md):
-
-        * Review and optimize sync rules to filter and reduce data retrieved from the source before syncing.
-
-    * **Use a self-managed connector** instead of a managed connector:
-
-        * Because self-managed connectors run on your infrastructure, they are not subject to the same RAM limitations of the Enterprise Search node.
-
-* **Upgrades from deployments running on versions earlier than 8.9.0 can cause sync job failures**
-
-    Due to a bug, the `job_type` field mapping will be missing after upgrading from deployments running on versions earlier than 8.9.0. Sync jobs won’t be displayed in the Kibana UI (job history) and the connector service won’t be able to start new sync jobs. **This will only occur if you have previously scheduled sync jobs.**
-
-    To resolve this issue, you can manually add the missing field with the following command and trigger a sync job:
-
-    ```console
-    PUT .elastic-connectors-sync-jobs-v1/_mapping
-    {
-      "properties": {
-        "job_type": {
-          "type": "keyword"
-        }
-      }
-    }
-    ```
-
-* **The connector service will fail to sync when the connector tries to fetch more more than 2,147,483,647 (*2^31-1*) documents from a data source**
-
-    A workaround is to manually partition the data to be synced using multiple search indices.
-
-* **Custom scheduling might break when upgrading from version 8.6 or earlier.**
-
-    If you encounter the error `'custom_schedule_triggered': undefined method 'each' for nil:NilClass (NoMethodError)`, it means the custom scheduling feature migration failed. You can use the following manual workaround:
-
-    ```console
-    POST /.elastic-connectors/_update/connector-id
-    {
-      "doc": {
-        "custom_scheduling": {}
-      }
-    }
-    ```
-
-    This error can appear on Connectors or Crawlers that aren’t the cause of the issue. If the error continues, try running the above command for every document in the `.elastic-connectors` index.
-
-* **Connectors upgrading from 8.7 or earlier can be missing configuration fields**
-
-    A connector that was created prior to 8.8 can sometimes be missing configuration fields. This is a known issue for the MySQL connector but could also affect other connectors.
-
-    If the self-managed connector raises the error `Connector for <connector_id> has missing configuration fields: <field_a>, <field_b>...`, you can resolve the error by manually adding the missing configuration fields via the Dev Tools. Only the following two field properties are required, as the rest will be autopopulated by the self-managed connector:
-
-    * `type`: one of `str`, `int`, `bool`, or `list`
-    * `value`: any value, as long as it is of the correct `type` (`list` type values should be saved as comma-separated strings)
-
-        ```console
-        POST /.elastic-connectors/_update/connector_id
-        {
-          "doc" : {
-            "configuration": {
-              "field_a": {
-                "type": "str",
-                "value": ""
-              },
-              "field_b": {
-                "type": "bool",
-                "value": false
-              },
-              "field_c": {
-                "type": "int",
-                "value": 1
-              },
-              "field_d": {
-                "type": "list",
-                "value": "a,b"
-              }
-            }
-          }
-        }
-        ```
-
-* **Python connectors that upgraded from 8.7.1 will report document volumes in gigabytes (GB) instead of megabytes (MB)**
-
-    As a result, true document volume will be under-reported by a factor of 1024.
-
-* **The following Elastic managed connectors will not run correctly on Elastic Cloud in 8.9.0.** They are still available as self-managed connectors.
-
-    * Azure Blob Storage
-    * Confluence Cloud & Server
-    * Jira Cloud & Server
-    * Network drives
-
-
-
-## Individual connector known issues [es-connectors-known-issues-specific]
-
-Individual connectors may have additional known issues. Refer to [each connector’s reference documentation](/reference/search-connectors/index.md) for connector-specific known issues.
+# Elasticsearch known issues [elasticsearch-known-issues]
+Known issues are significant defects or limitations that may impact your implementation. These issues are actively being worked on and will be addressed in a future release. Review the Elasticsearch known issues to help you make informed decisions, such as upgrading to a new version.
 

modules/data-streams/src/main/java/org/elasticsearch/datastreams/DataStreamsPlugin.java

@@ -10,8 +10,6 @@
 
 import org.apache.lucene.util.SetOnce;
 import org.elasticsearch.ElasticsearchException;
-import org.elasticsearch.action.ActionRequest;
-import org.elasticsearch.action.ActionResponse;
 import org.elasticsearch.action.datastreams.CreateDataStreamAction;
 import org.elasticsearch.action.datastreams.DataStreamsStatsAction;
 import org.elasticsearch.action.datastreams.DeleteDataStreamAction;
@@ -225,24 +223,24 @@ public Collection<?> createComponents(PluginServices services) {
     }
 
     @Override
-    public List<ActionHandler<? extends ActionRequest, ? extends ActionResponse>> getActions() {
-        List<ActionHandler<? extends ActionRequest, ? extends ActionResponse>> actions = new ArrayList<>();
-        actions.add(new ActionHandler<>(CreateDataStreamAction.INSTANCE, CreateDataStreamTransportAction.class));
-        actions.add(new ActionHandler<>(DeleteDataStreamAction.INSTANCE, DeleteDataStreamTransportAction.class));
-        actions.add(new ActionHandler<>(GetDataStreamAction.INSTANCE, TransportGetDataStreamsAction.class));
-        actions.add(new ActionHandler<>(DataStreamsStatsAction.INSTANCE, DataStreamsStatsTransportAction.class));
-        actions.add(new ActionHandler<>(MigrateToDataStreamAction.INSTANCE, MigrateToDataStreamTransportAction.class));
-        actions.add(new ActionHandler<>(PromoteDataStreamAction.INSTANCE, PromoteDataStreamTransportAction.class));
-        actions.add(new ActionHandler<>(ModifyDataStreamsAction.INSTANCE, ModifyDataStreamsTransportAction.class));
-        actions.add(new ActionHandler<>(PutDataStreamLifecycleAction.INSTANCE, TransportPutDataStreamLifecycleAction.class));
-        actions.add(new ActionHandler<>(GetDataStreamLifecycleAction.INSTANCE, TransportGetDataStreamLifecycleAction.class));
-        actions.add(new ActionHandler<>(DeleteDataStreamLifecycleAction.INSTANCE, TransportDeleteDataStreamLifecycleAction.class));
-        actions.add(new ActionHandler<>(ExplainDataStreamLifecycleAction.INSTANCE, TransportExplainDataStreamLifecycleAction.class));
-        actions.add(new ActionHandler<>(GetDataStreamLifecycleStatsAction.INSTANCE, TransportGetDataStreamLifecycleStatsAction.class));
+    public List<ActionHandler> getActions() {
+        List<ActionHandler> actions = new ArrayList<>();
+        actions.add(new ActionHandler(CreateDataStreamAction.INSTANCE, CreateDataStreamTransportAction.class));
+        actions.add(new ActionHandler(DeleteDataStreamAction.INSTANCE, DeleteDataStreamTransportAction.class));
+        actions.add(new ActionHandler(GetDataStreamAction.INSTANCE, TransportGetDataStreamsAction.class));
+        actions.add(new ActionHandler(DataStreamsStatsAction.INSTANCE, DataStreamsStatsTransportAction.class));
+        actions.add(new ActionHandler(MigrateToDataStreamAction.INSTANCE, MigrateToDataStreamTransportAction.class));
+        actions.add(new ActionHandler(PromoteDataStreamAction.INSTANCE, PromoteDataStreamTransportAction.class));
+        actions.add(new ActionHandler(ModifyDataStreamsAction.INSTANCE, ModifyDataStreamsTransportAction.class));
+        actions.add(new ActionHandler(PutDataStreamLifecycleAction.INSTANCE, TransportPutDataStreamLifecycleAction.class));
+        actions.add(new ActionHandler(GetDataStreamLifecycleAction.INSTANCE, TransportGetDataStreamLifecycleAction.class));
+        actions.add(new ActionHandler(DeleteDataStreamLifecycleAction.INSTANCE, TransportDeleteDataStreamLifecycleAction.class));
+        actions.add(new ActionHandler(ExplainDataStreamLifecycleAction.INSTANCE, TransportExplainDataStreamLifecycleAction.class));
+        actions.add(new ActionHandler(GetDataStreamLifecycleStatsAction.INSTANCE, TransportGetDataStreamLifecycleStatsAction.class));
         if (DataStream.isFailureStoreFeatureFlagEnabled()) {
-            actions.add(new ActionHandler<>(GetDataStreamOptionsAction.INSTANCE, TransportGetDataStreamOptionsAction.class));
-            actions.add(new ActionHandler<>(PutDataStreamOptionsAction.INSTANCE, TransportPutDataStreamOptionsAction.class));
-            actions.add(new ActionHandler<>(DeleteDataStreamOptionsAction.INSTANCE, TransportDeleteDataStreamOptionsAction.class));
+            actions.add(new ActionHandler(GetDataStreamOptionsAction.INSTANCE, TransportGetDataStreamOptionsAction.class));
+            actions.add(new ActionHandler(PutDataStreamOptionsAction.INSTANCE, TransportPutDataStreamOptionsAction.class));
+            actions.add(new ActionHandler(DeleteDataStreamOptionsAction.INSTANCE, TransportDeleteDataStreamOptionsAction.class));
         }
         return actions;
     }

modules/ingest-common/src/main/java/org/elasticsearch/ingest/common/IngestCommonPlugin.java

@@ -9,8 +9,6 @@
 
 package org.elasticsearch.ingest.common;
 
-import org.elasticsearch.action.ActionRequest;
-import org.elasticsearch.action.ActionResponse;
 import org.elasticsearch.cluster.metadata.IndexNameExpressionResolver;
 import org.elasticsearch.cluster.node.DiscoveryNodes;
 import org.elasticsearch.common.io.stream.NamedWriteableRegistry;
@@ -81,8 +79,8 @@ public Map<String, Processor.Factory> getProcessors(Processor.Parameters paramet
     }
 
     @Override
-    public List<ActionHandler<? extends ActionRequest, ? extends ActionResponse>> getActions() {
-        return List.of(new ActionHandler<>(GrokProcessorGetAction.INSTANCE, GrokProcessorGetAction.TransportAction.class));
+    public List<ActionHandler> getActions() {
+        return List.of(new ActionHandler(GrokProcessorGetAction.INSTANCE, GrokProcessorGetAction.TransportAction.class));
     }
 
     @Override

modules/ingest-common/src/main/java/org/elasticsearch/ingest/common/RemoveProcessor.java

@@ -96,24 +96,24 @@ public Factory(ScriptService scriptService) {
         @Override
         public RemoveProcessor create(
             Map<String, Processor.Factory> registry,
-            String processorTag,
+            String tag,
             String description,
             Map<String, Object> config,
             ProjectId projectId
         ) throws Exception {
-            final List<TemplateScript.Factory> compiledTemplatesToRemove = getTemplates(processorTag, config, "field");
-            final List<TemplateScript.Factory> compiledTemplatesToKeep = getTemplates(processorTag, config, "keep");
+            final List<TemplateScript.Factory> compiledTemplatesToRemove = getTemplates(tag, config, "field");
+            final List<TemplateScript.Factory> compiledTemplatesToKeep = getTemplates(tag, config, "keep");
 
             if (compiledTemplatesToRemove.isEmpty() && compiledTemplatesToKeep.isEmpty()) {
-                throw newConfigurationException(TYPE, processorTag, "keep", "or [field] must be specified");
+                throw newConfigurationException(TYPE, tag, "keep", "or [field] must be specified");
             }
 
             if (compiledTemplatesToRemove.isEmpty() == false && compiledTemplatesToKeep.isEmpty() == false) {
-                throw newConfigurationException(TYPE, processorTag, "keep", "and [field] cannot both be used in the same processor");
+                throw newConfigurationException(TYPE, tag, "keep", "and [field] cannot both be used in the same processor");
             }
 
-            boolean ignoreMissing = ConfigurationUtils.readBooleanProperty(TYPE, processorTag, config, "ignore_missing", false);
-            return new RemoveProcessor(processorTag, description, compiledTemplatesToRemove, compiledTemplatesToKeep, ignoreMissing);
+            boolean ignoreMissing = ConfigurationUtils.readBooleanProperty(TYPE, tag, config, "ignore_missing", false);
+            return new RemoveProcessor(tag, description, compiledTemplatesToRemove, compiledTemplatesToKeep, ignoreMissing);
         }
 
         private List<TemplateScript.Factory> getTemplates(String processorTag, Map<String, Object> config, String propertyName) {