diff --git a/docs/reference/migration/migrate_9_0.asciidoc b/docs/reference/migration/migrate_9_0.asciidoc index 71516fdd540d6..38685d9ac1d1f 100644 --- a/docs/reference/migration/migrate_9_0.asciidoc +++ b/docs/reference/migration/migrate_9_0.asciidoc @@ -21,9 +21,20 @@ and prevent them from operating normally. Before upgrading to 9.0, review these changes and take the described steps to mitigate the impact. +[discrete] +[[breaking_90_aggregations_changes]] +==== Aggregations changes -There are no notable breaking changes in {es} 9.0. -But there are some less critical breaking changes. +[[remove_date_histogram_boolean_support]] +.Remove date histogram boolean support +[%collapsible] +==== +*Details* + +Elasticsearch no longer allows running Date Histogram aggregations over boolean fields. Instead, use Terms aggregation for boolean fields. + +*Impact* + +We expect the impact to be minimal, as this never produced good results, and has been deprecated for years. +==== [discrete] [[breaking_90_analysis_changes]] @@ -73,11 +84,43 @@ Lucene 10 ships with an updated Korean dictionary (mecab-ko-dic-2.1.1). For det The change is small and should generally provide better analysis results. Existing indices for full-text use cases should be reindexed though. ==== - [discrete] [[breaking_90_cluster_and_node_setting_changes]] ==== Cluster and node setting changes +[[configuring_bind_dn_in_an_ldap_or_active_directory_ad_realm_without_corresponding_bind_password_will_prevent_node_from_starting]] +.-| Configuring a bind DN in an LDAP or Active Directory (AD) realm without a corresponding bind password will prevent node from starting +[%collapsible] +==== +*Details* + +-| For LDAP or AD authentication realms, setting a bind DN (via the `xpack.security.authc.realms.ldap.*.bind_dn` or `xpack.security.authc.realms.active_directory.*.bind_dn` realm settings) without a bind password is a misconfiguration that may prevent successful authentication to the node. Nodes will fail to start if a bind DN is specified without a password. + +*Impact* + +-| If you have a bind DN configured for an LDAP or AD authentication realm, set a bind password for {ref}/ldap-realm.html#ldap-realm-configuration[LDAP] or {ref}/active-directory-realm.html#ad-realm-configuration[Active Directory]. Configuring a bind DN without a password prevents the misconfigured node from starting. +==== + +[[deprecated_tracing_apm_settings_got_removed]] +.Deprecated tracing.apm.* settings got removed. +[%collapsible] +==== +*Details* + +Deprecated `tracing.apm.*` settings got removed, use respective `telemetry.*` / `telemetry.tracing.*` settings instead. + +*Impact* + +9.x nodes will refuse to start if any such setting (including secret settings) is still present. +==== + +[[limit_bytesizeunit_to_2_decimals]] +.Limit `ByteSizeUnit` to 2 decimals +[%collapsible] +==== +*Details* + +In the past, byte values like `1.25 mb` were allowed but deprecated. Now, values with up to two decimal places are allowed, unless the unit is bytes, in which case no decimals are allowed. Values with too many decimal places result in an error. + +*Impact* + +Values with more than two decimal places, like `0.123 mb` will be rejected as an error, where in the past, they'd be accepted with a deprecation warning. +==== + [[minimum_shard_balancer_threshold_1_0]] .Minimum shard balancer threshold is now 1.0 [%collapsible] @@ -89,6 +132,28 @@ Earlier versions of {es} accepted any non-negative value for `cluster.routing.al Do not set `cluster.routing.allocation.balance.threshold` to a value less than `1.0`. ==== +[[remove_tlsv1_1_from_default_protocols]] +.Remove TLSv1.1 from default protocols +[%collapsible] +==== +*Details* + +TLSv1.1 is no longer enabled by default. Prior to version 9.0, Elasticsearch would attempt to enable TLSv1.1 if the JDK supported it. In most cases, including all cases where Elasticsearch 8 was running with the bundled JDK, the JDK would not support TLSv1.1, so that protocol would not be available in Elasticsearch. However, if Elasticsearch was running on an old JDK or a JDK that have been reconfigured to support TLSv1.1, then the protocol would automatically be available within Elasticsearch. As of Elasticsearch 9.0, this is no longer true. If you wish to enable TLSv1.1 then you must enable it within the JDK and also enable it within Elasticsearch by using the `ssl.supported_protocols` setting. + +*Impact* + +Most users will not be impacted. If your Elastisearch 8 cluster was using a custom JDK and you relied on TLSv1.1, then you will need to explicitly enable TLSv1.1 within Elasticsearch (as well as enabling it within your JDK) +==== + +[[remove_client_type_setting]] +.Remove `client.type` setting +[%collapsible] +==== +*Details* + +The node setting `client.type` has been ignored since the node client was removed in 8.0. The setting is now removed. + +*Impact* + +Remove the `client.type` setting from `elasticsearch.yml` +==== + [[remove_cluster_routing_allocation_disk_watermark_enable_for_single_data_node_setting]] .Remove `cluster.routing.allocation.disk.watermark.enable_for_single_data_node` setting [%collapsible] @@ -122,6 +187,21 @@ Earlier versions of {es} had a `discovery.type` setting which permitted values t Remove any value for `discovery.type` from your `elasticsearch.yml` configuration file. ==== +[discrete] +[[breaking_90_index_setting_changes]] +==== Index setting changes + +[[remove_ability_to_read_frozen_indices]] +.Remove the ability to read frozen indices +[%collapsible] +==== +*Details* + +The ability to read frozen indices has been removed. (Frozen indices are no longer useful due to improvements in heap memory usage. The ability to freeze indices was removed in 8.0.) + +*Impact* + +Users must unfreeze any frozen indices before upgrading. +==== + [discrete] [[breaking_90_ingest_changes]] ==== Ingest changes @@ -148,6 +228,44 @@ The option fallback_to_default_databases on the geoip ingest processor has been Customers should stop remove the noop fallback_to_default_databases option on any geoip ingest processors. ==== +[discrete] +[[breaking_90_logging_changes]] +==== Logging changes + +[[deprecation_logging_value_change_for_data_stream_dataset_event_dataset]] +.Deprecation logging value change for "data_stream.dataset" and "event.dataset" +[%collapsible] +==== +*Details* + +This change modifies the "data_stream.dataset" and "event.dataset" value for deprecation logging +to use the value `elasticsearch.deprecation` instead of `deprecation.elasticsearch`. This is now +consistent with other values where the name of the service is the first part of the key. + +*Impact* + +If you are directly consuming deprecation logs for "data_stream.dataset" and "event.dataset" and filtering on +this value, you will need to update your filters to use `elasticsearch.deprecation` instead of +`deprecation.elasticsearch`. +==== + +[discrete] +[[breaking_90_logs_changes]] +==== Logs changes + +[[conditionally_enable_logsdb_by_default]] +.Conditionally enable logsdb by default +[%collapsible] +==== +*Details* + +Logsdb will be enabled by default for data streams matching with logs-*-* pattern. +If upgrading from 8.x to 9.x and data streams matching with log-*-* do exist, +then Logsdb will not be enabled by default. + +*Impact* + +Logsdb reduce storage footprint in Elasticsearch for logs, but there are side effects +to be taken into account that are described in the Logsdb docs: +https://www.elastic.co/guide/en/elasticsearch/reference/current/logs-data-stream.html#upgrade-to-logsdb-notes +==== + [discrete] [[breaking_90_mapping_changes]] ==== Mapping changes @@ -157,12 +275,38 @@ Customers should stop remove the noop fallback_to_default_databases option on an [%collapsible] ==== *Details* + -The type, fields, copy_to and boost parameters are no longer supported in metadata field definition +The type, fields, copy_to and boost parameters are no longer supported in metadata field definition starting with version 9. *Impact* + Users providing type, fields, copy_to or boost as part of metadata field definition should remove them from their mappings. ==== +[[turn_source_meta_fieldmappers_mode_attribute_into_no_op]] +.Turn `_source` meta fieldmapper's mode attribute into a no-op +[%collapsible] +==== +*Details* + +The `mode` mapping attribute of `_source` metadata field mapper has been turned into a no-op. Instead the `index.mapping.source.mode` index setting should be used to configure source mode. + +*Impact* + +Configuring the `mode` attribute for the `_source` meta field mapper will have no effect on indices created with Elasticsearch 9.0.0 or later. Note that `_source.mode` configured on indices before upgrading to 9.0.0 or later will remain efficive after upgrading. +==== + +[discrete] +[[breaking_90_packaging_changes]] +==== Packaging changes + +[[disable_machine_learning_on_macos_x86_64]] +.Disable machine learning on macOS x86_64 +[%collapsible] +==== +*Details* + +The machine learning plugin is permanently disabled on macOS x86_64. For the last three years Apple has been selling hardware based on the arm64 architecture, and support will increasingly focus on this architecture in the future. Changes to upstream dependencies of Elastic's machine learning functionality have made it unviable for Elastic to continue to build machine learning on macOS x86_64. + +*Impact* + +To continue to use machine learning functionality on macOS please switch to an arm64 machine (Apple silicon). Alternatively, it will still be possible to run Elasticsearch with machine learning enabled in a Docker container on macOS x86_64. +==== + [discrete] [[breaking_90_rest_api_changes]] ==== REST API changes @@ -178,6 +322,17 @@ Previously, the following classes of malformed input were deprecated but not rej Users must provide well-formed input when using the bulk API. (They can request REST API compatibility with v8 to get the previous behaviour back as an interim measure.) ==== +[[change_most_elasticsearch_timeouts_to_429_response_instead_of_5xx]] +.Change most Elasticsearch timeouts to 429 response instead of 5xx +[%collapsible] +==== +*Details* + +When a timeout occurs in most REST requests, whether via a per-request timeout, or a system default, the request would return a 5xx response code. The response code from those APIs when a timeout occurs is now 429. + +*Impact* + +Adjust any code relying on retrying on 5xx responses for timeouts to look for a 429 response code and inspect the response to determine whether a timeout occured. +==== + [[error_json_structure_has_changed_when_detailed_errors_are_disabled]] .Error JSON structure has changed when detailed errors are disabled [%collapsible] @@ -201,6 +356,17 @@ detailed errors are enabled. To use the previous structure for non-detailed errors, use the v8 REST API. ==== +[[remove_any_references_to_org_elasticsearch_core_restapiversion_v_7]] +.Remove any references to org.elasticsearch.core.RestApiVersion#V_7 +[%collapsible] +==== +*Details* + +This PR removes all references to V_7 in the Rest API. V7 features marked for deprecation have been removed. + +*Impact* + +This change is breaking for any external plugins/clients that rely on the V_7 enum or deprecated version 7 functionality +==== + [[remove_cluster_state_from_cluster_reroute_response]] .Remove cluster state from `/_cluster/reroute` response [%collapsible] @@ -234,6 +400,17 @@ The deprecated range query parameters `to`, `from`, `include_lower`, and `includ Users should use `lt`, `lte`, `gt`, and `gte` query parameters instead. ==== +[[remove_old_knn_search_tech_preview_api_in_v9]] +.Remove old `_knn_search` tech preview API in v9 +[%collapsible] +==== +*Details* + +The original, tech-preview api for vector search, `_knn_search`, has been removed in v9. For all vector search operations, you should utilize the `_search` endpoint. + +*Impact* + +The `_knn_search` API is now inaccessible without providing a compatible-with flag for v8. +==== + [[remove_support_for_deprecated_force_source_highlighting_parameter]] .Remove support for deprecated `force_source` highlighting parameter [%collapsible] @@ -245,23 +422,89 @@ The deprecated highlighting `force_source` parameter is no longer supported. Users should remove usages of the `force_source` parameter from their search requests. ==== +[[remove_unfreeze_rest_endpoint]] +.Remove unfreeze REST endpoint +[%collapsible] +==== +*Details* + +The `/{index}/_unfreeze` REST endpoint is no longer supported. This API was deprecated, and the corresponding `/{index}/_freeze` endpoint was removed in 8.0. + +*Impact* + +None, since it is not possible to have a frozen index in a version which is readable by Elasticsearch 9.0 +==== + +[[removing_support_for_types_field_in_watcher_search]] +.Removing support for types field in watcher search +[%collapsible] +==== +*Details* + +Previously, setting the `input.search.request.types` field in the payload when creating a watcher to an empty array was allowed, although it resulted in a deprecation warning and had no effect (and any value other than an empty array would result in an error). Now, support for this field is entirely removed, and the empty array will also result in an error. + +*Impact* + +Users should stop setting this field (which did not have any effect anyway). +==== + +[[restrict_connector_apis_to_manage_monitor_connector_privileges]] +.Restrict Connector APIs to manage/monitor_connector privileges +[%collapsible] +==== +*Details* + +Connector APIs now enforce the manage_connector and monitor_connector privileges (introduced in 8.15), replacing the previous reliance on index-level permissions for .elastic-connectors and .elastic-connectors-sync-jobs in API calls. + +*Impact* + +Connector APIs now require manage_connector and monitor_connector privileges +==== + +[[set_allow_partial_search_results_true_by_default]] +.Set allow_partial_search_results=true by default +[%collapsible] +==== +*Details* + +Before this change, in case of shard failures, EQL queries always returned an error. With this change, they will keep running and will return partial results. + +*Impact* + +EQL queries that would previously fail due to shard failures, will now succeed and return partial results. The previous defaults can be restored by setting `xpack.eql.default_allow_partial_results` cluster setting to `false` or setting with `allow_partial_search_results` to `false` in the query request. +==== + [discrete] -[[breaking_90_transforms_changes]] -==== {transforms-cap} changes +[[breaking_90_search_changes]] +==== Search changes -[[updating_deprecated_transform_roles]] -.Updating deprecated {transform} roles (`data_frame_transforms_admin` and `data_frame_transforms_user`) +[[adjust_random_score_default_field_to_seq_no_field]] +.Adjust `random_score` default field to `_seq_no` field [%collapsible] ==== *Details* + -The `data_frame_transforms_admin` and `data_frame_transforms_user` {transform} roles have been deprecated. +When providing a 'seed' parameter to a 'random_score' function in the 'function_score' query but NOT providing a 'field', the default 'field' is switched from '_id' to '_seq_no'. *Impact* + -Users must update any existing {transforms} that use deprecated {transform} roles (`data_frame_transforms_admin` or `data_frame_transforms_user`) to use the new equivalent {transform} roles (`transform_admin` or `transform_user`). -To update the {transform} roles: +The random scoring and ordering may change when providing a 'seed' and not providing a 'field' to a 'random_score' function. +==== + +[[change_semantic_text_to_act_like_normal_text_field]] +.Change semantic text to act like a normal text field +[%collapsible] +==== +*Details* + +The previous semantic_text format used a complex subfield structure in _source to store the embeddings. This complicated interactions/integrations with semantic_text fields and _source in general. This new semantic_text format treats it as a normal text field, where the field's value in _source is the value assigned by the user. + +*Impact* + +Users who parsed the subfield structure of the previous semantic_text format in _source will need to update their parsing logic. The new format does not directly expose the chunks and embeddings generated from the input text. The new format will be applied to all new indices, any existing indices will continue to use the previous format. +==== -1. Switch to a user with the `transform_admin` role (to replace `data_frame_transforms_admin`) or the `transform_user` role (to replace `data_frame_transforms_user`). -2. Call the <> with that user. +[discrete] +[[breaking_90_transform_changes]] +==== Transform changes + +[[remove_data_frame_transforms_roles]] +.Remove `data_frame_transforms` roles +[%collapsible] +==== +*Details* + +`data_frame_transforms_admin` and `data_frame_transforms_user` were deprecated in Elasticsearch 7 and are being removed in Elasticsearch 9. `data_frame_transforms_admin` is now `transform_admin`. `data_frame_transforms_user` is now `transform_user`. Users must call the `_update` API to replace the permissions on the Transform before the Transform can be started. + +*Impact* + +Transforms created with either the `data_frame_transforms_admin` or the `data_frame_transforms_user` role will fail to start. The Transform will remain in a `stopped` state, and its health will be red while displaying permission failures. ==== @@ -279,18 +522,33 @@ To find out if you are using any deprecated functionality, enable <>. [discrete] -[[deprecations_90_mapping]] -==== Mapping deprecations +[[deprecations_90_authorization]] +==== Authorization deprecations -[[deprecate_source_mode_in_mappings]] -.Deprecate `_source.mode` in mappings +[[deprecate_certificate_based_remote_cluster_security_model]] +.Deprecate certificate based remote cluster security model [%collapsible] ==== *Details* + -Configuring `_source.mode` in mappings is deprecated and will be removed in future versions. Use `index.mapping.source.mode` index setting instead. +-| <> is deprecated and will be removed in a future major version. Users are encouraged to <>. The <> is preferred way to configure remote clusters, as it allows to follow security best practices when setting up remote cluster connections and defining fine-grained access control. *Impact* + -Use `index.mapping.source.mode` index setting instead +-| If you have configured remote clusters with certificate-based security model, you should <>. Configuring a remote cluster using <>, generates a warning in the deprecation logs. +==== + +[discrete] +[[deprecations_90_es_ql]] +==== ES|QL deprecations + +[[drop_support_for_brackets_from_metadata_syntax]] +.Drop support for brackets from METADATA syntax +[%collapsible] +==== +*Details* + +Please describe the details of this change for the release notes. You can use asciidoc. + +*Impact* + +Please describe the impact of this change to users ==== [discrete] @@ -320,319 +578,32 @@ In the current version there is no impact. In a future version, users of the `el ==== [discrete] -[[breaking_90_anomaly_detection_results]] -=== Anomaly detection results migration - -The {anomaly-detect} result indices `.ml-anomalies-*` created in {es} 7.x must be either reindexed, marked read-only, or deleted before upgrading to 9.x. - -**Reindexing**: While anomaly detection results are being reindexed, jobs continue to run and process new data. -However, you cannot completely delete an {anomaly-job} that stores results in this index until the reindexing is complete. - -**Marking indices as read-only**: This is useful for large indexes that contain the results of only one or a few {anomaly-jobs}. -If you delete these jobs later, you will not be able to create a new job with the same name. - -**Deleting**: Delete jobs that are no longer needed in the {ml-app} in {kib}. -The result index is deleted when all jobs that store results in it have been deleted. - -[[which_indices_require_attention]] -.Which indices require attention? -[%collapsible] -==== - -To identify indices that require action, use the <>: - -[source,console] ------------------------------------------------------------- -GET /.ml-anomalies-*/_migration/deprecations ------------------------------------------------------------- -// TEST[skip:TBD] - -The response contains the list of critical deprecation warnings in the `index_settings` section: - -[source,console-result] ------------------------------------------------------------- -"index_settings": { - ".ml-anomalies-shared": [ - { - "level": "critical", - "message": "Index created before 8.0", - "url": "https://ela.st/es-deprecation-8-reindex", - "details": "This index was created with version 7.8.23 and is not compatible with 9.0. Reindex or remove the index before upgrading.", - "resolve_during_rolling_upgrade": false - } - ] - } ------------------------------------------------------------- -// NOTCONSOLE - - -==== - -[[reindex_anomaly_result_index]] -.Reindexing anomaly result indices -[%collapsible] -==== -For an index with less than 10GB that contains results from multiple jobs that are still required, we recommend reindexing into a new format using UI. -You can use the <> to obtain the size of an index: - -[source,console] ------------------------------------------------------------- -GET _cat/indices/.ml-anomalies-custom-example?v&h=index,store.size ------------------------------------------------------------- -// TEST[skip:TBD] - -The reindexing can be initiated in the Kibana Upgrade Assistant. - -If an index size is greater than 10 GB it is recommended to use the Reindex API. -Reindexing consists of the following steps: - -. Set the original index to read-only. -+ --- -[source,console] ------------------------------------------------------------- -PUT .ml-anomalies-custom-example/_block/read_only ------------------------------------------------------------- -// TEST[skip:TBD] --- - -. Create a new index from the legacy index. -+ --- -[source,console] ------------------------------------------------------------- -POST _create_from/.ml-anomalies-custom-example/.reindexed-v9-ml-anomalies-custom-example ------------------------------------------------------------- -// TEST[skip:TBD] --- - -. Reindex documents. -To accelerate the reindexing process, it is recommended that the number of replicas be set to `0` before the reindexing and then set back to the original number once it is completed. -.. Get the number of replicas. -+ --- -[source,console] ------------------------------------------------------------- -GET /.reindexed-v9-ml-anomalies-custom-example/_settings ------------------------------------------------------------- -// TEST[skip:TBD] -Note the number of replicas in the response. For example: -[source,console-result] ------------------------------------------------------------- -{ - ".reindexed-v9-ml-anomalies-custom-example": { - "settings": { - "index": { - "number_of_replicas": "1", - "number_of_shards": "1" - } - } - } -} ------------------------------------------------------------- -// NOTCONSOLE --- -.. Set the number of replicas to `0`. -+ --- -[source,console] ------------------------------------------------------------- -PUT /.reindexed-v9-ml-anomalies-custom-example/_settings -{ - "index": { - "number_of_replicas": 0 - } -} ------------------------------------------------------------- -// TEST[skip:TBD] --- -.. Start the reindexing process in asynchronous mode. -+ --- -[source,console] ------------------------------------------------------------- -POST _reindex?wait_for_completion=false -{ - "source": { - "index": ".ml-anomalies-custom-example" - }, - "dest": { - "index": ".reindexed-v9-ml-anomalies-custom-example" - } -} ------------------------------------------------------------- -// TEST[skip:TBD] -The response will contain a task_id. You can check when the task is completed using the following command: -[source,console] ------------------------------------------------------------- -GET _tasks/ ------------------------------------------------------------- -// TEST[skip:TBD] --- -.. Set the number of replicas to the original number when the reindexing is finished. -+ --- -[source,console] ------------------------------------------------------------- -PUT /.reindexed-v9-ml-anomalies-custom-example/_settings -{ - "index": { - "number_of_replicas": "" - } -} ------------------------------------------------------------- -// TEST[skip:TBD] --- - -. Get the aliases the original index is pointing to. -+ --- -[source,console] ------------------------------------------------------------- -GET .ml-anomalies-custom-example/_alias ------------------------------------------------------------- -// TEST[skip:TBD] - -The response may contain multiple aliases if the results of multiple jobs are stored in the same index. - -[source,console-result] ------------------------------------------------------------- -{ - ".ml-anomalies-custom-example": { - "aliases": { - ".ml-anomalies-example1": { - "filter": { - "term": { - "job_id": { - "value": "example1" - } - } - }, - "is_hidden": true - }, - ".ml-anomalies-example2": { - "filter": { - "term": { - "job_id": { - "value": "example2" - } - } - }, - "is_hidden": true - } - } - } -} ------------------------------------------------------------- -// NOTCONSOLE --- - -. Now you can reassign the aliases to the new index and delete the original index in one step. -Note that when adding the new index to the aliases, you must use the same filter and is_hidden parameters as for the original index. -+ --- -[source,console] ------------------------------------------------------------- -POST _aliases -{ - "actions": [ - { - "add": { - "index": ".reindexed-v9-ml-anomalies-custom-example", - "alias": ".ml-anomalies-example1", - "filter": { - "term": { - "job_id": { - "value": "example1" - } - } - }, - "is_hidden": true - } - }, - { - "add": { - "index": ".reindexed-v9-ml-anomalies-custom-example", - "alias": ".ml-anomalies-example2", - "filter": { - "term": { - "job_id": { - "value": "example2" - } - } - }, - "is_hidden": true - } - }, - { - "remove": { - "index": ".ml-anomalies-custom-example", - "aliases": ".ml-anomalies-*" - } - }, - { - "remove_index": { - "index": ".ml-anomalies-custom-example" - } - }, - { - "add": { - "index": ".reindexed-v9-ml-anomalies-custom-example", - "alias": ".ml-anomalies-custom-example", - "is_hidden": true - } - } - ] -} ------------------------------------------------------------- -// TEST[skip:TBD] --- -==== - -[[mark_anomaly_result_index_read_only]] -.Marking anomaly result indices as read-only -[%collapsible] -==== -Legacy indexes created in {es} 7.x can be made read-only and supported in {es} 9.x. -Making an index with a large amount of historical results read-only allows for a quick migration to the next major release, since you don't have to wait for the data to be reindexed into the new format. -However, it has the limitation that even after deleting an {anomaly-job}, the historical results associated with this job are not completely deleted. -Therefore, the system will prevent you from creating a new job with the same name. - -To set the index as read-only, add the `write` block to the index: - -[source,console] ------------------------------------------------------------- -PUT .ml-anomalies-custom-example/_block/write ------------------------------------------------------------- -// TEST[skip:TBD] - -Indices created in {es} 7.x that have a `write` block will not raise a critical deprecation warning. -==== - -[[delete_anomaly_result_index]] -.Deleting anomaly result indices -[%collapsible] -==== -If an index contains results of the jobs that are no longer required. -To list all jobs that stored results in an index, use the terms aggregation: - -[source,console] ------------------------------------------------------------- -GET .ml-anomalies-custom-example/_search -{ - "size": 0, - "aggs": { - "job_ids": { - "terms": { - "field": "job_id", - "size": 100 - } - } - } -} ------------------------------------------------------------- -// TEST[skip:TBD] - -The jobs can be deleted in the UI. -After the last job is deleted, the index will be deleted as well. +[[deprecations_90_rollup]] +==== Rollup deprecations + +[[emit_deprecation_warning_when_executing_one_of_rollup_apis]] +.Emit deprecation warning when executing one of the rollup APIs +[%collapsible] +==== +*Details* + +Rollup is already deprecated since 8.11.0 via documentation and since 8.15.0 it is no longer possible to create new rollup jobs in clusters without rollup usage. This change updates the rollup APIs to emit a deprecation warning. + +*Impact* + +Returning a deprecation warning when using one of the rollup APIs. ==== + +[discrete] +[[deprecations_90_transform]] +==== Transform deprecations + +[[removing_index_alias_creation_for_deprecated_transforms_notification_index]] +.Removing index alias creation for deprecated transforms notification index +[%collapsible] +==== +*Details* + +As part of the migration from 7.x to 8.x, the `.data-frame-notifications-1` index was deprecated and replaced with the `.transform-notifications-000002` index. The index is no longer created by default, all writes are directed to the new index, and any clusters with the deprecated index will have an alias created to ensure that reads are still retrieving data that was written to the index before the migration to 8.x. This change removes the alias from the deprecated index in 9.x. Any clusters with the alias present will retain it, but it will not be created on new clusters. + +*Impact* + +No known end user impact. +==== + diff --git a/docs/reference/release-notes/9.0.0.asciidoc b/docs/reference/release-notes/9.0.0.asciidoc index 93e5a30cb82f7..4d3d0746ac931 100644 --- a/docs/reference/release-notes/9.0.0.asciidoc +++ b/docs/reference/release-notes/9.0.0.asciidoc @@ -9,6 +9,9 @@ Also see <>. [float] === Breaking changes +Aggregations:: +* Remove date histogram boolean support {es-pull}118484[#118484] + Allocation:: * Increase minimum threshold in shard balancer {es-pull}115831[#115831] * Remove `cluster.routing.allocation.disk.watermark.enable_for_single_data_node` setting {es-pull}114207[#114207] @@ -20,15 +23,39 @@ Analysis:: * The 'persian' analyzer has stemmer by default {es-pull}113482[#113482] (issue: {es-issue}113050[#113050]) * The Korean dictionary for Nori has been updated {es-pull}114124[#114124] +Authentication:: +* Configuring a bind DN in an LDAP or Active Directory (AD) realm without a corresponding bind password +will prevent node from starting {es-pull}118366[#118366] + Cluster Coordination:: * Remove unsupported legacy value for `discovery.type` {es-pull}112903[#112903] +EQL:: +* Set allow_partial_search_results=true by default {es-pull}120267[#120267] + +Extract&Transform:: +* Restrict Connector APIs to manage/monitor_connector privileges {es-pull}119863[#119863] + Highlighting:: * Remove support for deprecated `force_source` highlighting parameter {es-pull}116943[#116943] Indices APIs:: * Apply more strict parsing of actions in bulk API {es-pull}115923[#115923] * Remove deprecated local attribute from alias APIs {es-pull}115393[#115393] +* Remove the ability to read frozen indices {es-pull}120108[#120108] +* Remove unfreeze REST endpoint {es-pull}119227[#119227] + +Infra/Core:: +* Change Elasticsearch timeouts to 429 response instead of 5xx {es-pull}116026[#116026] +* Limit `ByteSizeUnit` to 2 decimals {es-pull}120142[#120142] +* Remove `client.type` setting {es-pull}118192[#118192] (issue: {es-issue}104574[#104574]) +* Remove any references to org.elasticsearch.core.RestApiVersion#V_7 {es-pull}118103[#118103] + +Infra/Logging:: +* Change `deprecation.elasticsearch` keyword to `elasticsearch.deprecation` {es-pull}117933[#117933] (issue: {es-issue}83251[#83251]) + +Infra/Metrics:: +* Deprecated tracing.apm.* settings got removed. {es-pull}119926[#119926] Infra/REST API:: * Output a consistent format when generating error json {es-pull}90529[#90529] (issue: {es-issue}89387[#89387]) @@ -37,15 +64,36 @@ Ingest Node:: * Remove `ecs` option on `user_agent` processor {es-pull}116077[#116077] * Remove ignored fallback option on GeoIP processor {es-pull}116112[#116112] +Logs:: +* Conditionally enable logsdb by default for data streams matching with logs-*-* pattern. {es-pull}121049[#121049] (issue: {es-issue}106489[#106489]) + +Machine Learning:: +* Disable machine learning on macOS x86_64 {es-pull}104125[#104125] + Mapping:: -* Remove support for type, fields, `copy_to` and boost in metadata field definition {es-pull}116944[#116944] +* Remove support for type, fields, `copy_to` and boost in metadata field definition {es-pull}118825[#118825] +* Turn `_source` meta fieldmapper's mode attribute into a no-op {es-pull}119072[#119072] (issue: {es-issue}118596[#118596]) Search:: +* Adjust `random_score` default field to `_seq_no` field {es-pull}118671[#118671] +* Change Semantic Text To Act Like A Normal Text Field {es-pull}120813[#120813] * Remove legacy params from range query {es-pull}116970[#116970] Snapshot/Restore:: * Remove deprecated `xpack.searchable.snapshot.allocate_on_rolling_restart` setting {es-pull}114202[#114202] +TLS:: +* Remove TLSv1.1 from default protocols {es-pull}121731[#121731] + +Transform:: +* Remove `data_frame_transforms` roles {es-pull}117519[#117519] + +Vector Search:: +* Remove old `_knn_search` tech preview API in v9 {es-pull}118104[#118104] + +Watcher:: +* Removing support for types field in watcher search {es-pull}120748[#120748] + [[bug-9.0.0]] [float] === Bug fixes @@ -54,244 +102,469 @@ Aggregations:: * Handle with `illegalArgumentExceptions` negative values in HDR percentile aggregations {es-pull}116174[#116174] (issue: {es-issue}115777[#115777]) Analysis:: -* Adjust analyze limit exception to be a `bad_request` {es-pull}116325[#116325] +* Analyze API to return 400 for wrong custom analyzer {es-pull}121568[#121568] (issue: {es-issue}121443[#121443]) -CCS:: -* Fix long metric deserialize & add - auto-resize needs to be set manually {es-pull}117105[#117105] (issue: {es-issue}116914[#116914]) +CAT APIs:: +* Fix cat_component_templates documentation {es-pull}120487[#120487] CRUD:: * Preserve thread context when waiting for segment generation in RTG {es-pull}114623[#114623] -* Standardize error code when bulk body is invalid {es-pull}114869[#114869] +* Preserve thread context when waiting for segment generation in RTG {es-pull}117148[#117148] Data streams:: -* Add missing header in `put_data_lifecycle` rest-api-spec {es-pull}116292[#116292] +* Avoid updating settings version in `MetadataMigrateToDataStreamService` when settings have not changed {es-pull}118704[#118704] +* Block-writes cannot be added after read-only {es-pull}119007[#119007] (issue: {es-issue}119002[#119002]) +* Ensure removal of index blocks does not leave key with null value {es-pull}122246[#122246] +* Match dot prefix of migrated DS backing index with the source index {es-pull}120042[#120042] +* Refresh source index before reindexing data stream index {es-pull}120752[#120752] (issue: {es-issue}120314[#120314]) +* `ReindexDataStreamIndex` bug in assertion caused by reference equality {es-pull}121325[#121325] + +Downsampling:: +* Copy metrics and `default_metric` properties when downsampling `aggregate_metric_double` {es-pull}121727[#121727] (issues: {es-issue}119696[#119696], {es-issue}96076[#96076]) EQL:: -* Don't use a `BytesStreamOutput` to copy keys in `BytesRefBlockHash` {es-pull}114819[#114819] (issue: {es-issue}114599[#114599]) +* Fix JOIN command validation (not supported) {es-pull}122011[#122011] ES|QL:: -* Added stricter range type checks and runtime warnings for ENRICH {es-pull}115091[#115091] (issues: {es-issue}107357[#107357], {es-issue}116799[#116799]) -* Don't return TEXT type for functions that take TEXT {es-pull}114334[#114334] (issues: {es-issue}111537[#111537], {es-issue}114333[#114333]) -* ESQL: Fix sorts containing `_source` {es-pull}116980[#116980] (issue: {es-issue}116659[#116659]) -* ESQL: fix the column position in errors {es-pull}117153[#117153] -* ES|QL: Fix stats by constant expression {es-pull}114899[#114899] -* Fix NPE in `EnrichLookupService` on mixed clusters with <8.14 versions {es-pull}116583[#116583] (issues: {es-issue}116529[#116529], {es-issue}116544[#116544]) +* Allow the data type of `null` in filters {es-pull}118324[#118324] (issue: {es-issue}116351[#116351]) +* Correct line and column numbers of missing named parameters {es-pull}120852[#120852] +* Drop null columns in text formats {es-pull}117643[#117643] (issue: {es-issue}116848[#116848]) +* Fix ENRICH validation for use of wildcards {es-pull}121911[#121911] +* Fix ROUND() with unsigned longs throwing in some edge cases {es-pull}119536[#119536] * Fix TDigestState.read CB leaks {es-pull}114303[#114303] (issue: {es-issue}114194[#114194]) -* Fixing remote ENRICH by pushing the Enrich inside `FragmentExec` {es-pull}114665[#114665] (issue: {es-issue}105095[#105095]) -* Use `SearchStats` instead of field.isAggregatable in data node planning {es-pull}115744[#115744] (issue: {es-issue}115737[#115737]) -* [ESQL] Fix Binary Comparisons on Date Nanos {es-pull}116346[#116346] -* [ES|QL] To_DatePeriod and To_TimeDuration return better error messages on `union_type` fields {es-pull}114934[#114934] +* Fix TopN row size estimate {es-pull}119476[#119476] (issue: {es-issue}106956[#106956]) +* Fix `AbstractShapeGeometryFieldMapperTests` {es-pull}119265[#119265] (issue: {es-issue}119201[#119201]) +* Fix a bug in TOP {es-pull}121552[#121552] +* Fix async stop sometimes not properly collecting result {es-pull}121843[#121843] (issue: {es-issue}121249[#121249]) +* Fix attribute set equals {es-pull}118823[#118823] +* Fix double lookup failure on ESQL {es-pull}115616[#115616] (issue: {es-issue}111398[#111398]) +* Fix queries with document level security on lookup indexes {es-pull}120617[#120617] (issue: {es-issue}120509[#120509]) +* Fix writing for LOOKUP status {es-pull}119296[#119296] (issue: {es-issue}119086[#119086]) +* Limit memory usage of `fold` {es-pull}118602[#118602] +* Limit size of query {es-pull}117898[#117898] +* Remove redundant sorts from execution plan {es-pull}121156[#121156] +* Revert unwanted ES|QL lexer changes from PR #120354 {es-pull}120538[#120538] -Infra/CLI:: -* Fix NPE on plugin sync {es-pull}115640[#115640] (issue: {es-issue}114818[#114818]) +Health:: +* Do not recommend increasing `max_shards_per_node` {es-pull}120458[#120458] + +Indices APIs:: +* Add `?master_timeout` to `POST /_ilm/migrate_to_data_tiers` {es-pull}120883[#120883] +* Fix broken yaml test `30_create_from` {es-pull}120662[#120662] +* Include hidden indices in `DeprecationInfoAction` {es-pull}118035[#118035] (issue: {es-issue}118020[#118020]) + +Inference:: +* [Inference API] Put back legacy EIS URL setting {es-pull}121207[#121207] + +Infra/Core:: +* Epoch Millis Rounding Down and Not Up 2 {es-pull}118353[#118353] +* Wrap jackson exception on malformed json string {es-pull}114445[#114445] (issue: {es-issue}114142[#114142]) + +Infra/Logging:: +* Move `SlowLogFieldProvider` instantiation to node construction {es-pull}117949[#117949] Infra/Metrics:: * Make `randomInstantBetween` always return value in range [minInstant, `maxInstant]` {es-pull}114177[#114177] +Infra/Plugins:: +* Remove unnecessary entitlement {es-pull}120959[#120959] +* Restrict agent entitlements to the system classloader unnamed module {es-pull}120546[#120546] + Infra/REST API:: * Fixed a `NullPointerException` in `_capabilities` API when the `path` parameter is null. {es-pull}113413[#113413] (issue: {es-issue}113413[#113413]) +Infra/Scripting:: +* Register mustache size limit setting {es-pull}119291[#119291] + Infra/Settings:: * Don't allow secure settings in YML config (109115) {es-pull}115779[#115779] (issue: {es-issue}109115[#109115]) Ingest Node:: * Add warning headers for ingest pipelines containing special characters {es-pull}114837[#114837] (issue: {es-issue}104411[#104411]) -* Reducing error-level stack trace logging for normal events in `GeoIpDownloader` {es-pull}114924[#114924] +* Fix geoip databases index access after system feature migration {es-pull}121196[#121196] Logs:: * Always check if index mode is logsdb {es-pull}116922[#116922] -* Prohibit changes to index mode, source, and sort settings during resize {es-pull}115812[#115812] Machine Learning:: -* Fix bug in ML autoscaling when some node info is unavailable {es-pull}116650[#116650] -* Fix deberta tokenizer bug caused by bug in normalizer {es-pull}117189[#117189] -* Hides `hugging_face_elser` service from the `GET _inference/_services API` {es-pull}116664[#116664] (issue: {es-issue}116644[#116644]) -* Mitigate IOSession timeouts {es-pull}115414[#115414] (issues: {es-issue}114385[#114385], {es-issue}114327[#114327], {es-issue}114105[#114105], {es-issue}114232[#114232]) -* Propagate scoring function through random sampler {es-pull}116957[#116957] (issue: {es-issue}110134[#110134]) -* Update Deberta tokenizer {es-pull}116358[#116358] +* Change format for Unified Chat {es-pull}121396[#121396] +* Fix get all inference endponts not returning multiple endpoints sharing model deployment {es-pull}121821[#121821] +* Fixing bedrock event executor terminated cache issue {es-pull}118177[#118177] (issue: {es-issue}117916[#117916]) +* Fixing bug setting index when parsing Google Vertex AI results {es-pull}117287[#117287] +* Updating Inference Update API documentation to have the correct PUT method {es-pull}121048[#121048] * Wait for up to 2 seconds for yellow status before starting search {es-pull}115938[#115938] (issues: {es-issue}107777[#107777], {es-issue}105955[#105955], {es-issue}107815[#107815], {es-issue}112191[#112191]) +* [Inference API] Fix unique ID message for inference ID matches trained model ID {es-pull}119543[#119543] (issue: {es-issue}111312[#111312]) Mapping:: -* Change synthetic source logic for `constant_keyword` {es-pull}117182[#117182] (issue: {es-issue}117083[#117083]) -* Ignore conflicting fields during dynamic mapping update {es-pull}114227[#114227] (issue: {es-issue}114228[#114228]) +* Enable New Semantic Text Format Only On Newly Created Indices {es-pull}121556[#121556] +* Fix propagation of dynamic mapping parameter when applying `copy_to` {es-pull}121109[#121109] (issue: {es-issue}113049[#113049]) +* Fix realtime get of nested fields with synthetic source {es-pull}119575[#119575] (issue: {es-issue}119553[#119553]) +* Merge field mappers when updating mappings with [subobjects:false] {es-pull}120370[#120370] (issue: {es-issue}120216[#120216]) +* Tweak `copy_to` handling in synthetic `_source` to account for nested objects {es-pull}120974[#120974] (issue: {es-issue}120831[#120831]) Network:: -* Use underlying `ByteBuf` `refCount` for `ReleasableBytesReference` {es-pull}116211[#116211] - -Ranking:: -* Propagating nested `inner_hits` to the parent compound retriever {es-pull}116408[#116408] (issue: {es-issue}116397[#116397]) - -Relevance:: -* Fix handling of bulk requests with semantic text fields and delete ops {es-pull}116942[#116942] +* Remove ChunkedToXContentBuilder {es-pull}119310[#119310] (issue: {es-issue}118647[#118647]) Search:: * Catch and handle disconnect exceptions in search {es-pull}115836[#115836] -* Fields caps does not honour ignore_unavailable {es-pull}116021[#116021] (issue: {es-issue}107767[#107767]) -* Fix handling of time exceeded exception in fetch phase {es-pull}116676[#116676] * Fix leak in `DfsQueryPhase` and introduce search disconnect stress test {es-pull}116060[#116060] (issue: {es-issue}115056[#115056]) +* Fix/QueryBuilderBWCIT_muted_test {es-pull}117831[#117831] +* In this pr, a 400 error is returned when _source / _seq_no / _feature / _nested_path / _field_names is requested, rather a 5xx {es-pull}117229[#117229] * Inconsistency in the _analyzer api when the index is not included {es-pull}115930[#115930] -* Semantic text simple partial update {es-pull}116478[#116478] +* Remove duplicate code in ESIntegTestCase {es-pull}120799[#120799] +* SearchStatesIt failures reported by CI {es-pull}117618[#117618] (issues: {es-issue}116617[#116617], {es-issue}116618[#116618]) +* Skip fetching _inference_fields field in legacy semantic_text format {es-pull}121720[#121720] +* Test/107515 restore template with match only text mapper it fail {es-pull}120392[#120392] (issue: {es-issue}107515[#107515]) * Updated Date Range to Follow Documentation When Assuming Missing Values {es-pull}112258[#112258] (issue: {es-issue}111484[#111484]) -* Validate missing shards after the coordinator rewrite {es-pull}116382[#116382] -* _validate does not honour ignore_unavailable {es-pull}116656[#116656] (issue: {es-issue}116594[#116594]) +* `CrossClusterIT` `testCancel` failure {es-pull}117750[#117750] (issue: {es-issue}108061[#108061]) +* `SearchServiceTests.testParseSourceValidation` failure {es-pull}117963[#117963] Snapshot/Restore:: +* Add undeclared Azure settings, modify test to exercise them {es-pull}118634[#118634] +* Fork post-snapshot-delete cleanup off master thread {es-pull}122047[#122047] * Retry throttled snapshot deletions {es-pull}113237[#113237] +* Use the system index descriptor in the snapshot blob cache cleanup task {es-pull}120937[#120937] (issue: {es-issue}120518[#120518]) + +Store:: +* Do not capture `ClusterChangedEvent` in `IndicesStore` call to #onClusterStateShardsClosed {es-pull}120193[#120193] Vector Search:: -* Update Semantic Query To Handle Zero Size Responses {es-pull}116277[#116277] (issue: {es-issue}116083[#116083]) +* Apply default k for knn query eagerly {es-pull}118774[#118774] +* Fix `bbq_hnsw` merge file cleanup on random IO exceptions {es-pull}119691[#119691] (issue: {es-issue}119392[#119392]) Watcher:: -* Watch Next Run Interval Resets On Shard Move or Node Restart {es-pull}115102[#115102] (issue: {es-issue}111433[#111433]) +* Watcher history index has too many indexed fields - {es-pull}117701[#117701] (issue: {es-issue}71479[#71479]) [[deprecation-9.0.0]] [float] === Deprecations +ES|QL:: +* Drop support for brackets from METADATA syntax {es-pull}119846[#119846] (issue: {es-issue}115401[#115401]) + Ingest Node:: * Fix `_type` deprecation on simulate pipeline API {es-pull}116259[#116259] Machine Learning:: +* Removing index alias creation for deprecated transforms notification index {es-pull}117583[#117583] * [Inference API] Deprecate elser service {es-pull}113216[#113216] -Mapping:: -* Deprecate `_source.mode` in mappings {es-pull}116689[#116689] +Rollup:: +* Emit deprecation warning when executing one of the rollup APIs {es-pull}113131[#113131] + +Security:: +* Deprecate certificate based remote cluster security model {es-pull}120806[#120806] [[enhancement-9.0.0]] [float] === Enhancements Allocation:: +* Add a not-master state for desired balance {es-pull}116904[#116904] * Only publish desired balance gauges on master {es-pull}115383[#115383] +* Reset relocation/allocation failure counter on node join/shutdown {es-pull}119968[#119968] + +Authentication:: +* Allow `SSHA-256` for API key credential hash {es-pull}120997[#120997] Authorization:: -* Add a `monitor_stats` privilege and allow that privilege for remote cluster privileges {es-pull}114964[#114964] +* Allow kibana_system user to manage .reindexed-v8-internal.alerts indices {es-pull}118959[#118959] +* Grant necessary Kibana application privileges to `reporting_user` role {es-pull}118058[#118058] +* Make reserved built-in roles queryable {es-pull}117581[#117581] * [Security Solution] Add `create_index` to `kibana_system` role for index/DS `.logs-endpoint.action.responses-*` {es-pull}115241[#115241] +* [Security Solution] allows `kibana_system` user to manage .reindexed-v8-* Security Solution indices {es-pull}119054[#119054] + +CCS:: +* Resolve/cluster allows querying for cluster info only (no index expression required) {es-pull}119898[#119898] CRUD:: +* Remove INDEX_REFRESH_BLOCK after index becomes searchable {es-pull}120807[#120807] * Suppress merge-on-recovery for older indices {es-pull}113462[#113462] +Cluster Coordination:: +* Include `clusterApplyListener` in long cluster apply warnings {es-pull}120087[#120087] + Data streams:: -* Adding a deprecation info API warning for data streams with old indices {es-pull}116447[#116447] -* Apm-data: disable date_detection for all apm data streams {es-pull}116995[#116995] +* Add action to create index from a source index {es-pull}118890[#118890] +* Add index and reindex request settings to speed up reindex {es-pull}119780[#119780] +* Add rest endpoint for `create_from_source_index` {es-pull}119250[#119250] +* Add sanity check to `ReindexDatastreamIndexAction` {es-pull}120231[#120231] +* Adding a migration reindex cancel API {es-pull}118291[#118291] +* Adding get migration reindex status {es-pull}118267[#118267] +* Consistent mapping for OTel log and event bodies {es-pull}120547[#120547] +* Filter deprecated settings when making dest index {es-pull}120163[#120163] +* Ignore closed indices for reindex {es-pull}120244[#120244] +* Improve how reindex data stream index action handles api blocks {es-pull}120084[#120084] +* Initial work on `ReindexDatastreamIndexAction` {es-pull}116996[#116996] +* Make `requests_per_second` configurable to throttle reindexing {es-pull}120207[#120207] +* Optimized index sorting for OTel logs {es-pull}119504[#119504] +* Report Deprecated Indices That Are Flagged To Ignore Migration Reindex As A Warning {es-pull}120629[#120629] +* Update data stream deprecations warnings to new format and filter searchable snapshots from response {es-pull}118562[#118562] Distributed:: +* Make various alias retrieval APIs wait for cluster to unblock {es-pull}117230[#117230] * Metrics for incremental bulk splits {es-pull}116765[#116765] * Use Azure blob batch API to delete blobs in batches {es-pull}114566[#114566] +EQL:: +* Add support for partial shard results {es-pull}116388[#116388] +* Optional named arguments for function in map {es-pull}118619[#118619] + ES|QL:: -* Add ES|QL `bit_length` function {es-pull}115792[#115792] -* ESQL: Honor skip_unavailable setting for nonmatching indices errors at planning time {es-pull}116348[#116348] (issue: {es-issue}114531[#114531]) -* ESQL: Remove parent from `FieldAttribute` {es-pull}112881[#112881] -* ESQL: extract common filter from aggs {es-pull}115678[#115678] -* ESQL: optimise aggregations filtered by false/null into evals {es-pull}115858[#115858] -* ES|QL CCS uses `skip_unavailable` setting for handling disconnected remote clusters {es-pull}115266[#115266] (issue: {es-issue}114531[#114531]) -* ES|QL: add metrics for functions {es-pull}114620[#114620] -* Esql Enable Date Nanos (tech preview) {es-pull}117080[#117080] +* Add ES|QL cross-cluster query telemetry collection {es-pull}119474[#119474] +* Add a `LicenseAware` interface for licensed Nodes {es-pull}118931[#118931] (issue: {es-issue}117405[#117405]) +* Add a `PostAnalysisAware,` distribute verification {es-pull}119798[#119798] +* Add a standard deviation aggregating function: STD_DEV {es-pull}116531[#116531] +* Add cluster level reduction {es-pull}117731[#117731] +* Add nulls support to Categorize {es-pull}117655[#117655] +* Async search responses have CCS metadata while searches are running {es-pull}117265[#117265] +* Check for early termination in Driver {es-pull}118188[#118188] +* Do not serialize `EsIndex` in plan {es-pull}119580[#119580] +* ESQL - Remove restrictions for disjunctions in full text functions {es-pull}118544[#118544] +* ESQL - enabling scoring with METADATA `_score` {es-pull}113120[#113120] +* ESQL Add esql hash function {es-pull}117989[#117989] +* ESQL Support IN operator for Date nanos {es-pull}119772[#119772] (issue: {es-issue}118578[#118578]) +* ESQL: CATEGORIZE as a `BlockHash` {es-pull}114317[#114317] +* ESQL: Enterprise license enforcement for CCS {es-pull}118102[#118102] +* ES|QL: Partial result on demand for async queries {es-pull}118122[#118122] +* Enable KQL function as a tech preview {es-pull}119730[#119730] +* Enable LOOKUP JOIN in non-snapshot builds {es-pull}121193[#121193] (issue: {es-issue}121185[#121185]) +* Enable node-level reduction by default {es-pull}119621[#119621] +* Enable physical plan verification {es-pull}118114[#118114] +* Ensure cluster string could be quoted {es-pull}120355[#120355] +* Esql - Support date nanos in date extract function {es-pull}120727[#120727] (issue: {es-issue}110000[#110000]) +* Esql - support date nanos in date format function {es-pull}120143[#120143] (issue: {es-issue}109994[#109994]) +* Esql Support date nanos on date diff function {es-pull}120645[#120645] (issue: {es-issue}109999[#109999]) +* Esql bucket function for date nanos {es-pull}118474[#118474] (issue: {es-issue}118031[#118031]) +* Esql compare nanos and millis {es-pull}118027[#118027] (issue: {es-issue}116281[#116281]) +* Esql implicit casting for date nanos {es-pull}118697[#118697] (issue: {es-issue}118476[#118476]) +* Extend `TranslationAware` to all pushable expressions {es-pull}120192[#120192] +* Hash functions {es-pull}118938[#118938] +* Implement a `MetricsAware` interface {es-pull}121074[#121074] +* LOOKUP JOIN using field-caps for field mapping {es-pull}117246[#117246] +* Lookup join on multiple join fields not yet supported {es-pull}118858[#118858] +* Move scoring in ES|QL out of snapshot {es-pull}120354[#120354] +* Optimize ST_EXTENT_AGG for `geo_shape` and `cartesian_shape` {es-pull}119889[#119889] +* Push down filter passed lookup join {es-pull}118410[#118410] +* Resume Driver on cancelled or early finished {es-pull}120020[#120020] +* Rewrite TO_UPPER/TO_LOWER comparisons {es-pull}118870[#118870] (issue: {es-issue}118304[#118304]) +* ST_EXTENT_AGG optimize envelope extraction from doc-values for cartesian_shape {es-pull}118802[#118802] +* Smarter field caps with subscribable listener {es-pull}116755[#116755] * Support partial sort fields in TopN pushdown {es-pull}116043[#116043] (issue: {es-issue}114515[#114515]) -* [ES|QL] Implicit casting string literal to intervals {es-pull}115814[#115814] (issue: {es-issue}115352[#115352]) +* Support some stats on aggregate_metric_double {es-pull}120343[#120343] (issue: {es-issue}110649[#110649]) +* Take named parameters for identifier and pattern out of snapshot {es-pull}121850[#121850] +* Term query for ES|QL {es-pull}117359[#117359] +* Update grammar to rely on `indexPattern` instead of identifier in join target {es-pull}120494[#120494] +* `_score` should not be a reserved attribute in ES|QL {es-pull}118435[#118435] (issue: {es-issue}118460[#118460]) + +Engine:: +* Defer unpromotable shard refreshes until index refresh blocks are cleared {es-pull}120642[#120642] +* POC mark read-only {es-pull}119743[#119743] + +Experiences:: +* Integrate IBM watsonx to Inference API for re-ranking task {es-pull}117176[#117176] + +Geo:: +* Optimize indexing points with index and doc values set to true {es-pull}120271[#120271] Health:: * Increase `replica_unassigned_buffer_time` default from 3s to 5s {es-pull}112834[#112834] +ILM+SLM:: +* Add a `replicate_for` option to the ILM `searchable_snapshot` action {es-pull}119003[#119003] + Indices APIs:: -* Ensure class resource stream is closed in `ResourceUtils` {es-pull}116437[#116437] +* Add `remove_index_block` arg to `_create_from` api {es-pull}120548[#120548] +* Remove index blocks by default in `create_from` {es-pull}120643[#120643] +* Run `TransportGetComponentTemplateAction` on local node {es-pull}116868[#116868] +* Run `TransportGetComposableIndexTemplate` on local node {es-pull}119830[#119830] +* Run `TransportGetIndexTemplateAction` on local node {es-pull}119837[#119837] +* introduce new categories for deprecated resources in deprecation API {es-pull}120505[#120505] Inference:: * Add version prefix to Inference Service API path {es-pull}117095[#117095] +* Remove Elastic Inference Service feature flag and deprecated setting {es-pull}120842[#120842] +* Update sparse text embeddings API route for Inference Service {es-pull}118025[#118025] +* [Elastic Inference Service] Add ElasticInferenceService Unified ChatCompletions Integration {es-pull}118871[#118871] + +Infra/CLI:: +* Strengthen encryption for elasticsearch-keystore tool to AES 256 {es-pull}119749[#119749] Infra/Circuit Breakers:: * Add link to Circuit Breaker "Data too large" exception message {es-pull}113561[#113561] Infra/Core:: -* Support for unsigned 64 bit numbers in Cpu stats {es-pull}114681[#114681] (issue: {es-issue}112274[#112274]) +* Add support for specifying reindexing script for system index migration {es-pull}119001[#119001] +* Change default Docker image to be based on UBI minimal instead of Ubuntu {es-pull}116739[#116739] Infra/Metrics:: * Add `ensureGreen` test method for use with `adminClient` {es-pull}113425[#113425] +Infra/REST API:: +* A new query parameter `?include_source_on_error` was added for create / index, update and bulk REST APIs to control +if to include the document source in the error response in case of parsing errors. The default value is `true`. {es-pull}120725[#120725] + Infra/Scripting:: * Add a `mustache.max_output_size_bytes` setting to limit the length of results from mustache scripts {es-pull}114002[#114002] +Infra/Settings:: +* Introduce `IndexSettingDeprecatedInV8AndRemovedInV9` Setting property {es-pull}120334[#120334] +* Run `TransportClusterGetSettingsAction` on local node {es-pull}119831[#119831] + Ingest Node:: -* Add postal_code support to the City and Enterprise databases {es-pull}114193[#114193] -* Add support for registered country fields for maxmind geoip databases {es-pull}114521[#114521] -* Adding support for additional mapping to simulate ingest API {es-pull}114742[#114742] -* Adding support for simulate ingest mapping adddition for indices with mappings that do not come from templates {es-pull}115359[#115359] -* Support IPinfo database configurations {es-pull}114548[#114548] -* Support more maxmind fields in the geoip processor {es-pull}114268[#114268] +* Optimize `IngestCtxMap` construction {es-pull}120833[#120833] +* Optimize `IngestDocMetadata` `isAvailable` {es-pull}120753[#120753] +* Optimize `IngestDocument` `FieldPath` allocation {es-pull}120573[#120573] +* Optimize some per-document hot paths in the geoip processor {es-pull}120824[#120824] +* Returning ignored fields in the simulate ingest API {es-pull}117214[#117214] +* Run `GetPipelineTransportAction` on local node {es-pull}120445[#120445] +* Run `TransportGetEnrichPolicyAction` on local node {es-pull}121124[#121124] +* Run template simulation actions on local node {es-pull}120038[#120038] + +License:: +* Bump `TrialLicenseVersion` to allow starting new trial on 9.0 {es-pull}120198[#120198] Logs:: -* Add logsdb telemetry {es-pull}115994[#115994] -* Add num docs and size to logsdb telemetry {es-pull}116128[#116128] -* Feature: re-structure document ID generation favoring _id inverted index compression {es-pull}104683[#104683] +* Add LogsDB option to route on sort fields {es-pull}116687[#116687] +* Add a new index setting to skip recovery source when synthetic source is enabled {es-pull}114618[#114618] +* Configure index sorting through index settings for logsdb {es-pull}118968[#118968] (issue: {es-issue}118686[#118686]) +* Optimize loading mappings when determining synthetic source usage and whether host.name can be sorted on. {es-pull}120055[#120055] Machine Learning:: * Add DeBERTa-V2/V3 tokenizer {es-pull}111852[#111852] -* Add special case for elastic reranker in inference API {es-pull}116962[#116962] -* Adding inference endpoint validation for `AzureAiStudioService` {es-pull}113713[#113713] -* Adds support for `input_type` field to Vertex inference service {es-pull}116431[#116431] -* Enable built-in Inference Endpoints and default for Semantic Text {es-pull}116931[#116931] -* Increase default `queue_capacity` to 10_000 and decrease max `queue_capacity` to 100_000 {es-pull}115041[#115041] +* Add Inference Unified API for chat completions for OpenAI {es-pull}117589[#117589] +* Add Jina AI API to do inference for Embedding and Rerank models {es-pull}118652[#118652] +* Add enterprise license check for Inference API actions {es-pull}119893[#119893] +* Adding chunking settings to `IbmWatsonxService` {es-pull}114914[#114914] +* Adding default endpoint for Elastic Rerank {es-pull}117939[#117939] +* Adding endpoint creation validation for all task types to remaining services {es-pull}115020[#115020] +* Check for presence of error object when validating streaming responses from integrations in the inference API {es-pull}118375[#118375] +* Ignore failures from renormalizing buckets in read-only index {es-pull}118674[#118674] * Inference duration and error metrics {es-pull}115876[#115876] +* Migrate stream to core error parsing {es-pull}120722[#120722] * Remove all mentions of eis and gateway and deprecate flags that do {es-pull}116692[#116692] -* [Inference API] Add API to get configuration of inference services {es-pull}114862[#114862] -* [Inference API] Improve chunked results error message {es-pull}115807[#115807] +* Remove deprecated sort from reindex operation within dataframe analytics procedure {es-pull}117606[#117606] +* Retry on `ClusterBlockException` on transform destination index {es-pull}118194[#118194] +* [Inference API] fix spell words: covertToString to convertToString {es-pull}119922[#119922] + +Mapping:: +* Add Optional Source Filtering to Source Loaders {es-pull}113827[#113827] Network:: * Allow http unsafe buffers by default {es-pull}116115[#116115] +* Http stream activity tracker and exceptions handling {es-pull}119564[#119564] +* Remove HTTP content copies {es-pull}117303[#117303] +* `ConnectTransportException` returns retryable BAD_GATEWAY {es-pull}118681[#118681] (issue: {es-issue}118320[#118320]) + +Ranking:: +* Set default reranker for text similarity reranker to Elastic reranker {es-pull}120551[#120551] Recovery:: -* Attempt to clean up index before remote transfer {es-pull}115142[#115142] (issue: {es-issue}104473[#104473]) +* Allow archive and searchable snapshots indices in N-2 version {es-pull}118941[#118941] * Trigger merges after recovery {es-pull}113102[#113102] Reindex:: * Change Reindexing metrics unit from millis to seconds {es-pull}115721[#115721] -Relevance:: -* Add query rules retriever {es-pull}114855[#114855] -* Add tracking for query rule types {es-pull}116357[#116357] - Search:: -* Add Search Phase APM metrics {es-pull}113194[#113194] -* Add `docvalue_fields` Support for `dense_vector` Fields {es-pull}114484[#114484] (issue: {es-issue}108470[#108470]) -* Add initial support for `semantic_text` field type {es-pull}113920[#113920] -* Adds access to flags no_sub_matches and no_overlapping_matches to hyphenation-decompounder-tokenfilter {es-pull}115459[#115459] (issue: {es-issue}97849[#97849]) -* Better sizing `BytesRef` for Strings in Queries {es-pull}115655[#115655] -* Enable `_tier` based coordinator rewrites for all indices (not just mounted indices) {es-pull}115797[#115797] +* Add match support for `semantic_text` fields {es-pull}117839[#117839] +* Add support for `sparse_vector` queries against `semantic_text` fields {es-pull}118617[#118617] +* Add support for knn vector queries on `semantic_text` fields {es-pull}119011[#119011] +* Adding linear retriever to support weighted sums of sub-retrievers {es-pull}120222[#120222] +* Address and remove any references of RestApiVersion version 7 {es-pull}117572[#117572] +* Feat: add a user-configurable timeout parameter to the `_resolve/cluster` API {es-pull}120542[#120542] +* Make semantic text part of the text family {es-pull}119792[#119792] * Only aggregations require at least one shard request {es-pull}115314[#115314] +* Prevent data nodes from sending stack traces to coordinator when `error_trace=false` {es-pull}118266[#118266] +* Propagate status codes from shard failures appropriately {es-pull}118016[#118016] (issue: {es-issue}118482[#118482]) Security:: * Add refresh `.security` index call between security migrations {es-pull}114879[#114879] Snapshot/Restore:: -* Improve message about insecure S3 settings {es-pull}116915[#116915] +* Add IMDSv2 support to `repository-s3` {es-pull}117748[#117748] (issue: {es-issue}105135[#105135]) +* Expose operation and request counts separately in repository stats {es-pull}117530[#117530] (issue: {es-issue}104443[#104443]) * Retry `S3BlobContainer#getRegister` on all exceptions {es-pull}114813[#114813] -* Split searchable snapshot into multiple repo operations {es-pull}116918[#116918] +* Retry internally when CAS upload is throttled [GCS] {es-pull}120250[#120250] (issue: {es-issue}116546[#116546]) * Track shard snapshot progress during node shutdown {es-pull}112567[#112567] +Suggesters:: +* Extensible Completion Postings Formats {es-pull}111494[#111494] + +TSDB:: +* Increase field limit for OTel metrics to 10 000 {es-pull}120591[#120591] + +Transform:: +* Add support for `extended_stats` {es-pull}120340[#120340] +* Auto-migrate `max_page_search_size` {es-pull}119348[#119348] +* Create upgrade mode {es-pull}117858[#117858] +* Wait while index is blocked {es-pull}119542[#119542] +* [Deprecation] Add `transform_ids` to outdated index {es-pull}120821[#120821] + Vector Search:: -* Add support for bitwise inner-product in painless {es-pull}116082[#116082] +* Even better(er) binary quantization {es-pull}117994[#117994] +* Speed up bit compared with floats or bytes script operations {es-pull}117199[#117199] [[feature-9.0.0]] [float] === New features -Data streams:: -* Add default ILM policies and switch to ILM for apm-data plugin {es-pull}115687[#115687] +CRUD:: +* Metrics for indexing failures due to version conflicts {es-pull}119067[#119067] ES|QL:: -* Add support for `BYTE_LENGTH` scalar function {es-pull}116591[#116591] -* Esql/lookup join grammar {es-pull}116515[#116515] -* Remove snapshot build restriction for match and qstr functions {es-pull}114482[#114482] +* ESQL - Add Match function options {es-pull}120360[#120360] +* ESQL - Allow full text functions disjunctions for non-full text functions {es-pull}120291[#120291] +* ESQL: Enable async get to support formatting {es-pull}111104[#111104] (issue: {es-issue}110926[#110926]) +* Expand type compatibility for match function and operator {es-pull}117555[#117555] +* ST_EXTENT aggregation {es-pull}117451[#117451] (issue: {es-issue}104659[#104659]) +* Support ST_ENVELOPE and related (ST_XMIN, ST_XMAX, ST_YMIN, ST_YMAX) functions {es-pull}116964[#116964] (issue: {es-issue}104875[#104875]) + +Extract&Transform:: +* [Connector API] Support hard deletes with new URL param in delete endpoint {es-pull}120200[#120200] +* [Connector API] Support soft-deletes of connectors {es-pull}118669[#118669] +* [Connector APIs] Enforce index prefix for managed connectors {es-pull}117778[#117778] -Search:: -* ESQL - Add match operator (:) {es-pull}116819[#116819] +Highlighting:: +* Add Highlighter for Semantic Text Fields {es-pull}118064[#118064] + +Infra/Core:: +* Infrastructure for assuming cluster features in the next major version {es-pull}118143[#118143] + +Machine Learning:: +* ES|QL categorize with multiple groupings {es-pull}118173[#118173] +* Support mTLS for the Elastic Inference Service integration inside the inference API {es-pull}119679[#119679] +* [Inference API] Add node-local rate limiting for the inference API {es-pull}120400[#120400] + +Mapping:: +* Add option to store `sparse_vector` outside `_source` {es-pull}117917[#117917] + +Ranking:: +* Add a generic `rescorer` retriever based on the search request's rescore functionality {es-pull}118585[#118585] (issue: {es-issue}118327[#118327]) + +Relevance:: +* Add Multi-Field Support for Semantic Text Fields {es-pull}120128[#120128] + +Vector Search:: +* Add new experimental `rank_vectors` mapping for late-interaction second order ranking {es-pull}118804[#118804] +* KNN vector rescoring for quantized vectors {es-pull}116663[#116663] +* Mark bbq indices as GA and add rolling upgrade integration tests {es-pull}121105[#121105] [[upgrade-9.0.0]] [float] === Upgrades +Infra/Core:: +* Bump major version for feature migration system indices {es-pull}117243[#117243] +* Update ASM 9.7 -> 9.7.1 to support JDK 24 {es-pull}118094[#118094] + +Machine Learning:: +* Automatically rollover legacy .ml-anomalies indices {es-pull}120913[#120913] +* Automatically rollover legacy ml indices {es-pull}120405[#120405] +* Change the auditor to write via an alias {es-pull}120064[#120064] + Search:: * Upgrade to Lucene 10 {es-pull}114741[#114741] +* Upgrade to Lucene 10.1.0 {es-pull}119308[#119308] diff --git a/docs/reference/release-notes/highlights.asciidoc b/docs/reference/release-notes/highlights.asciidoc index b87081639c684..5b1cdb450a367 100644 --- a/docs/reference/release-notes/highlights.asciidoc +++ b/docs/reference/release-notes/highlights.asciidoc @@ -10,13 +10,29 @@ For detailed information about this release, see the <> and endif::[] -// The notable-highlights tag marks entries that -// should be featured in the Stack Installation and Upgrade Guide: // tag::notable-highlights[] -// [discrete] -// === Heading -// -// Description. + +[discrete] +[[add_new_experimental_rank_vectors_mapping_for_late_interaction_second_order_ranking]] +=== Add new experimental `rank_vectors` mapping for late-interaction second order ranking +Late-interaction models are powerful rerankers. While their size and overall cost doesn't lend itself for HNSW indexing, utilizing them as second order reranking can provide excellent boosts in relevance. The new `rank_vectors` mapping allows for rescoring over new and novel multi-vector late-interaction models like ColBERT or ColPali. + +{es-pull}118804[#118804] + +[discrete] +[[enable_lookup_join_in_non_snapshot_builds]] +=== Enable LOOKUP JOIN in non-snapshot builds +This effectively releases LOOKUP JOIN into tech preview. Docs will +follow in a separate PR. + +- Enable the lexing/grammar for LOOKUP JOIN in non-snapshot builds. +- Remove the grammar for the unsupported `| JOIN ...` command (without `LOOKUP` as first keyword). The way the lexer modes work, otherwise we'd also have to enable `| JOIN ...` syntax on non-snapshot builds and would have to add additional validation to provide appropriate error messages. +- Remove grammar for `LOOKUP JOIN index AS ...` because qualifiers are not yet supported. Otherwise we'd have to put in additional validation as well to prevent such queries. + +Also fix https://github.com/elastic/elasticsearch/issues/121185 + +{es-pull}121193[#121193] + // end::notable-highlights[]