diff --git a/docs/reference/migration/migrate_8_15.asciidoc b/docs/reference/migration/migrate_8_15.asciidoc new file mode 100644 index 0000000000000..1961230da1bbf --- /dev/null +++ b/docs/reference/migration/migrate_8_15.asciidoc @@ -0,0 +1,140 @@ +[[migrating-8.15]] +== Migrating to 8.15 +++++ +8.15 +++++ + +This section discusses the changes that you need to be aware of when migrating +your application to {es} 8.15. + +See also <> and <>. + +coming::[8.15.0] + + +[discrete] +[[breaking-changes-8.15]] +=== Breaking changes + +The following changes in {es} 8.15 might affect your applications +and prevent them from operating normally. +Before upgrading to 8.15, review these changes and take the described steps +to mitigate the impact. + +[discrete] +[[breaking_815_cluster_and_node_setting_changes]] +==== Cluster and node setting changes + +[[change_skip_unavailable_remote_cluster_setting_default_value_to_true]] +.Change `skip_unavailable` remote cluster setting default value to true +[%collapsible] +==== +*Details* + +The default value of the `skip_unavailable` setting is now set to true. All existing and future remote clusters that do not define this setting will use the new default. This setting only affects cross-cluster searches using the _search or _async_search API. + +*Impact* + +Unavailable remote clusters in a cross-cluster search will no longer cause the search to fail unless skip_unavailable is configured to be `false` in elasticsearch.yml or via the `_cluster/settings` API. Unavailable clusters with `skip_unavailable`=`true` (either explicitly or by using the new default) are marked as SKIPPED in the search response metadata section and do not fail the entire search. If users want to ensure that a search returns a failure when a particular remote cluster is not available, `skip_unavailable` must be now be set explicitly. +==== + +[discrete] +[[breaking_815_rollup_changes]] +==== Rollup changes + +[[disallow_new_rollup_jobs_in_clusters_with_no_rollup_usage]] +.Disallow new rollup jobs in clusters with no rollup usage +[%collapsible] +==== +*Details* + +The put rollup API will fail with an error when a rollup job is created in a cluster with no rollup usage + +*Impact* + +Clusters with no rollup usage (either no rollup job or index) can not create new rollup jobs +==== + +[discrete] +[[breaking_815_rest_api_changes]] +==== REST API changes + +[[interpret_timeout_1_as_infinite_ack_timeout]] +.Interpret `?timeout=-1` as infinite ack timeout +[%collapsible] +==== +*Details* + +Today {es} accepts the parameter `?timeout=-1` in many APIs, but interprets +this to mean the same as `?timeout=0`. From 8.15 onwards `?timeout=-1` will +mean to wait indefinitely, aligning the behaviour of this parameter with +other similar parameters such as `?master_timeout`. + +*Impact* + +Use `?timeout=0` to force relevant operations to time out immediately +instead of `?timeout=-1` +==== + +[[replace_model_id_with_inference_id]] +.Replace `model_id` with `inference_id` in GET inference API +[%collapsible] +==== +*Details* + +From 8.15 onwards the <> response will return an +`inference_id` field instead of a `model_id`. + +*Impact* + +If your application uses the `model_id` in a GET inference API response, +switch it to use `inference_id` instead. +==== + + +[discrete] +[[deprecated-8.15]] +=== Deprecations + +The following functionality has been deprecated in {es} 8.15 +and will be removed in a future version. +While this won't have an immediate impact on your applications, +we strongly encourage you to take the described steps to update your code +after upgrading to 8.15. + +To find out if you are using any deprecated functionality, +enable <>. + +[discrete] +[[deprecations_815_cluster_and_node_setting]] +==== Cluster and node setting deprecations + +[[deprecate_absolute_size_values_for_indices_breaker_total_limit_setting]] +.Deprecate absolute size values for `indices.breaker.total.limit` setting +[%collapsible] +==== +*Details* + +Previously, the value of `indices.breaker.total.limit` could be specified as an absolute size in bytes. This setting controls the overal amount of memory the server is allowed to use before taking remedial actions. Setting this to a specific number of bytes led to strange behaviour when the node maximum heap size changed because the circut breaker limit would remain unchanged. This would either leave the value too low, causing part of the heap to remain unused; or it would leave the value too high, causing the circuit breaker to be ineffective at preventing OOM errors. The only reasonable behaviour for this setting is that it scales with the size of the heap, and so absolute byte limits are now deprecated. + +*Impact* + +Users must change their configuration to specify a percentage instead of an absolute number of bytes for `indices.breaker.total.limit`, or else accept the default, which is already specified as a percentage. +==== + +[discrete] +[[deprecations_815_rest_api]] +==== REST API deprecations + +[[deprecate_text_expansion_weighted_tokens_queries]] +.Deprecate `text_expansion` and `weighted_tokens` queries +[%collapsible] +==== +*Details* + +The `text_expansion` and `weighted_tokens` queries have been replaced by `sparse_vector`. + +*Impact* + +Please update your existing `text_expansion` and `weighted_tokens` queries to use `sparse_vector.` +==== + +[[deprecate_using_slm_privileges_to_access_ilm]] +.Deprecate using slm privileges to access ilm +[%collapsible] +==== +*Details* + +The `read_slm` privilege can get the ILM status, and the `manage_slm` privilege can start and stop ILM. Access to these APIs should be granted using the `read_ilm` and `manage_ilm` privileges instead. Access to ILM APIs will be removed from SLM privileges in a future major release, and is now deprecated. + +*Impact* + +Users that need access to the ILM status API should now use the `read_ilm` privilege. Users that need to start and stop ILM, should use the `manage_ilm` privilege. +==== + diff --git a/docs/reference/release-notes/8.15.4.asciidoc b/docs/reference/release-notes/8.15.4.asciidoc new file mode 100644 index 0000000000000..61a54363f3c6d --- /dev/null +++ b/docs/reference/release-notes/8.15.4.asciidoc @@ -0,0 +1,33 @@ +[[release-notes-8.15.4]] +== {es} version 8.15.4 + +Also see <>. + +[[bug-8.15.4]] +[float] +=== Bug fixes + +Aggregations:: +* Always check the parent breaker with zero bytes in `PreallocatedCircuitBreakerService` {es-pull}115181[#115181] + +ES|QL:: +* ESQL: Disable pushdown of WHERE past STATS {es-pull}115308[#115308] (issue: {es-issue}115281[#115281]) + +Geo:: +* Try to simplify geometries that fail with `TopologyException` {es-pull}115834[#115834] + +Infra/Core:: +* Guard blob store local directory creation with `doPrivileged` {es-pull}115459[#115459] + +License:: +* Fix lingering license warning header in IP filter {es-pull}115510[#115510] (issue: {es-issue}114865[#114865]) + +Search:: +* Do not exclude empty arrays or empty objects in source filtering {es-pull}112250[#112250] (issue: {es-issue}109668[#109668]) +* Fix synthetic source handling for `bit` type in `dense_vector` field {es-pull}114407[#114407] (issue: {es-issue}114402[#114402]) + +Vector Search:: +* Fix dim validation for bit `element_type` {es-pull}114533[#114533] +* Support semantic_text in object fields {es-pull}114601[#114601] (issue: {es-issue}114401[#114401]) + + diff --git a/docs/reference/release-notes/highlights.asciidoc b/docs/reference/release-notes/highlights.asciidoc index edecd4f727583..160e852a7a5f4 100644 --- a/docs/reference/release-notes/highlights.asciidoc +++ b/docs/reference/release-notes/highlights.asciidoc @@ -1,6 +1,3 @@ -// THIS IS A GENERATED FILE. DO NOT EDIT DIRECTLY. -// The content generated here are is not correct and most has been manually commented out until it can be fixed. -// See ES-9931 for more details. [[release-highlights]] == What's new in {minor-version} @@ -12,163 +9,163 @@ For detailed information about this release, see the <> and <>. endif::[] -// -// // tag::notable-highlights[] -// -// [discrete] -// [[esql_inlinestats]] -// === ESQL: INLINESTATS -// This adds the `INLINESTATS` command to ESQL which performs a STATS and -// then enriches the results into the output stream. So, this query: -// -// [source,esql] -// ---- -// FROM test -// | INLINESTATS m=MAX(a * b) BY b -// | WHERE m == a * b -// | SORT a DESC, b DESC -// | LIMIT 3 -// ---- -// -// Produces output like: -// -// | a | b | m | -// | --- | --- | ----- | -// | 99 | 999 | 98901 | -// | 99 | 998 | 98802 | -// | 99 | 997 | 98703 | -// -// {es-pull}109583[#109583] -// -// [discrete] -// [[always_allow_rebalancing_by_default]] -// === Always allow rebalancing by default -// In earlier versions of {es} the `cluster.routing.allocation.allow_rebalance` setting defaults to -// `indices_all_active` which blocks all rebalancing moves while the cluster is in `yellow` or `red` health. This was -// appropriate for the legacy allocator which might do too many rebalancing moves otherwise. Today's allocator has -// better support for rebalancing a cluster that is not in `green` health, and expects to be able to rebalance some -// shards away from over-full nodes to avoid allocating shards to undesirable locations in the first place. From -// version 8.16 `allow_rebalance` setting defaults to `always` unless the legacy allocator is explicitly enabled. -// -// {es-pull}111015[#111015] -// -// [discrete] -// [[add_global_retention_in_data_stream_lifecycle]] -// === Add global retention in data stream lifecycle -// Data stream lifecycle now supports configuring retention on a cluster level, -// namely global retention. Global retention \nallows us to configure two different -// retentions: -// -// - `data_streams.lifecycle.retention.default` is applied to all data streams managed -// by the data stream lifecycle that do not have retention defined on the data stream level. -// - `data_streams.lifecycle.retention.max` is applied to all data streams managed by the -// data stream lifecycle and it allows any data stream \ndata to be deleted after the `max_retention` has passed. -// -// {es-pull}111972[#111972] -// -// [discrete] -// [[enable_zstandard_compression_for_indices_with_index_codec_set_to_best_compression]] -// === Enable ZStandard compression for indices with index.codec set to best_compression -// Before DEFLATE compression was used to compress stored fields in indices with index.codec index setting set to -// best_compression, with this change ZStandard is used as compression algorithm to stored fields for indices with -// index.codec index setting set to best_compression. The usage ZStandard results in less storage usage with a -// similar indexing throughput depending on what options are used. Experiments with indexing logs have shown that -// ZStandard offers ~12% lower storage usage and a ~14% higher indexing throughput compared to DEFLATE. -// -// {es-pull}112665[#112665] -// -// [discrete] -// [[esql_introduce_per_agg_filter]] -// === ESQL: Introduce per agg filter -// Add support for aggregation scoped filters that work dynamically on the -// data in each group. -// -// [source,esql] -// ---- -// | STATS success = COUNT(*) WHERE 200 <= code AND code < 300, -// redirect = COUNT(*) WHERE 300 <= code AND code < 400, -// client_err = COUNT(*) WHERE 400 <= code AND code < 500, -// server_err = COUNT(*) WHERE 500 <= code AND code < 600, -// total_count = COUNT(*) -// ---- -// -// Implementation wise, the base AggregateFunction has been extended to -// allow a filter to be passed on. This is required to incorporate the -// filter as part of the aggregate equality/identity which would fail with -// the filter as an external component. -// As part of the process, the serialization for the existing aggregations -// had to be fixed so AggregateFunction implementations so that it -// delegates to their parent first. -// -// {es-pull}113735[#113735] -// -// // end::notable-highlights[] -// -// -// [discrete] -// [[esql_multi_value_fields_supported_in_geospatial_predicates]] -// === ESQL: Multi-value fields supported in Geospatial predicates -// Supporting multi-value fields in `WHERE` predicates is a challenge due to not knowing whether `ALL` or `ANY` -// of the values in the field should pass the predicate. -// For example, should the field `age:[10,30]` pass the predicate `WHERE age>20` or not? -// This ambiguity does not exist with the spatial predicates -// `ST_INTERSECTS` and `ST_DISJOINT`, because the choice between `ANY` or `ALL` -// is implied by the predicate itself. -// Consider a predicate checking a field named `location` against a test geometry named `shape`: -// -// * `ST_INTERSECTS(field, shape)` - true if `ANY` value can intersect the shape -// * `ST_DISJOINT(field, shape)` - true only if `ALL` values are disjoint from the shape -// -// This works even if the shape argument is itself a complex or compound geometry. -// -// Similar logic exists for `ST_CONTAINS` and `ST_WITHIN` predicates, but these are not as easily solved -// with `ANY` or `ALL`, because a collection of geometries contains another collection if each of the contained -// geometries is within at least one of the containing geometries. Evaluating this requires that the multi-value -// field is first combined into a single geometry before performing the predicate check. -// -// * `ST_CONTAINS(field, shape)` - true if the combined geometry contains the shape -// * `ST_WITHIN(field, shape)` - true if the combined geometry is within the shape -// -// {es-pull}112063[#112063] -// -// [discrete] -// [[enhance_sort_push_down_to_lucene_to_cover_references_to_fields_st_distance_function]] -// === Enhance SORT push-down to Lucene to cover references to fields and ST_DISTANCE function -// The most used and likely most valuable geospatial search query in Elasticsearch is the sorted proximity search, -// finding items within a certain distance of a point of interest and sorting the results by distance. -// This has been possible in ES|QL since 8.15.0, but the sorting was done in-memory, not pushed down to Lucene. -// Now the sorting is pushed down to Lucene, which results in a significant performance improvement. -// -// Queries that perform both filtering and sorting on distance are supported. For example: -// -// [source,esql] -// ---- -// FROM test -// | EVAL distance = ST_DISTANCE(location, TO_GEOPOINT("POINT(37.7749, -122.4194)")) -// | WHERE distance < 1000000 -// | SORT distance ASC, name DESC -// | LIMIT 10 -// ---- -// -// In addition, the support for sorting on EVAL expressions has been extended to cover references to fields: -// -// [source,esql] -// ---- -// FROM test -// | EVAL ref = field -// | SORT ref ASC -// | LIMIT 10 -// ---- -// -// {es-pull}112938[#112938] -// -// [discrete] -// [[cross_cluster_search_telemetry]] -// === Cross-cluster search telemetry -// The cross-cluster search telemetry is collected when cross-cluster searches -// are performed, and is returned as "ccs" field in `_cluster/stats` output. -// It also add a new parameter `include_remotes=true` to the `_cluster/stats` API -// which will collect data from connected remote clusters. -// -// {es-pull}113825[#113825] + +// tag::notable-highlights[] + +[discrete] +[[esql_inlinestats]] +=== ESQL: INLINESTATS +This adds the `INLINESTATS` command to ESQL which performs a STATS and +then enriches the results into the output stream. So, this query: + +[source,esql] +---- +FROM test +| INLINESTATS m=MAX(a * b) BY b +| WHERE m == a * b +| SORT a DESC, b DESC +| LIMIT 3 +---- + +Produces output like: + +| a | b | m | +| --- | --- | ----- | +| 99 | 999 | 98901 | +| 99 | 998 | 98802 | +| 99 | 997 | 98703 | + +{es-pull}109583[#109583] + +[discrete] +[[always_allow_rebalancing_by_default]] +=== Always allow rebalancing by default +In earlier versions of {es} the `cluster.routing.allocation.allow_rebalance` setting defaults to +`indices_all_active` which blocks all rebalancing moves while the cluster is in `yellow` or `red` health. This was +appropriate for the legacy allocator which might do too many rebalancing moves otherwise. Today's allocator has +better support for rebalancing a cluster that is not in `green` health, and expects to be able to rebalance some +shards away from over-full nodes to avoid allocating shards to undesirable locations in the first place. From +version 8.16 `allow_rebalance` setting defaults to `always` unless the legacy allocator is explicitly enabled. + +{es-pull}111015[#111015] + +[discrete] +[[add_global_retention_in_data_stream_lifecycle]] +=== Add global retention in data stream lifecycle +Data stream lifecycle now supports configuring retention on a cluster level, +namely global retention. Global retention \nallows us to configure two different +retentions: + +- `data_streams.lifecycle.retention.default` is applied to all data streams managed +by the data stream lifecycle that do not have retention defined on the data stream level. +- `data_streams.lifecycle.retention.max` is applied to all data streams managed by the +data stream lifecycle and it allows any data stream \ndata to be deleted after the `max_retention` has passed. + +{es-pull}111972[#111972] + +[discrete] +[[enable_zstandard_compression_for_indices_with_index_codec_set_to_best_compression]] +=== Enable ZStandard compression for indices with index.codec set to best_compression +Before DEFLATE compression was used to compress stored fields in indices with index.codec index setting set to +best_compression, with this change ZStandard is used as compression algorithm to stored fields for indices with +index.codec index setting set to best_compression. The usage ZStandard results in less storage usage with a +similar indexing throughput depending on what options are used. Experiments with indexing logs have shown that +ZStandard offers ~12% lower storage usage and a ~14% higher indexing throughput compared to DEFLATE. + +{es-pull}112665[#112665] + +[discrete] +[[esql_introduce_per_agg_filter]] +=== ESQL: Introduce per agg filter +Add support for aggregation scoped filters that work dynamically on the +data in each group. + +[source,esql] +---- +| STATS success = COUNT(*) WHERE 200 <= code AND code < 300, + redirect = COUNT(*) WHERE 300 <= code AND code < 400, + client_err = COUNT(*) WHERE 400 <= code AND code < 500, + server_err = COUNT(*) WHERE 500 <= code AND code < 600, + total_count = COUNT(*) +---- + +Implementation wise, the base AggregateFunction has been extended to +allow a filter to be passed on. This is required to incorporate the +filter as part of the aggregate equality/identity which would fail with +the filter as an external component. +As part of the process, the serialization for the existing aggregations +had to be fixed so AggregateFunction implementations so that it +delegates to their parent first. + +{es-pull}113735[#113735] + +// end::notable-highlights[] + + +[discrete] +[[esql_multi_value_fields_supported_in_geospatial_predicates]] +=== ESQL: Multi-value fields supported in Geospatial predicates +Supporting multi-value fields in `WHERE` predicates is a challenge due to not knowing whether `ALL` or `ANY` +of the values in the field should pass the predicate. +For example, should the field `age:[10,30]` pass the predicate `WHERE age>20` or not? +This ambiguity does not exist with the spatial predicates +`ST_INTERSECTS` and `ST_DISJOINT`, because the choice between `ANY` or `ALL` +is implied by the predicate itself. +Consider a predicate checking a field named `location` against a test geometry named `shape`: + +* `ST_INTERSECTS(field, shape)` - true if `ANY` value can intersect the shape +* `ST_DISJOINT(field, shape)` - true only if `ALL` values are disjoint from the shape + +This works even if the shape argument is itself a complex or compound geometry. + +Similar logic exists for `ST_CONTAINS` and `ST_WITHIN` predicates, but these are not as easily solved +with `ANY` or `ALL`, because a collection of geometries contains another collection if each of the contained +geometries is within at least one of the containing geometries. Evaluating this requires that the multi-value +field is first combined into a single geometry before performing the predicate check. + +* `ST_CONTAINS(field, shape)` - true if the combined geometry contains the shape +* `ST_WITHIN(field, shape)` - true if the combined geometry is within the shape + +{es-pull}112063[#112063] + +[discrete] +[[enhance_sort_push_down_to_lucene_to_cover_references_to_fields_st_distance_function]] +=== Enhance SORT push-down to Lucene to cover references to fields and ST_DISTANCE function +The most used and likely most valuable geospatial search query in Elasticsearch is the sorted proximity search, +finding items within a certain distance of a point of interest and sorting the results by distance. +This has been possible in ES|QL since 8.15.0, but the sorting was done in-memory, not pushed down to Lucene. +Now the sorting is pushed down to Lucene, which results in a significant performance improvement. + +Queries that perform both filtering and sorting on distance are supported. For example: + +[source,esql] +---- +FROM test +| EVAL distance = ST_DISTANCE(location, TO_GEOPOINT("POINT(37.7749, -122.4194)")) +| WHERE distance < 1000000 +| SORT distance ASC, name DESC +| LIMIT 10 +---- + +In addition, the support for sorting on EVAL expressions has been extended to cover references to fields: + +[source,esql] +---- +FROM test +| EVAL ref = field +| SORT ref ASC +| LIMIT 10 +---- + +{es-pull}112938[#112938] + +[discrete] +[[cross_cluster_search_telemetry]] +=== Cross-cluster search telemetry +The cross-cluster search telemetry is collected when cross-cluster searches +are performed, and is returned as "ccs" field in `_cluster/stats` output. +It also add a new parameter `include_remotes=true` to the `_cluster/stats` API +which will collect data from connected remote clusters. + +{es-pull}113825[#113825]