From 850467e10e14d73e6f1b98ada5e5f98fa092ea19 Mon Sep 17 00:00:00 2001 From: elasticsearchmachine Date: Thu, 29 May 2025 17:08:07 +0000 Subject: [PATCH 1/3] Forward port release notes for v8.18.2 --- .../reference/migration/migrate_8_18.asciidoc | 14 +++ docs/reference/release-notes.asciidoc | 2 + docs/reference/release-notes/8.18.2.asciidoc | 67 ++++++++++++ .../release-notes/highlights.asciidoc | 102 ++++++++++++++++-- 4 files changed, 179 insertions(+), 6 deletions(-) create mode 100644 docs/reference/release-notes/8.18.2.asciidoc diff --git a/docs/reference/migration/migrate_8_18.asciidoc b/docs/reference/migration/migrate_8_18.asciidoc index d1b9040022523..c809d25969852 100644 --- a/docs/reference/migration/migrate_8_18.asciidoc +++ b/docs/reference/migration/migrate_8_18.asciidoc @@ -81,6 +81,20 @@ Before this change, in case of shard failures, EQL queries always returned an er EQL queries that would previously fail due to shard failures, will now succeed and return partial results. The previous defaults can be restored by setting `xpack.eql.default_allow_partial_results` cluster setting to `false` or setting with `allow_partial_search_results` to `false` in the query request. ==== +[discrete] +[[breaking_818_esql_changes]] +==== ES|QL changes + +[[limit_scope_skip_unavailable_setting_remote_clusters]] +.Limit the scope of the skip_unavailable setting for remote clusters +[%collapsible] +==== +*Details* + +Before this change, ES|QL would honor the skip_unavailable setting for nonmatching indices errors at planning time, meaning that when a query with a concrete (non-wildcarded) index was issued to a cluster where skip_unavailable=true, the index would be skipped, allowing the query to proceed on any other specified clusters rather than returning an error. + +*Impact* + +Remote clusters, regardless of their skip_unavailable setting, will now use the same logic as the local cluster for index expression analysis at plan time. As a result, any missing concrete index in your index expression will return an error rather than partial results. +==== [discrete] [[deprecated-8.18]] diff --git a/docs/reference/release-notes.asciidoc b/docs/reference/release-notes.asciidoc index 3622d65546850..5041d7c259307 100644 --- a/docs/reference/release-notes.asciidoc +++ b/docs/reference/release-notes.asciidoc @@ -7,6 +7,7 @@ This section summarizes the changes in each release. * <> +* <> * <> * <> * <> @@ -93,6 +94,7 @@ This section summarizes the changes in each release. -- include::release-notes/8.19.0.asciidoc[] +include::release-notes/8.18.2.asciidoc[] include::release-notes/8.18.1.asciidoc[] include::release-notes/8.18.0.asciidoc[] include::release-notes/8.17.6.asciidoc[] diff --git a/docs/reference/release-notes/8.18.2.asciidoc b/docs/reference/release-notes/8.18.2.asciidoc new file mode 100644 index 0000000000000..57be512158e88 --- /dev/null +++ b/docs/reference/release-notes/8.18.2.asciidoc @@ -0,0 +1,67 @@ +[[release-notes-8.18.2]] +== {es} version 8.18.2 + +Also see <>. + +[[breaking-8.18.2]] +[float] +=== Breaking changes + +Snapshot/Restore:: +* Make S3 custom query parameter optional {es-pull}128043[#128043] + +[[bug-8.18.2]] +[float] +=== Bug fixes + +Aggregations:: +* Fix a bug in `significant_terms` {es-pull}127975[#127975] + +Audit:: +* Handle streaming request body in audit log {es-pull}127798[#127798] + +Data streams:: +* Fix system data streams incorrectly showing up in the list of template validation problems {es-pull}128161[#128161] + +Downsampling:: +* Downsampling does not consider passthrough fields as dimensions {es-pull}127752[#127752] (issue: {es-issue}125156[#125156]) + +ES|QL:: +* Ensure ordinal builder emit ordinal blocks {es-pull}127949[#127949] +* Fix union types in CCS {es-pull}128111[#128111] + +Infra/Core:: +* Add missing `outbound_network` entitlement to x-pack-core {es-pull}126992[#126992] (issue: {es-issue}127003[#127003]) +* Check hidden frames in entitlements {es-pull}127877[#127877] + +Infra/Scripting:: +* Avoid nested docs in painless execute api {es-pull}127991[#127991] (issue: {es-issue}41004[#41004]) + +Logs:: +* Fix an authorization error when LogsPatternUsageService attempts to update `logsdb.prior_logs_usage` cluster setting. {es-pull}128050[#128050] + +Machine Learning:: +* Append all data to Chat Completion buffer {es-pull}127658[#127658] +* Fix services API Google Vertex AI Rerank location field requirement {es-pull}127856[#127856] +* Use internal user for internal inference action {es-pull}128327[#128327] + +Relevance:: +* Fix: Add `NamedWriteable` for `RuleQueryRankDoc` {es-pull}128153[#128153] (issue: {es-issue}126071[#126071]) + +Security:: +* Remove dangling spaces wherever found {es-pull}127475[#127475] + +Snapshot/Restore:: +* Add missing entitlement to `repository-azure` {es-pull}128047[#128047] (issue: {es-issue}128046[#128046]) + +TSDB:: +* Skip the validation when retrieving the index mode during reindexing a time series data stream {es-pull}127824[#127824] + +[[enhancement-8.18.2]] +[float] +=== Enhancements + +Authentication:: +* Http proxy support in JWT realm {es-pull}127337[#127337] (issue: {es-issue}114956[#114956]) + + diff --git a/docs/reference/release-notes/highlights.asciidoc b/docs/reference/release-notes/highlights.asciidoc index 9ae4dbffb325a..615e5c1a4d3d0 100644 --- a/docs/reference/release-notes/highlights.asciidoc +++ b/docs/reference/release-notes/highlights.asciidoc @@ -33,12 +33,102 @@ Other versions: endif::[] -// The notable-highlights tag marks entries that -// should be featured in the Stack Installation and Upgrade Guide: // tag::notable-highlights[] -// [discrete] -// === Heading -// -// Description. + +[discrete] +[[upgrade_repository_s3_to_aws_sdk_v2]] +=== Upgrade `repository-s3` to AWS SDK v2 +In earlier versions of {es} the `repository-s3` plugin was based on the AWS SDK v1. AWS will withdraw support for this SDK before the end of the life of {es} {minor-version} so we have migrated this plugin to the newer AWS SDK v2. +The two SDKs are not quite compatible, so please check the breaking changes documentation and test the new version thoroughly before upgrading any production workloads. + +{es-pull}126843[#126843] + +[discrete] +[[add_ability_to_redirect_ingestion_failures_on_data_streams_to_failure_store]] +=== Add ability to redirect ingestion failures on data streams to a failure store +Documents that encountered ingest pipeline failures or mapping conflicts +would previously be returned to the client as errors in the bulk and +index operations. Many client applications are not equipped to respond +to these failures. This leads to the failed documents often being +dropped by the client which cannot hold the broken documents +indefinitely. In many end user workloads, these failed documents +represent events that could be critical signals for observability or +security use cases. + +To help mitigate this problem, data streams can now maintain a "failure +store" which is used to accept and hold documents that fail to be +ingested due to preventable configuration errors. The data stream's +failure store operates like a separate set of backing indices with their +own mappings and access patterns that allow Elasticsearch to accept +documents that would otherwise be rejected due to unhandled ingest +pipeline exceptions or mapping conflicts. + +Users can enable redirection of ingest failures to the failure store on +new data streams by specifying it in the new `data_stream_options` field +inside of a component or index template: + +[source,yaml] +---- +PUT _index_template/my-template +{ + "index_patterns": ["logs-test-*"], + "data_stream": {}, + "template": { + "data_stream_options": { + "failure_store": { + "enabled": true + } + } + } +}' +---- + +Existing data streams can be configured with the new data stream +`_options` endpoint: + +[source,yaml] +---- +PUT _data_stream/logs-test-apache/_options +{ + "failure_store": { + "enabled": "true" + } +} +---- + +When redirection is enabled, any ingestion related failures will be +captured in the failure store if the cluster is able to, along with the +timestamp that the failure occurred, details about the error +encountered, and the document that could not be ingested. Since failure +stores are a kind of Elasticsearch index, we can search the data stream +for the failures that it has collected. The failures are not shown by +default as they are stored in different indices than the normal data +stream data. In order to retrieve the failures, we use the `_search` API +along with a new bit of index pattern syntax, the `::` selector. + +[source,yaml] +---- +POST logs-test-apache::failures/_search +---- + +This index syntax informs the search operation to target the indices in +its failure store instead of its backing indices. It can be mixed in a +number of ways with other index patterns to include their failure store +indices in the search operation: + +[source,yaml] +---- +POST logs-*::failures/_search +POST logs-*,logs-*::failures/_search +POST *::failures/_search +POST _query +{ + "query": "FROM my_data_stream*::failures" +} +---- + +{es-pull}126973[#126973] + // end::notable-highlights[] + From c8585b15d904cd6381f440f2d56fcdcb9ea5aaf5 Mon Sep 17 00:00:00 2001 From: Liam Thompson <32779855+leemthompo@users.noreply.github.com> Date: Mon, 2 Jun 2025 09:12:29 +0200 Subject: [PATCH 2/3] Revert highlights.asciidoc changes --- .../release-notes/highlights.asciidoc | 100 ++---------------- 1 file changed, 7 insertions(+), 93 deletions(-) diff --git a/docs/reference/release-notes/highlights.asciidoc b/docs/reference/release-notes/highlights.asciidoc index 615e5c1a4d3d0..2bade69388402 100644 --- a/docs/reference/release-notes/highlights.asciidoc +++ b/docs/reference/release-notes/highlights.asciidoc @@ -35,99 +35,13 @@ endif::[] // tag::notable-highlights[] -[discrete] -[[upgrade_repository_s3_to_aws_sdk_v2]] -=== Upgrade `repository-s3` to AWS SDK v2 -In earlier versions of {es} the `repository-s3` plugin was based on the AWS SDK v1. AWS will withdraw support for this SDK before the end of the life of {es} {minor-version} so we have migrated this plugin to the newer AWS SDK v2. -The two SDKs are not quite compatible, so please check the breaking changes documentation and test the new version thoroughly before upgrading any production workloads. - -{es-pull}126843[#126843] - -[discrete] -[[add_ability_to_redirect_ingestion_failures_on_data_streams_to_failure_store]] -=== Add ability to redirect ingestion failures on data streams to a failure store -Documents that encountered ingest pipeline failures or mapping conflicts -would previously be returned to the client as errors in the bulk and -index operations. Many client applications are not equipped to respond -to these failures. This leads to the failed documents often being -dropped by the client which cannot hold the broken documents -indefinitely. In many end user workloads, these failed documents -represent events that could be critical signals for observability or -security use cases. - -To help mitigate this problem, data streams can now maintain a "failure -store" which is used to accept and hold documents that fail to be -ingested due to preventable configuration errors. The data stream's -failure store operates like a separate set of backing indices with their -own mappings and access patterns that allow Elasticsearch to accept -documents that would otherwise be rejected due to unhandled ingest -pipeline exceptions or mapping conflicts. - -Users can enable redirection of ingest failures to the failure store on -new data streams by specifying it in the new `data_stream_options` field -inside of a component or index template: - -[source,yaml] ----- -PUT _index_template/my-template -{ - "index_patterns": ["logs-test-*"], - "data_stream": {}, - "template": { - "data_stream_options": { - "failure_store": { - "enabled": true - } - } - } -}' ----- - -Existing data streams can be configured with the new data stream -`_options` endpoint: - -[source,yaml] ----- -PUT _data_stream/logs-test-apache/_options -{ - "failure_store": { - "enabled": "true" - } -} ----- - -When redirection is enabled, any ingestion related failures will be -captured in the failure store if the cluster is able to, along with the -timestamp that the failure occurred, details about the error -encountered, and the document that could not be ingested. Since failure -stores are a kind of Elasticsearch index, we can search the data stream -for the failures that it has collected. The failures are not shown by -default as they are stored in different indices than the normal data -stream data. In order to retrieve the failures, we use the `_search` API -along with a new bit of index pattern syntax, the `::` selector. - -[source,yaml] ----- -POST logs-test-apache::failures/_search ----- - -This index syntax informs the search operation to target the indices in -its failure store instead of its backing indices. It can be mixed in a -number of ways with other index patterns to include their failure store -indices in the search operation: - -[source,yaml] ----- -POST logs-*::failures/_search -POST logs-*,logs-*::failures/_search -POST *::failures/_search -POST _query -{ - "query": "FROM my_data_stream*::failures" -} ----- - -{es-pull}126973[#126973] +// The notable-highlights tag marks entries that +// should be featured in the Stack Installation and Upgrade Guide: +// tag::notable-highlights[] +// [discrete] +// === Heading +// +// Description. // end::notable-highlights[] From 90869d4c050e1b36acd47d92594cfcea55785f3c Mon Sep 17 00:00:00 2001 From: Liam Thompson <32779855+leemthompo@users.noreply.github.com> Date: Mon, 2 Jun 2025 09:17:22 +0200 Subject: [PATCH 3/3] Revert reversion to highlights.asciidoc I'm not awake --- .../release-notes/highlights.asciidoc | 33 +++++++++++++++---- 1 file changed, 26 insertions(+), 7 deletions(-) diff --git a/docs/reference/release-notes/highlights.asciidoc b/docs/reference/release-notes/highlights.asciidoc index 2bade69388402..32998b9655ad6 100644 --- a/docs/reference/release-notes/highlights.asciidoc +++ b/docs/reference/release-notes/highlights.asciidoc @@ -35,13 +35,32 @@ endif::[] // tag::notable-highlights[] -// The notable-highlights tag marks entries that -// should be featured in the Stack Installation and Upgrade Guide: -// tag::notable-highlights[] -// [discrete] -// === Heading -// -// Description. +[discrete] +[[add_new_experimental_rank_vectors_mapping_for_late_interaction_second_order_ranking]] +=== `rank_vectors` field type is now available for late-interaction ranking + +<> is a new field type released as an experimental feature in {es} {minor-version}. It is designed to be used with dense vectors and allows for late-interaction second order ranking. + +Late-interaction models are powerful rerankers. While their size and overall cost doesn't lend itself for HNSW indexing, utilizing them as second order reranking can provide excellent boosts in relevance. The new `rank_vectors` mapping allows for rescoring over new and novel multi-vector late-interaction models like ColBERT or ColPali. + +[discrete] +[[enable_lookup_join_in_non_snapshot_builds]] +=== {esql} LOOKUP JOIN is now available in technical preview + +<> is now available in technical preview. +LOOKUP JOIN combines data from your {esql} queries with matching records from a lookup index, enabling you to: + +* Enrich your search results with reference data +* Speed up root-cause analysis and security investigations +* Join data across indices without complex queries +* Reduce operational overhead when correlating events + +[discrete] +[[release_semantic_text_as_ga_feature]] +=== The `semantic_text` field type is now GA + +<> is now an official GA (generally available) feature! This field type allows you to easily set up and perform semantic search with minimal ramp up time. + // end::notable-highlights[]