|
| 1 | +[[release-highlights]] |
| 2 | +== What's new in {minor-version} |
| 3 | + |
| 4 | +coming::[{minor-version}] |
| 5 | + |
| 6 | +Here are the highlights of what's new and improved in {es} {minor-version}! |
| 7 | +ifeval::["{release-state}"!="unreleased"] |
| 8 | +For detailed information about this release, see the <<es-release-notes>> and |
| 9 | +<<breaking-changes>>. |
| 10 | + |
| 11 | +// Add previous release to the list |
| 12 | +Other versions: |
| 13 | + |
| 14 | +{ref-bare}/8.18/release-highlights.html[8.18] |
| 15 | +| {ref-bare}/8.17/release-highlights.html[8.17] |
| 16 | +| {ref-bare}/8.16/release-highlights.html[8.16] |
| 17 | +| {ref-bare}/8.15/release-highlights.html[8.15] |
| 18 | +| {ref-bare}/8.14/release-highlights.html[8.14] |
| 19 | +| {ref-bare}/8.13/release-highlights.html[8.13] |
| 20 | +| {ref-bare}/8.12/release-highlights.html[8.12] |
| 21 | +| {ref-bare}/8.11/release-highlights.html[8.11] |
| 22 | +| {ref-bare}/8.10/release-highlights.html[8.10] |
| 23 | +| {ref-bare}/8.9/release-highlights.html[8.9] |
| 24 | +| {ref-bare}/8.8/release-highlights.html[8.8] |
| 25 | +| {ref-bare}/8.7/release-highlights.html[8.7] |
| 26 | +| {ref-bare}/8.6/release-highlights.html[8.6] |
| 27 | +| {ref-bare}/8.5/release-highlights.html[8.5] |
| 28 | +| {ref-bare}/8.4/release-highlights.html[8.4] |
| 29 | +| {ref-bare}/8.3/release-highlights.html[8.3] |
| 30 | +| {ref-bare}/8.2/release-highlights.html[8.2] |
| 31 | +| {ref-bare}/8.1/release-highlights.html[8.1] |
| 32 | +| {ref-bare}/8.0/release-highlights.html[8.0] |
| 33 | + |
| 34 | +endif::[] |
| 35 | + |
| 36 | +// tag::notable-highlights[] |
| 37 | + |
| 38 | +[discrete] |
| 39 | +[[upgrade_repository_s3_to_aws_sdk_v2]] |
| 40 | +=== Upgrade `repository-s3` to AWS SDK v2 |
| 41 | +In earlier versions of {es} the `repository-s3` plugin was based on the AWS SDK v1. AWS will withdraw support for this SDK before the end of the life of {es} {minor-version} so we have migrated this plugin to the newer AWS SDK v2. |
| 42 | +The two SDKs are not quite compatible, so please check the breaking changes documentation and test the new version thoroughly before upgrading any production workloads. |
| 43 | + |
| 44 | +{es-pull}126843[#126843] |
| 45 | + |
| 46 | +[discrete] |
| 47 | +[[add_ability_to_redirect_ingestion_failures_on_data_streams_to_failure_store]] |
| 48 | +=== Add ability to redirect ingestion failures on data streams to a failure store |
| 49 | +Documents that encountered ingest pipeline failures or mapping conflicts |
| 50 | +would previously be returned to the client as errors in the bulk and |
| 51 | +index operations. Many client applications are not equipped to respond |
| 52 | +to these failures. This leads to the failed documents often being |
| 53 | +dropped by the client which cannot hold the broken documents |
| 54 | +indefinitely. In many end user workloads, these failed documents |
| 55 | +represent events that could be critical signals for observability or |
| 56 | +security use cases. |
| 57 | + |
| 58 | +To help mitigate this problem, data streams can now maintain a "failure |
| 59 | +store" which is used to accept and hold documents that fail to be |
| 60 | +ingested due to preventable configuration errors. The data stream's |
| 61 | +failure store operates like a separate set of backing indices with their |
| 62 | +own mappings and access patterns that allow Elasticsearch to accept |
| 63 | +documents that would otherwise be rejected due to unhandled ingest |
| 64 | +pipeline exceptions or mapping conflicts. |
| 65 | + |
| 66 | +Users can enable redirection of ingest failures to the failure store on |
| 67 | +new data streams by specifying it in the new `data_stream_options` field |
| 68 | +inside of a component or index template: |
| 69 | + |
| 70 | +[source,yaml] |
| 71 | +---- |
| 72 | +PUT _index_template/my-template |
| 73 | +{ |
| 74 | + "index_patterns": ["logs-test-*"], |
| 75 | + "data_stream": {}, |
| 76 | + "template": { |
| 77 | + "data_stream_options": { |
| 78 | + "failure_store": { |
| 79 | + "enabled": true |
| 80 | + } |
| 81 | + } |
| 82 | + } |
| 83 | +}' |
| 84 | +---- |
| 85 | + |
| 86 | +Existing data streams can be configured with the new data stream |
| 87 | +`_options` endpoint: |
| 88 | + |
| 89 | +[source,yaml] |
| 90 | +---- |
| 91 | +PUT _data_stream/logs-test-apache/_options |
| 92 | +{ |
| 93 | + "failure_store": { |
| 94 | + "enabled": "true" |
| 95 | + } |
| 96 | +} |
| 97 | +---- |
| 98 | + |
| 99 | +When redirection is enabled, any ingestion related failures will be |
| 100 | +captured in the failure store if the cluster is able to, along with the |
| 101 | +timestamp that the failure occurred, details about the error |
| 102 | +encountered, and the document that could not be ingested. Since failure |
| 103 | +stores are a kind of Elasticsearch index, we can search the data stream |
| 104 | +for the failures that it has collected. The failures are not shown by |
| 105 | +default as they are stored in different indices than the normal data |
| 106 | +stream data. In order to retrieve the failures, we use the `_search` API |
| 107 | +along with a new bit of index pattern syntax, the `::` selector. |
| 108 | + |
| 109 | +[source,yaml] |
| 110 | +---- |
| 111 | +POST logs-test-apache::failures/_search |
| 112 | +---- |
| 113 | + |
| 114 | +This index syntax informs the search operation to target the indices in |
| 115 | +its failure store instead of its backing indices. It can be mixed in a |
| 116 | +number of ways with other index patterns to include their failure store |
| 117 | +indices in the search operation: |
| 118 | + |
| 119 | +[source,yaml] |
| 120 | +---- |
| 121 | +POST logs-*::failures/_search |
| 122 | +POST logs-*,logs-*::failures/_search |
| 123 | +POST *::failures/_search |
| 124 | +POST _query |
| 125 | +{ |
| 126 | + "query": "FROM my_data_stream*::failures" |
| 127 | +} |
| 128 | +---- |
| 129 | + |
| 130 | +{es-pull}126973[#126973] |
| 131 | + |
| 132 | +// end::notable-highlights[] |
| 133 | + |
| 134 | + |
0 commit comments