diff --git a/docs/reference/release-notes.asciidoc b/docs/reference/release-notes.asciidoc index dbb0c349891a8..a0b06aadd9646 100644 --- a/docs/reference/release-notes.asciidoc +++ b/docs/reference/release-notes.asciidoc @@ -7,6 +7,7 @@ This section summarizes the changes in each release. * <> +* <> * <> * <> * <> @@ -73,8 +74,9 @@ This section summarizes the changes in each release. -- include::release-notes/8.16.0.asciidoc[] -include::release-notes/8.15.0.asciidoc[] +include::release-notes/8.15.2.asciidoc[] include::release-notes/8.15.1.asciidoc[] +include::release-notes/8.15.0.asciidoc[] include::release-notes/8.14.3.asciidoc[] include::release-notes/8.14.2.asciidoc[] include::release-notes/8.14.1.asciidoc[] diff --git a/docs/reference/release-notes/8.15.2.asciidoc b/docs/reference/release-notes/8.15.2.asciidoc new file mode 100644 index 0000000000000..d2853215098cd --- /dev/null +++ b/docs/reference/release-notes/8.15.2.asciidoc @@ -0,0 +1,44 @@ +[[release-notes-8.15.2]] +== {es} version 8.15.2 + +Also see <>. + +[[bug-8.15.2]] +[float] +=== Bug fixes + +Authorization:: +* Fix remote cluster credential secure settings reload {es-pull}111535[#111535] + +ES|QL:: +* ESQL: Don't mutate the `BoolQueryBuilder` in plan {es-pull}111519[#111519] +* ES|QL: Fix `ResolvedEnrichPolicy` serialization (bwc) in v 8.15 {es-pull}112985[#112985] (issue: {es-issue}112968[#112968]) +* Fix union-types where one index is missing the field {es-pull}111932[#111932] (issue: {es-issue}111912[#111912]) +* Support widening of numeric types in union-types {es-pull}112610[#112610] (issue: {es-issue}111277[#111277]) +* Shorten error messages for UnsupportedAttributes {es-pull}112819[#112819] +* Reduce memory footprint of serialized query execution plans {es-pull}112865[#112865] + +Infra/Core:: +* JSON parse failures should be 4xx codes {es-pull}112703[#112703] +* Json parsing exceptions should not cause 500 errors {es-pull}111548[#111548] (issue: {es-issue}111542[#111542]) +* Make sure file accesses in `DnRoleMapper` are done in stack frames with permissions {es-pull}112400[#112400] + +Ingest Node:: +* Fix missing header in `put_geoip_database` JSON spec {es-pull}112581[#112581] + +Logs:: +* Fix encoding of dynamic arrays in ignored source {es-pull}112713[#112713] + +Mapping:: +* Full coverage of ECS by ecs@mappings when `date_detection` is disabled {es-pull}112444[#112444] (issue: {es-issue}112398[#112398]) + +Search:: +* Fix parsing error in `_terms_enum` API {es-pull}112872[#112872] (issue: {es-issue}94378[#94378]) + +Security:: +* Allowlist `tracestate` header on remote server port {es-pull}112649[#112649] + +Vector Search:: +* Fix NPE in `dense_vector` stats {es-pull}112720[#112720] + + diff --git a/docs/reference/release-notes/highlights.asciidoc b/docs/reference/release-notes/highlights.asciidoc index 007dd740f34cf..664b006b9e8e1 100644 --- a/docs/reference/release-notes/highlights.asciidoc +++ b/docs/reference/release-notes/highlights.asciidoc @@ -32,42 +32,6 @@ endif::[] // tag::notable-highlights[] -[discrete] -[[stored_fields_are_compressed_with_zstandard_instead_of_lz4_deflate]] -=== Stored fields are now compressed with ZStandard instead of LZ4/DEFLATE -Stored fields are now compressed by splitting documents into blocks, which -are then compressed independently with ZStandard. `index.codec: default` -(default) uses blocks of at most 14kB or 128 documents compressed with level -0, while `index.codec: best_compression` uses blocks of at most 240kB or -2048 documents compressed at level 3. On most datasets that we tested -against, this yielded storage improvements in the order of 10%, slightly -faster indexing and similar retrieval latencies. - -{es-pull}103374[#103374] - -[discrete] -[[stricter_failure_handling_in_multi_repo_get_snapshots_request_handling]] -=== Stricter failure handling in multi-repo get-snapshots request handling -If a multi-repo get-snapshots request encounters a failure in one of the -targeted repositories then earlier versions of Elasticsearch would proceed -as if the faulty repository did not exist, except for a per-repository -failure report in a separate section of the response body. This makes it -impossible to paginate the results properly in the presence of failures. In -versions 8.15.0 and later this API's failure handling behaviour has been -made stricter, reporting an overall failure if any targeted repository's -contents cannot be listed. - -{es-pull}107191[#107191] - -[discrete] -[[add_new_int4_quantization_to_dense_vector]] -=== Add new int4 quantization to dense_vector -New int4 (half-byte) scalar quantization support via two knew index types: `int4_hnsw` and `int4_flat`. -This gives an 8x reduction from `float32` with some accuracy loss. In addition to less memory required, this -improves query and merge speed significantly when compared to raw vectors. - -{es-pull}109317[#109317] - [discrete] [[esql_inlinestats]] === ESQL: INLINESTATS @@ -93,50 +57,6 @@ Produces output like: {es-pull}109583[#109583] -[discrete] -[[mark_query_rules_as_ga]] -=== Mark Query Rules as GA -This PR marks query rules as Generally Available. All APIs are no longer -in tech preview. - -{es-pull}110004[#110004] - -[discrete] -[[adds_new_bit_element_type_for_dense_vectors]] -=== Adds new `bit` `element_type` for `dense_vectors` -This adds `bit` vector support by adding `element_type: bit` for -vectors. This new element type works for indexed and non-indexed -vectors. Additionally, it works with `hnsw` and `flat` index types. No -quantization based codec works with this element type, this is -consistent with `byte` vectors. - -`bit` vectors accept up to `32768` dimensions in size and expect vectors -that are being indexed to be encoded either as a hexidecimal string or a -`byte[]` array where each element of the `byte` array represents `8` -bits of the vector. - -`bit` vectors support script usage and regular query usage. When -indexed, all comparisons done are `xor` and `popcount` summations (aka, -hamming distance), and the scores are transformed and normalized given -the vector dimensions. - -For scripts, `l1norm` is the same as `hamming` distance and `l2norm` is -`sqrt(l1norm)`. `dotProduct` and `cosineSimilarity` are not supported. - -Note, the dimensions expected by this element_type are always to be -divisible by `8`, and the `byte[]` vectors provided for index must be -have size `dim/8` size, where each byte element represents `8` bits of -the vectors. - -{es-pull}110059[#110059] - -[discrete] -[[redact_processor_generally_available]] -=== The Redact processor is Generally Available -The Redact processor uses the Grok rules engine to obscure text in the input document matching the given Grok patterns. The Redact processor was initially released as Technical Preview in `8.7.0`, and is now released as Generally Available. - -{es-pull}110395[#110395] - [discrete] [[always_allow_rebalancing_by_default]] === Always allow rebalancing by default @@ -149,76 +69,65 @@ version 8.16 `allow_rebalance` setting defaults to `always` unless the legacy al {es-pull}111015[#111015] -// end::notable-highlights[] - - [discrete] -[[new_custom_parser_for_iso_8601_datetimes]] -=== New custom parser for ISO-8601 datetimes -This introduces a new custom parser for ISO-8601 datetimes, for the `iso8601`, `strict_date_optional_time`, and -`strict_date_optional_time_nanos` built-in date formats. This provides a performance improvement over the -default Java date-time parsing. Whilst it maintains much of the same behaviour, -the new parser does not accept nonsensical date-time strings that have multiple fractional seconds fields -or multiple timezone specifiers. If the new parser fails to parse a string, it will then use the previous parser -to parse it. If a large proportion of the input data consists of these invalid strings, this may cause -a small performance degradation. If you wish to force the use of the old parsers regardless, -set the JVM property `es.datetime.java_time_parsers=true` on all ES nodes. - -{es-pull}106486[#106486] +[[add_global_retention_in_data_stream_lifecycle]] +=== Add global retention in data stream lifecycle +Data stream lifecycle now supports configuring retention on a cluster level, +namely global retention. Global retention \nallows us to configure two different +retentions: -[discrete] -[[new_custom_parser_for_more_iso_8601_date_formats]] -=== New custom parser for more ISO-8601 date formats -Following on from #106486, this extends the custom ISO-8601 datetime parser to cover the `strict_year`, -`strict_year_month`, `strict_date_time`, `strict_date_time_no_millis`, `strict_date_hour_minute_second`, -`strict_date_hour_minute_second_millis`, and `strict_date_hour_minute_second_fraction` date formats. -As before, the parser will use the existing java.time parser if there are parsing issues, and the -`es.datetime.java_time_parsers=true` JVM property will force the use of the old parsers regardless. +- `data_streams.lifecycle.retention.default` is applied to all data streams managed +by the data stream lifecycle that do not have retention defined on the data stream level. +- `data_streams.lifecycle.retention.max` is applied to all data streams managed by the +data stream lifecycle and it allows any data stream \ndata to be deleted after the `max_retention` has passed. -{es-pull}108606[#108606] +{es-pull}111972[#111972] [discrete] -[[preview_support_for_connection_type_domain_isp_databases_in_geoip_processor]] -=== Preview: Support for the 'Connection Type, 'Domain', and 'ISP' databases in the geoip processor -As a Technical Preview, the {ref}/geoip-processor.html[`geoip`] processor can now use the commercial -https://dev.maxmind.com/geoip/docs/databases/connection-type[GeoIP2 'Connection Type'], -https://dev.maxmind.com/geoip/docs/databases/domain[GeoIP2 'Domain'], -and -https://dev.maxmind.com/geoip/docs/databases/isp[GeoIP2 'ISP'] -databases from MaxMind. - -{es-pull}108683[#108683] +[[enable_zstandard_compression_for_indices_with_index_codec_set_to_best_compression]] +=== Enable ZStandard compression for indices with index.codec set to best_compression +Before DEFLATE compression was used to compress stored fields in indices with index.codec index setting set to +best_compression, with this change ZStandard is used as compression algorithm to stored fields for indices with +index.codec index setting set to best_compression. The usage ZStandard results in less storage usage with a +similar indexing throughput depending on what options are used. Experiments with indexing logs have shown that +ZStandard offers ~12% lower storage usage and a ~14% higher indexing throughput compared to DEFLATE. -[discrete] -[[update_elasticsearch_to_lucene_9_11]] -=== Update Elasticsearch to Lucene 9.11 -Elasticsearch is now updated using the latest Lucene version 9.11. -Here are the full release notes: -But, here are some particular highlights: -- Usage of MADVISE for better memory management: https://github.com/apache/lucene/pull/13196 -- Use RWLock to access LRUQueryCache to reduce contention: https://github.com/apache/lucene/pull/13306 -- Speedup multi-segment HNSW graph search for nested kNN queries: https://github.com/apache/lucene/pull/13121 -- Add a MemorySegment Vector scorer - for scoring without copying on-heap vectors: https://github.com/apache/lucene/pull/13339 - -{es-pull}109219[#109219] +{es-pull}112665[#112665] [discrete] -[[synthetic_source_improvements]] -=== Synthetic `_source` improvements -There are multiple improvements to synthetic `_source` functionality: +[[8_x_remove_zstd_feature_flag_for_index_codec_best_compression]] +=== [8.x] Remove zstd feature flag for index codec best compression +Backports the following commits to 8.x: - Remove zstd feature flag for +index codec best compression. (#112665) -* Synthetic `_source` is now supported for all field types including `nested` and `object`. `object` fields are supported with `enabled` set to `false`. +{es-pull}112857[#112857] -* Synthetic `_source` can be enabled together with `ignore_malformed` and `ignore_above` parameters for all field types that support them. +// end::notable-highlights[] -{es-pull}109501[#109501] [discrete] -[[index_sorting_on_indexes_with_nested_fields]] -=== Index sorting on indexes with nested fields -Index sorting is now supported for indexes with mappings containing nested objects. -The index sort spec (as specified by `index.sort.field`) can't contain any nested -fields, still. - -{es-pull}110251[#110251] +[[esql_multi_value_fields_supported_in_geospatial_predicates]] +=== ESQL: Multi-value fields supported in Geospatial predicates +Supporting multi-value fields in `WHERE` predicates is a challenge due to not knowing whether `ALL` or `ANY` +of the values in the field should pass the predicate. +For example, should the field `age:[10,30]` pass the predicate `WHERE age>20` or not? +This ambiguity does not exist with the spatial predicates +`ST_INTERSECTS` and `ST_DISJOINT`, because the choice between `ANY` or `ALL` +is implied by the predicate itself. +Consider a predicate checking a field named `location` against a test geometry named `shape`: + +* `ST_INTERSECTS(field, shape)` - true if `ANY` value can intersect the shape +* `ST_DISJOINT(field, shape)` - true only if `ALL` values are disjoint from the shape + +This works even if the shape argument is itself a complex or compound geometry. + +Similar logic exists for `ST_CONTAINS` and `ST_WITHIN` predicates, but these are not as easily solved +with `ANY` or `ALL`, because a collection of geometries contains another collection if each of the contained +geometries is within at least one of the containing geometries. Evaluating this requires that the multi-value +field is first combined into a single geometry before performing the predicate check. + +* `ST_CONTAINS(field, shape)` - true if the combined geometry contains the shape +* `ST_WITHIN(field, shape)` - true if the combined geometry is within the shape + +{es-pull}112063[#112063]