Skip to content

Conversation

joegallo
Copy link
Contributor

Backport of #114847

gregkalapos and others added 30 commits October 4, 2024 23:24
…#114049) (elastic#114125)

* Add @timestamp as sort field for logs and traces

* Change order of sort fields

* Adapt tests
) (elastic#114087)

* Assert that REST params are consumed iff supported (elastic#114040)

REST APIs which declare their supported parameters must consume exactly
those parameters: consuming an unsupported parameter means that requests
including that parameter will be rejected, whereas failing to consume a
supported parameter means that this parameter has no effect and should
be removed.

This commit adds an assertion to verify that we are consuming the
correct parameters.

Closes elastic#113854

* CI poke
…astic#113961) (elastic#114130)

While working on Date Nanos, I noticed that Least and Greatest didn't have support for datetime. This PR corrects that and adds tests for it.

It seems to me that resolveType() is doing the wrong thing for these functions, as it accepts types that then do not have evaluator mappings, but refactoring that seems out of scope right now.

---------

Co-authored-by: Elastic Machine <[email protected]>
…stic#114136)

Fixes a validation step that might prevent creating empty aggregations iif the output format does not 
allows negative numbers.
…lastic#114140)

This change exposes query, fetch and indexing metrics
for each index mode.
Today there are a handful of integer settings for `repository-s3`
repositories whose docs link to the page about numeric field types. Yet
these settings are not fields, and do not support floating-point values
either. The convention throughout the rest of the docs is to just call
these things `integer` without linking to anything. This commit aligns
the `repository-s3` docs with this convention.

Backport of elastic#114093 to `8.x`
This reworks `EvaluatorMapper` to use a named interface instead of a
`Function<Expression, EvalOperator.ExpressionEvaluator.Factory>`. It's
shorter to type and it allows us to add extra information to the
builder.
This fixes tests so that they can work with multiple shards.
…operations (elastic#114075) (elastic#114153)

* Took time and cluster details get updated for coordinator only query operations

The ComputeService.runCompute pathway for coordinator only operations (such as 
`FROM foo | LIMIT 0` or a ROW command) get updated with overall took time.
This also includes support for cross-cluster coordinator only operations, which come
about with queries like `FROM foo,remote:foo | LIMIT 0`. The _clusters metadata is
now properly updated for those cases as well.

Fixes elastic#114014
Currently we have a relatively basic decider about when to throttling
indexing. This commit adds two levels of watermarks with configurable
bulk size deciders. Additionally, adds additional settings to control
primary, coordinating, and replica rejection limits.
It is possible in the incremental high watermark test that the data is
submitted causing a corruption of the bulk request. This commit fixes
the issue to ensure we only send new data after it has been requested.
Additionally, it adds an assertion to prevent this error from happening
again.
… UI (elastic#114042) (elastic#114173)

* Hide aggregated metrics generated for the APM UI

* Update 30_aggregated_metrics_tests.yml

* Review feedback

- Introduced templates for 1 minute aggregations
- Moved dynamic templates `ecs_ip` and `all_strings_to_keywords` into a dedicated file and now I pull the file in instead of repeating them
- Introduced `metrics-[x]m.otel@custom`
- Added tests with terms aggregation that assert by default 1 bucket (only 1m) with metricsset.interval, and with allowing hidden indices it's 3 buckets (1m, 10m, 60m)

* Update 30_aggregated_metrics_tests.yml

Simplify - no need to separate tests for the 3bucket queries.

* Rename metrics-otel-fixed@mappings to ecs-tsdb@mappings

---------

Co-authored-by: Elastic Machine <[email protected]>
Spells out that third-party EOL schedules don't affect our support. Also
reorders the information to talk about the benefits of the bundled JDK
before mentioning alternatives, and clarifies the division of
responsibilities for "supported" JDKs other than the bundled one.
…stic#113936)

Backports the following commits to 8.x:
 - Fix MemoryTracking...ArraysTests bulkSet errors (elastic#113934)
Small refactor that makes CellBoundary immutable.
…elastic#114212)

The leftover usages of the deprecated  search(Query, TotalHitCountCollector)
have been replaced with search(Query, TotalHitCountCollectorManager)
…lastic#114216)

This example request will succeed, but follow-up requests to run a sync on a connector with this language value will fail.
…ic#114224)

* Fix BWC for file settings based role mappings
---------

Co-authored-by: Johannes Freden Jansson <[email protected]>
(cherry picked from commit 763764c)

Co-authored-by: Nikolaj Volgushev <[email protected]>
kkrik-es and others added 18 commits October 16, 2024 00:05
The list with two options doesn't get rendered as a list, due to the
snippet in between.

https://www.elastic.co/guide/en/elasticsearch/reference/master/passthrough.html#passthrough-conflicts
(cherry picked from commit 8cf2cb3)
…114725)

Fix rank doc query when some segments have no ranked docs
…) (elastic#114826)

* [TEST] Migrated ccs-unavailable-clusters QA tests (elastic#114764)

Ccs-unavailable-clusters QA tests migrated to the new REST testing
framework, using 'elasticsearch.internal-java-rest-test' Gradle plugin

(cherry picked from commit 551a7d6)

* [TEST] Fix ccs-unavailable-clusters QA tests build (elastic#114833)

Properly use `configureEach` on the task configuration to postpone the
tasks creation and configuration in the build process
* ESQL: Introduce per agg filter (elastic#113735)

Add support for aggregation scoped filters that work dynamically on the
 data in each group.

| STATS
    success = COUNT(*) WHERE 200 <= code AND code < 300,
   redirect = COUNT(*) WHERE 300 <= code AND code < 400,
 client_err = COUNT(*) WHERE 400 <= code AND code < 500,
 server_err = COUNT(*) WHERE 500 <= code AND code < 600,
 total_count = COUNT(*)

Implementation wise, the base AggregateFunction has been extended to
 allow a filter to be passed on. This is required to incorporate the
 filter as part of the aggregate equality/identify which would fail with
 the filter as an external component.

As part of the process, the serialization for the existing aggregations
 had to be fixed so AggregateFunction implementations so that it
 delegates to their parent first.

(cherry picked from commit d102659)

* Update docs/changelog/114842.yaml

* Delete docs/changelog/114842.yaml
…14843)

Data Stream Lifecycle has GA'ed in 8.14, so we can safely mark these as
stable.
) (elastic#114841)

`ThreadContext#stashContext` does not yield a completely fresh context:
it preserves headers related to tracing the original request. That may
be appropriate in many situations, but sometimes we really do want to
detach processing entirely from the original task. This commit
introduces new utilities to do that.
…lastic#114829) (elastic#114846)

* Set min number of allocations for ElasticSearchInternalService to 0

* Updating IT tests with new min allocations value

---------

Co-authored-by: Elastic Machine <[email protected]>
…lastic#114853)

Remove dev flag left in grammar for agg filtering
Related to elastic#113735

(cherry picked from commit 5e59ab5)
…c#114648)

* Allow stored source in logsdb and tsdb (elastic#114454)

(cherry picked from commit a62228a)

# Conflicts:
#	modules/aggregations/build.gradle
#	modules/data-streams/src/javaRestTest/java/org/elasticsearch/datastreams/logsdb/LogsIndexModeCustomSettingsIT.java
#	rest-api-spec/build.gradle

* Fix tests

* Fix tests

---------

Co-authored-by: Elastic Machine <[email protected]>
…c#114836) (elastic#114860)

In elastic#112063 we added support for multivalued fields to the compute engine for ST_INTERSECTS and relatives, but not for ST_DISTANCE. In elastic#114729 it was discovered that, at least for the common case of a field and a constant, this support was not needed due to ST_DISTANCE being re-written to ST_INTERSECTS. However, for many other cases, like ST_DISTANCE used on the coordinator node, or between two fields, this lack of support would result in null values.

This PR fixes those cases, making sure ST_DISTANCE uses the Block-Builder approach similar to what was done for ST_INTERSECTS et al.
…stic#114832) (elastic#114851)

We actually don't need a cluster feature, a capability added if the
feature flag is enabled is enough for testing.

closes elastic#114787

(cherry picked from commit e87b894)

Co-authored-by: Elastic Machine <[email protected]>
@joegallo joegallo added :Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP backport Team:Data Management Meta label for data/management team auto-merge-without-approval Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) v8.16.0 labels Oct 15, 2024
@joegallo joegallo requested review from a team as code owners October 15, 2024 23:48
Copy link
Contributor

Documentation preview:

@joegallo
Copy link
Contributor Author

OOPS

@joegallo joegallo closed this Oct 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto-merge-without-approval Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) backport :Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP Team:Data Management Meta label for data/management team v8.16.0 v9.0.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.