Run field extraction concurrently in TS #128643
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dnhatn
force-pushed
the
ts-parallel-extract
branch
from
ff7256a to
fdbf5c9
Compare
elasticsearchmachine
added
Team:Analytics
|
Pinging @elastic/es-analytical-engine (Team:Analytics) |
|
Pinging @elastic/es-storage-engine (Team:StorageEngine) |
28 tasks
kkrik-es
reviewed
May 30, 2025
| @@ -94,22 +89,14 @@ private PhysicalPlan addFieldExtract( | |||
| attributesToExtract.addAll(extract.attributesToExtract()); | |||
| } | |||
| List<Attribute> attrs = query.attrs(); | |||
| if (keepDocAttribute == false) { | |||
| attrs = attrs.stream().filter(a -> EsQueryExec.isSourceAttribute(a) == false).toList(); | |||
There was a problem hiding this comment.
That's one of the overheads of running them separately. Previously, if all fields were loaded directly in the source operator, we could skip emitting the doc vector. Now, however, we need to emit it for ValuesSourceReaderOperator.
kkrik-es
approved these changes
May 30, 2025
|
Thanks Kostas! |
mridula-s109
pushed a commit
that referenced
this pull request
Jun 2, 2025
This change extracts the field extraction logic previously run inside `TimeSeriesSourceOperator` into a separate operator, executed in a separate driver. We should consider consolidating this operator with `ValuesSourceReaderOperator`. I tried to extend `ValuesSourceReaderOperator` to support this, but it may take some time to complete. Our current goal is to move quickly with experimental support for querying time-series data in ES|QL, so I am proceeding with a separate operator. I will spend more time on combining these two operators later. With #128419 and this PR, the query time for the example below decreased from 41ms to 27ms. ``` POST /_query { "profile": true, "query": "TS metrics-hostmetricsreceiver.otel-default | WHERE @timestamp >= \"2025-05-08T18:00:08.001Z\" | STATS cpu = avg(rate(`metrics.process.cpu.time`)) BY host.name, BUCKET(@timestamp, 5 minute)" } ```
mridula-s109
pushed a commit
to mridula-s109/elasticsearch
that referenced
this pull request
Jun 3, 2025
This change extracts the field extraction logic previously run inside `TimeSeriesSourceOperator` into a separate operator, executed in a separate driver. We should consider consolidating this operator with `ValuesSourceReaderOperator`. I tried to extend `ValuesSourceReaderOperator` to support this, but it may take some time to complete. Our current goal is to move quickly with experimental support for querying time-series data in ES|QL, so I am proceeding with a separate operator. I will spend more time on combining these two operators later. With elastic#128419 and this PR, the query time for the example below decreased from 41ms to 27ms. ``` POST /_query { "profile": true, "query": "TS metrics-hostmetricsreceiver.otel-default | WHERE @timestamp >= \"2025-05-08T18:00:08.001Z\" | STATS cpu = avg(rate(`metrics.process.cpu.time`)) BY host.name, BUCKET(@timestamp, 5 minute)" } ```
joshua-adams-1
pushed a commit
to joshua-adams-1/elasticsearch
that referenced
this pull request
Jun 3, 2025
This change extracts the field extraction logic previously run inside `TimeSeriesSourceOperator` into a separate operator, executed in a separate driver. We should consider consolidating this operator with `ValuesSourceReaderOperator`. I tried to extend `ValuesSourceReaderOperator` to support this, but it may take some time to complete. Our current goal is to move quickly with experimental support for querying time-series data in ES|QL, so I am proceeding with a separate operator. I will spend more time on combining these two operators later. With elastic#128419 and this PR, the query time for the example below decreased from 41ms to 27ms. ``` POST /_query { "profile": true, "query": "TS metrics-hostmetricsreceiver.otel-default | WHERE @timestamp >= \"2025-05-08T18:00:08.001Z\" | STATS cpu = avg(rate(`metrics.process.cpu.time`)) BY host.name, BUCKET(@timestamp, 5 minute)" } ```
Samiul-TheSoccerFan
pushed a commit
to Samiul-TheSoccerFan/elasticsearch
that referenced
this pull request
Jun 5, 2025
This change extracts the field extraction logic previously run inside `TimeSeriesSourceOperator` into a separate operator, executed in a separate driver. We should consider consolidating this operator with `ValuesSourceReaderOperator`. I tried to extend `ValuesSourceReaderOperator` to support this, but it may take some time to complete. Our current goal is to move quickly with experimental support for querying time-series data in ES|QL, so I am proceeding with a separate operator. I will spend more time on combining these two operators later. With elastic#128419 and this PR, the query time for the example below decreased from 41ms to 27ms. ``` POST /_query { "profile": true, "query": "TS metrics-hostmetricsreceiver.otel-default | WHERE @timestamp >= \"2025-05-08T18:00:08.001Z\" | STATS cpu = avg(rate(`metrics.process.cpu.time`)) BY host.name, BUCKET(@timestamp, 5 minute)" } ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change extracts the field extraction logic previously run inside
TimeSeriesSourceOperatorinto a separate operator, executed in a separate driver. We should consider consolidating this operator withValuesSourceReaderOperator. I tried to extendValuesSourceReaderOperatorto support this, but it may take some time to complete. Our current goal is to move quickly with experimental support for querying time-series data in ES|QL, so I am proceeding with a separate operator. I will spend more time on combining these two operators later.With #128419 and this PR, the query time for the example below decreased from 41ms to 27ms.