ESQL: Enable visualizing a query profile #124361

alex-spies · 2025-03-07T17:24:34Z

To understand query performance, we often peruse the output of _query-requests run with "profile": true.

This is difficult when the query runs in a large cluster with many nodes and shards, or in case of CCQ.

This adds an option to visualize a query using Chromium's/Chrome's builtin about:tracing - or, for even better visuals and querying the different drivers via SQL, perfetto (c.f. https://ui.perfetto.dev/).

To use, save the JSON output of a query run with "profile": true to a file, like output.json and then invoke the following Gradle task:

./gradlew x-pack:plugin:esql:tools:parseProfile --args='~/output.json ~/parsed_profile.json'

Either open about:tracing in Chromium/Chrome

Or head over to https://ui.perfetto.dev (build locally in case of potentially sensitive data in the profille):

Every slice is a driver, the colors indicating the ratio of cpu time over total time.

In Perfetto, essentials like duration, cpu duration, timestamp and a few others can be queried via SQL - this allows e.g. querying for all drivers that spent more than 50% of their time waiting and other fun things.
Details about a driver, esp. which operators it ran, are available when clicking the driver's slice.

Invoke it like this: ./gradlew x-pack:plugin:esql:qa:testFixtures:parseProfile --args='~/elasticsearch/profile.json ~/elasticsearch/output.json' Then it can be imported into e.g. perfetto or into Chromes trace viewer (about:tracing)

elasticsearchmachine · 2025-03-07T17:25:12Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

idegtiarenko · 2025-03-10T07:56:45Z

x-pack/plugin/esql/tools/src/main/java/org/elasticsearch/xpack/esql/tools/ProfileParser.java

+        Path outputFileName = Path.of(args[1].replaceFirst("^~", System.getProperty("user.home"))).toAbsolutePath();
+
+        Map<String, Object> map;
+        try (InputStream input = Files.newInputStream(inputFileName)) {


I believe you originally started with jq. What was the final reason to replace it with java?

It was too messy to properly emit metadata events for nodes, assign correct tid and pids etc.

If we want to evolve either the visualization/profile parsing or the profile itself, it's much easier to do so in Java.

I can nicely test this.

idegtiarenko · 2025-03-10T07:58:55Z

x-pack/plugin/esql/tools/src/main/java/org/elasticsearch/xpack/esql/tools/ProfileParser.java

+        Map<String, Object> map;
+        try (InputStream input = Files.newInputStream(inputFileName)) {
+            logger.info("Starting to parse {}", inputFileName);
+            map = XContentHelper.convertToMap(JsonXContent.jsonXContent, input, true);


I wonder if we should consider model rather than a map?
Possibly something like:

public record Response(Profile profile) {} public record Profile(List<Driver> drivers) {} public record Driver( @JsonProperty("task_description") String taskDescription, @JsonProperty("cluster_name") String cluster, @JsonProperty("node_name") String node, @JsonProperty("start_millis") long startMillis, @JsonProperty("stop_millis") long stopMillis, List<Operator> operators ) {} public record Operator(String operator) {}

Thank you, with Jackson it's indeed much clearer. I use this approach now + updated the tests accordingly.

elasticsearchmachine · 2025-03-13T14:12:04Z

💔 Backport failed

Status	Branch	Result
❌	8.x	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 124361

alex-spies · 2025-03-13T14:36:16Z

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Questions ?

Please refer to the Backport tool documentation

To understand query performance, we often peruse the output of `_query`-requests run with `"profile": true`. This is difficult when the query runs in a large cluster with many nodes and shards, or in case of CCQ. This adds an option to visualize a query using Chromium's/Chrome's builtin `about:tracing` - or, for even better visuals and querying the different drivers via SQL, perfetto (c.f. https://ui.perfetto.dev/). To use, save the JSON output of a query run with `"profile": true` to a file, like `output.json` and then invoke the following Gradle task: ``` ./gradlew x-pack:plugin:esql:tools:parseProfile --args='~/output.json ~/parsed_profile.json' ``` Either open `about:tracing` in Chromium/Chrome ![image](https://github.com/user-attachments/assets/75e17ddf-f032-4aa1-bf3e-61b985b4e0b6) Or head over to https://ui.perfetto.dev (build locally in case of potentially sensitive data in the profille): ![image](https://github.com/user-attachments/assets/b3372b7d-fbec-45aa-a68c-b24e62a8c704) Every slice is a driver, the colors indicating the ratio of cpu time over total time. - In Perfetto, essentials like duration, cpu duration, timestamp and a few others can be queried via SQL - this allows e.g. querying for all drivers that spent more than 50% of their time waiting and other fun things. ![image](https://github.com/user-attachments/assets/4a0ab2ce-3585-4953-b2eb-71991777b3fa) - Details about a driver, esp. which operators it ran, are available when clicking the driver's slice. ![image](https://github.com/user-attachments/assets/e1c0b30d-0a31-468c-9ff4-27ca452716fc) (cherry picked from commit fc4d8d6) # Conflicts: # x-pack/plugin/esql/qa/server/single-node/src/javaRestTest/java/org/elasticsearch/xpack/esql/qa/single_node/RestEsqlIT.java

To understand query performance, we often peruse the output of `_query`-requests run with `"profile": true`. This is difficult when the query runs in a large cluster with many nodes and shards, or in case of CCQ. This adds an option to visualize a query using Chromium's/Chrome's builtin `about:tracing` - or, for even better visuals and querying the different drivers via SQL, perfetto (c.f. https://ui.perfetto.dev/). To use, save the JSON output of a query run with `"profile": true` to a file, like `output.json` and then invoke the following Gradle task: ``` ./gradlew x-pack:plugin:esql:tools:parseProfile --args='~/output.json ~/parsed_profile.json' ``` Either open `about:tracing` in Chromium/Chrome ![image](https://github.com/user-attachments/assets/75e17ddf-f032-4aa1-bf3e-61b985b4e0b6) Or head over to https://ui.perfetto.dev (build locally in case of potentially sensitive data in the profille): ![image](https://github.com/user-attachments/assets/b3372b7d-fbec-45aa-a68c-b24e62a8c704) Every slice is a driver, the colors indicating the ratio of cpu time over total time. - In Perfetto, essentials like duration, cpu duration, timestamp and a few others can be queried via SQL - this allows e.g. querying for all drivers that spent more than 50% of their time waiting and other fun things. ![image](https://github.com/user-attachments/assets/4a0ab2ce-3585-4953-b2eb-71991777b3fa) - Details about a driver, esp. which operators it ran, are available when clicking the driver's slice. ![image](https://github.com/user-attachments/assets/e1c0b30d-0a31-468c-9ff4-27ca452716fc)

* ESQL: Enable visualizing a query profile (#124361) To understand query performance, we often peruse the output of `_query`-requests run with `"profile": true`. This is difficult when the query runs in a large cluster with many nodes and shards, or in case of CCQ. This adds an option to visualize a query using Chromium's/Chrome's builtin `about:tracing` - or, for even better visuals and querying the different drivers via SQL, perfetto (c.f. https://ui.perfetto.dev/). To use, save the JSON output of a query run with `"profile": true` to a file, like `output.json` and then invoke the following Gradle task: ``` ./gradlew x-pack:plugin:esql:tools:parseProfile --args='~/output.json ~/parsed_profile.json' ``` Either open `about:tracing` in Chromium/Chrome ![image](https://github.com/user-attachments/assets/75e17ddf-f032-4aa1-bf3e-61b985b4e0b6) Or head over to https://ui.perfetto.dev (build locally in case of potentially sensitive data in the profille): ![image](https://github.com/user-attachments/assets/b3372b7d-fbec-45aa-a68c-b24e62a8c704) Every slice is a driver, the colors indicating the ratio of cpu time over total time. - In Perfetto, essentials like duration, cpu duration, timestamp and a few others can be queried via SQL - this allows e.g. querying for all drivers that spent more than 50% of their time waiting and other fun things. ![image](https://github.com/user-attachments/assets/4a0ab2ce-3585-4953-b2eb-71991777b3fa) - Details about a driver, esp. which operators it ran, are available when clicking the driver's slice. ![image](https://github.com/user-attachments/assets/e1c0b30d-0a31-468c-9ff4-27ca452716fc) (cherry picked from commit fc4d8d6) # Conflicts: # x-pack/plugin/esql/qa/server/single-node/src/javaRestTest/java/org/elasticsearch/xpack/esql/qa/single_node/RestEsqlIT.java * Account for missing driver descr., node, cluster

alex-spies added 10 commits March 7, 2025 17:50

Add profile parser

a384444

Invoke it like this: ./gradlew x-pack:plugin:esql:qa:testFixtures:parseProfile --args='~/elasticsearch/profile.json ~/elasticsearch/output.json' Then it can be imported into e.g. perfetto or into Chromes trace viewer (about:tracing)

Add more data to the traceEvents

5d6bb55

Turn into complete events and add cpu duration

a5ded31

Organize drivers by node and cluster

5852d95

Move profile parser into its own module

225d23d

Cleanup

700579c

Move to more appropriate package + add stub test

19c26e8

Start adding a rest test

e5a71ba

Fix missing driver description

55e66b2

Finish test

c4a0f75

alex-spies added >non-issue auto-backport Automatically create backport pull requests when merged :Analytics/ES|QL AKA ESQL v8.19.0 v9.1.0 labels Mar 7, 2025

alex-spies requested review from idegtiarenko and nik9000 March 7, 2025 17:24

elasticsearchmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Mar 7, 2025

[CI] Auto commit changes from spotless

83a8293

idegtiarenko reviewed Mar 10, 2025

View reviewed changes

idegtiarenko approved these changes Mar 10, 2025

View reviewed changes

alex-spies added 2 commits March 12, 2025 12:39

Use Jackson instead of XContentHelper

8458af5

Merge remote-tracking branch 'upstream/main' into profiling_parser

46f233d

alex-spies added the auto-merge-without-approval Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) label Mar 12, 2025

alex-spies added 3 commits March 13, 2025 10:32

Fix single-node test for Serverless

f8fa8e9

Merge remote-tracking branch 'upstream/main' into profiling_parser

1e91fd6

Make test also apply to multi-nodes for Serverless

d7b7fd0

elasticsearchmachine merged commit fc4d8d6 into elastic:main Mar 13, 2025
17 checks passed

alex-spies deleted the profiling_parser branch March 13, 2025 14:10

elasticsearchmachine added the backport pending label Mar 13, 2025

alex-spies mentioned this pull request Mar 13, 2025

[8.x] ESQL: Enable visualizing a query profile (#124361) #124759

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

ESQL: Enable visualizing a query profile #124361

ESQL: Enable visualizing a query profile #124361

Uh oh!

alex-spies commented Mar 7, 2025

Uh oh!

elasticsearchmachine commented Mar 7, 2025

Uh oh!

idegtiarenko Mar 10, 2025

Uh oh!

alex-spies Mar 10, 2025

Uh oh!

idegtiarenko Mar 10, 2025

Uh oh!

alex-spies Mar 12, 2025

Uh oh!

Uh oh!

elasticsearchmachine commented Mar 13, 2025

Uh oh!

alex-spies commented Mar 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ESQL: Enable visualizing a query profile #124361

ESQL: Enable visualizing a query profile #124361

Uh oh!

Conversation

alex-spies commented Mar 7, 2025

Uh oh!

elasticsearchmachine commented Mar 7, 2025

Uh oh!

idegtiarenko Mar 10, 2025

Choose a reason for hiding this comment

Uh oh!

alex-spies Mar 10, 2025

Choose a reason for hiding this comment

Uh oh!

idegtiarenko Mar 10, 2025

Choose a reason for hiding this comment

Uh oh!

alex-spies Mar 12, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elasticsearchmachine commented Mar 13, 2025

💔 Backport failed

Uh oh!

alex-spies commented Mar 13, 2025

💚 All backports created successfully

Questions ?

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants