[rocprofiler-sdk] Enable region argument annotation in perfetto output trace by mcao59 · Pull Request #3854 · ROCm/rocm-systems

mcao59 · 2026-03-07T04:12:56Z

Motivation

This PR enables function argument annotation in Perfetto with --annotate-args.
Without info like event ID arg value, it may be hard to follow event dependencies and debug HIP event synchronization.
The goal is disambiguating waits across multiple events/streams: when a stream waits on different events originating from different streams, having the event ID allows them to correlate hipStreamWaitEvent with the corresponding hipEventRecord calls and identify which event (and thus which upstream stream/work) is actually blocking execution.

With this PR: When --annotate-args (introduced in #340) is used in rocpd to perfetto conversion, the function arguments get annotated

Technical Details

Enabling arg annotation in perfetto.cpp:
Tests
- added separate execution and validation tests for perfetto output converted with --annotate-args
- Test checks for annotations. Test will fail if expected annotations are not found

JIRA ID

Resolves AIPROFSDK-195

Test Plan

Local testing:

$ pytest -s validate_annotations.py   --pftrace-input=rocpd-output-data-annotations/out_results.pftrace 
===== test session starts =====
platform linux -- Python 3.10.12, pytest-9.0.2, pluggy-1.6.0
rootdir: /home/mcao/my_proj/rocm-systems/projects/rocprofiler-sdk/tests/rocprofv3/rocpd
configfile: pytest.ini
plugins: metadata-3.1.1, json-report-1.5.0, allure-pytest-2.15.3, rerunfailures-16.1
collected 1 item                                                                                                                                                                                                                                                     

../../../../projects/rocprofiler-sdk/tests/rocprofv3/rocpd/validate_annotations.py 
Validation passed: Found 14019 argument annotations
  - 11 unique API calls annotated
  - 1 categories: ['hip_api']
  - 12 unique argument types
  - Sample argument names: ['debug.count', 'debug.deviceId', 'debug.stream', 'debug.event', 'debug.function_address', 'debug.numBlocks', 'debug.dimBlocks', 'debug.args', 'debug.sharedMemBytes', 'debug.ms']
.

====== PASSES =====
 ======slowest 20 durations =========
3.25s setup    validate_annotations.py::test_arg_annotations
2.96s call     validate_annotations.py::test_arg_annotations

(1 durations < 0.005s hidden.  Use -vv to show these durations.)
===== short test summary info =====
PASSED ../../../../projects/rocprofiler-sdk/tests/rocprofv3/rocpd/validate_annotations.py::test_arg_annotations
====== 1 passed in 6.21s =====

Test Result

Submission Checklist

Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

yhuiYH · 2026-03-12T20:28:25Z

projects/rocprofiler-sdk/tests/rocprofv3/rocpd/CMakeLists.txt

+rocprofiler_add_integration_execute_test(
+    rocprofv3-test-rocpd-perfetto-generation-annotations
+    COMMAND
+        ${Python3_EXECUTABLE} -m rocpd convert -f pftrace --kernel-rename --annotate-args
+        -d ${CMAKE_CURRENT_BINARY_DIR}/rocpd-output-data-annotations -i
+        ${CMAKE_CURRENT_BINARY_DIR}/rocpd-input-data/out_results.db
+    DEPENDS rocprofiler-sdk::rocprofv3
+    TIMEOUT 120
+    LABELS "integration-tests;rocpd"
+    PRELOAD "${ROCPROFILER_MEMCHECK_PRELOAD_ENV_VALUE}"
+    ENVIRONMENT "${rocprofv3-rocpd-env}"
+    FIXTURES_SETUP rocprofv3-test-rocpd-generation-annotations
+    FIXTURES_REQUIRED rocprofv3-test-rocpd)
+


This looks like just a duplication of the prior execution_test, with the added --annotate-args param. Do you want to just add --annotate-args to the original test, and call your validate_annotations.py using that file?

I was thinking of keeping this separate because --annotate-args is optional mode rather than the default path. Also, if we add more options for rocpd convert and more tests in future, it may be cleaner to keep separate execute tests so it’s easier to debug issues. But I do see your point about avoiding duplications. I’m fine with whichever direction we want to standardize on.

OK, my thought is that your validation is going to check the annotations later anyway. So we can save 1 test & time by just lumping the 2 generation tests into 1. Just thought I'd call it out, I'm ok either way on this one too.

Yeah, we should minimize the generation tests.

Wait disregard the previous comment. I thought this was running an app on the GPU

projects/rocprofiler-sdk/tests/pytest-packages/tests/rocprofv3.py

projects/rocprofiler-sdk/source/lib/python/rocpd/source/perfetto.cpp

yhuiYH

LGTM. CI tests are queued because the pool has been re-purposed. But it was passing before and I checked the tests locally, seems to be passing.

jrmadsen

Why not just annotate all the rocpd_args associated with the event_id? Wouldn't this include the event handle?

mcao59 · 2026-03-17T22:30:49Z

Why not just annotate all the rocpd_args associated with the event_id? Wouldn't this include the event handle?

@jrmadsen sure, please see enable arg annotation for all instead of only event ID. Updated tests as well to validate --annotate-args

jrmadsen · 2026-03-19T17:42:37Z

projects/rocprofiler-sdk/tests/pytest-packages/tests/rocprofv3.py

            _perform_csv_json_match(a, b, keys_mapping[category], json_data)
+
+
+def test_perfetto_arg_annotations(pftrace_data, pftrace_filename):


Isn't pftrace_data the PerfettoReader of pftrace_filename?

Please see cleanup: remove unnecessary PerfettoReader call

jrmadsen · 2026-03-19T17:45:50Z

projects/rocprofiler-sdk/tests/rocprofv3/rocpd/CMakeLists.txt

+rocprofiler_add_integration_execute_test(
+    rocprofv3-test-rocpd-perfetto-generation-annotations
+    COMMAND
+        ${Python3_EXECUTABLE} -m rocpd convert -f pftrace --kernel-rename --annotate-args
+        -d ${CMAKE_CURRENT_BINARY_DIR}/rocpd-output-data-annotations -i
+        ${CMAKE_CURRENT_BINARY_DIR}/rocpd-input-data/out_results.db
+    DEPENDS rocprofiler-sdk::rocprofv3
+    TIMEOUT 120
+    LABELS "integration-tests;rocpd"
+    PRELOAD "${ROCPROFILER_MEMCHECK_PRELOAD_ENV_VALUE}"
+    ENVIRONMENT "${rocprofv3-rocpd-env}"
+    FIXTURES_SETUP rocprofv3-test-rocpd-generation-annotations
+    FIXTURES_REQUIRED rocprofv3-test-rocpd)
+


Yeah, we should minimize the generation tests.

jrmadsen · 2026-03-19T17:51:09Z

projects/rocprofiler-sdk/tests/rocprofv3/rocpd/validate_annotations.py

+import pytest
+
+
+def test_arg_annotations(pftrace_data, request):


Suggested change

def test_arg_annotations(pftrace_data, request):

def test_arg_annotations(pftrace_reader):

Just make a fixture that drops .read()[0] from pftrace_data so that we aren't creating two readers of the same file.

Sure, done in cleanup: remove unnecessary PerfettoReader call

mcao59 requested review from a team as code owners March 7, 2026 04:12

github-actions bot added the project: rocprofiler-sdk label Mar 7, 2026

mcao59 added organization: ROCm WIP labels Mar 7, 2026

mcao59 force-pushed the users/mcao/perfetto_event_id branch 2 times, most recently from fd77ad6 to bb0e3fe Compare March 11, 2026 23:36

mcao59 added rocprofiler-sdk: Ready for 1st code review and removed WIP labels Mar 12, 2026

mcao59 force-pushed the users/mcao/perfetto_event_id branch from bb0e3fe to 88e23bf Compare March 12, 2026 20:10

yhuiYH reviewed Mar 12, 2026

View reviewed changes

mcao59 force-pushed the users/mcao/perfetto_event_id branch from 88e23bf to 239c4c9 Compare March 16, 2026 21:46

yhuiYH approved these changes Mar 17, 2026

View reviewed changes

yhuiYH added ready for 2nd review rocprofiler-sdk: Ready for 2nd code review and removed rocprofiler-sdk: Ready for 1st code review labels Mar 17, 2026

mcao59 changed the title ~~[rocprofiler-sdk] Add Event ID in perfetto output trace~~ [rocprofiler-sdk] Add Event ID in perfetto output trace for HIP event APIs Mar 17, 2026

jrmadsen reviewed Mar 17, 2026

View reviewed changes

mcao59 force-pushed the users/mcao/perfetto_event_id branch from 4a2cee2 to 8137764 Compare March 17, 2026 21:56

mcao59 changed the title ~~[rocprofiler-sdk] Add Event ID in perfetto output trace for HIP event APIs~~ [rocprofiler-sdk] Enable argument annotation in perfetto output trace Mar 17, 2026

mcao59 changed the title ~~[rocprofiler-sdk] Enable argument annotation in perfetto output trace~~ [rocprofiler-sdk] Enable region argument annotation in perfetto output trace Mar 17, 2026

mcao59 force-pushed the users/mcao/perfetto_event_id branch from 8137764 to 1fe0a46 Compare March 17, 2026 23:29

mcao59 added 7 commits March 18, 2026 10:33

add event id in rocpd to perfetto conversion

553a5b7

run clang-format

3b5ae5c

enable tests for HIP event API argument annotations

478008b

run formatters

fb5466c

move annotation test (w/ --annotate-args) separate from original test

f058079

improve var naming

475d411

clean up rocprofv3.py

688dc9c

enable arg annotation for all instead of only event ID

ca1ea6e

mcao59 force-pushed the users/mcao/perfetto_event_id branch from 1fe0a46 to ca1ea6e Compare March 18, 2026 15:33

mcao59 mentioned this pull request Mar 19, 2026

[rocprofiler-sdk] Fix categories filtering in rocpd analysis tool #4161

Open

1 task

jrmadsen requested changes Mar 19, 2026

View reviewed changes

cleanup: remove unnecessary PerfettoReader call

5299811

		_perform_csv_json_match(a, b, keys_mapping[category], json_data)


		def test_perfetto_arg_annotations(pftrace_data, pftrace_filename):

		import pytest


		def test_arg_annotations(pftrace_data, request):

	def test_arg_annotations(pftrace_data, request):
	def test_arg_annotations(pftrace_reader):

Conversation

mcao59 commented Mar 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Technical Details

JIRA ID

Test Plan

Test Result

Submission Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

yhuiYH left a comment

Choose a reason for hiding this comment

Uh oh!

jrmadsen left a comment

Choose a reason for hiding this comment

Uh oh!

mcao59 commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mcao59 Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mcao59 commented Mar 7, 2026 •

edited

Loading

mcao59 commented Mar 17, 2026 •

edited

Loading

mcao59 Mar 20, 2026 •

edited

Loading