[Feature request] Make it easier to show node assignments

[justinchuby]

Make it easier to show node assignments for the official python build. Right now it seems a debug build is required, which is very cumbersome. For the python distribution I think we are more lenient on wheel size and should include more profiling capabilities to boost productivity.

Additionally, it would be good to have a way to produce an artifact that records node assignment, beyond the log messages that we have now, to enable better benchmarking and performance analysis.

@tianleiwu @devang-ml

[tianleiwu]

It's easy to build a package to do that. Just append '--cmake_extra_defines onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS=ON' to the build.sh or build.py command line.

Then set environment variable export ORT_DEBUG_NODE_IO_DUMP_NODE_PLACEMENT=1 before running your model. The node placement will print to stdout like:

Placement: CUDAExecutionProvider
-----------
GroupQueryAttention node: GroupQueryAttention_0
Input 0 Name: query
 Shape: {1,3,4096}
OrtMemoryInfo:[name:Cuda OrtMemType:0 OrtAllocatorType:0 Device:[DeviceType:1 MemoryType:0 VendorId:4318 DeviceId:0 Alignment:0]]
-0.018493652, -0.0085067749, -0.052886963, 0.0029029846, -0.0024166107, ...

This is not in official package. Maybe we can enable this build flag in nightly package to help developers. Need to change the default to not print debug information unless explicitly turned on.

For benchmarking and performance analysis, I can see that ORT has gap there that we can improve.

[justinchuby]

I am inclined to propose that we measure the overhead and find ways to integrate the features into the official builds. Building ort, especially with cuda and other eps (webgpu, etc) and figuring out all the options to use is may not most straight forward thing or the ideal dev experience imo.