[Bug]: Undefined symbol: cuProfilerStart on Jetson AGX Orin

[ken4647]

I find some applications will failed with lack of symbol when I tested it in Jetson platform. Most of them(written in python) failed just when importing packages.

But note that the examples in the project work well. The version of jetpack is 6.2 in my machine, which has these symbols indeed.

Thanks in advance for you help! I will also attempt to analyze the build script for libshimcuda.so (where I think the promble might be here).

[ken4647]

I found that the issue appears to be related to the CUDA version—Jetson likely includes some platform-specific libraries, such as cudaEGL.h. I also noticed the README.md file in the platforms/example directory, which provides guidance on adapting the codebase to new architectures. (Note: the documentation contains a minor error—in Section 2, the command for merge_headers.py seems to require a -d flag.)
Following this guide, I generated the necessary symbol redirections and updated the corresponding entries in xsched/platforms/cuda/shim/src/intercept.cpp and xsched/cuda/hal/common/driver.h. After recompiling, I was able to successfully load PyCUDA, OpenCV (cv2), and TensorRT. This confirms that the scheduling mechanism is now functioning correctly.

[ken4647]

However, there are still a few minor issues in practice. For instance, cudaMallocAsync consistently fails with an "out-of-memory" error in TensorRT scenarios. Additionally, it is essential to ensure that the application explicitly uses CUDA streams; otherwise, execution falls back to xsched::cuda::DirectLaunch. In that code path, there is a conditional check based on the GPU architecture. Since I am using a Jetson AGX Orin (compute capability SM 8.7), adding a corresponding entry for sm87 at level 1 resolves the issue.

[Naplesoul]

Hi @ken4647,
Thank you for the detailed report and for digging into the codebase! Your analysis and the manual fixes are absolutely correct.

We really appreciate your contribution to making XSched work on the Jetson platform. Based on your feedback, we will prioritize the following updates in the next few days:

• Symbol Interception: We will add the symbol redirections for cuProfiler and cuEGL interfaces to ensure out-of-the-box compatibility for Jetson/JetPack users.

• DirectLaunch Fallback: We will let new SM version to fallback to Lv1 by default so that it doesn't fail silently or require manual architecture entries for every new SM version.

• Documentation: We will fix the -d flag in the merge_headers.py instructions.

About the TensorRT OOM issue: We previously conducted tests on JetPack 5.1.4 and did not encounter OOM errors with cudaMallocAsync. To help us investigate and reproduce this on JetPack 6.2, could you provide more details? Specifically, a minimal reproducible example (code) and the model you are using would be helpful.

We really appreciate your contribution to making XSched better!

[Naplesoul]

Hi @ken4647,

Please check PR #20 to see if it resolves your symbol and DirectLaunch issues. BTW, I notice that your generated intercept.cpp treats CUresult as int, and EGL types as void *. This may because you didn't pass -i arguments to gen.py, so that clang does not know where to find these EGL symbols. I recommend to use our updated driver.h and intercept.cpp to prevent potential bugs.

[ken4647]

Hi @ken4647,

Please check PR #20 to see if it resolves your symbol and DirectLaunch issues. BTW, I notice that your generated intercept.cpp treats CUresult as int, and EGL types as void *. This may because you didn't pass -i arguments to gen.py, so that clang does not know where to find these EGL symbols. I recommend to use our updated driver.h and intercept.cpp to prevent potential bugs.

Thank you for your reply, and I sincerely apologize for my delayed response!

I’ve re-cloned the latest version of the code and redeployed it. I can now confirm that all the issues I previously encountered have been resolved.

However, I’m still running into the TensorRT out-of-memory (OOM) error. Attached is a minimal reproducible code snippet(https://drive.google.com/file/d/1gqqJ_JePX4OSHxPVYpHLH1rKMk2yXC1E/view?usp=drive_link): it’s an extremely lightweight MLP-based policy model (only 0.6 million parameters—so small that it’s arguably better suited for CPU deployment) used for bipedal robot motion control. When running this on an Orin AGX with 64GB RAM, I encounter the following error:

[Naplesoul]

Hi, the code seems to be a private google drive link. I have requested access to it, please check it out. Great thanks to your response!

[ken4647]

Hi, the code seems to be a private google drive link. I have requested access to it, please check it out. Great thanks to your response!

I forgot to update the permissions—should be accessible now.

[Naplesoul]

Hi @ken4647,

We are sorry that we were unable to reproduce the OOM issue on our Orin. Our Orin AGX has 32GB of memory and is installed with JetPack 6.2.1 (REV. 1). For testing, we used the system TensorRT included in JetPack, version 10.3.0. The results after running the script are as follows:

# convert onnx to engine, since we have different TensorRT versions
/usr/src/tensorrt/bin/trtexec --onnx=./policy.onnx --saveEngine=./rl_policy.engine
...

# in another shell
./output/bin/xserver 
[INFO @ T24932 @ 13:48:08.409295] using default: output/bin/xserver HPF 50000
[INFO @ T24932 @ 13:48:08.409521] scheduler created with policy HPF
[INFO @ T24932 @ 13:48:08.409710] pidfd_open is supported, using pidfd_wait method
[INFO @ T24932 @ 13:48:08.410200] events receiver started
[INFO @ T24935 @ 13:48:08.410227] HTTP server listening on port 50000
[INFO @ T24932 @ 13:48:50.356145] client process 25171 connected
[INFO @ T24933 @ 13:48:50.743939] XQueue (0x6253000033dc73f0) from process 25171 created
[INFO @ T24933 @ 13:48:50.744033] set priority 4 for XQueue 0x6253000033dc73f0
[INFO @ T24933 @ 13:48:50.745760] XQueue (0x6253000033dd5e20) from process 25171 created
[INFO @ T24933 @ 13:48:50.745927] set priority 4 for XQueue 0x6253000033dd5e20
[INFO @ T24933 @ 13:48:50.745947] XQueue (0x6253000033dd5e20) from process 25171 destroyed
[INFO @ T24933 @ 13:48:50.746221] XQueue (0x6253000033dc73f0) from process 25171 destroyed
[INFO @ T24933 @ 13:48:50.746878] XQueue (0x6253000033dc73f0) from process 25171 created
[INFO @ T24933 @ 13:48:50.746896] set priority 4 for XQueue 0x6253000033dc73f0
[INFO @ T24933 @ 13:48:50.751588] XQueue (0x6253000033df8390) from process 25171 created
[INFO @ T24933 @ 13:48:50.751635] set priority 4 for XQueue 0x6253000033df8390
[INFO @ T24933 @ 13:48:50.775333] XQueue (0x6253000033e6d590) from process 25171 created
[INFO @ T24933 @ 13:48:50.775396] set priority 4 for XQueue 0x6253000033e6d590

# in the first shell
XSCHED_PATH=/home/xxx/ bash ./xsched_policy.sh 
[INFO @ T25171 @ 13:48:50.355614] using global scheduler
[01/30/2026-13:48:50] [TRT] [I] Loaded engine size: 1 MiB
[01/30/2026-13:48:50] [TRT] [I] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +0, now: CPU 0, GPU 1 (MiB)
Starting inference loop with explicit CUDA stream...
[Step 1000] Mean: 0.31 ms, Var: 0.0083 ms², P99: 0.71 ms
[Step 2000] Mean: 0.30 ms, Var: 0.0016 ms², P99: 0.40 ms
[Step 3000] Mean: 0.29 ms, Var: 0.0022 ms², P99: 0.39 ms
...

We think the script is running normally on our machine.

We noticed in the logs you provided: "[W] [TRT] Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors." You might be using a TensorRT plan that was generated and optimized for another machine, which could lead to issues like incorrect workspace size. It is recommended to use /usr/src/tensorrt/bin/trtexec to regenerate the plan.

At the same time, it is suggested to compile xsched using the system gcc in the system environment (not in a virtual environment, e.g., conda). We know that libc++ compatibility can sometimes lead to unexpected errors.

Although our environment is slightly different from yours (Jetpack 6.2.1 vs. 6.2), we don’t think this is the cause of the problem. If the issue persists after you have tried the above, we may need to downgrade our machine. Given that it is currently the holiday season, this may take longer.