Low Real-Time Factor Running add_sensors_on_robot.py demo on EC2 g5.8xlarge

[Amin173]

Hello,

I have been benchmarking the performance of one of the IsaacLab standalone demos—specifically, source/standalone/tutorials/04_sensors/add_sensors_on_robot.py. Running this on a g5.8xlarge EC2 instance with a single robot environment, I observed a real-time factor of ~0.2, which is much lower than expected. I'm curious if there are configuration steps I might have overlooked on my instance that could be impacting the performance. Any insights from those with experience in similar setups would be appreciated.

-------------------------------
Simulation step time: 0.005[s]
Loop execution time (running average): 0.021250667256047216[s]
Execution time to sim time step ratio (running average): 4.250133451209443
Real time ratio (running average): 0.23528672957679342
GPU Utilization: 64%
GPU Memory Usage: 10854/23028 MB
CPU Usage: 1.8%
Memory Usage: 17.8%
Total Memory: 124.46 GB
Used Memory: 19.07 GB
Available Memory: 102.26 GB
/isaac-sim/exts/omni.isaac.ml_archive/pip_prebundle/torch/nn/modules/module.py:1520: UserWarning: RNN module weights are not part of single contiguous chunk of memory. This means they need to be compacted at every call, possibly greatly increasing memory usage. To compact weights again call flatten_parameters(). (Triggered internally at ../aten/src/ATen/native/cudnn/RNN.cpp:982.)
  return forward_call(*args, **kwargs)
-------------------------------
Camera @ '/World/envs/env_.*/Robot/base/front_cam': 
	data types   : ['distance_to_image_plane', 'rgb'] 
	semantic filter : *:*
	colorize semantic segm.   : True
	colorize instance segm.   : True
	colorize instance id segm.: True
	update period (s): 0.2
	shape        : (480, 640)
	number of sensors : 1
Received shape of rgb   image:  torch.Size([1, 480, 640, 4])
Received shape of depth image:  torch.Size([1, 480, 640])
-------------------------------
Ray-caster @ '/World/envs/env_.*/Robot/base': 
	view type            : <class 'omni.physics.tensors.impl.api.ArticulationView'>
	update period (s)    : 0.02
	number of meshes     : 1
	number of sensors    : 1
	number of rays/sensor: 187
	total number of rays : 187
Received max height value:  3.814697265625e-06
-------------------------------
Contact sensor @ '/World/envs/env_.*/Robot/.*_FOOT': 
	view type         : <class 'omni.physics.tensors.impl.api.RigidBodyView'>
	update period (s) : 0.0
	number of bodies  : 4
	body names        : ['LF_FOOT', 'LH_FOOT', 'RF_FOOT', 'RH_FOOT']

Received max contact force of:  132.93650817871094
-------------------------------

I’ve attached the test script I’m using. It’s based on the standalone demo mentioned above, with a few additional lines to measure and display performance metrics. I’ve also attached the nvidia-smi output for further context.

One potential issue I’m aware of is that the ECC setting on the GPU driver is enabled and I've had some issues with disabling it. I’m not certain how much performance degradation I should expect that.

Attached is the full server configuration.

nvidia-smi:

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 555.58.02              Driver Version: 555.58.02      CUDA Version: 12.5     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA A10G                    Off |   00000000:00:1E.0 Off |                    0 |
|  0%   34C    P0             83W /  300W |   10839MiB /  23028MiB |     63%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A      1420      G   /usr/lib/xorg/Xorg                             79MiB |
|    0   N/A  N/A      3895      G   /usr/lib/xorg/Xorg                            212MiB |
|    0   N/A  N/A      4050      G   /usr/bin/gnome-shell                          145MiB |
|    0   N/A  N/A      4379    C+G   /usr/lib/x86_64-linux-gnu/dcv/dcvagent        985MiB |
|    0   N/A  N/A      4562      G   ...ures=SpareRendererForSitePerProcess         27MiB |
|    0   N/A  N/A      4805      G   /usr/bin/nvidia-settings                        0MiB |
|    0   N/A  N/A      4880      G   ...erProcess --variations-seed-version        143MiB |
|    0   N/A  N/A     13751    C+G   ...b/_isaac_sim/kit/python/bin/python3       4516MiB |
|    0   N/A  N/A     24061    C+G   ...b/_isaac_sim/kit/python/bin/python3       4548MiB |
+-----------------------------------------------------------------------------------------+

Server Config:
g5.8xlarge

Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Byte Order:                           Little Endian
Address sizes:                        48 bits physical, 48 bits virtual
CPU(s):                               32
On-line CPU(s) list:                  0-31
Thread(s) per core:                   2
Core(s) per socket:                   16
Socket(s):                            1
NUMA node(s):                         1
Vendor ID:                            AuthenticAMD
CPU family:                           23
Model:                                49
Model name:                           AMD EPYC 7R32
Stepping:                             0
CPU MHz:                              2800.000
BogoMIPS:                             5600.00
Hypervisor vendor:                    KVM
Virtualization type:                  full
L1d cache:                            512 KiB
L1i cache:                            512 KiB
L2 cache:                             8 MiB
L3 cache:                             64 MiB

                
             total          used        free      shared  buff/cache   available
Mem:          124Gi        19Gi        98Gi       1.9Gi       7.2Gi       102Gi

add_sensors_on_robot standalone demo

[Dhoeller19]

Hi @Amin173,
we have new scripts for benchmarking: https://isaac-sim.github.io/IsaacLab/source/overview/reinforcement-learning/performance_benchmarks.html#benchmark-scripts.
Could you try running Isaac-Cartpole-RGB-Camera-Direct-v0 and compare with what is reported on the tables?