[Question] Dealing with out of GPU memory errors and closing the process automatically

[StoneT2000]

Question

I tried increasing the number of envs and camera width/height to large values to see what the limit I can hit given my GPU memory that is free. However in trying to test this I will sometimes get

2024-06-05 01:29:43 [8,172ms] [Error] [carb.graphics-vulkan.plugin] Out of GPU memory allocating resource 'MemoryManager-allocate' [size: unknown]
2024-06-05 01:29:43 [8,172ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_OUT_OF_DEVICE_MEMORY
2024-06-05 01:29:43 [8,172ms] [Error] [carb.graphics-vulkan.plugin] vkAllocateMemory failed for flags: 2.
2024-06-05 01:29:43 [8,172ms] [Error] [gpu.foundation.plugin] Unable to allocate buffer
2024-06-05 01:29:43 [8,172ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-06-05 01:29:43 [8,172ms] [Error] [rtx.rtxsensor.plugin] RTXSensor Buffer Allocation: Allocation failed for buffer RTXSensor Returns.shadingNormal. Assuming device was lost.
2024-06-05 01:29:43 [8,172ms] [Error] [gpu.foundation.plugin] A GPU crash occurred. Exiting the application...
Reasons for the failure: a device lost, out of memory, or an unexpected bug.

While I understand the error (out of memory), the process does not automatically close, and I cannot call .close on the simulation app because no error is raised somehow? Any way to make the process close? Thanks!

[StoneT2000]

This is no longer an issue on the latest release v1.2.0

[sapan-ostic]

I can encountered the same issue with v1.2.0.
Error:

2024-11-21 01:43:07 [15,837ms] [Error] [carb.graphics-vulkan.plugin] Out of GPU memory allocating resource 'Foundation Resource Allocator' [size: unknown]
2024-11-21 01:43:07 [15,837ms] [Error] [carb.graphics-vulkan.plugin] Failure injector rule to repro:
{
    debugName="Foundation Resource Allocator",
}
2024-11-21 01:43:07 [15,837ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_OUT_OF_DEVICE_MEMORY
2024-11-21 01:43:07 [15,837ms] [Error] [carb.graphics-vulkan.plugin] vkAllocateMemory failed for flags: 0.
2024-11-21 01:43:07 [15,837ms] [Error] [gpu.foundation.plugin] Failed to allocate memory for resource allocator Foundation Resource Allocator
2024-11-21 01:43:07 [15,837ms] [Error] [gpu.foundation.plugin] Failed to create a heap for pool Foundation Resource Allocator memory pool
2024-11-21 01:43:07 [15,837ms] [Error] [gpu.foundation.plugin] Releasing memory that isn't owned by the sub-allocator Foundation Resource Allocator

I am trying to run one of the tutorial workflow:

./isaaclab.sh -p source/standalone/workflows/skrl/train.py --task Isaac-Reach-Franka-v0 --video --num_envs=1