Training/simulation crashing, possibly due to NaN values

[jackzeng-robotics]

Hi developers,

I am working on a setup where multiple drones carry an object together with strings. The strings are modeled as 7 thin links with ball joints between them. The ball joints are modeled as 3 continuous joints in x, y and z direction. It looks like this:

The links have a very low mass and inertia and before I thought this could create some stability issues. What I observed: given big forces and torques, the setup makes very wild movements and then the setup disappears from the simulation. Probably due to some NaN values. This also caused my training to crash. My hypothesis before was that the cable links have very low mass and inertia causing them to fly away when under high loads, for example when compressed. So, I added numerous termination terms that stopped this from happening, for example limiting the angle between the drone and the links, setting terminations on high velocity values, angular rates etc.

This pretty much solved the issue for a long time until I implemented a low level controller for the drones to go with it. The behaviour of the low level controller seems fine, similar to what we get in real life. Without the RL policy is does not exhibit any strange behaviour, moreover the actions (forces on the drone rotors) are clamped as well. Now, the error has come back and I get this error when training with SKRL:

[Error] [omni.physx.plugin] PhysX error: The application needs to increase PxGpuDynamicsMemoryConfig::foundLostAggregatePairsCapacity to -1423966208, otherwise, the simulation will miss interactions, FILE /builds/omniverse/physics/physx/source/gpubroadphase/src/PxgAABBManager.cpp, LINE 1269

And this when using RSL-RL:
File "/home/isaac-sim/.local/share/ov/pkg/isaac-sim-4.2.0/exts/omni.isaac.ml_archive/pip_prebundle/torch/distributions/normal.py", line 71, in sample
return torch.normal(self.loc.expand(shape), self.scale.expand(shape))
RuntimeError: normal expects all elements of std >= 0.0

Is there any way to figure out where the crash comes from? Like finding the last state of the drones in some log somewhere? Or do you have any idea on what it could be?

Thanks in advance!

Jack

[StrainFlow]

Have you reported this in the Issues section? That would be the better place to post a crash bug like this.

In order for them to solve this they'll need to be able to reproduce the issue, so the more info you have ready to provide the better.