Actuator Neural Network using MLP/LSTM

[Asura66]

We are implementing the approach outlined in the paper 'Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning' and have designed an actuator neural network using LSTM. Our input parameters are position error and velocity, and the outputs are torques. While we successfully trained the neural network, when we replace the original actuator neural network in the simulation with our NN_model, the robot fails to receive any joint commands. Any insights or suggestions on what might be causing this issue would be greatly appreciated!

[dxyy1]

when you say failed to receive any joint commands, do you mean there is no output at all or that the outputs are gibberish?

If it is the former then I'd think your implementation perhaps incorrectly implemented something. If it's the latter, the thing I'd look out for when I played around with acutator_net was that the ordering of the joints of the actuator_net.

[Asura66]

When I begin training my robot, the actuator model fails to produce any output, causing the robot to collapse to the ground and terminate the simulation. During training, I provided position error and velocity as inputs, expecting the model to predict the corresponding torques. However, when I replace the default actuator model with my trained one, the robot falls immediately. Could the issue be related to the architecture of my model or the choice of inputs? I would appreciate a more detailed description of the training procedure and any suggestions for improving the model to ensure better performance.

[Asura66]

I trained an LSTM-based actuator model to predict joint torques for a robot in Isaac Sim. The model takes position error and velocity as inputs and outputs torques for all joints as a single tensor. I normalized the data and used an LSTM with 2 input features, 8 hidden units, and outputs matching the number of joints. During training, I initialized the hidden and cell states (h0, c0) as zeros and ensured the output shape was [batch_size, num_joints]. when i used anymal actuator i got data logged in my terminal as

ANYmal actuator model log of 1 iteration:
ActuatorNetLSTM - Inputs:
Position Error: tensor([ 0.6764, 1.1550, 0.9174, -0.2560, -0.9965, 0.4943, -0.2204, 0.4052,
1.7233, 0.1542, -0.2298, -0.2773], device='cuda:0')
Velocity: tensor([ 0.4099, 3.8640, 10.9939, 4.2951, -0.6245, -5.7991, 1.5379, 8.8142,
-1.5825, 9.3693, 0.7480, -1.5918], device='cuda:0')
ActuatorNetLSTM - Outputs:
Predicted Torque: tensor([ 26.5869, 66.6058, -9.7557, -26.2441, -18.5697, 47.7313, -17.4783,
4.2502, 80.6506, -28.1040, -9.1082, 6.1764], device='cuda:0')
joint effort: tensor([[ 26.5869, 58.1765, -9.7557, -26.2441, -18.5697, 47.7313, -17.4783,
0.0000, 80.0000, -28.1040, -9.1082, 6.1764]], device='cuda:0')

My own LSTM actuator model output :
ActuatorNetLSTM - Inputs:
Position Error: tensor([-0.1499, -0.7921, 0.4945, -1.1634, 0.3670, 0.6246, -0.3349, -1.0253,
-0.2994, -2.3928, -1.7586, -1.2150], device='cuda:0')
Velocity: tensor([-4.2830e+00, -3.2080e-06, 4.1087e-02, 1.9328e-01, 5.0635e+00,
-4.9914e+00, 4.1166e+00, 3.8100e+00, 7.4311e+00, 1.1551e+01,
7.5290e+00, 7.4712e+00], device='cuda:0')
ActuatorNetLSTM - Outputs:
Predicted Torque: tensor([-1.0272, 1.3006, 1.2277, 1.2501, 0.6426, 0.6476, 0.5881, 0.5821,
0.5573, 0.6293, 0.5659, 0.5549], device='cuda:0')
joint effort: tensor([[-1.0272, 1.3006, 1.2277, 1.2501, 0.6426, 0.6476, 0.5881, 0.5821,
0.5573, 0.0000, 0.0000, 0.4602]], device='cuda:0')