Compute action for humanoid env based on mocap data

[jabrantley]

Hi,

I am trying to implement behavioral cloning using the humanoid and mocap data. I know it is not as sophisticated as DeepMimic but its a simpler starting point. I have been playing with lots of the humanoid examples and I am struggling to figure out how I can compute values for the action space. I played with several examples in bullet3/examples/pybullet/gym/

• ../pybullet_envs/deep_mimic/env/testHumanoid.py
• ../pybullet_envs/deep_mimic/mocap/ { inverse_kinematics.py , render_reference.py, etc.} (location of mocap data i've been playing with)
• ../pybullet_examples/humanoidMotionCapture.py

I have also been using the humanoid from: gym_env.DeepMimicHumanoidGymEnv:DeepMimicHumanoidGymEnv

I know how to define networks with the correct dimensions for action_space and observation_space if I want to train from scratch or use the DeepMimic approach. For behavioral cloning, I need (state, action) pairs that I can use as expert data. My issue is that the action space is only (17, ), but everything being passed to the joints seems to be in the observation_space dimensions. I tried using the output of the various PD calculations, such as taus = stablePD.computePD(...), but this is also based on a larger joint space.

Do I need to use something like calculateInverseDynamics to compute the joint torques? Or, is the action_space (17, ) just a subset of the joints and there is a simple way to do it?

I am still quite new to pybullet so I am still working through everything.

[jabrantley]

Just an update on this.

I have been working off of ../pybullet_envs/deep_mimic/mocap/render_reference.py as my base code. I edited Humanoid.ApplyPose to include a PD controller from PDControllerStableMultiDof(bc) (from pybullet_examples.pdControllerStable import PDControllerStableMultiDof). I then get the torque values using

taus = stablePD.computePD(bodyUniqueId=humanoid,
                                jointIndices=jointIndicesAll,
                                desiredPositions=pose_list,
                                desiredVelocities=[0] * totalDofs,
                                kps=kpOrg,
                                kds=kdOrg,
                                maxForces=[maxForce] * totalDofs,
                                timeStep=timeStep)

I then saved these values along with the states for the walking data in the main render_reference.py code in the call to humanoid.RenderReference. My code looks like this starting around line 100.

humanoid = Humanoid(bc, motion, [0, 0, 0]) 

print(p.getBasePositionAndOrientation(humanoid._humanoid))

simTime = 0
keyFrameDuration = motion.KeyFrameDuraction()
print("keyFrameDuration=", keyFrameDuration)

all_torque  = []     #<---- MY EDIT
all_state = []       #<---- MY EDIT
all_nextstate = []   #<---- MY EDIT

for utNum in range(motion.NumFrames()):
  bc.stepSimulation()
  torque, frameData, frameDataNext = humanoid.RenderReference(utNum * keyFrameDuration) #<---- MY EDIT TO INCLUDE OUTPUT OF RenderReference-->ApplyPose
  all_torque.append([torque])            #<---- MY EDIT
  all_state.append([frameData]).         #<---- MY EDIT
  all_nextstate.append([frameDataNext])  #<---- MY EDIT
  
   if draw_gt:
       draw_ground_truth(coord_seq=ground_truth,
                      frame=utNum,
                      duration=keyFrameDuration,
                      shift=[-1.0, 0.0, 1.0])
  time.sleep(0.001)
stage = 0

def Reset(humanoid):
  global simTime
  humanoid.Reset()
  simTime = 0
  humanoid.SetSimTime(simTime)
  pose = humanoid.InitializePoseFromMotionData()
  humanoid.ApplyPose(pose, True, True, humanoid._humanoid, bc)


np.savez("state_action_data.npz",torque=all_torque,state=all_state,nextstate=all_nextstate) #<---- MY EDIT

Reset(humanoid)
p.disconnect()

I then save the state and torque data and use them with a humanoid built with gym.make( ). The action space is bounded [-1 , 1] so values above or below get clipped. I tried rescaling each column to this range, but I don't think this is correct either.

import os
import numpy as np
import gym
import pybullet_envs
from gym import wrappers
import matplotlib.pyplot as plt
from IPython import display
%matplotlib inline

# Humanoid
from pybullet_envs.deep_mimic.env.pybullet_deep_mimic_env import PyBulletDeepMimicEnv
gym.envs.registration.register('DeepMimicHumanoidEnv-v0',
                                   entry_point='gym_env.DeepMimicHumanoidGymEnv:DeepMimicHumanoidGymEnv',
                                   max_episode_steps=1000,
                                   kwargs={
                                       'render': False
                                   })
# Rescale columns
def scale_cols(X, x_min, x_max):
    nom = (X-X.min(axis=0))*(x_max-x_min)
    denom = X.max(axis=0) - X.min(axis=0)
    denom[denom==0] = 1
    return x_min + nom/denom 

# Load data 
with np.load(os.path.join(os.getcwd(),'mocap','state_action_data.npz'),allow_pickle=True) as data:
    action    = scale_cols(data['torque'].squeeze(),-1,1)
    state     = scale_cols(data['state'].squeeze(),-1,1)
    nextstate = scale_cols(data['nextstate'].squeeze(),-1,1)

# Simulate agent in environment
frames = []
this_state = env.reset()
joints_to_keep = np.array([8,9,7,15,17,16,19,29,31,30,33,24,25,28,38,39,42]) # based on https://github.com/openai/gym/wiki/Humanoid-V1
plt.figure(figsize=((10,8)))

for ii in range(100): #range(action.shape[0]):
    next_state, reward, done, _ = env.step(action[ii,joints_to_keep])
    plt.imshow(env.render(mode='rgb_array'))
    display.display(plt.gcf())    
    display.clear_output(wait=True)
    if done:
        print('done.')
        break

The commands are "working" in that the humanoid moves, but it just slowly falls to the ground. I tried debugging by setting the action to a fixed value.

• env.step(17*[1]) causes it to slowly fall to the ground
• env.step(17[-1]) causes it to quickly collapse in on itself and fall to the ground.

I cannot figure out how to the get all of the values scaled within the correct range. I am also not sure if I am choosing the joints correctly. The rotations listed in the wiki for the gym humanoid shows the joint rotations in order of x-z-y. I am still unclear if this is x position of rotation (assumed to be the first index) or rotation about x (could be y or z depending on coordinate system). I also see that the Humanoid class has a call to Humanoid.ApplyAction, but it does not seem that this is used anywhere in the repo directly.

Any help is appreciated! Thanks! @erwincoumans Are you able to provide some insight?

[erwincoumans]

This command should work, if all dependencies are installed properly

python -m pybullet_envs.deep_mimic.testrl --arg_file run_humanoid3d_walk_args.txt

Once that works, you can check what files are used and see the action bounds etc.

[jabrantley]

Hi Erwin,

Thanks so much for your reply. It was indeed helpful to walk through the deepmimic example and I enjoyed exploring the world, agent, and humanoid classes in substantial detail. The only follow up question I have is about PD control for the humanoid. Is it possible for the humanoid to achieve walking using only PD control? In the example humanoidMotionCapture.py, the movement of the humanoid is stepped through so it only shows if the desired position is achieved. In the examples in ../pybullet_envs/deep_mimic/mocap/, the replication of the movement is done using position control so it makes sense that it is exact.

In the testrl.py implementation, it seems that the update is done in a two step process following the call to world.update:

 self._update_agents(timestep)
 self._update_env(timestep)

(First) I see how the action is being predicted through update calls from world to the PPO agent. Once the predicted action is obtained, the pose is estimated and stored in the call self.desiredPose = self._humanoid.convertActionToPose(action), where self is the env class. (Second) Then, a direct update call is made self._update_env(timestep). and the env.desiredPosition is used in the call self._humanoid.computeAndApplyPDForces(self.desiredPose,maxForces=maxForces). However, in humanoidMotionCapture.py, the same approach is being used to compute the torques, only the desired position is computed directly from the output of slerp, which relies on frameData and frameDataNext from the motion capture data at each time point. I tried using the joint computations from slerp directly (e.g., _poseInterpolator._chestRot, _poseInterpolator._rightKneeRot) for the PD computation and it "works" for controlling the humanoid but it does not quite walk. It sort of steps forward and slowly collapses, but never really walks.

Is it simply the case that PD control is insufficient for achieving stable walking? I know this was long and I hope it is not confusing to follow, but I really appreciate the feedback.

Thanks!

EDIT: By the way, this is not a question about deepmimic or RL in general, just about PD control for the humanoid.

[jabrantley]

Hi @erwincoumans. Sorry to bother about this again, but I wanted to close out this final thought. Any chance you have thoughts on the PD control? BTW, I was able to get the behavioral cloning working by generating expert (state, action) pairs from the trained implementation. Thanks for your advice on that.

[zhangy76]

Hi @jabrantley,

I'm also interested in what you were doing. For a given motion sequence, I'd like to infer the underlying forces that match the motion. I wonder if you have figured them out?