Deep Mimic: get_state_space() : Check-point graph/graph-building

[rohit-kumar-j]

Hi :D
I was trying to implement deep_mimic walking policies on a custom robot that is structurally similar.

Upon running in a virtual env, there was a graph-shape mismatch error.

I had to change the definition of the function def get_state_size(): within pybullet_deep_mimic_env.py (In order to deal with a previous error when state_size for the robot was required to be 288 instead of 197)

def get_state_size(self, agent_id):
    return 197

to

def get_state_size(self, agent_id):
    return 288

The json_data or agent_file referenced below is linked here

Any suggestions as to how to tackle this error?
This is the output_file:

(park) rkj@Strix:~/Work/RJAX/Experimental/pybullet_envs/deep_mimic$ python3 -m testrl --arg_file run_humanoid3d_walk_args.txt
parentdir= /home/rkj/Work/RJAX/Experimental
/home/rkj/envs/park/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/rkj/envs/park/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/rkj/envs/park/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/rkj/envs/park/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/rkj/envs/park/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/rkj/envs/park/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
/home/rkj/envs/park/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/rkj/envs/park/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/rkj/envs/park/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/rkj/envs/park/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/rkj/envs/park/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/rkj/envs/park/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
pybullet build time: Dec 23 2020 01:46:51
run_humanoid3d_walk_args.txt
enable_draw= True
Initialization strategy: InitializationStrategy.RANDOM
startThreads creating 1 threads.
starting thread 0
started thread 0 
argc=2
argv[0] = --unused
argv[1] = --start_demo_name=Physics Server
ExampleBrowserThreadFunc started
X11 functions dynamically loaded using dlopen/dlsym OK!
X11 functions dynamically loaded using dlopen/dlsym OK!
Creating context
Created GL 3.3 context
Direct GLX rendering context obtained
Making context current
GL_VENDOR=Intel
GL_RENDERER=Mesa Intel(R) UHD Graphics 630 (CFL GT2)
GL_VERSION=4.6 (Core Profile) Mesa 20.2.6
GL_SHADING_LANGUAGE_VERSION=4.60
pthread_getconcurrency()=0
Version = 4.6 (Core Profile) Mesa 20.2.6
Vendor = Intel
Renderer = Mesa Intel(R) UHD Graphics 630 (CFL GT2)
b3Printf: Selected demo: Physics Server
startThreads creating 1 threads.
starting thread 0
started thread 0 
MotionThreadFunc thread started
ven = Intel
Workaround for some crash in the Intel OpenGL driver on Linux/Ubuntu
ven = Intel
Workaround for some crash in the Intel OpenGL driver on Linux/Ubuntu
planeId= 0
motion_file= data/motions/humanoid3d_walk.txt
num_agents= 1

Num Agents: 1
len(agent_files)= 1
Agent 0: data/agents/ct_agent_humanoid_ppo.txt
2021-02-10 20:16:10.568006: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2021-02-10 20:16:10.589460: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2599990000 Hz
2021-02-10 20:16:10.590101: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x558d80a8cbc0 executing computations on platform Host. Devices:
2021-02-10 20:16:10.590125: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
WARNING:tensorflow:From /home/rkj/envs/park/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
state_offset= [ 0.5 -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.
 -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.
 -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.
 -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.
 -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.
 -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.
 -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.
 -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.
 -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.
 -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.
 -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.
 -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.
 -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.
 -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.
 -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.
 -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.
 -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.
 -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.
 -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.
 -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0.
 -0.  -0.  -0.  -0.  -0.  -0.  -0.  -0. ]
state_scale= [0.5 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.
 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.
 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.
 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.
 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.
 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.
 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.
 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.
 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.
 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.
 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.
 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.
 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.
 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.
 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.
 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1. ]
2021-02-10 20:16:10.652753: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
WARNING:tensorflow:From /home/rkj/Work/RJAX/Experimental/pybullet_envs/deep_mimic/learning/tf_util.py:47: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.dense instead.
WARNING:tensorflow:Entity <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x7f3df03a8e80>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x7f3df03a8e80>>: AssertionError: Bad argument number for Name: 3, expecting 4
WARNING:tensorflow:Entity <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x7f3df03a8ef0>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x7f3df03a8ef0>>: AssertionError: Bad argument number for Name: 3, expecting 4
WARNING:tensorflow:Entity <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x7f3df0392a58>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x7f3df0392a58>>: AssertionError: Bad argument number for Name: 3, expecting 4
WARNING:tensorflow:Entity <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x7f3df03a8e10>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x7f3df03a8e10>>: AssertionError: Bad argument number for Name: 3, expecting 4
WARNING:tensorflow:Entity <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x7f3df03a8e10>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x7f3df03a8e10>>: AssertionError: Bad argument number for Name: 3, expecting 4
WARNING:tensorflow:Entity <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x7f3df0392e80>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x7f3df0392e80>>: AssertionError: Bad argument number for Name: 3, expecting 4
Built actor net: fc_2layers_1024units
Built critic net: fc_2layers_1024units
WARNING:tensorflow:From /home/rkj/Work/RJAX/Experimental/pybullet_envs/deep_mimic/learning/tf_util.py:68: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
WARNING:tensorflow:From /home/rkj/envs/park/lib/python3.6/site-packages/tensorflow/python/ops/math_grad.py:1250: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
{
"ID": 0,
 "Type": "PPO",
 "ActionSpace": "Continuous",
 "StateDim": 288,
 "GoalDim": 0,
 "ActionDim": 36
}
WARNING:tensorflow:From /home/rkj/envs/park/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
Traceback (most recent call last):
  File "/home/rkj/envs/park/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1356, in _do_call
    return fn(*args)
  File "/home/rkj/envs/park/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1341, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/home/rkj/envs/park/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [288] rhs shape= [197]
	 [[{{node agent_2/save/Assign_26}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/rkj/envs/park/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1286, in restore
    {self.saver_def.filename_tensor_name: save_path})
  File "/home/rkj/envs/park/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 950, in run
    run_metadata_ptr)
  File "/home/rkj/envs/park/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1173, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/rkj/envs/park/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1350, in _do_run
    run_metadata)
  File "/home/rkj/envs/park/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1370, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [288] rhs shape= [197]
	 [[node agent_2/save/Assign_26 (defined at home/rkj/Work/RJAX/Experimental/pybullet_envs/deep_mimic/learning/tf_agent.py:141) ]]

Errors may have originated from an input operation.
Input Source operations connected to node agent_2/save/Assign_26:
 agent/resource/s_norm/mean (defined at home/rkj/Work/RJAX/Experimental/pybullet_envs/deep_mimic/learning/tf_normalizer.py:56)

Original stack trace for 'agent_2/save/Assign_26':
  File "usr/local/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "usr/local/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "home/rkj/Work/RJAX/Experimental/pybullet_envs/deep_mimic/testrl.py", line 98, in <module>
    world = build_world(args, True)
  File "home/rkj/Work/RJAX/Experimental/pybullet_envs/deep_mimic/testrl.py", line 69, in build_world
    world = RLWorld(env, arg_parser)
  File "home/rkj/Work/RJAX/Experimental/pybullet_envs/deep_mimic/learning/rl_world.py", line 20, in __init__
    self.build_agents()
  File "home/rkj/Work/RJAX/Experimental/pybullet_envs/deep_mimic/learning/rl_world.py", line 75, in build_agents
    curr_agent = self._build_agent(i, curr_file)
  File "home/rkj/Work/RJAX/Experimental/pybullet_envs/deep_mimic/learning/rl_world.py", line 141, in _build_agent
    agent = AgentBuilder.build_agent(self, id, agent_file)
  File "home/rkj/Work/RJAX/Experimental/pybullet_envs/deep_mimic/learning/agent_builder.py", line 18, in build_agent
    agent = PPOAgent(world, id, json_data)
  File "home/rkj/Work/RJAX/Experimental/pybullet_envs/deep_mimic/learning/ppo_agent.py", line 32, in __init__
    super().__init__(world, id, json_data)
  File "home/rkj/Work/RJAX/Experimental/pybullet_envs/deep_mimic/learning/pg_agent.py", line 43, in __init__
    super().__init__(world, id, json_data)
  File "home/rkj/Work/RJAX/Experimental/pybullet_envs/deep_mimic/learning/tf_agent.py", line 23, in __init__
    self._build_graph(json_data)
  File "home/rkj/Work/RJAX/Experimental/pybullet_envs/deep_mimic/learning/tf_agent.py", line 68, in _build_graph
    self._build_saver()
  File "home/rkj/Work/RJAX/Experimental/pybullet_envs/deep_mimic/learning/tf_agent.py", line 141, in _build_saver
    self.saver = tf.train.Saver(vars, max_to_keep=0)
  File "home/rkj/envs/park/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 825, in __init__
    self.build()
  File "home/rkj/envs/park/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 837, in build
    self._build(self._filename, build_save=True, build_restore=True)
  File "home/rkj/envs/park/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 875, in _build
    build_restore=build_restore)
  File "home/rkj/envs/park/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 508, in _build_internal
    restore_sequentially, reshape)
  File "home/rkj/envs/park/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 350, in _AddRestoreOps
    assign_ops.append(saveable.restore(saveable_tensors, shapes))
  File "home/rkj/envs/park/lib/python3.6/site-packages/tensorflow/python/training/saving/saveable_object_util.py", line 72, in restore
    self.op.get_shape().is_fully_defined())
  File "home/rkj/envs/park/lib/python3.6/site-packages/tensorflow/python/ops/state_ops.py", line 227, in assign
    validate_shape=validate_shape)
  File "home/rkj/envs/park/lib/python3.6/site-packages/tensorflow/python/ops/gen_state_ops.py", line 66, in assign
    use_locking=use_locking, name=name)
  File "home/rkj/envs/park/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
    op_def=op_def)
  File "home/rkj/envs/park/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "home/rkj/envs/park/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
    op_def=op_def)
  File "home/rkj/envs/park/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2005, in __init__
    self._traceback = tf_stack.extract_stack()


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/rkj/Work/RJAX/Experimental/pybullet_envs/deep_mimic/testrl.py", line 98, in <module>
    world = build_world(args, True)
  File "/home/rkj/Work/RJAX/Experimental/pybullet_envs/deep_mimic/testrl.py", line 69, in build_world
    world = RLWorld(env, arg_parser)
  File "/home/rkj/Work/RJAX/Experimental/pybullet_envs/deep_mimic/learning/rl_world.py", line 20, in __init__
    self.build_agents()
  File "/home/rkj/Work/RJAX/Experimental/pybullet_envs/deep_mimic/learning/rl_world.py", line 85, in build_agents
    curr_agent.load_model(pybullet_data.getDataPath() + "/" + curr_model_file)
  File "/home/rkj/Work/RJAX/Experimental/pybullet_envs/deep_mimic/learning/tf_agent.py", line 42, in load_model
    self.saver.restore(self.sess, in_path)
  File "/home/rkj/envs/park/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1322, in restore
    err, "a mismatch between the current graph and the graph")
tensorflow.python.framework.errors_impl.InvalidArgumentError: Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Assign requires shapes of both tensors to match. lhs shape= [288] rhs shape= [197]
	 [[node agent_2/save/Assign_26 (defined at home/rkj/Work/RJAX/Experimental/pybullet_envs/deep_mimic/learning/tf_agent.py:141) ]]

Errors may have originated from an input operation.
Input Source operations connected to node agent_2/save/Assign_26:
 agent/resource/s_norm/mean (defined at home/rkj/Work/RJAX/Experimental/pybullet_envs/deep_mimic/learning/tf_normalizer.py:56)

Original stack trace for 'agent_2/save/Assign_26':
  File "usr/local/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "usr/local/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "home/rkj/Work/RJAX/Experimental/pybullet_envs/deep_mimic/testrl.py", line 98, in <module>
    world = build_world(args, True)
  File "home/rkj/Work/RJAX/Experimental/pybullet_envs/deep_mimic/testrl.py", line 69, in build_world
    world = RLWorld(env, arg_parser)
  File "home/rkj/Work/RJAX/Experimental/pybullet_envs/deep_mimic/learning/rl_world.py", line 20, in __init__
    self.build_agents()
  File "home/rkj/Work/RJAX/Experimental/pybullet_envs/deep_mimic/learning/rl_world.py", line 75, in build_agents
    curr_agent = self._build_agent(i, curr_file)
  File "home/rkj/Work/RJAX/Experimental/pybullet_envs/deep_mimic/learning/rl_world.py", line 141, in _build_agent
    agent = AgentBuilder.build_agent(self, id, agent_file)
  File "home/rkj/Work/RJAX/Experimental/pybullet_envs/deep_mimic/learning/agent_builder.py", line 18, in build_agent
    agent = PPOAgent(world, id, json_data)
  File "home/rkj/Work/RJAX/Experimental/pybullet_envs/deep_mimic/learning/ppo_agent.py", line 32, in __init__
    super().__init__(world, id, json_data)
  File "home/rkj/Work/RJAX/Experimental/pybullet_envs/deep_mimic/learning/pg_agent.py", line 43, in __init__
    super().__init__(world, id, json_data)
  File "home/rkj/Work/RJAX/Experimental/pybullet_envs/deep_mimic/learning/tf_agent.py", line 23, in __init__
    self._build_graph(json_data)
  File "home/rkj/Work/RJAX/Experimental/pybullet_envs/deep_mimic/learning/tf_agent.py", line 68, in _build_graph
    self._build_saver()
  File "home/rkj/Work/RJAX/Experimental/pybullet_envs/deep_mimic/learning/tf_agent.py", line 141, in _build_saver
    self.saver = tf.train.Saver(vars, max_to_keep=0)
  File "home/rkj/envs/park/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 825, in __init__
    self.build()
  File "home/rkj/envs/park/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 837, in build
    self._build(self._filename, build_save=True, build_restore=True)
  File "home/rkj/envs/park/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 875, in _build
    build_restore=build_restore)
  File "home/rkj/envs/park/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 508, in _build_internal
    restore_sequentially, reshape)
  File "home/rkj/envs/park/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 350, in _AddRestoreOps
    assign_ops.append(saveable.restore(saveable_tensors, shapes))
  File "home/rkj/envs/park/lib/python3.6/site-packages/tensorflow/python/training/saving/saveable_object_util.py", line 72, in restore
    self.op.get_shape().is_fully_defined())
  File "home/rkj/envs/park/lib/python3.6/site-packages/tensorflow/python/ops/state_ops.py", line 227, in assign
    validate_shape=validate_shape)
  File "home/rkj/envs/park/lib/python3.6/site-packages/tensorflow/python/ops/gen_state_ops.py", line 66, in assign
    use_locking=use_locking, name=name)
  File "home/rkj/envs/park/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
    op_def=op_def)
  File "home/rkj/envs/park/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "home/rkj/envs/park/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
    op_def=op_def)
  File "home/rkj/envs/park/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2005, in __init__
    self._traceback = tf_stack.extract_stack()

numActiveThreads = 0
stopping threads
Thread with taskId 0 exiting
Thread TERMINATED
destroy semaphore
semaphore destroyed
destroy main semaphore
main semaphore destroyed
finished
numActiveThreads = 0
btShutDownExampleBrowser stopping threads
Thread with taskId 0 exiting
Thread TERMINATED
destroy semaphore
semaphore destroyed
destroy main semaphore
main semaphore destroyed
(park) rkj@Strix:~/Work/RJAX/Experimental/pybullet_envs/deep_mimic$ 

Warm Regards,
Rohit Kumar

[rohit-kumar-j]

Solved it. The mismatch was from HumanoidBulletEnv