Dimensions of inputs should match

[J-You]

Has anyone ever had a similar question about mismatch of dimensions of inputs?
I use deep-md v2.0, gpu version.
Maybe it is caused by the distribution of dataset?

DEEPMD INFO    initialize model from scratch
DEEPMD INFO    start training at lr 1.00e-03 (== 1.00e-03), decay_step 5000, decay_rate 0.980995, final lr will be 1.00e-08
2021-08-29 18:13:13.973960: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11
2021-08-29 18:13:15.482269: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11
Traceback (most recent call last):
  File "/data/home/scv0645/run/software/deepmd-kit/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1375, in _do_call
    return fn(*args)
  File "/data/home/scv0645/run/software/deepmd-kit/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1359, in _run_fn
    return self._call_tf_sessionrun(options, feed_dict, fetch_list,
  File "/data/home/scv0645/run/software/deepmd-kit/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1451, in _call_tf_sessionrun
    return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict,
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument: ConcatOp : Dimensions of inputs should match: shape[0] = [132000,9] vs. shape[1] = [26400,8]
         [[{{node filter_type_all/concat_2}}]]
         [[l2_atom_ener_test/_49]]
  (1) Invalid argument: ConcatOp : Dimensions of inputs should match: shape[0] = [132000,9] vs. shape[1] = [26400,8]
         [[{{node filter_type_all/concat_2}}]]
0 successful operations.
0 derived errors ignored.

[shishaochen]

@J-You BUG Fix(#1074) could solve your problem.

[J-You]

Thanks.

[J-You]

@shishaochen Hi, the testing error can also reproduce with using type_embedding and disabled type_ne_side, batch_size equals to 1.
I test my model by running dp test -m graph.pb -s ../data/validation_set -n 100 --shuffle-test -d TEST_validation
I installed dp-kit through the off-line package, cuda 11.3 version.
By the way, LAMMPS runs with the model works well.
Thanks for your attention.

2021-09-04 09:03:41.981180: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-09-04 09:03:41.982241: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-09-04 09:03:41.983264: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30993 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-32GB, pci bus id: 0000:00:0c.0, compute capability: 7.0)
Traceback (most recent call last):
  File "/data/home/scv0645/run/software/deepmd-kit/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1375, in _do_call
    return fn(*args)
  File "/data/home/scv0645/run/software/deepmd-kit/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1359, in _run_fn
    return self._call_tf_sessionrun(options, feed_dict, fetch_list,
  File "/data/home/scv0645/run/software/deepmd-kit/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1451, in _call_tf_sessionrun
    return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict,
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument: ConcatOp : Dimensions of inputs should match: shape[0] = [3600000,9] vs. shape[1] = [36000,8]
         [[{{node load/filter_type_all/concat_2}}]]
         [[load/o_energy/_37]]
  (1) Invalid argument: ConcatOp : Dimensions of inputs should match: shape[0] = [3600000,9] vs. shape[1] = [36000,8]
         [[{{node load/filter_type_all/concat_2}}]]
0 successful operations.
0 derived errors ignored.

[J-You]

In fact, the model was trained by the previous version of dp-kit, that is, #1074 has not been fixed in that version. Does it matter?