Segmentation fault when running LAMMPS

[MoleOrbitalHybridAnalyst]

Hi,

I am running the examples/water. I installed the gpu version of deepmd-kit via conda.

The training seemed running smoothly so I Ctrl-C'ed it after a checkpoint was written. I froze the model and started running lammps but got a seg fault. The detailed input of lammps was as follows but it did not look very helpful to me.

LAMMPS (3 Mar 2020)

OMP_NUM_THREADS environment is not set. Defaulting to 1 thread. (src/comm.cpp:94)
  using 1 OpenMP thread(s) per MPI task
Reading data file ...
  triclinic box = (0 0 0) to (12.4447 12.4447 12.4447) with tilt (0 0 0)
  1 by 1 by 1 MPI processor grid
  reading atoms ...
  192 atoms
  read_data CPU = 0.012475 secs
Summary of lammps deepmd module ...
  >>> Info of deepmd-kit:
  installed to:       /project2/gavoth/chhli/miniconda3
  source:             v1.2.1-3-g30922e7
  source brach:       HEAD
  source commit:      30922e7
  source commit at:   2020-09-14 22:49:07 +0800
  build float prec:   double
  build with tf inc:  /project2/gavoth/chhli/miniconda3/include;/project2/gavoth/chhli/miniconda3/include
  build with tf lib:  /project2/gavoth/chhli/miniconda3/lib/libtensorflow_cc.so;/project2/gavoth/chhli/miniconda3/lib/libtensorflow_framework.so
  set tf intra_op_parallelism_threads: 0
  set tf inter_op_parallelism_threads: 0
  >>> Info of lammps module:
  use deepmd-kit at:  /project2/gavoth/chhli/miniconda3
  source:             v1.2.1-3-g30922e7
  source branch:      HEAD
  source commit:      30922e7
  source commit at:   2020-09-14 22:49:07 +0800
  build float prec:   double
  build with tf inc:  /project2/gavoth/chhli/miniconda3/include;/project2/gavoth/chhli/miniconda3/include
  build with tf lib:  /project2/gavoth/chhli/miniconda3/lib/libtensorflow_cc.so;/project2/gavoth/chhli/miniconda3/lib/libtensorflow_framework.so
2020-09-27 19:48:18.726949: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2020-09-27 19:48:18.728602: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-09-27 19:48:18.731264: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
pciBusID: 0000:02:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.582GHz coreCount: 28 deviceMemorySize: 10.92GiB deviceMemoryBandwidth: 451.17GiB/s
2020-09-27 19:48:18.732155: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-09-27 19:48:18.737550: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-09-27 19:48:18.741355: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-09-27 19:48:18.742078: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-09-27 19:48:18.746377: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-09-27 19:48:18.748384: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-09-27 19:48:18.757527: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-09-27 19:48:18.760656: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
2020-09-27 19:48:18.760708: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-09-27 19:48:18.917716: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-09-27 19:48:18.917771: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102]      0
2020-09-27 19:48:18.917790: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 0:   N
2020-09-27 19:48:18.921554: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10060 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:02:00.0, compute capability: 6.1)
  >>> Info of model(s):
  using   1 model(s): ../train/graph.pb
  rcut in model:      6
  ntypes in model:    2
Neighbor list info ...
  update every 10 steps, delay 0 steps, check no
  max neighbors/atom: 2000, page size: 100000
  master list distance cutoff = 8
  ghost atom cutoff = 8
  binsize = 4, bins = 4 4 4
  1 neighbor lists, perpetual/occasional/extra = 1 0 0
  (1) pair deepmd, perpetual
      attributes: , newton on
      pair build: full/bin/atomonly
      stencil: full/bin/3d
      bin: standard
Setting up Verlet run ...
  Unit style    : metal
  Current step  : 0
  Time step     : 0.0005
2020-09-27 19:48:19.255161: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
Segmentation fault

A naive check by valgrind was attached. Not sure if it will be helpful.
valgrind.log

[njzjz]

I repeated it successfully on Google Colab, see:
https://nbviewer.jupyter.org/urls/drive.google.com/uc/%3Fexport%3Ddownload%26id%3D16e5c1GONUBqZmAeE_U7KmAL1UQT027-z
or https://colab.research.google.com/drive/16e5c1GONUBqZmAeE_U7KmAL1UQT027-z if you can visit Google.

Please check if there is anything different.

[MoleOrbitalHybridAnalyst]

I don't see any difference. Actually, I was able to go through without any error later on when I switched to the CPU version. Not sure what's wrong with my GPU environment. Thank you anyway!