neighbor-stat seems to take a very long time with large datasets

[asedova]

It seems that in the latest versions of deepmd, the automatic neighbor-stat must be run in order for model compression to be enabled. To me this seems to mean that the model cannot be compressed even if you pre-calculate the neighbor-stat with dp before training? I have some very large datasets, and it seems that the neighbor-stat calculation takes an extremely long time, unless something is not working on my system. Basically, it has never finished, the 10 or so times I have tried. This means I can only run training with this option turned off. Is there no way to allow the user to pass in a sel and also enable model compression?

[njzjz]

Indeed, the compression requires min_nbor_dist (the nearest distance between neighbor atoms) instead of sel.

[asedova]

From the manual, --skip-neighbor-stat ...model compression will be disabled. Also, is there a reason that this takes so long? Is there any way to make it faster?

[njzjz]

To be clear, data stating returns both sel and min_nbor_dist.

This calculation has not been implemented on GPUs yet. Contribution is welcome.

[asedova]

There does seem to be some parallelism, if I am reading the code correctly. Should I be using threads for this (either OMP or TF)? If so, wouldn't that need to be the same threading config I use for the rest of the training, since I can't pass the results to the training command?

[njzjz]

Both TF and OMP should be set. Read the FAQ for details.

You can also do neighbor stat when executing dp compress by adding the -t TRAINING_SCRIPT argument. See dp compress.

[asedova]

I set OMP_NUM_THREADS to 1 for training on our system. I worked with NVIDIA and this seemed to be the best way. I am running training on nodes with 6 GPUs each. So if I need to have a different setting for the neighbor calculation, this won't work. Running the neighbor stat during compression seems to be what I need.