DP Compress Error with conflicting tensorflow and deepmd-kit compilation

[kimiaGF]

I am trying to compress a model to run a large (~100,000 atom) LAMMPS simulation, but when I try to compress the final frozen model I get this error:

WARNING:tensorflow:From /home/kimia.gh/mambaforge/envs/deepmd/lib/python3.10/site-packages/tensorflow/python/compat/v2_compat.py:107: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term
WARNING:root:To get the best performance, it is recommended to adjust the number of threads by setting the environment variables OMP_NUM_THREADS, TF_INTRA_OP_PARALLELISM_THREADS, and TF_INTER_OP_PARALLELISM_THREADS.
Traceback (most recent call last):
  File "/home/kimia.gh/.local/lib/python3.10/site-packages/deepmd/env.py", line 226, in get_module
    module = tf.load_op_library(str(module_file))
  File "/home/kimia.gh/mambaforge/envs/deepmd/lib/python3.10/site-packages/tensorflow/python/framework/load_library.py", line 54, in load_op_library
    lib_handle = py_tf.TF_LoadLibrary(library_filename)
tensorflow.python.framework.errors_impl.NotFoundError: /home/kimia.gh/.local/lib/python3.10/site-packages/deepmd/op/libop_abi.so: undefined symbol: _ZNK10tensorflow8OpKernel11TraceStringERKNS_15OpKernelContextEb

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/kimia.gh/mambaforge/envs/deepmd/bin/dp", line 7, in <module>
    from deepmd.entrypoints.main import main
  File "/home/kimia.gh/.local/lib/python3.10/site-packages/deepmd/__init__.py", line 7, in <module>
    import deepmd.utils.network as network
  File "/home/kimia.gh/.local/lib/python3.10/site-packages/deepmd/utils/__init__.py", line 2, in <module>
    from .data import DeepmdData
  File "/home/kimia.gh/.local/lib/python3.10/site-packages/deepmd/utils/data.py", line 10, in <module>
    from deepmd.env import GLOBAL_NP_FLOAT_PRECISION
  File "/home/kimia.gh/.local/lib/python3.10/site-packages/deepmd/env.py", line 303, in <module>
    op_module = get_module("libop_abi")
  File "/home/kimia.gh/.local/lib/python3.10/site-packages/deepmd/env.py", line 236, in get_module
    raise RuntimeError(
RuntimeError: This deepmd-kit package was compiled with CXX11_ABI_FLAG=0, but TensorFlow runtime was compiled with CXX11_ABI_FLAG=1. These two library ABIs are incompatible and thus an error is raised when loading libop_abi. You need to rebuild deepmd-kit against this TensorFlow runtime.

I understand it's an issue with how tensorflow and deepmd-kit compilation, but I'm not sure how to fix this. I created this environment with mamba and followed the cpu-version installation procedure for conda.

[njzjz]

You loaded two different Python environments, one is /home/kimia.gh/mambaforge/envs/deepmd/lib/python3.10/site-packages, and another one is /home/kimia.gh/.local/lib/python3.10/site-packages.

[njzjz]

You can execute env to see whether there is any environmental variable for those paths.

[kimiaGF]

there are no environment variables set to either of those paths

[kimiaGF]

After looking through the source code and following the given link, I think it might be an issue with my GCC version as it is controlled by our HPC admin and is g++ (GCC) 4.8.5 20150623. I have attempted to install older deepmd-kit versions but this still does not fix anything. Any advice?

[njzjz]

~/.local/lib/ can be disabled by PYTHONUSERBASE. Did you install the package into this directory?

[kimiaGF]

Yes! it turns out it was defaulting to a deepmd that was installed in .local. I renamed .local to something else to see if that was the issue and I no longer get the error. Thank you!

[ZaibulAnam]

How did you solve this issue? In my case, my package is in "/home/manam/.conda/envs/deepmd3/lib/python3.12/site-packages/deepmd/...". I am getting the same error. Can you please help me?