Is there any memory leak in DeePMD-kit C interface?

[Cloudac7]

I am dealing with integration to CP2K from DeePMD-kit. With C interface released, there is now a rather easy way to get them altogether.

While everything seems to be OK with ssmp version, in sdbg version (with -fsanitize=leak to detect possible memory leak), it always run into an EXIT issue, showing there is still memory leaking. (cp2k/cp2k#3145)

By compiling DeePMD-kit C interface also with -fsanitize=leak by adding the following into CMakeLists.txt, we finally found the leak is due to some implementation of C interface or even deeper dependencies like tensorflow.

add_compile_options(-fsanitize=leak -g)
add_link_options(-fsanitize=leak)

Logs of regression test for sdbg version

 -------------------------------------------------------------------------------
  **** **** ******  **  PROGRAM ENDED AT                 2023-12-08 13:51:52.513
 ***** ** ***  *** **   PROGRAM RAN ON                          chenglab-MS-7B09
 **    ****   ******    PROGRAM RAN BY                                     ypliu
 ***** **    ** ** **   PROGRAM PROCESS ID                               1794399
  **** **  *******  **  PROGRAM STOPPED IN 

==1794399==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 72704 byte(s) in 1 object(s) allocated from:
    #0 0x7faa2bc14302 in __interceptor_malloc ../../../../src/libsanitizer/lsan/lsan_interceptors.cpp:75
    #1 0x7fa97b654d95  (<unknown module>)
    #2 0x7fa97b5da8da  (<unknown module>)

Direct leak of 416 byte(s) in 1 object(s) allocated from:
    #0 0x7faa2bc12a92 in operator new(unsigned long) ../../../../src/libsanitizer/lsan/lsan_interceptors.cpp:248
    #1 0x7faa2c217ca4 in DP_NewDeepPot /home/ypliu/playground/deepmd-kit/source/api_c/src/c_api.cc:37

Indirect leak of 32 byte(s) in 1 object(s) allocated from:
    #0 0x7faa2bc12a92 in operator new(unsigned long) ../../../../src/libsanitizer/lsan/lsan_interceptors.cpp:248
    #1 0x55f7ce104ad9 in __gnu_cxx::new_allocator<int>::allocate(unsigned long, void const*) /usr/include/c++/11/ext/new_allocator.h:127
    #2 0x55f7ce104ad9 in std::allocator_traits<std::allocator<int> >::allocate(std::allocator<int>&, unsigned long) /usr/include/c++/11/bits/alloc_traits.h:464
    #3 0x55f7ce104ad9 in std::_Vector_base<int, std::allocator<int> >::_M_allocate(unsigned long) /usr/include/c++/11/bits/stl_vector.h:346
    #4 0x55f7ce104ad9 in std::vector<int, std::allocator<int> >::_M_default_append(unsigned long) /usr/include/c++/11/bits/vector.tcc:635
    #5 0x7faa2ab504c7 in std::vector<int, std::allocator<int> >::resize(unsigned long) /usr/include/c++/11/bits/stl_vector.h:940
    #6 0x7faa2ab504c7 in deepmd::AtomMap::AtomMap(__gnu_cxx::__normal_iterator<int const*, std::vector<int, std::allocator<int> > >, __gnu_cxx::__normal_iterator<int const*, std::vector<int, std::allocator<int> > >) /home/ypliu/playground/deepmd-kit/source/api_cc/src/AtomMap.cc:22

Indirect leak of 32 byte(s) in 1 object(s) allocated from:
    #0 0x7faa2bc12a92 in operator new(unsigned long) ../../../../src/libsanitizer/lsan/lsan_interceptors.cpp:248
    #1 0x55f7ce104ad9 in __gnu_cxx::new_allocator<int>::allocate(unsigned long, void const*) /usr/include/c++/11/ext/new_allocator.h:127
    #2 0x55f7ce104ad9 in std::allocator_traits<std::allocator<int> >::allocate(std::allocator<int>&, unsigned long) /usr/include/c++/11/bits/alloc_traits.h:464
    #3 0x55f7ce104ad9 in std::_Vector_base<int, std::allocator<int> >::_M_allocate(unsigned long) /usr/include/c++/11/bits/stl_vector.h:346
    #4 0x55f7ce104ad9 in std::vector<int, std::allocator<int> >::_M_default_append(unsigned long) /usr/include/c++/11/bits/vector.tcc:635
    #5 0x7faa2ab504dd in std::vector<int, std::allocator<int> >::resize(unsigned long) /usr/include/c++/11/bits/stl_vector.h:940
    #6 0x7faa2ab504dd in deepmd::AtomMap::AtomMap(__gnu_cxx::__normal_iterator<int const*, std::vector<int, std::allocator<int> > >, __gnu_cxx::__normal_iterator<int const*, std::vector<int, std::allocator<int> > >) /home/ypliu/playground/deepmd-kit/source/api_cc/src/AtomMap.cc:21

Indirect leak of 32 byte(s) in 1 object(s) allocated from:
    #0 0x7faa2bc12a92 in operator new(unsigned long) ../../../../src/libsanitizer/lsan/lsan_interceptors.cpp:248
    #1 0x55f7ce104ad9 in __gnu_cxx::new_allocator<int>::allocate(unsigned long, void const*) /usr/include/c++/11/ext/new_allocator.h:127
    #2 0x55f7ce104ad9 in std::allocator_traits<std::allocator<int> >::allocate(std::allocator<int>&, unsigned long) /usr/include/c++/11/bits/alloc_traits.h:464
    #3 0x55f7ce104ad9 in std::_Vector_base<int, std::allocator<int> >::_M_allocate(unsigned long) /usr/include/c++/11/bits/stl_vector.h:346
    #4 0x55f7ce104ad9 in std::vector<int, std::allocator<int> >::_M_default_append(unsigned long) /usr/include/c++/11/bits/vector.tcc:635
    #5 0x7faa2ab5037e in std::vector<int, std::allocator<int> >::resize(unsigned long) /usr/include/c++/11/bits/stl_vector.h:940
    #6 0x7faa2ab5037e in deepmd::AtomMap::AtomMap(__gnu_cxx::__normal_iterator<int const*, std::vector<int, std::allocator<int> > >, __gnu_cxx::__normal_iterator<int const*, std::vector<int, std::allocator<int> > >) /home/ypliu/playground/deepmd-kit/source/api_cc/src/AtomMap.cc:14

SUMMARY: LeakSanitizer: 73216 byte(s) leaked in 5 allocation(s).


Runtime failure with code 23.

I found a few implementation of new xxx when creating a pointer, like this in c_api.cc:

DP_DeepPot* DP_NewDeepPot(const char* c_model) {
  std::string model(c_model);
  DP_NEW_OK(DP_DeepPot, deepmd::DeepPot dp(model);
            DP_DeepPot* new_dp = new DP_DeepPot(dp); return new_dp;)
}

Is it safe and might it be the reason of memory leaking?

[njzjz]

As I pointed out in #1828, most memory leaks happen inside TensorFlow.

I agree new xxx in DeePMD-kit may also cause memory leaks, but considering the huge memory leak in TensorFlow, it makes little difference. PR is welcome.

[njzjz]

This is what TF C API does:

https://github.com/tensorflow/tensorflow/blob/b018a4a0ee3a9be5b6144e2c470930d8cdb8a0ca/tensorflow/c/c_api.cc#L1581-L1590

They have a Delete API.

[njzjz]

I tried to free DP_DeepPot and got the following SEGV in the tests:

LeakSanitizer:DEADLYSIGNAL
==4407==ERROR: LeakSanitizer: SEGV on unknown address (pc 0x7fde7733f152 bp 0x7ffe8dfe7120 sp 0x7ffe8dfe70f0 T0)
==4407==The signal is caused by a READ memory access.
==4407==Hint: this fault was caused by a dereference of a high value address (see register values below).  Disassemble the provided pc to learn which register was used.
    #0 0x7fde7733f152 in google::protobuf::internal::ThreadSafeArena::~ThreadSafeArena() (/opt/hostedtoolcache/Python/3.11.7/x64/lib/python3.11/site-packages/tensorflow/libtensorflow_framework.so.2+0x1d3f152) (BuildId: 02798b28b8d875b28b92a268aa44f887)
    #1 0x7fde775f1d31 in tensorflow::GraphDef::~GraphDef() (/opt/hostedtoolcache/Python/3.11.7/x64/lib/python3.11/site-packages/tensorflow/libtensorflow_framework.so.2+0x1ff1d31) (BuildId: 02798b28b8d875b28b92a268aa44f887)
    #2 0x7fde775f1ded in tensorflow::GraphDef::~GraphDef() (/opt/hostedtoolcache/Python/3.11.7/x64/lib/python3.11/site-packages/tensorflow/libtensorflow_framework.so.2+0x1ff1ded) (BuildId: 02798b28b8d875b28b92a268aa44f887)
    #3 0x7fdea8494496 in deepmd::DeepPot::~DeepPot() /home/runner/work/deepmd-kit/deepmd-kit/source/api_cc/src/DeepPot.cc:410
    #4 0x7fdea85bac43 in DP_DeepPot::~DP_DeepPot() /home/runner/work/deepmd-kit/deepmd-kit/source/api_c/include/c_api_internal.h:36
    #5 0x7fdea85b54ba in DP_DeleteDeepPot /home/runner/work/deepmd-kit/deepmd-kit/source/api_c/src/c_api.cc:64
    #6 0x55db8a7bb1d5 in deepmd::hpp::DeepPot::~DeepPot() /home/runner/work/deepmd-kit/deepmd-kit/source/api_c/include/deepmd.hpp:567
    #7 0x55db8a7ef220 in TestInferDeepPotAFParamAParam<double>::~TestInferDeepPotAFParamAParam() /home/runner/work/deepmd-kit/deepmd-kit/source/api_c/tests/test_deeppot_a_fparam_aparam.cc:16
    #8 0x55db8a7efc56 in TestInferDeepPotAFParamAParam_cpu_build_nlist_Test<double>::~TestInferDeepPotAFParamAParam_cpu_build_nlist_Test() /home/runner/work/deepmd-kit/deepmd-kit/source/api_c/tests/test_deeppot_a_fparam_aparam.cc:101
    #9 0x55db8a7efc88 in TestInferDeepPotAFParamAParam_cpu_build_nlist_Test<double>::~TestInferDeepPotAFParamAParam_cpu_build_nlist_Test() /home/runner/work/deepmd-kit/deepmd-kit/source/api_c/tests/test_deeppot_a_fparam_aparam.cc:101
    #10 0x55db8a96fa27 in testing::Test::DeleteSelf_() (/home/runner/work/deepmd-kit/deepmd-kit/source/build_tests/api_c/tests/runUnitTests_c+0x2e1a27) (BuildId: 9a3481221514c030d4d7402f8b0a8f7a35b3df19)
    #11 0x55db8a981cdc in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/home/runner/work/deepmd-kit/deepmd-kit/source/build_tests/api_c/tests/runUnitTests_c+0x2f3cdc) (BuildId: 9a3481221514c030d4d7402f8b0a8f7a35b3df19)
    #12 0x55db8a97a836 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/home/runner/work/deepmd-kit/deepmd-kit/source/build_tests/api_c/tests/runUnitTests_c+0x2ec836) (BuildId: 9a3481221514c030d4d7402f8b0a8f7a35b3df19)
    #13 0x55db8a956d02 in testing::TestInfo::Run() (/home/runner/work/deepmd-kit/deepmd-kit/source/build_tests/api_c/tests/runUnitTests_c+0x2c8d02) (BuildId: 9a3481221514c030d4d7402f8b0a8f7a35b3df19)
    #14 0x55db8a9575e6 in testing::TestSuite::Run() (/home/runner/work/deepmd-kit/deepmd-kit/source/build_tests/api_c/tests/runUnitTests_c+0x2c95e6) (BuildId: 9a3481221514c030d4d7402f8b0a8f7a35b3df19)
    #15 0x55db8a967818 in testing::internal::UnitTestImpl::RunAllTests() (/home/runner/work/deepmd-kit/deepmd-kit/source/build_tests/api_c/tests/runUnitTests_c+0x2d9818) (BuildId: 9a3481221514c030d4d7402f8b0a8f7a35b3df19)
    #16 0x55db8a982bcd in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (/home/runner/work/deepmd-kit/deepmd-kit/source/build_tests/api_c/tests/runUnitTests_c+0x2f4bcd) (BuildId: 9a3481221514c030d4d7402f8b0a8f7a35b3df19)
    #17 0x55db8a97b8e6 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (/home/runner/work/deepmd-kit/deepmd-kit/source/build_tests/api_c/tests/runUnitTests_c+0x2ed8e6) (BuildId: 9a3481221514c030d4d7402f8b0a8f7a35b3df19)
    #18 0x55db8a965e3a in testing::UnitTest::Run() (/home/runner/work/deepmd-kit/deepmd-kit/source/build_tests/api_c/tests/runUnitTests_c+0x2d7e3a) (BuildId: 9a3481221514c030d4d7402f8b0a8f7a35b3df19)
    #19 0x55db8a941db8 in RUN_ALL_TESTS() (/home/runner/work/deepmd-kit/deepmd-kit/source/build_tests/api_c/tests/runUnitTests_c+0x2b3db8) (BuildId: 9a3481221514c030d4d7402f8b0a8f7a35b3df19)
    #20 0x55db8a941d31 in main (/home/runner/work/deepmd-kit/deepmd-kit/source/build_tests/api_c/tests/runUnitTests_c+0x2b3d31) (BuildId: 9a3481221514c030d4d7402f8b0a8f7a35b3df19)
    #21 0x7fdea7c29d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: 203de0ae33b53fee1578b117cb4123e85d0534f0)
    #22 0x7fdea7c29e3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e3f) (BuildId: 203de0ae33b53fee1578b117cb4123e85d0534f0)
    #23 0x55db8a782824 in _start (/home/runner/work/deepmd-kit/deepmd-kit/source/build_tests/api_c/tests/runUnitTests_c+0xf4824) (BuildId: 9a3481221514c030d4d7402f8b0a8f7a35b3df19)

LeakSanitizer can not provide additional info.
SUMMARY: LeakSanitizer: SEGV (/opt/hostedtoolcache/Python/3.11.7/x64/lib/python3.11/site-packages/tensorflow/libtensorflow_framework.so.2+0x1d3f152) (BuildId: 02798b28b8d875b28b92a268aa44f887) in google::protobuf::internal::ThreadSafeArena::~ThreadSafeArena()
==4407==ABORTING

I don't understand what's wrong with protobuf.

[njzjz]

Resolved in #3223.