[BUG] GPU Memory usage increases over time with cuopt.linear_programming

[DenizKizaroglu]

Bug Report: GPU Memory usage increases over time with cuopt.linear_programming

Bug Description

When calling cuopt.linear_programming.Solve repeatedly in a loop, the allocated GPU memory appears to increase over time, suggesting a potential memory accumulation. Even when explicitly deleting solver and data model objects and calling gc.collect(), the memory usage does not remain stable.

Steps/Code to Reproduce

Run the python script attached

# Run the script
python3 reproduce_gpu.py

Expected Behavior

The GPU memory usage should remain stable and constant across iterations, as the same problem is being solved repeatedly and resources are (seemingly) being freed after each solve.

Environment Details

• cuopt-cu12: 25.8.0

• cudf-cu12: 25.8.0

• cuopt_mps_parser: 25.8.0

• cuopt_server-cu12: 25.8.0

• cuopt_sh_client: 25.8.0

• cupy-cuda12x: 13.4.1

Installation Method:

pip install --extra-index-url=https::/pypi.nvidia.com 'cuopt-cu12==25.8*'

reproduce_gpu.py

[nguidotti]

This is caused by a memory leak in PDLP:

Direct leak of 9703680 byte(s) in 303240 object(s) allocated from:
    #0 0xf2cf2567ff58 in malloc ../../../../libsanitizer/asan/asan_malloc_linux.cpp:69
    #1 0xf2cf109b97b0  (/home/nguidotti/miniconda/envs/cuopt_dev/lib/libcusparse.so.12+0x8697b0)
    #2 0xf2cf1022be3c in cusparseCreateConstDnVec (/home/nguidotti/miniconda/envs/cuopt_dev/lib/libcusparse.so.12+0xdbe3c)
    #3 0xf2cf1022ca40 in cusparseCreateDnVec (/home/nguidotti/miniconda/envs/cuopt_dev/lib/libcusparse.so.12+0xdca40)
    #4 0xf2cf20ad24b8 in cusparseStatus_t raft::sparse::detail::cusparsecreatednvec<double>(cusparseDnVecDescr**, long, double*) /home/nguidotti/miniconda/envs/cuopt_dev/include/raft/sparse/detail/cusparse_wrappers.h:280
    #5 0xf2cf20ad24b8 in cuopt::linear_programming::detail::pdhg_solver_t<int, double>::update_solution(cuopt::linear_programming::detail::cusparse_view_t<int, double>&) /cuopt-dev/cuopt/cpp/src/linear_programming/pdhg.cu:479
    #6 0xf2cf20acf5c0 in cuopt::linear_programming::detail::pdlp_solver_t<int, double>::run_solver(cuopt::timer_t const&) /cuopt-dev/cuopt/cpp/src/linear_programming/pdlp.cu:1315
    #7 0xf2cf2154e3b4 in cuopt::linear_programming::optimization_problem_solution_t<int, double> cuopt::linear_programming::detail::get_relaxed_lp_solution<int, double>(cuopt::linear_programming::detail::problem_t<int, double>&, rmm::device_uvector<double>&, cuopt::linear_programming::detail::lp_state_t<int, double>&, cuopt::linear_programming::detail::relaxed_lp_settings_t const&) /cuopt-dev/cuopt/cpp/src/mip/relaxed_lp/relaxed_lp.cu:89
    #8 0xf2cf2154e95c in cuopt::linear_programming::optimization_problem_solution_t<int, double> cuopt::linear_programming::detail::get_relaxed_lp_solution<int, double>(cuopt::linear_programming::detail::problem_t<int, double>&, cuopt::linear_programming::detail::solution_t<int, double>&, cuopt::linear_programming::detail::relaxed_lp_settings_t const&) /cuopt-dev/cuopt/cpp/src/mip/relaxed_lp/relaxed_lp.cu:32
    #9 0xf2cf2157027c in cuopt::linear_programming::detail::feasibility_pump_t<int, double>::linear_project_onto_polytope(cuopt::linear_programming::detail::solution_t<int, double>&, double, bool) /cuopt-dev/cuopt/cpp/src/mip/local_search/feasibility_pump/feasibility_pump.cu:220
    #10 0xf2cf21573bd0 in cuopt::linear_programming::detail::feasibility_pump_t<int, double>::run_single_fp_descent(cuopt::linear_programming::detail::solution_t<int, double>&) /cuopt-dev/cuopt/cpp/src/mip/local_search/feasibility_pump/feasibility_pump.cu:489
    #11 0xf2cf215517a4 in cuopt::linear_programming::detail::local_search_t<int, double>::run_fp(cuopt::linear_programming::detail::solution_t<int, double>&, cuopt::timer_t, cuopt::linear_programming::detail::population_t<int, double>*) /cuopt-dev/cuopt/cpp/src/mip/local_search/local_search.cu:739
    #12 0xf2cf2152f244 in cuopt::linear_programming::detail::diversity_manager_t<int, double>::run_fp_alone() /cuopt-dev/cuopt/cpp/src/mip/diversity/diversity_manager.cu:271
    #13 0xf2cf215375d4 in cuopt::linear_programming::detail::diversity_manager_t<int, double>::run_solver() /cuopt-dev/cuopt/cpp/src/mip/diversity/diversity_manager.cu:415
    #14 0xf2cf215277e8 in cuopt::linear_programming::detail::mip_solver_t<int, double>::run_solver() /cuopt-dev/cuopt/cpp/src/mip/solver.cu:215
    #15 0xf2cf2152096c in cuopt::linear_programming::mip_solution_t<int, double> cuopt::linear_programming::run_mip<int, double>(cuopt::linear_programming::detail::problem_t<int, double>&, cuopt::linear_programming::mip_solver_settings_t<int, double> const&, cuopt::timer_t&) /cuopt-dev/cuopt/cpp/src/mip/solve.cu:127
    #16 0xf2cf21521664 in cuopt::linear_programming::mip_solution_t<int, double> cuopt::linear_programming::solve_mip<int, double>(cuopt::linear_programming::optimization_problem_t<int, double>&, cuopt::linear_programming::mip_solver_settings_t<int, double> const&) /cuopt-dev/cuopt/cpp/src/mip/solve.cu:235
    #17 0xf2cf21523394 in cuopt::linear_programming::mip_solution_t<int, double> cuopt::linear_programming::solve_mip<int, double>(raft::handle_t const*, cuopt::mps_parser::mps_data_model_t<int, double> const&, cuopt::linear_programming::mip_solver_settings_t<int, double> const&) /cuopt-dev/cuopt/cpp/src/mip/solve.cu:301
    #18 0xb2e293ba9804 in run_single_file(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int, int, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::optional<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, bool, int, bool, bool, double) /cuopt-dev/cuopt/benchmarks/linear_programming/cuopt/run_mip.cpp:210
    #19 0xb2e293b7575c in main /cuopt-dev/cuopt/benchmarks/linear_programming/cuopt/run_mip.cpp:496
    #20 0xf2cf1e4a60d8 in __libc_start_call_main (/lib64/libc.so.6+0x260d8) (BuildId: 3e8a85223cce4c8863fe93a9ca2395636660eac3)
    #21 0x7af2cf1e4a61b8  (<unknown module>)
    #22 0x5bb2e293b7f29c  (<unknown module>)

The matrix and vectors descriptors for cusparse are allocated in update_solution, but are never deallocated. Can we convert this to a C++ object in order to use RAII (maybe a shared_ptr or unique_ptr with custom allocator and deallocator)? @Kh4ster @aliceb-nv

[Kh4ster]

Thanks for opening an issue for this, it's being fixed here: #590