rtabmap with libtorch error

[calmonpray]

I use edited rtabmap in docker ubuntu 18 from https://github.com/MI-BioLab/continual-learning-regions
It adds continual learning module and need libtorch to build. I try to reproduce the experiment following the readme:
using libtorch-shared-with-deps-1.13.1 and cmake -DOpenCV_DIR=/usr/local/lib/cmake/opencv4 from the readme to install rtabmap but fails.
almost every lib cannot link.
while using libtorch with cxx11 ABI version and build successfully.

cmake -DWITH_TORCH=ON \
      -DCMAKE_PREFIX_PATH=/root/SLAM/programs/rtabmap/libtorch \
      -DOpenCV_DIR=/usr/local/lib/cmake/opencv4 ..

while running the launch or just run rtabmap, error:
gdb result

Thread 1 "rtabmap" received signal SIGSEGV, Segmentation fault.
0x00007fff89786c1b in llvm::PMTopLevelManager::schedulePass(llvm::Pass*) ()
from /usr/lib/x86_64-linux-gnu/libLLVM-10.so.1
(gdb) bt
#0 0x00007fff89786c1b in llvm::PMTopLevelManager::schedulePass(llvm::Pass*) ()
at /usr/lib/x86_64-linux-gnu/libLLVM-10.so.1
https://github.com/MI-BioLab/continual-learning-regions/issues/1 0x00007fff8dab3bab in () at /usr/lib/x86_64-linux-gnu/dri/swrast_dri.so
#2 0x00007fff8dab3c2f in () at /usr/lib/x86_64-linux-gnu/dri/swrast_dri.so
#3 0x00007fff8db1195b in () at /usr/lib/x86_64-linux-gnu/dri/swrast_dri.so
#4 0x00007fff8db05f18 in () at /usr/lib/x86_64-linux-gnu/dri/swrast_dri.so
#5 0x00007fff8daf1750 in () at /usr/lib/x86_64-linux-gnu/dri/swrast_dri.so
#6 0x00007fff8d51ff2f in () at /usr/lib/x86_64-linux-gnu/dri/swrast_dri.so
#7 0x00007fff8d6e0a8e in () at /usr/lib/x86_64-linux-gnu/dri/swrast_dri.so
#8 0x00007fff8d5ab152 in () at /usr/lib/x86_64-linux-gnu/dri/swrast_dri.so
#9 0x00007fff8d5cd925 in () at /usr/lib/x86_64-linux-gnu/dri/swrast_dri.so
#10 0x00007fffeffcd70a in vtkOpenGLDisplayListPainter::RenderInternal(vtkRenderer*, vtkActor*, unsigned long, bool) () at /usr/lib/x86_64-linux-gnu/libvtkRenderingOpenGL-6.3.so.6.3
#11 0x00007fffeffc73eb in vtkOpenGLClipPlanesPainter::RenderInternal(vtkRenderer*, vtkActor*, unsigned long, bool) () at /usr/lib/x86_64-linux-gnu/libvtkRenderingOpenGL-6.3.so.6.3
#12 0x00007ffff003294e in vtkOpenGLScalarsToColorsPainter::RenderInternal(vtkRenderer*, vtkActor*, unsigned long, bool) () at /usr/lib/x86_64-linux-gnu/libvtkRenderingOpenGL-6.3.so.6.3
#13 0x00007ffff004c2f9 in vtkPainterPolyDataMapper::RenderPiece(vtkRenderer*, vtkActor*) ()
at /usr/lib/x86_64-linux-gnu/libvtkRenderingOpenGL-6.3.so.6.3
#14 0x00007fffef320c41 in vtkPolyDataMapper::Render(vtkRenderer*, vtkActor*) ()
at /usr/lib/x86_64-linux-gnu/libvtkRenderingCore-6.3.so.6.3
#15 0x00007fffeffb801b in vtkOpenGLActor::Render(vtkRenderer*, vtkMapper*) ()
at /usr/lib/x86_64-linux-gnu/libvtkRenderingOpenGL-6.3.so.6.3
#16 0x00007ffff0a75729 in vtkLODActor::Render(vtkRenderer*, vtkMapper*) ()
at /usr/lib/x86_64-linux-gnu/libvtkRenderingLOD-6.3.so.6.3
#17 0x00007ffff0a74c0d in vtkLODActor::RenderOpaqueGeometry(vtkViewport*) ()
at /usr/lib/x86_64-linux-gnu/libvtkRenderingLOD-6.3.so.6.3

what can i do? comply the libtorch from source?

[matlabbe]

Are you running this from the docker container? It looks like a display driver issue when launching it from docker. You may try building without libtorch to confirm it is not libtorch related. See also https://github.com/introlab/rtabmap/wiki/Installation#docker to launch with nvidia driver.

[calmonpray]

sorry for my reply late. i run this in docker container. The part related to rtabmap in dockerfile is

RUN /bin/bash -c "cd ~/ && \
                 git clone https://github.com/MI-BioLab/continual-learning-regions.git && \
                 cp -r continual-learning-regions/rtabmap ~/SLAM/programs && \
                 cp -r continual-learning-regions/rtabmap_ros ~/catkin_ws/src && \
                 rm -rf ~/continual-learning-regions"

# RUN /bin/bash -c "cd ~/SLAM/programs/rtabmap && \
#                  mkdir build && \
#                  cd build && \
#                  cmake -DOpenCV_DIR=/usr/local/lib/cmake/opencv4 .. && \
#                  make -j4 && \
#                  make install"

# cv_bridge rebuilded with opencv 4
RUN /bin/bash -c "cd ~/catkin_ws/src && \
                 git clone https://github.com/fizyr-forks/vision_opencv.git && \
                 cd vision_opencv && \
                 git checkout opencv4 && \
                 cd ~/catkin_ws/ && \
                 . /opt/ros/melodic/setup.bash && \
                 catkin_make -j4"

# RUN /bin/bash -c "cd ~/catkin_ws/src/rtabmap_ros && \
#                  . /opt/ros/melodic/setup.bash && \
#                  cd ~/catkin_ws && \
#                  catkin_make -DOpenCV_DIR=/usr/local/lib/cmake/opencv4 -DRTABMAP_SYNC_MULTI_RGBD=ON -DRTABMAP_SYNC_USER_DATA=ON -j4"

RUN ldconfig

it cannot build the image successfully because the rtabmap is not installed yet so i skip the cv_bridge rebuilded step and build the image and the container.

In the container, i install the recommended libtorch(shared-with-deps)1.13.1 and start to build the modified rtabmap, it cannot work. The program builds rtabmap without libtorch originally with

cd path/to/rtabmap
mkdir build
cd build
cmake -DOpenCV_DIR=/usr/local/lib/cmake/opencv4 .. 
make -j4 
make install 

but it cannot work and pose error:

In file included from /home/axolotl/SLAM/programs/rtabmap/corelib/src/Memory.cpp:35:0:
/home/axolotl/SLAM/programs/rtabmap/corelib/src/../include/rtabmap/core/Memory.h:62:7: note: forward declaration of ‘class rtabmap::Region’
 class Region;
       ^~~~~~
/home/axolotl/SLAM/programs/rtabmap/corelib/src/Memory.cpp:6676:42: error: invalid use of incomplete type ‘class rtabmap::Region’
      signature->setRegionId(initialRegion->id());
                                          ^~

It is obvious a modified cpp with new header and you may see the change

#ifdef RTABMAP_TORCH
#include "rtabmap/core/region/clustering/Region.h"
#include "rtabmap/core/region/ContinualLearning.h"
#endif

So I have to add and use libtorch 1.13.1(cxx11 ABI)

cmake -DWITH_TORCH=ON \
      -DCMAKE_PREFIX_PATH=/root/libtorch \
      -DOpenCV_DIR=/usr/local/lib/cmake/opencv4 ..

i have no idea why i have to use -DWITH_TORCH=ON

[matlabbe]

It looks like rtabmap is not built with libtorch, then this preprocessor define RTABMAP_TORCH doesn't exist and cannot include the Region definition.

The current way to know that rtabmap will build with libtorch is to look at the output of cmake:

-- --------------------------------------------
-- Info :
--   RTAB-Map Version =     0.23.2
--   CMAKE_VERSION =        3.22.1
--   CMAKE_INSTALL_PREFIX = /usr/local
--   CMAKE_BUILD_TYPE =     Release
--   CMAKE_INSTALL_LIBDIR = lib
--   BUILD_APP =            ON
--   BUILD_TOOLS =          ON
--   BUILD_EXAMPLES =       ON
--   BUILD_SHARED_LIBS =    ON
--   CMAKE_CXX_FLAGS =  -fmessage-length=0 -fopenmp -std=c++17
--   FLANN_KDTREE_MEM_OPT = OFF
--   PCL_DEFINITIONS = -DDISABLE_OPENNI2;-DDISABLE_PCAP;-DDISABLE_PNG
--   PCL_VERSION = 1.12.1
-- 
-- Optional dependencies ('*' affects some default parameters) :
--  *With OpenCV 4.5.4 xfeatures2d = YES, nonfree = YES (License: Non commercial)
--   With Qt 5.15.3            = YES (License: Open Source or Commercial)
--   With VTK 9.1              = YES (License: BSD)
--   With external SQLite3     = YES (License: Public Domain)
--   With ORB OcTree           = YES (License: GPLv3)
--   With SuperPoint          = NO (WITH_TORCH=OFF)         <----------------------------------------------
...

WITH_TORCH option is OFF by default, you should indeed do cmake -DWITH_TORCH=ON, then make sure it says YES there afterwards. You may have to setup the Torch_DIR like this:

rtabmap/docker/frontiers2022/Dockerfile

Line 74 in f44a4fc

cmake -DTorch_DIR=/opt/conda/lib/python3.8/site-packages/torch/share/cmake/Torch -DWITH_TORCH=ON -DWITH_PYTHON=ON -DWITH_QT=ON .. && \

(adjust actual torch path with the folder containing TorchConfig.cmake)