Table of Contents

1. Recommended Environment

2. Dataset Preparation

3. Docker Env Setup

4. Train and Test

5. Models

6. Add Custom Model, Loss Function, and Dataset

7. Registration results

8. Useful Commands

9. Acknowledgments

Recommended Environment

System Requirements

• Ubuntu 22.04
• Python >= 3.11
• CUDA 12.4
• Rest all of the environment requirements are as mentioned in the Dockerfile

Dataset preparation

Store all the datasets in one folder data

Dataset used in this thesis:

• MANTruckScenes

Other Datasets that can be used

• A2D2
• NuScenes
• KITTI

MANTruckScenes dataset folder structure

data
├── truckscenes
│   ├── samples
│   │   ├──LIDAR_LEFT
│   │   ├──LIDAR_RIGHT
│   ├── sweeps
│   ├── v1.0-trainval
├── truckscenes_test
│   ├── samples
│   │   ├──LIDAR_LEFT
│   │   ├──LIDAR_RIGHT
│   ├── sweeps
│   ├── v1.0-test
├── truckscenes_mini
│   ├── samples
│   │   ├──LIDAR_LEFT
│   │   ├──LIDAR_RIGHT
│   ├── sweeps
│   ├── v1.0-mini

Docker Env Setup

cd ~/pcd_reg_hregnet

docker build -t pcd_calib:latest -f Dockerfile .

cd ..

The following docker command creates a container named pcd_reg with follwing volumes attached.

• pcd_reg_hregnet
• data (the folder with all the datasets) Replace your_dataset_path with your local dataset folder path

docker run --privileged --name pcd_reg --gpus device='all,"capabilities=compute,utility,graphics"' -it --network host --shm-size 64GB -v $(pwd)/pcd_reg_hregnet:/workspace -v /your_dataset_path/:/workspace/data pcd_calib:latest /bin/bash

Once inside the contatiner, install point_utils package

cd /workspace/Models/HRegNet/PointUtils

pip install .

Login to wandb from terminal to setup logging -

wandb login

Train and Test

To train a model, run the following command from /workspace:

sh scripts/train_man_registration.sh

To test the model on a test dataset, use:

sh scripts/test_man.sh

Before running, ensure the relevant hyperparameters and directory paths are updated in the respective .sh files.

Models

• HRegNet - Original HRegNet model

• Model V1 - HRegNet with MI added before coarse registration layer

• Model V2 - HRegNet with MI added after coarse registration layer (aka Adaption 1 / A1)

• Model V3 - Model V2 with regression head

• Model V4 - Model V3 with Overlap loss + MI Loss

• Model V5 - Unused custom model with self and cross attention between point clouds

• Model V6 - Backbone changed to Point Transformer V3(Ptv3) (aka Adaption 2 / A2, Encoder only Ptv3 used with no pooling)

• Model V6a - Backbone changed to Ptv3 (Encoder Decoder setup with Serialized Pooling)

• Caution: Ptv3 uses flash attention which works only on GPUs newer than Ampere architecture. If working with GPU older than Ampere, set enable_flash=False inside __init__() of Class PointTransformerV3 in models/model_v6/ptv3_mod.py

Registration results

Initial decalibration of ±20°, ±0.50m on the source point cloud

Model	Rotation [deg]		Translation [m]
	MAE	STD	MAE	STD
HRegNet	0.133	0.053	0.058	0.144
A1	0.142	0.046	0.054	0.212
A2	0.144	0.042	0.049	0.139
A2 + Point-to-Point ICP	0.100	0.583	0.286	0.120
A2 + Point-to-Plane ICP	0.078	0.109	0.191	0.057
Point-to-Point ICP	0.394	1.301	0.386	0.698
Point-to-Plane ICP	0.302	1.401	0.354	0.787

Sample Scene

Target(left) and source(right) point clouds of the above sample scene from Bird's Eye View

Add Custom Model, Loss Function, and Dataset

This framework supports easy integration of custom models, loss functions, and datasets. Follow these steps:

1. Add Model:

   • Create a folder for your model under models/ (e.g., models/custom_model/).
   • Implement your model in this folder.

2. Add Loss Function:

   • Implement your custom loss function in the losses/ directory.

3. Add Dataset:

   • Place your dataset in the data/ folder.
   • Create a dataloader in dataset/ and update dataset/config.json with the dataset configuration.
   • Import your dataloader in data_loader.py.

4. Update Training Script:

   • Copy train_reg_vx.py and modify it to include your custom model, loss function, and dataloader.
   • Adjust the training logic as needed.

Useful Commands:

• Check GPU:
  nvidia-smi, watch nvidia-smi, nvcc --version

• Test Internet Connection inside container:
  ping google.com

• Restart Docker:
  sudo systemctl restart docker

• Verify CUDA:

   import torch
   torch.cuda.is_available()

• Disk Usage:
  df -h, du -sh /home

Using tmux for Multiple Training Sessions

When running a container, it typically operates with a single terminal in the background (not the VS Code terminal), even after closing the VS Code GUI. To run multiple training sessions simultaneously, tmux can be very helpful.

Common tmux Commands:

• Create a new session:

   tmux new -s <session_name>

• Detach from a session without terminating it:
  Press Ctrl + B, then press D.
• List all active sessions:

   tmux list-sessions

• Reattach to a detached session:

   tmux attach -t <session_name>

• Terminate a session completely:

   tmux kill-session -t <session_name>

Acknowledgments

• HRegNet: HRegNet

• PointTransformerV3: PTv3

• Mutual Information Loss: MMI_PCN

• Chamfer Distance: otaheri/chamfer_distance

• Overlap Loss: GeoTransformer

• Metrics and Visualize: CalibViT