Skip to content

Commit 24809d1

Browse files
author
Benedikt Mersch
authored
Major Upgrade after HeLiMOS (#38)
* First draft * update readme * update readme * WIP: visualizer * Add polyscope visualizer * Propagate parameters * Fix this * Small change * Limit mos range * Update readme * Update * This fixes error * Minor changes * Format * Renaming * Fix
1 parent b04651a commit 24809d1

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

64 files changed

+3590
-2036
lines changed

‎.gitignore‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# Project related
2-
pretrained/
3-
predictions/
4-
logs/
2+
results/
3+
.polyscope.ini
4+
imgui.ini
55

66
# Byte-compiled / optimized / DLL files
77
__pycache__/

‎CITATION.cff‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
cff-version: 1.2.0
2+
preferred-citation:
3+
title: "Receding Moving Object Segmentation in 3D LiDAR Data Using Sparse 4D Convolutions"
4+
doi: "110.1109/LRA.2022.3183245"
5+
year: "2022"
6+
type: article
7+
journal: "IEEE Robotics and Automation Letters (RA-L)"
8+
url: https://www.ipb.uni-bonn.de/pdfs/mersch2022ral.pdf
9+
codeurl: https://github.com/PRBonn/4DMOS
10+
authors:
11+
- family-names: Mersch
12+
given-names: Benedikt
13+
- family-names: Chen
14+
given-names: Xieyuanli
15+
- family-names: Vizzo
16+
given-names: Ignacio
17+
- family-names: Nunes
18+
given-names: Lucas
19+
- family-names: Behley
20+
given-names: Jens
21+
- family-names: Stachniss
22+
given-names: Cyrill

‎Dockerfile‎

Lines changed: 0 additions & 34 deletions
This file was deleted.

‎Makefile‎

Lines changed: 9 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -1,27 +1,12 @@
1-
export USER_ID:=$(shell id -u)
2-
export GROUP_ID:=$(shell id -g)
1+
install:
2+
@pip install -v .
33

4-
build:
5-
@echo Build docker image...
6-
@DOCKER_BUILDKIT=0 docker compose build project
4+
install-all:
5+
@pip install -v ".[all]"
76

8-
test: check-env
9-
@echo NVIDIA and CUDA setup
10-
@docker compose run project nvidia-smi
11-
@echo Pytorch CUDA setup installed?
12-
@docker compose run project python3 -c "import torch; print(torch.cuda.is_available())"
13-
@echo MinkowskiEngine installed?
14-
@docker compose run project python3 -c "import MinkowskiEngine as ME; print(ME.__version__)"
7+
uninstall:
8+
@pip -v uninstall mos4d
159

16-
run: check-env
17-
@docker compose run project
18-
19-
clean:
20-
@echo Removing docker image...
21-
@docker compose rm project
22-
23-
24-
check-env:
25-
ifndef DATA
26-
$(error Please specify where your data is located, export DATA=<path>)
27-
endif
10+
editable:
11+
@pip install scikit-build-core pyproject_metadata pathspec pybind11
12+
@pip install --no-build-isolation -ve .

‎README.md‎

Lines changed: 160 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,160 @@
1+
<div align="center">
2+
<h1>Receding Moving Object Segmentation in 3D LiDAR Data Using Sparse 4D Convolutions</h1>
3+
<a href="https://github.com/PRBonn/4DMOS#how-to-use-it"><img src="https://img.shields.io/badge/python-3670A0?style=flat-square&logo=python&logoColor=ffdd54" /></a>
4+
<a href="https://github.com/PRBonn/4DMOS#installation"><img src="https://img.shields.io/badge/Linux-FCC624?logo=linux&logoColor=black" /></a>
5+
<a href="https://www.ipb.uni-bonn.de/pdfs/mersch2022ral.pdf"><img src="https://img.shields.io/badge/Paper-pdf-<COLOR>.svg?style=flat-square" /></a>
6+
<a href="LICENSE"><img src="https://img.shields.io/badge/License-MIT-blue.svg?style=flat-square" /></a>
7+
</div>
8+
9+
![example](docs/4dmos.gif)
10+
*Our moving object segmentation on the unseen SemanticKITTI test sequences 18 and 21. Red points are predicted as moving.*
11+
12+
Please find the corresponding video [here](https://youtu.be/5aWew6caPNQ).
13+
14+
<p align="center">
15+
<img src="docs/introduction.png" width="600">
16+
</p>
17+
18+
*Given a sequence of point clouds, our method segments moving (red) from non-moving (black) points.*
19+
20+
<p align="center">
21+
<img src="docs/architecture.png">
22+
</p>
23+
24+
*We first create a sparse 4D point cloud of all points in a given receding window. We use sparse 4D convolutions from the [MinkowskiEngine](https://github.com/NVIDIA/MinkowskiEngine) to extract spatio-temporal features and predict per-points moving object scores.*
25+
26+
## Important Update
27+
The current state of the repository is improved by internally aligning the scans using [KISS-ICP](https://github.com/PRBonn/kiss-icp). Also, the build system and pipeline are inspired from [MapMOS](https://github.com/PRBonn/MapMOS), so you can run it on most point cloud data formats. If you want to reproduce the original results from the paper, this version is tagged under `0.1`. You can checkout by
28+
29+
```bash
30+
git checkout v0.1
31+
```
32+
33+
## Installation
34+
First, make sure the [MinkowskiEngine](https://github.com/NVIDIA/MinkowskiEngine) is installed on your system, see [here](https://github.com/NVIDIA/MinkowskiEngine#installation) for more details.
35+
36+
Next, clone our repository
37+
```bash
38+
git clone [email protected]:PRBonn/4DMOS && cd 4DMOS
39+
```
40+
41+
and install with
42+
```bash
43+
make install
44+
```
45+
46+
**or**
47+
```bash
48+
make install-all
49+
```
50+
if you want to install the project with all optional dependencies (needed for the visualizer). In case you want to edit the Python code, install in editable mode:
51+
```bash
52+
make editable
53+
```
54+
55+
## How to Use It
56+
Just type
57+
58+
```bash
59+
mos4d_pipeline --help
60+
```
61+
to see how to run 4DMOS.
62+
63+
Check the [Download](#downloads) section for a pre-trained model. Like [KISS-ICP](https://github.com/PRBonn/kiss-icp), our pipeline runs on a variety of point cloud data formats like `bin`, `pcd`, `ply`, `xyz`, `rosbags`, and more. To visualize these, just type
64+
65+
```bash
66+
mos4d_pipeline --visualize /path/to/weights.ckpt /path/to/data
67+
```
68+
69+
<details>
70+
<summary>Want to evaluate with ground truth labels?</summary>
71+
72+
Because these labels come in all shapes, you need to specify a dataloader. This is currently available for SemanticKITTI, NuScenes, HeLiMOS, and our labeled KITTI Tracking sequence 19 and Apollo sequences (see [Downloads](#downloads)).
73+
74+
</details>
75+
76+
## Training
77+
To train our approach, you need to first cache your data. To see how to do that, just `cd` into the `4DMOS` repository and type
78+
79+
```bash
80+
python3 scripts/precache.py --help
81+
```
82+
83+
After this, you can run the training script. Again, `--help` shows you how:
84+
```bash
85+
python3 scripts/train.py --help
86+
```
87+
88+
<details>
89+
<summary>Want to verify the cached data?</summary>
90+
91+
You can inspect the cached training samples by using the script `python3 scripts/cache_to_ply.py --help`.
92+
93+
</details>
94+
95+
<details>
96+
<summary>Want to change the logging directory?</summary>
97+
98+
The training log and checkpoints will be saved by default to the current working directory. To change that, export the `export LOGS=/your/path/to/logs` environment variable before running the training script.
99+
100+
</details>
101+
102+
## HeLiMOS
103+
To train on the HeLiMOS data with different sensor configurations, use the following commands:
104+
105+
```shell
106+
python3 scripts/precache.py /path/to/HeLiMOS helimos /path/to/cache --config config/helimos/*_training.yaml
107+
python3 scripts/train.py /path/to/HeLiMOS helimos /path/to/cache --config config/helimos/*_training.yaml
108+
```
109+
110+
by replacing the paths and the config file names. To evaluate for example on the Velodyne test data, run
111+
112+
```shell
113+
mos4d_pipeline /path/to/weights.ckpt /path/to/HeLiMOS --dataloader helimos -s Velodyne/test.txt
114+
```
115+
116+
## Evaluation and Visualization
117+
We use the [SemanticKITTI API](https://github.com/PRBonn/semantic-kitti-api) to evaluate the intersection-over-union (IOU) of the moving class as well as to visualize the predictions. Clone the repository in your workspace, install the dependencies and then run the following command to visualize your predictions for e.g. sequence 8:
118+
119+
```
120+
cd semantic-kitti-api
121+
./visualize_mos.py --sequence 8 --dataset /path/to/dataset --predictions /path/to/4DMOS/predictions/ID/POSES/labels/STRATEGY/
122+
```
123+
124+
## Benchmark
125+
To submit the results to the LiDAR-MOS benchmark, please follow the instructions [here](https://competitions.codalab.org/competitions/28894).
126+
127+
## Downloads
128+
<p align="center">
129+
<img src="docs/table.png" width="600">
130+
</p>
131+
132+
* [Model [A]: 5 scans @ 0.1s](https://www.ipb.uni-bonn.de/html/projects/4DMOS/5_scans.zip)
133+
* [Model [B]: 5 scans @ 0.2s](https://www.ipb.uni-bonn.de/html/projects/4DMOS/5_scans_dt_0p2.zip)
134+
* [Model [C]: 5 scans @ 0.3s](https://www.ipb.uni-bonn.de/html/projects/4DMOS/5_scans_dt_0p3.zip)
135+
* [Model [D]: 5 scans, no poses](https://www.ipb.uni-bonn.de/html/projects/4DMOS/5_scans_no_poses.zip)
136+
* [Model [E]: 5 scans input, 1 scan output](https://www.ipb.uni-bonn.de/html/projects/4DMOS/5_scans_single_output.zip)
137+
* [Model [F]: 2 scans](https://www.ipb.uni-bonn.de/html/projects/4DMOS/2_scans.zip)
138+
* [Model [G]: 10 scans](https://www.ipb.uni-bonn.de/html/projects/4DMOS/10_scans.zip)
139+
140+
## Publication
141+
If you use our code in your academic work, please cite the corresponding [paper](https://www.ipb.uni-bonn.de/pdfs/mersch2022ral.pdf):
142+
143+
```bibtex
144+
@article{mersch2022ral,
145+
author = {B. Mersch and X. Chen and I. Vizzo and L. Nunes and J. Behley and C. Stachniss},
146+
title = {{Receding Moving Object Segmentation in 3D LiDAR Data Using Sparse 4D Convolutions}},
147+
journal={IEEE Robotics and Automation Letters (RA-L)},
148+
year = 2022,
149+
volume = {7},
150+
number = {3},
151+
pages = {7503--7510},
152+
codeurl = {https://github.com/PRBonn/4DMOS},
153+
}
154+
```
155+
156+
## Acknowledgments
157+
This implementation is heavily inspired by [KISS-ICP](https://github.com/PRBonn/kiss-icp).
158+
159+
## License
160+
This project is free software made available under the MIT License. For details see the LICENSE file.

0 commit comments

Comments
 (0)