You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Point cloud computation has become an increasingly more important workload for autonomous driving and other applications. Unlike dense 2D computation, point cloud convolution has **sparse** and **irregular** computation patterns and thus requires dedicated inference system support with specialized high-performance kernels. While existing point cloud deep learning libraries have developed different dataflows for convolution on point clouds, they assume a single dataflow throughout the execution of the entire model. In this work, we systematically analyze and improve existing dataflows. Our resulting system, TorchSparse, achieves **2.9x**, **3.3x**, **2.2x** and **1.7x** measured end-to-end speedup on an NVIDIA A100 GPU over the state-of-the-art MinkowskiEngine, SpConv 1.2, TorchSparse (MLSys) and SpConv v2 in inference respectively.
6
11
7
12
## Installation
8
13
9
-
TorchSparse depends on the [Google Sparse Hash](https://github.com/sparsehash/sparsehash) library.
14
+
We provide pre-built torchsparse v2.1.0 packages (recommended) with different PyTorch and CUDA versions to simplify the building for the Linux system.
- You can also compile the library locally (if you do not have the sudo permission) and add the library path to the environment variable `CPLUS_INCLUDE_PATH`.
30
+
1. Then the right TorchSparse wheel can be found and installed by running the installation script:
24
31
25
-
The latest released TorchSparse (v1.4.0) can then be installed by
If Pypi server does not work as expected, no worries, you can still manually download the wheels. The wheels are listed in [this website](http://pypi.hanlab.ai/simple/torchsparse). One can utilize our installation script to automatically determine the version number used to index the wheels. For example, if you use PyTorch 1.11.0, CUDA 11.5, the version number will end up to be 2.1.0+torch111cu115. You can then select the proper wheel according to your Python version.
30
38
31
-
If you use TorchSparse in your code, please remember to specify the exact version in your dependencies.
39
+
## Benchmarks
32
40
33
-
For installation help and troubleshooting, please consult the [Frequently Asked Questions](./docs/FAQ.md) before posting an issue.
We compare TorchSparse with [MinkowskiEngine](https://github.com/NVIDIA/MinkowskiEngine) (where the latency is measured on NVIDIA GTX 1080Ti):
45
+
TorchSparse significantly outperforms existing point cloud inference engines in both 3D object detection and LiDAR segmentation benchmarks across three generations of GPU architecture (Pascal, Turing and Ampere) and all precisions (FP16, TF32, FP32). It is up to **1.7x** faster than state-of-the-art SpConv 2.3.5 and is up to **2.2x** faster than
46
+
TorchSparse-MLsys on cloud GPUs. It also improves the latency of SpConv 2.3.5 by **1.25×** on Orin.
TorchSparse achieves superior mixed-precision training speed compared with MinkowskiEngine, TorchSparse-MLSys and SpConv 2.3.5. Specifically, it is **1.16x** faster on Tesla A100, **1.27x** faster on RTX 2080 Ti than state-of-the-art SpConv 2.3.5. It also significantly outperforms MinkowskiEngine by **4.6-4.8x*** across seven benchmarks on A100 and 2080 Ti. Measured with batch size = 2.
49
53
50
-
Sparse tensor (`SparseTensor`) is the main data structure for point cloud, which has two data fields:
51
54
52
-
- Coordinates (`coords`): a 2D integer tensor with a shape of N x 4, where the first three dimensions correspond to quantized x, y, z coordinates, and the last dimension denotes the batch index.
53
-
- Features (`feats`): a 2D tensor with a shape of N x C, where C is the number of feature channels.
55
+
## Team
54
56
55
-
Most existing datasets provide raw point cloud data with float coordinates. We can use `sparse_quantize` (provided in `torchsparse.utils.quantize`) to voxelize x, y, z coordinates and remove duplicates:
57
+
TorchSparse is developed by the following wonderful team:
56
58
57
-
```python
58
-
coords -= np.min(coords, axis=0, keepdims=True)
59
-
coords, indices = sparse_quantize(coords, voxel_size, return_index=True)
-[Yujun Lin](https://yujunlin.com/): Ph.D. student (2018-) at MIT EECS, v2.0 core developer;
66
+
-[Guohao Dai](https://scholar.google.com/citations?user=gz3Tkl0AAAAJ&hl=en): Associate Professor at Shanghai Jiao Tong University, mentor of the project;
67
+
-[Yu Wang](http://nicsefc.ee.tsinghua.edu.cn/): Professor at Tsinghua University, mentor of the project;
68
+
-[Song Han](https://songhan.mit.edu): Associate Professor at MIT EECS, mentor of the project.
64
69
65
-
We can then use `sparse_collate_fn` (provided in `torchsparse.utils.collate`) to assemble a batch of `SparseTensor`'s (and add the batch dimension to `coords`). Please refer to [this example](./examples/example.py) for more details.
66
70
67
-
### Sparse Neural Network
71
+
##Citation
68
72
69
-
The neural network interface in TorchSparse is very similar to PyTorch:
73
+
If you use TorchSparse, please use the following BibTeX entries to cite:
70
74
71
-
```python
72
-
from torch import nn
73
-
from torchsparse import nn as spnn
75
+
Preliminary version of TorchSparse++ (TorchSparse v2.1) is presented at CVPR Workshops 2023:
title = {{TorchSparse++: Efficient Point Cloud Engine}},
80
+
author = {Tang, Haotian and Yang, Shang and Liu, Zhijian and Hong, Ke and Yu, Zhongming and Li, Xiuyu and Dai, Guohao and Wang, Yu and Han, Song},
81
+
booktitle = {Computer Vision and Pattern Recognition Workshops (CVPRW)},
82
+
year = {2023}
83
+
}
80
84
```
81
85
82
-
## Citation
83
-
84
-
If you use TorchSparse in your research, please use the following BibTeX entries:
86
+
TorchSparse is presented at MLSys 2022:
85
87
86
88
```bibtex
87
89
@inproceedings{tang2022torchsparse,
@@ -92,6 +94,8 @@ If you use TorchSparse in your research, please use the following BibTeX entries
92
94
}
93
95
```
94
96
97
+
Initial version of TorchSparse is part of the SPVNAS paper at ECCV 2020:
98
+
95
99
```bibtex
96
100
@inproceedings{tang2020searching,
97
101
title = {{Searching Efficient 3D Architectures with Sparse Point-Voxel Convolution}},
@@ -101,6 +105,19 @@ If you use TorchSparse in your research, please use the following BibTeX entries
101
105
}
102
106
```
103
107
104
-
## Acknowledgements
108
+
PCEngine paper is accepted by MLSys 2023:
109
+
110
+
```bibtex
111
+
@inproceedings{hong2023pcengine,
112
+
title={{Exploiting Hardware Utilization and Adaptive Dataflow for Efficient Sparse Convolution in 3D Point Clouds}},
113
+
author={Hong, Ke and Yu, Zhongming and Dai, Guohao and Yang, Xinhao and Lian, Yaoxiu and Liu, Zehao and Xu, Ningyi and Wang, Yu},
114
+
booktitle={Sixth Conference on Machine Learning and Systems (MLSys)},
115
+
year={2023}
116
+
}
117
+
```
118
+
119
+
## Acknowledgement
120
+
121
+
We thank Yan Yan from TuSimple for helpful discussions.
105
122
106
-
TorchSparse is inspired by many existing open-source libraries, including (but not limited to) [MinkowskiEngine](https://github.com/NVIDIA/MinkowskiEngine), [SECOND](https://github.com/traveller59/second.pytorch)and [SparseConvNet](https://github.com/facebookresearch/SparseConvNet).
123
+
Please also have a look at the [dgSparse](https://dgsparse.github.io/) library, which is designed for fast and efficient sparse computation on graphs and point clouds. The work from PCEngine (MLSys 2023) team is also highly related to us.
0 commit comments