Skip to content

Commit 148a085

Browse files
JingweiZhang12JingweiZhang12
authored
[Feature] Support downloading datasets from OpenDataLab using mim (#2593)
* init commit * add dataset unzip scripts * polish docs * polish docs
1 parent f40d8d2 commit 148a085

File tree

8 files changed

+118
-8
lines changed

8 files changed

+118
-8
lines changed

‎MANIFEST.in

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
include mmdet3d/.mim/model-index.yml
2+
include mmdet3d/.mim/dataset-index.yml
23
include requirements/*.txt
34
recursive-include mmdet3d/.mim/ops *.cpp *.cu *.h *.cc
45
recursive-include mmdet3d/.mim/configs *.py *.yml

‎dataset-index.yml

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
kitti:
2+
# The name of dataset in OpenDataLab referring to
3+
# https://opendatalab.com/KITTI_Object/cli. You can also download it
4+
# by running `odl get ${dataset}` independently
5+
dataset: KITTI_Object
6+
download_root: data
7+
data_root: data/kitti
8+
# Scripts for unzipping datasets
9+
script: tools/dataset_converters/kitti_unzip.sh
10+
11+
nuscenes:
12+
# The name of dataset in OpenDataLab referring to
13+
# https://opendatalab.com/nuScenes/cli. You can also download it
14+
# by running `odl get ${dataset}` independently
15+
dataset: nuScenes
16+
download_root: data
17+
data_root: data/nuscenes
18+
# Scripts for unzipping datasets
19+
script: tools/dataset_converters/nuscenes_unzip.sh
20+
21+
semantickitti:
22+
# The name of dataset in OpenDataLab referring to
23+
# https://opendatalab.com/SemanticKITTI/cli. You can also download it
24+
# by running `odl get ${dataset}` independently
25+
dataset: SemanticKITTI
26+
download_root: data
27+
data_root: data/semantickitti
28+
# Scripts for unzipping datasets
29+
script: tools/dataset_converters/semantickitti_unzip.sh

‎docs/en/user_guides/dataset_prepare.md

Lines changed: 42 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -86,7 +86,20 @@ mmdetection3d
8686

8787
### KITTI
8888

89-
Download KITTI 3D detection data [HERE](http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=3d). Prepare KITTI data splits by running:
89+
1. Download KITTI 3D detection data [HERE](http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=3d). Alternatively, you
90+
can download the dataset from [OpenDataLab](https://opendatalab.com/) using MIM. The command scripts are the following:
91+
92+
```bash
93+
# install OpenDataLab CLI tools
94+
pip install -U opendatalab
95+
# log in OpenDataLab. Note that you should register an account on [OpenDataLab](https://opendatalab.com/) before.
96+
pip install odl
97+
odl login
98+
# download and preprocess by MIM
99+
mim download mmdet3d --dataset kitti
100+
```
101+
102+
2. Prepare KITTI data splits by running:
90103

91104
```bash
92105
mkdir ./data/kitti/ && mkdir ./data/kitti/ImageSets
@@ -98,7 +111,7 @@ wget -c https://raw.githubusercontent.com/traveller59/second.pytorch/master/sec
98111
wget -c https://raw.githubusercontent.com/traveller59/second.pytorch/master/second/data/ImageSets/trainval.txt --no-check-certificate --content-disposition -O ./data/kitti/ImageSets/trainval.txt
99112
```
100113

101-
Then generate info files by running:
114+
3. Generate info files by running:
102115

103116
```bash
104117
python tools/create_data.py kitti --root-path ./data/kitti --out-dir ./data/kitti --extra-tag kitti
@@ -160,7 +173,20 @@ Note that:
160173

161174
### NuScenes
162175

163-
Download nuScenes V1.0 full dataset data [HERE](https://www.nuscenes.org/download). Prepare nuscenes data by running:
176+
1. Download nuScenes V1.0 full dataset data [HERE](https://www.nuscenes.org/download). Alternatively, you
177+
can download the dataset from [OpenDataLab](https://opendatalab.com/) using MIM. The downloading and unzipping command scripts are the following:
178+
179+
```bash
180+
# install OpenDataLab CLI tools
181+
pip install -U opendatalab
182+
# log in OpenDataLab. Note that you should register an account on [OpenDataLab](https://opendatalab.com/) before.
183+
pip install odl
184+
odl login
185+
# download and preprocess by MIM
186+
mim download mmdet3d --dataset nuscenes
187+
```
188+
189+
2. Prepare nuscenes data by running:
164190

165191
```bash
166192
python tools/create_data.py nuscenes --root-path ./data/nuscenes --out-dir ./data/nuscenes --extra-tag nuscenes
@@ -187,9 +213,20 @@ Note that we follow the original folder names for clear organization. Please ren
187213

188214
### SemanticKITTI
189215

190-
Download SemanticKITTI dataset [HERE](http://semantic-kitti.org/dataset.html#download) and unzip all zip files.
216+
1. Download SemanticKITTI dataset [HERE](http://semantic-kitti.org/dataset.html#download) and unzip all zip files. Alternatively, you
217+
can download the dataset from [OpenDataLab](https://opendatalab.com/) using MIM. The downloading and unzipping command scripts are the following:
218+
219+
```bash
220+
# install OpenDataLab CLI tools
221+
pip install -U opendatalab
222+
# log in OpenDataLab. Note that you should register an account on [OpenDataLab](https://opendatalab.com/) before.
223+
pip install odl
224+
odl login
225+
# download and preprocess by MIM
226+
mim download mmdet3d --dataset semantickitti
227+
```
191228

192-
Then generate info files by running:
229+
2. Generate info files by running:
193230

194231
```bash
195232
python ./tools/create_data.py semantickitti --root-path ./data/semantickitti --out-dir ./data/semantickitti --extra-tag semantickitti

‎projects/BEVFusion/configs/bevfusion_lidar-cam_voxel0075_second_secfpn_8xb4-cyclic-20e_nus-3d.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -127,7 +127,7 @@
127127
'ori_lidar2img', 'img_aug_matrix', 'box_type_3d', 'sample_idx',
128128
'lidar_path', 'img_path', 'transformation_3d_flow', 'pcd_rotation',
129129
'pcd_scale_factor', 'pcd_trans', 'img_aug_matrix',
130-
'lidar_aug_matrix'
130+
'lidar_aug_matrix', 'num_pts_feats'
131131
])
132132
]
133133

@@ -168,7 +168,7 @@
168168
meta_keys=[
169169
'cam2img', 'ori_cam2img', 'lidar2cam', 'lidar2img', 'cam2lidar',
170170
'ori_lidar2img', 'img_aug_matrix', 'box_type_3d', 'sample_idx',
171-
'lidar_path', 'img_path'
171+
'lidar_path', 'img_path', 'num_pts_feats'
172172
])
173173
]
174174

‎setup.py

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -158,7 +158,9 @@ def add_mim_extention():
158158
else:
159159
return
160160

161-
filenames = ['tools', 'configs', 'demo', 'model-index.yml']
161+
filenames = [
162+
'tools', 'configs', 'demo', 'model-index.yml', 'dataset-index.yml'
163+
]
162164
repo_path = osp.dirname(__file__)
163165
mim_path = osp.join(repo_path, 'mmdet3d', '.mim')
164166
os.makedirs(mim_path, exist_ok=True)

‎tools/dataset_converters/kitti_unzip.sh

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
#!/usr/bin/env bash
2+
3+
DOWNLOAD_DIR=$1 # The directory where the downloaded data set is stored
4+
DATA_ROOT=$2 # The root directory of the converted dataset
5+
6+
for zip_file in $DOWNLOAD_DIR/KITTI_Object/raw/*.zip; do
7+
echo "Unzipping $zip_file to $DATA_ROOT ......"
8+
unzip -oq $zip_file -d $DATA_ROOT
9+
echo "[Done] Unzip $zip_file to $DATA_ROOT"
10+
# delete the original files
11+
rm -f $zip_file
12+
done

‎tools/dataset_converters/nuscenes_unzip.sh

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
#!/usr/bin/env bash
2+
3+
DOWNLOAD_DIR=$1 # The directory where the downloaded data set is stored
4+
DATA_ROOT=$2 # The root directory of the converted dataset
5+
6+
for split in $DOWNLOAD_DIR/nuScenes/raw/*; do
7+
for tgz_file in $split/*; do
8+
if [[ $tgz_file == *.tgz ]]
9+
then
10+
echo "Unzipping $tgz_file to $DATA_ROOT ......"
11+
unzip -oq $tgz_file -d $DATA_ROOT/
12+
echo "[Done] Unzip $tgz_file to $DATA_ROOT"
13+
fi
14+
# delete the original files
15+
rm -f $tgz_file
16+
done
17+
done

‎tools/dataset_converters/semantickitti_unzip.sh

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
#!/usr/bin/env bash
2+
3+
DOWNLOAD_DIR=$1 # The directory where the downloaded data set is stored
4+
DATA_ROOT=$2 # The root directory of the converted dataset
5+
6+
for zip_file in $DOWNLOAD_DIR/SemanticKITTI/raw/*.zip; do
7+
echo "Unzipping $zip_file to $DATA_ROOT ......"
8+
unzip -oq $zip_file -d $DATA_ROOT
9+
echo "[Done] Unzip $zip_file to $DATA_ROOT"
10+
# delete the original files
11+
rm -f $zip_file
12+
done

0 commit comments

Comments
 (0)