Skip to content

Commit 943237d

Browse files
voldemortXvoldemortX
committed
Merge branch 'master' of github.com:voldemortX/pytorch-auto-drive
2 parents 571373c + 5ed7a9f commit 943237d

File tree

6 files changed

+27
-21
lines changed

6 files changed

+27
-21
lines changed

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -49,7 +49,7 @@ Models from this repo are faster to train (**single card trainable**) and often
4949
| lane detection | ERFNet, ENet | [SAD](https://arxiv.org/abs/1908.00821) ([*Postponed*](https://github.com/voldemortX/pytorch-auto-drive/wiki/Notes)) |
5050
| lane detection | ERFNet | [PRNet](http://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123630698.pdf) (*In progress*) |
5151
| lane detection | ResNets (18, 34, 50, 101), ResNet18-reduced | [LSTR](https://arxiv.org/abs/2011.04233) |
52-
| lane detection | ResNets (18, 34) | [LaneAtt](https://arxiv.org/abs/2010.12035) |
52+
| lane detection | ResNets (18, 34) | [LaneATT](https://arxiv.org/abs/2010.12035) |
5353
| lane detection | ResNets (18, 34) | [BézierLaneNet](/configs/lane_detection/bezierlanenet) |
5454

5555
## Model Zoo

‎docs/DEPLOY.md‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,12 @@ In this version, TensorRT may use CUDA runtime >= 11, you might avoid using cond
2424

2525
Or you can incrementally install **Extra Dependencies** through the tutorial.
2626

27+
## Important Note
28+
29+
Note that we only convert the model `forward()` function,
30+
post-processing (i.e., `inference()`) is not included. Typical post-processing includes: segmentation map interpolation,
31+
line NMS for anchor-based lane detection, sigmoid/softmax activations, etc.
32+
2733
## PyTorch -> ONNX:
2834

2935
**PyTorch version >= 1.6.0 is recommended for this feature.**
@@ -52,7 +58,7 @@ Same config mechanism and commandline overwrite by `--cfg-options` as in trainin
5258
For detailed instructions and commandline shortcuts available, run:
5359

5460
```
55-
python tools/profiling.py --help
61+
python tools/to_onnx.py --help
5662
```
5763

5864
### Currently Unsupported Models:
@@ -89,4 +95,4 @@ You'll then see the saved `ckpt.engine` file and a report on the conversion qual
8995
- SCNN (lane detection)
9096
- Swin backbone (lane detection)
9197
- DCNv2 in BézierLaneNet (lane detection, could support if built custom op from mmcv and directly convert from PyTorch to TensorRT)
92-
- LaneATT (lane detection)
98+
- LaneATT (supported if TensorRT >= 8.4.1.5)

‎docs/MODEL_ZOO.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@
77
- **level 0**: only small rotation and resize
88
- **level 1a**: the LSTR augmentations
99
- **level 1b**: the BézierLaneNet augmentations
10-
- **level 1c**: the LaneAtt augmentations
10+
- **level 1c**: the LaneATT augmentations
1111

1212
| method | backbone | data<br>augmentation | resolution | mixed precision? | dataset | metric | average | best | training time <br> *(2080 Ti)* |
1313
| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
@@ -73,8 +73,8 @@
7373
| LSTR | ResNet18s-2X<sup>#</sup> | level 0 | 288 x 800 | *no* | CULane | F1 | 36.27 | 39.77 | 28.5h* |
7474
| LSTR | ResNet18s-2X<sup>#</sup> | level 1a | 288 x 800 | *no* | CULane | F1 | 68.35 | 68.72 | 31.5h* |
7575
| LSTR | ResNet34 | level 1a | 288 x 800 | *no* | CULane | F1 | 72.17 | 72.48 | 45.0h* |
76-
| LaneAtt | ResNet18 | level 1c | 360 x 640 | *no* | CULane | F1 | 74.71 | 74.87 | 3.6h** |
77-
| LaneAtt | ResNet34 | level 1c | 360 x 640 | *no* | CULane | F1 | 75.76 | 75.82 | 4.0h** |
76+
| LaneATT | ResNet18 | level 1c | 360 x 640 | *no* | CULane | F1 | 74.71 | 74.87 | 3.6h** |
77+
| LaneATT | ResNet34 | level 1c | 360 x 640 | *no* | CULane | F1 | 75.76 | 75.82 | 4.0h** |
7878
| BézierLaneNet | ResNet18 | level 1b | 288 x 800 | *yes* | CULane | F1 | 73.36 | 73.67 | 9.9h |
7979
| BézierLaneNet | ResNet34 | level 1b | 288 x 800 | *yes* | CULane | F1 | 75.30 | 75.57 | 11.0h |
8080
| Baseline | ERFNet | level 0 | 360 x 640 | *yes* | LLAMAS | F1 | 95.94 | 96.13 | 10.9h<sup>+</sup> |
@@ -169,8 +169,8 @@
169169
| LSTR | ResNet18s-2X | level 0 | 56.17 | 39.10 | 22.90 | 25.62 | 25.49 | 52.09 | 40.21 | 30.33 | 1690 | 39.77 | [model](https://drive.google.com/file/d/1vdYwM0xDcQLjMAibjmls8hX-IsUe0xcq/view?usp=sharing) \| [shell](../tools/shells/resnet18s_lstr_culane.sh) |
170170
| LSTR | ResNet18s-2X | level 1a | 86.78 | 67.34 | 59.92 | 40.10 | 59.82 | 78.66 | 56.63 | 56.64 | 1166 | 68.72 | [model](https://drive.google.com/file/d/11Tv_nowlWmQtTYQfhGsziDIzb20kPo8o/view?usp=sharing) \| [shell](../tools/shells/resnet18s_lstr-aug_culane.sh) |
171171
| LSTR | ResNet34 | level 1a | 89.73 | 69.77 | 66.72 | 45.32 | 68.16 | 85.03 | 64.34 | 64.13 | 1247 | 72.48 | [model](https://drive.google.com/file/d/1KfmXubuAtUoE9MO8iViMyB_3XhTxWnwH/view?usp=sharing) \| [shell](../tools/shells/resnet34_lstr-aug_culane.sh) |
172-
| LaneAtt | ResNet18 | level 1c | 90.74 | 72.63 | 69.53 | 47.71 | 70.38 | 86.55 | 65.02 | 65.73 | 1036 | 74.87 | [model](https://drive.google.com/file/d/17Ve7RiqxHK4aEXOt7MJzycj6mvvrieZ1/view?usp=sharing) \| [shell](../tools/shells/resnet18_laneatt_culane.sh) |
173-
| LaneAtt | ResNet34 | level 1c | 91.36 | 73.72 | 70.71 | 48.40 | 73.69 | 86.86 | 68.95 | 66.00 | 965 | 75.82 | [model](https://drive.google.com/file/d/1AEyNZFAskPg2MKcD3KCaL95JsBeJG_48/view?usp=sharing) \| [shell](../tools/shells/resnet34_laneatt_culane.sh) |
172+
| LaneATT | ResNet18 | level 1c | 90.74 | 72.63 | 69.53 | 47.71 | 70.38 | 86.55 | 65.02 | 65.73 | 1036 | 74.87 | [model](https://drive.google.com/file/d/17Ve7RiqxHK4aEXOt7MJzycj6mvvrieZ1/view?usp=sharing) \| [shell](../tools/shells/resnet18_laneatt_culane.sh) |
173+
| LaneATT | ResNet34 | level 1c | 91.36 | 73.72 | 70.71 | 48.40 | 73.69 | 86.86 | 68.95 | 66.00 | 965 | 75.82 | [model](https://drive.google.com/file/d/1AEyNZFAskPg2MKcD3KCaL95JsBeJG_48/view?usp=sharing) \| [shell](../tools/shells/resnet34_laneatt_culane.sh) |
174174
| BézierLaneNet | ResNet18 | level 1b | 90.22 | 71.55 | 68.70 | 45.30 | 70.91 | 84.09 | 62.49 | 58.98 | 996 | 73.67 | [model](https://drive.google.com/file/d/1IpfusHvFeMEGe8wv0fer6KF3pH4X2Tj3/view?usp=sharing) \| [shell](../tools/shells/resnet18_bezierlanenet_culane-aug1b.sh) |
175175
| BézierLaneNet | ResNet34 | level 1b | 91.59 | 73.20 | 69.90 | 48.05 | 76.74 | 87.16 | 69.20 | 62.45 | 888 | 75.57 | [model](https://drive.google.com/file/d/1342FQeDQKRHMo283jW2T1WDgfgsYbR5q/view?usp=sharing) \| [shell](../tools/shells/resnet34_bezierlanenet_culane-aug1b.sh) |
176176

‎docs/MODEL_ZOO_J.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ The whole [MODEL_ZOO](./MODEL_ZOO.md) with specifications.
66
- SCNN
77
- RESA
88
- LSTR
9-
- LaneAtt
9+
- LaneATT
1010
- [BézierLaneNet](/configs/lane_detection/bezierlanenet)
1111

1212
## Semantic Segmentation

‎utils/models/lane_detection/laneatt.py‎

Lines changed: 10 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,8 @@ def __init__(self,
4242
anchor_feat_channels=None,
4343
conf_thres=None,
4444
nms_thres=0,
45-
nms_topk=3000):
45+
nms_topk=3000,
46+
trace_arg=None):
4647
super().__init__()
4748
self.backbone = MODELS.from_dict(backbone_cfg)
4849
self.backbone_channels = backbone_channels
@@ -61,6 +62,10 @@ def __init__(self,
6162
self.nms_thres = nms_thres
6263
self.nms_topk = nms_topk
6364

65+
if trace_arg is not None: # Pre-compute
66+
attention_matrix = torch.eye(topk_anchors).repeat(trace_arg['bs'], 1, 1)
67+
self.pre_non_diag_inds = torch.nonzero(attention_matrix == 0., as_tuple=False)
68+
6469
# generate anchors
6570
self.anchors, self.anchors_cut = self.generate_anchors(lateral_n=72, bottom_n=128)
6671
# Filter masks if `anchors_freq_path` is provided
@@ -209,18 +214,12 @@ def forward(self, x):
209214
attention = softmax(scores).reshape(x.shape[0], len(self.anchors), -1)
210215
attention_matrix = torch.eye(attention.shape[1], device=x.device).repeat(x.shape[0], 1, 1)
211216
if is_tracing():
212-
# TODO: this also triggers nonzero, and where can't be used
213-
mask = attention_matrix < 1
214-
attention_matrix[mask] = attention.flatten()
215-
attention_matrix *= mask
216-
# 3 0 1 2
217-
# 0 3 1 2
218-
# 0 1 3 2
219-
# 0 1 2 3
217+
# Use pre-computed nonzero results
218+
non_diag_inds = self.pre_non_diag_inds.to(attention_matrix.device)
220219
else:
221220
non_diag_inds = torch.nonzero(attention_matrix == 0., as_tuple=False)
222-
attention_matrix[:] = 0
223-
attention_matrix[non_diag_inds[:, 0], non_diag_inds[:, 1], non_diag_inds[:, 2]] = attention.flatten()
221+
attention_matrix[:] = 0
222+
attention_matrix[non_diag_inds[:, 0], non_diag_inds[:, 1], non_diag_inds[:, 2]] = attention.flatten()
224223
batch_anchor_features = batch_anchor_features.reshape(x.shape[0], len(self.anchors), -1)
225224
attention_features = torch.bmm(torch.transpose(batch_anchor_features, 1, 2),
226225
torch.transpose(attention_matrix, 1, 2)).transpose(1, 2)

‎utils/onnx_utils.py‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,8 @@
1818
TRACE_REQUIRE_PREPROCESSING = [
1919
'LSTR',
2020
'RESA',
21-
'RESA_Net'
21+
'RESA_Net',
22+
'LaneAtt'
2223
]
2324

2425

0 commit comments

Comments
 (0)