Skip to content

Commit 0dc8e3c

Browse files
fuyiqunJackFu123
authored andcommitted
prediction: add docs for VectorNet-TNT-interactive model
Change-Id: I9b39f8bdae81fc6e1e6e6a7bbb5b3d68bf64f62f
1 parent 15f82f8 commit 0dc8e3c

File tree

2 files changed

+39
-7
lines changed

2 files changed

+39
-7
lines changed
518 KB
Loading

docs/technical_documents/jointly_prediction_planning_evaluator.md

Lines changed: 39 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,16 @@
1-
# JOINTLY PREDICTION PLANNING EVALUATOR
1+
# Jointly VectorNet-TNT-Interaction Evaluator
22

33
# Introduction
44

5-
The prediction module comprises 4 main functionalities: Container, Scenario, Evaluator and Predictor.
5+
The prediction module comprises 4 main functionalities: Container, Scenario, Evaluator and Predictor.
66

7-
The Evaluator predicts path and speed separately for any given obstacle. An evaluator evaluates a path by outputting a probability for it (lane sequence) using the given model stored in prediction/data/.
7+
An Evaluator predicts paths and speeds for any given obstacles. An evaluator evaluates a path(lane sequence) with a probability by the given model stored in prediction/data/.
88

9-
Jointly prediction planning evaluator is used in the new Interactive Obstacle(vehicle-type) model to generate short term trajectory points which are calculated using Vectornet and LSTM. By considering ADC's trajectory info, the obstacle trajectory prediction can be more accurate under interaction scenario.
9+
In Apollo 7.0, a new model named Jointly VectorNet-TNT-Interaction Evaluator is
10+
used to generate short term trajectory points. This model uses VectorNet as
11+
encoder and TNT as decoder, in which the planning information of ADC is inserted
12+
at the end. For scenarios at junction, the obstacle trajectory prediction by the
13+
new model is more accurate according to the results of our experiments and road tests.
1014

1115
![Diagram](images/interaction_model_fig_1.png)
1216

@@ -22,9 +26,37 @@ Please refer [interaction filter](https://github.com/ApolloAuto/apollo/tree/mast
2226

2327
2. Such interactive obstacles will be labeled.
2428

25-
```cpp
29+
```cpp
2630
void AssignInteractiveTag();
2731
```
2832

29-
## Model inference
30-
1. The encoder of jointly prediction planning evaluator is Vectornet, before model inference, we need to process obstacle and map data into the correct format.
33+
# Algorithm Detail
34+
## Structure
35+
![Diagram](images/VectorNet-TNT-Interaction.png)
36+
37+
## Encoder
38+
Basically, the encoder is mainly using an [VectorNet](https://arxiv.org/abs/2005.04259). The ADC trajectory and all obstacle trajectories in the form of coordinate points are transformed into polylines. For each polyline, vectors are connected sequentially, where it contains start point, end point, obstacle length and some other attributes of vector. All points are transformed in the coordinate of the ADC, with North as the direction and (0, 0) as the position for ADC at time 0.
39+
40+
After that, map information is extracted from HDMap files. As structures of lane/road/junction/crosswalk are depicted in points in HDMap, they are also processed as polylines whose vectors contain the types of traffic elements. In order to speed up calculation in Network, only information in ROI is taken into consideration.
41+
42+
A Subgraph is used to deal with each polyline and the feature of polyline is acquired. After that, the polyline features are inputted into a Global Graph, which is a GNN Network substantially.
43+
44+
The encoding feature is gained after encoding of VectorNet. For more information in detail, please find the References.
45+
46+
## Decoder
47+
The structure of Decoder is a [TNT model](https://arxiv.org/abs/2008.08294).
48+
49+
There are three parts in TNT.
50+
1)Target Prediction. N points around the ADC is grid-sampled and M points are selected as target points. These target points are used as the final points of prediction trajectories in the next step.
51+
2)Motion Estimation. With VectorNet feature and target points, a prediction trajectory is generated for each selected target point. To train more efficiently, a teaching method using true trajectory is also used in this step.
52+
3)Scoring and Selection. A score is calculated for each motion trajectory in the second step, which is used to select final trajectories.
53+
54+
## Interactive Planning
55+
After three steps in TNT, K predicted trajectories are assumed to be outputted. There is a latest planning trajectory with the same form, with which the distance of position and velocity can be calculated, named as (cost1, cost2) respectively. Meanwhile, weights of (cost1, cost2) are also outputted by VectorNet, outputting the final cost by multiplication.
56+
57+
Note we can also find a cost with the truth obstacle trajectory and ADC planning, thus producing the true value of cost. That's how the loss is calculated in this step.
58+
59+
60+
# References
61+
1. Gao, Jiyang, et al. "Vectornet: Encoding hd maps and agent dynamics from vectorized representation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.
62+
2.Zhao, Hang, et al. "Tnt: Target-driven trajectory prediction." arXiv preprint arXiv:2008.08294 (2020).

0 commit comments

Comments
 (0)