Ground truth of safety-critical events

[xudongren-seu]

I have a question regarding the ground truth label of safety-critical events, as described in Section 5.2 of your manuscript.
My questions are:
1.Is it correct that the negative labels in the safe period will significantly outnumber the positive labels in the danger period?
If so, this could lead to a severe class imbalance in the ground truth dataset.
2.Do we need to address this class imbalance when computing performance metrics such as the AUPRC or other related evaluation measures?
For example, should we consider using balanced sampling, class weights, or other methods to mitigate any potential bias in the evaluation, given the imbalance between positive and negative samples?
I would appreciate your insights.
Best regards

[Yiru-Jiao]

Hi @rxd199682, thanks for your question.
As described in Section 5.2:

Then we can create positive ground truth (i.e., potential collision) between the subject vehicle and a conflicting object in the danger period; and negative ground truth (i.e., safe interactions) between the subject vehicle and other surrounding objects in the safe period.

Assuming that there are $N$ events with well-recognised danger and safe periods under evaluation, then the number of positive cases is $N$, and the number of negative cases is $N\times\bar{M}$ where $\bar{M}$ denotes the average number of surrounding objects that meet the requirements to be considered as in a safe period. In this paper's second-stage experiments, $\bar{M}\approx 1.46$.

Therefore, the problem of class imbalance does not exist in this paper's evaluation, and the accuracy metrics do not need extra modification. However, careful consideration of data imbalance is indeed required with other datasets in other future work. Thanks for raising this problem.

[xudongren-seu]

Thank you again for your insights.