Running time is a bit too long

[WangYong921]

Thank you so much for this great job. I am using the evaluation script to output evaluation results for one of the data files, how can I speed up the output evaluation time as currently it takes an estimated 52 hours to load 6322 files, thanks for the clarification! I'm using an 80GB A100, in addition the training database gives the need to expect around 200 hours, is this normal?

Train:

Evaluation:

[jirvin16]

Thanks for the kind words!

Train-time is slow. We got around this by using 10 GPUs at once, but you could also increase the batch size if you have more VRAM. We also tried ensuring the examples sampled every step were of the same image sequence length (so that the steps are much faster on average as many sequences much shorter than 8 images) which was considerably faster, but this also led to a drop in performance. We didn't investigate it further but it could be worth seeing whether performance could be maintained with this sequence length batching.

For evaluation, the object detection datasets are particularly slow as the models tend to output many more tokens. One option is to try to get a larger batch size working as right now it runs inference with one example at a time. This should theoretically be possible but it was not straightforward to get working when I tried.

[WangYong921]

Thank you for your reply, and one more question, I'm not quite sure what is the purpose of the presence of ‘ploygon’ in some of the change detection datasets?

[jirvin16]

That is the original polygon from which we derived the bounding boxes. We use that to correctly evaluate on the canonical detection tasks, because the metrics used for those evaluate against the polygon.

[WangYong921]

Thank you for sharing the dataset, in the XBD dataset you posted for training, we see that the image is divided into 16 blocks, I guess this may be to increase the amount of data and also to increase the detection ability to recognise small objects. I have a doubt how the Q&A pairs are constructed for each block, especially for labelling the location of the bounding box, like ‘Classify the level of damage experienced by the building at location [0, 4, 17, 53] in the second image’, how to know the truth of the corresponding bounding box, in fact, we see that the XBD open source data are labelled with polygonal coordinates. Similarly, for the assessment phase, we see that you have used polygonal coordinates to calculate some metrics, but I find that the coordinates don't seem to be consistent with the values of the coordinates in the labels in the open source data, what is the reason for this, and I hope to get your reply!

[jirvin16]

Re: dividing the images into 16 patches, that is correct.

Re: obtaining the bounding box, we convert each polygon to a bounding box by computing the tightest encompassing axis-aligned bounding box around the polygon.

Re: the polygon coordinates, would you mind sharing an example?