How are region-level descriptions obtained?

[UcanSee]

Thanks for your great work！
In your paper, the label of the semantic classification branch is mask cropped embedding obtained by CLIP, then how is the GT of the caption branch generated from SA-1B?

[PhyscalX]

Hi, @UcanSee

1. Caption branch (i.e., TextDecoder) is randomly initialized, but is NOT trained during SA-1B pre-training.

2. Caption branch is then trained only on VG data, with the frozen ImageEncoder & ImageDecoder.

3. Further e2e fine-tuning for Caption branch on a mixed SA/VG dataset (set caption loss to zero for SA data),
   could improve VG CIDEr from 154.7 to 164.7.