Training Dataset Sampling/Loading

[VickyYOX]

Hi,

Thank you for your work.

I wanted to ask: are you only using the first question in each QAs.json entry as the instruction during training, as indicated here:

SegEarth-R1/segearth_r1/train/train_dataset.py

Line 321 in f0c382c

question = QAs["questions"][0]

If so, wouldn’t that underutilize the dataset, since most samples contain 5–6 questions? Have you tried incorporating all questions during training?

Additionally, I noticed that the evaluation metric you used is IoU as mentioned in the paper, which only considers the segmentation masks. In that case, the answers in QAs.json are not directly involved in training. Why, then, are they loaded into the data_dict here:

SegEarth-R1/segearth_r1/train/train_dataset.py

Lines 327 to 328 in f0c382c

	# answer_idx = random.randint(0, answer_num - 1)
	answer = QAs["answer"][0]

Or am I missing something and the answers are also used during training? If so, I noticed you tried both random sampling and using answer[0] as the line above, could you clarify which approach you ultimately used, and whether using different strategies affected performance?

Regards,
Vicky

[xcarl1]

Hi @VickyYOX , thanks for raising this issue! 🙌

1. Use of questions during training:

   Regarding the full utilization of the dataset, during training we actually randomly select one question from each sample for the forward pass and gradient computation. Since we train for a total of 10 epochs, this ensures that the dataset is sufficiently utilized. In the dataset code, this should actually be:

   question = random.choice(QAs["questions"])
   

2. Role of answers in training:
   Regarding the use of answers in the dataset, during training we actually applied a cross-entropy loss to allow the large language model to learn textual explanations. However, in our evaluation, we used segmentation-based metrics, namely mIoU and gIoU, to assess the quality of the segmentation results, and we did not evaluate the quality of the generated textual explanations.

3. Answer selection strategy and performance:

   During training, we actually sampled the answer randomly, i.e.,

   answer = random.choice(QAs["answer"])
   

   The choice strategy of the answer had little impact on the results.

[VickyYOX]

Hi @xcarl1, Thx for your prompt reply and appreciate your clarification.

I understand from your paper that you used focal loss and dice loss for the mask supervision and evaluated the model with only mask-related metric. Since you mentioned cross entropy loss is also applied for textual generation, I am wondering how much it effects the performance w/wo the cross entropy loss?

Regards,
Vicky

[xcarl1]

Hi @VickyYOX , Since our task is defined as requiring the model to output an explanation for the question, we by default added cross entropy loss during training. We did not conduct experiments under the setting without cross entropy loss.