Data Loading issue during training

[lhycms]

If i have two LabeledSystem (LabeledSystem_1, LabeledSystem_2), the number of atoms in LabeledSystem_1 is 108, and the number of atoms in LabeledSystem_2 is 72. What is loading order during training process? Does they are loaded seperately in sequence? Maybe just like the following code?

for tmp_sys in [LabeledSystem_1, LabeledSystem_2]:
     for x, y in tmp_sys:
          y = model(x)
          err = MSELoss(y, y_pred)
          ...

Or deepmd-kit uses something similar to probability to select the data system, but it is difficult to ensure that all frames are traversed in one cycle. I

The above issues have been bothering me for a long time. Looking forward to everyone joining the discussion

[omaraek]

I think it picks randomly a system (the probability depends on the number of batches in each system) and then it picks randomly one batch of that system.

[lhycms]

Wow! That means that not traversing all frames per epoch. It's pretty amazing.

But if it pick randomly system and batch, what is the sign of the ending of this epoch?

[njzjz]

The system is picked randomly. The frames in a system are traversed once per epoch. Different systems don't share the same epoch.

[lhycms]

Thanks for the help.