Best way to separate multi-system data into validation and training set?

[kimiaGF]

I have multiple systems, each with different number of frames and number of atoms. When using the set_size option, I can only give it a fixed number of frames to include in each set, thus some frames might have 10 sets while others only have 1-2 sets. Is there a way to do this more uniformly?

[wanghan-iapcm]

It is safe to set set_size to a very large number so all systems has only one set.

[kimiaGF]

would I then manually separate the frames in the raw files for training and validation splitting?

[njzjz]

I don't think there is a existing way to split the dataset. You are welcome to contribute a method to dpdata

[paradro1d]

Here is a bash script for recursively checking sub-folders with systems and separating sets into training and validation datasets. It is created for 4 sets in each system, but might be modified.
test_valid.zip

[kimiaGF]

This is perfect thank you so much! Is the split 75-25 then? For training-validation