How to manipulate raw data more flexible

[J-You]

Hi all,

I'm wondering if there exists any way to prepare raw data more flexible?
I have notice that in #460, the kit could shuffles the order of frames automatically.

In my calculation, I collect output data of a quenching process, from 1500K to 500K, at an interval 200K.
Total frame numbers are 12,0000 steps, that is each temperature contain 2,0000.

The pre-process of raw data is used by dpdata package. Here, I use set_size=2,0000 to divide my raw data to 5 training sets and 1 testing set.
I want to know if the "shuffle" done by dpMD-kit could contain all of temperature data in testing set? That is, whether my this simple setting is reasonable?

By the way, the data of cooling process, e.g., 1500-1300K, whether should contain in training? To my best knowledge, it seems not, at least in the literature I have read.

Any feedback is welcome.

[amcadmus]

The data is shuffled within a "set".

In your case it means that the data at different temperatures will not be mixed.

If you wants to mix data from different temperatures you may use the shuffle method in the dpdata.LabeledSystem.
https://github.com/deepmodeling/dpdata/blob/cf4c99bdfdd938176c37f0910d0bd90b45c5dc77/dpdata/system.py#L1413

[J-You]

Thanks for your response.

I use script below to convert data:

from dpdata import LabeledSystem,MultiSystems
from glob import glob
fs=glob('/home/why/dptrain/raw_data/*.xml')  # remeber to change here !!!
ms=MultiSystems()
for f in fs:
    try:
        ls=LabeledSystem(f, fmt='vasp/xml').shuffle()
    except:
        print(f)
    if len(ls)>0:
        ms.append(ls)
ms.to_deepmd_npy(folder='/home/why/dptrain/data', set_size=20000)

However, if I call shuffle method, it would return RuntimeError("Object must be System or MultiSystems!")
If I don't use shuffle method, I would generate one system, including 6 sets. But it can't contain different temperature in the testing set.

Thanks for your time and attention.

[njzjz]

Your script should be

ls=LabeledSystem(f, fmt='vasp/xml')
ls.shuffle()

[J-You]

Thanks for your reply. It certainly works.
I misunderstood class syntax, should create instance object at first.