np.int64 not suitable for scalar (nan value)?

[daviddanan]

Describe the bug
When i tried to load a dataset i saved on disk, i was not able to retrieve properly a scalar. After digging a bit, i found out such a thing happen with np.int64.

To Reproduce

from pathlib import Path

from plaid import Sample
from plaid.containers.dataset import Dataset
from plaid.storage.writer import _check_folder

import numpy as np

if __name__ == "__main__":
    sample  = Sample()
    scalars = {"scalar1":np.int64(5),"scalar2":5}
    for simulationScalarsName, simulationScalarsValue in scalars.items():
        sample.add_scalar(simulationScalarsName, simulationScalarsValue)

    dataset = Dataset(samples=[sample])
    print(dataset.get_scalars_to_tabular())

    targetLocation = Path("RandomDir")
    _check_folder(targetLocation, True)
    dataset.save_to_dir(targetLocation)
    datasetReloaded = Dataset.load_from_file(targetLocation)
    print(datasetReloaded.get_scalars_to_tabular())

Expected behavior
This is the output
{'scalar1': array([5.]), 'scalar2': array([5.])}
{'scalar1': array([nan]), 'scalar2': array([5.])}

I expected to have the same before and after saving the dataset. For now, my fix is to cast the value as a float to be as generic as possible.
sample.add_scalar(simulationScalarsName, float(simulationScalarsValue))
Would it be relevant it do that within the method itself with proper error handling if the casting is not possible?

Desktop (please complete the following information):

• OS: Windows 11
• Version current main

[casenave]

Thank for your issue. Did you try with the writing/reading functions from storage ? Only these will be kept in the future. Until plaid v1, we keep all previous code to guarantee retrocompatibility, but the Dataset class will be depreciated

[daviddanan]

Yes, since you told me that only storage will be supported in the future, i am getting rid of code that rely on Dataset on my end.
It was simply faster to produce a MWE with Dataset.
With storage, while my test also failed, i do get a different behaviour: Instead of nan, i get None.

I did not try to produce a MWE for this very case but the float fix work for both (Dataset and storage).

[casenave]

Thanks for digging: there is something.

I tried:

from plaid import Sample
from plaid.storage import save_to_disk, init_from_disk

import numpy as np

if __name__ == "__main__":

    def gen():
        for i in range(2):
            sample  = Sample()
            scalars = {"scalar1":np.int64(i*5),"scalar2":i*5,"scalar3":i*5.}
            for simulationScalarsName, simulationScalarsValue in scalars.items():
                sample.add_scalar(simulationScalarsName, simulationScalarsValue)
            yield sample



    save_to_disk(".test_dataset", {"train":gen}, overwrite=True)

    ds, conv = init_from_disk(".test_dataset")

    sample = conv["train"].to_plaid(ds["train"], 1)

    print("scalar1 =", sample.get_global("scalar1"), "type:", type(sample.get_global("scalar1")))
    print("scalar2 =", sample.get_global("scalar2"), "type:", type(sample.get_global("scalar2")))
    print("scalar3 =", sample.get_global("scalar3"), "type:", type(sample.get_global("scalar3")))

    print(type(np.array([5], dtype=np.int64).item()))
    print(type(np.int64(5)))

which outputs:

scalar1 = 5 type: <class 'int'>
scalar2 = 5 type: <class 'int'>
scalar3 = 5.0 type: <class 'float'>
<class 'int'>
<class 'numpy.int64'>

np.array([5], dtype=np.int64).item()) is exactly what is done when retrieving 0D globals (namely scalars) here.
I'll correct this be ensuring input type is kept, thanks !

FYI, for fields, we keep the types for ints (required for CGNS) and cast all floats to float32 by design, do gain space, cf here