improve code efficiency to handle large amount of data#158

Open

JoachimPiret wants to merge 6 commits intoODINN-SciML:mainfrom

JoachimPiret:main

Commits on Jan 8, 2026

improve code efficiency of two functions of class AggregatedDataset() : init() and mapSplitsToDataset()
JoachimPiret
committed
allows to record dataframe in parquet format in addition to csv format
JoachimPiret
committed
add possibility to divide between test and train absed on subregion (c-region) as well as the possibility to have randomness and different sampling from sampling to sampling in set_train_test_split…
JoachimPiret
committed

Commits on Jan 9, 2026

adapation of dataset.py to choose output format of _get_output_filename() between csv and parquet
JoachimPiret
committed

Commits on Jan 14, 2026