-
|
Can we stream the data? |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment
-
|
Some datasets (e.g. |
Beta Was this translation helpful? Give feedback.
{{ message }}
Answered by
LTMeyer
Dec 8, 2024
0 replies
Answer selected by
LTMeyer
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Some datasets (e.g.
active_matter,helmholtz_staircase,rayleigh_benard, etc.) are already available on HuggingFace. However, due to the large size of the dataset and the reliance on HDF5 file format to store simulation output, which is hardly optimized for the cloud, we recommend downloading the data locally (see instructions).Some datasets have single frames with large memory footprint. For instance, for the
euler_quadrantdataset a single time step represents 33MB. It would require an extremely fast network with high bandwidth to train efficiently while streaming these data.