Copying data between stores / arrays

[BruAPAHE]

Can i write data from FilesystemStore to MemoryStore?
full code

    let store = Arc::new(FilesystemStore::new("gfs_t2m_030_040.zarr".to_string())?);

    let lat_arr = Array::open(store.clone(), "/latitude")?;
    let lon_arr = Array::open(store.clone(), "/longitude")?;
    let lat_vals: Vec<f64> = lat_arr
        .retrieve_array_subset_ndarray(&lat_arr.subset_all())?
        .iter()
        .cloned()
        .collect();
    let lon_vals: Vec<f64> = lon_arr
        .retrieve_array_subset_ndarray(&lon_arr.subset_all())?
        .iter()
        .cloned()
        .collect();

    let time_arr = Array::open(store.clone(), "/time")?;
    let nt = time_arr.shape()[0];
    let t2m = Array::open(store.clone(), "/t2m")?;
    
    let target_lat = 45.0;
    let target_lon = 135.0;
    let i_lat = nearest_index(&lat_vals, target_lat);
    let i_lon = nearest_index(&lon_vals, target_lon);


    let subset = ArraySubset::new_with_ranges(&[
        0..nt,
        (i_lat as u64)..(i_lat as u64 + 1),
        (i_lon as u64)..(i_lon as u64 + 1),
    ]);

    let start = Instant::now();
    
    let t2m_data: ArrayD<f32> = t2m.retrieve_array_subset_ndarray(&subset)?;

    let duration = start.elapsed();

    println!("Time elapsed in expensive_function() is: {:?}", duration);

get 10 ms one request get data by lat/lng

[LDeakin]

Out of interest, what is your overall objective to copy from a FilesystemStore a MemoryStore?

zarrs doesn't have any helper methods for this kind of operation. Partially because there is a lot of nuance to copying potentially large amounts of data from one store to another. E.g. how to deal with failure, whether the source data should be validated, etc.

Store Copying

If you just want to straight up copy an entire hierarchy from one store to another, it isn't too complicated:

1. Get all source store keys that you want to copy with a relevant method from https://docs.rs/zarrs_storage/0.3.4/zarrs_storage/trait.ListableStorageTraits.html
2. Read source keys with https://docs.rs/zarrs_storage/0.3.4/zarrs_storage/trait.ReadableStorageTraits.html#method.get
3. Write target keys with https://docs.rs/zarrs_storage/0.3.4/zarrs_storage/trait.WritableStorageTraits.html#tymethod.set
4. Parallelise the above if needed

Array Copying

If you want to copy arrays individually, that can be done with validation

1. Get the metadata of your source array https://docs.rs/zarrs/latest/zarrs/array/struct.Array.html#method.metadata
2. Create target array with https://docs.rs/zarrs/latest/zarrs/array/struct.Array.html#method.new_with_metadata
3. Store the target array metadata
4. Iterate over chunks (in parallel if needed)
   • a. copy via https://docs.rs/zarrs/latest/zarrs/array/struct.Array.html#method.retrieve_chunk and https://docs.rs/zarrs/latest/zarrs/array/struct.Array.html#method.store_chunk or similar
   • b. alternatively, copy via https://docs.rs/zarrs/latest/zarrs/array/struct.Array.html#method.retrieve_encoded_chunk and https://docs.rs/zarrs/latest/zarrs/array/struct.Array.html#method.store_encoded_chunk if you trust that your chunks are encoded correctly

[BruAPAHE]

@LDeakin Thank you so much for response!!