Test speeding up backtest output saving to a single zarr

[Sukh-P]

Currently our backtest scripts output a .nc file for each forecast t0 specified in the backtest range, sometimes that can mean many 1000s of files, I have found that when opening all of these .nc files into a single xarray dataset like this: xr.open_mfdataset(f"{output_dir}/*.nc", parallel=True) can sometimes be very slow (even with the parallel=True parameter) when it's a large number of files (in my case this was around ~35000 files), I had some success speeding this up using python multiprocessing after following the advice here https://stackoverflow.com/questions/65587633/ways-to-speed-up-open-mfdataset-in-xarray this issue is to benchmark the different ways of doing this and see how much quicker we can make it

[Sukh-P]

Done some quick benchmarking on a machine with 16 CPUs doing this for 2000 backtest forecast .nc files, tried 3 methods, xr.open_mfdataset, with and without parallel=True and a multiprocessing method which opens files simultaneously using xr.open_dataset and then does an xr.concat at the end, the results:

• xr.open_mfdataset(parallel=False) - 90 seconds
• xr.open_mfdataset(parallel=True) - 147 seconds
• multiprocessing approach - 28 seconds

Surprising that using parallel was slower for open_mfdataset, multiprocessing approach was 3.2x-5.3x than the respective open_mfdataset approaches, given that this would likely be quicker with more cpus and we sometimes deal with 10,000s of .nc files from our backtest outputs seems worth putting this into a PR.

Example code snippet showing the multiprocessing approach:

from multiprocessing import Pool, cpu_count

def open_ds(file_path: str) -> xr.Dataset:
        with xr.open_dataset(file_path, decode_timedelta=True) as ds:
            return ds
            
def multiprocessing_open(file_path: str):
    with Pool(processes=cpu_count()) as pool:
           
        results = pool.map(open_ds, file_path)
        
        batch_ds = xr.concat(results, dim="init_time_utc")
        return batch_ds

[AUdaltsova]

I also keep thinking about not saving every t0 individually and instead doing some concatting (e. g. save all of one day's t0s into one file, or just in groups of 16 or 32 or what have you), which will reduce the number of files significantly? We never use individual files anyway as they are immediately resaved as a zarr and wiped.