Optimizations for Zarr loading and processing in ESMValTool

[valeriupredoi]

Since #2785 we now have backend support for Zarr loading in ESMValTool ie one can simply:

    from esmvalcore.preprocessor._io import load


    zarr_path = (
        "https://uor-aces-o.s3-ext.jc.rl.ac.uk/"
        "esmvaltool-zarr/example_field_0.zarr3"
    )
    cubes = load(zarr_path)

and they get Iris cubes.

Front facing (ie passing instructions via recipe) will be in conjunction with #2765 - we want to load these files based on what catalogs they appear in etc. But before then, there are number of investigations still to be done:

• optimal memory handling: it appears that ncdata needs quite a bit of memory, when all things should be lazy, see How to improve performance: ncdata conversion from Xarray(zarr) to Cube is not ideal pp-mo/ncdata#139 - we are working with Patrick on that one;
• possible issues with locking, see Indefinite hang when consolidated=True pp-mo/ncdata#138 - again, we are orking with Patrick and Martin on this one as well;
• it'd also be good for folks to start having a crack at their favourite Zarr data and see if they can spot any optimization issues

🍻

[valeriupredoi]

that issue quoted above boils down to this in Xarray pydata/xarray#10612 - unfortunately not much @pp-mo nor I can do at the moment, if Xarray don't fix their data realization problem (or we can help there?); I'll have a chat with Patrick after some reaction to the Xarray issue, I may have an alternative to loading Zarr files with Xarray in ncdata

[valeriupredoi]

main problem fixed ie memory use and not realizing data at load; there is still this:

possible issues with locking, see pp-mo/ncdata#138 - again, we are working with Patrick and Martin on this one as well

But I will do some "private" investigations with Patrick (and not reopen this issue) since it's a little too obscure at the moment