`ECCODarwin` download fails

[simone-silvestri]

Apparently the tests cannot find the :PO₄ key
see https://buildkite.com/clima/climaocean-ci/builds/4852#01999ca3-f1c2-4c9e-826a-67fa32cdc911

@seamanticscience

[navidcy]

related: #639

[simone-silvestri]

As a follow-up, we should not close this issue since #637 is dropping the tests. Therefore, this issue is not addressed by #637

[seamanticscience]

I believe the ECCO2DarwinMonthly tests work, but the interpolation from LLC270 "binary" grid to the 0.25° lat-lon "native" grid takes a relatively long time, and the CI test just stops before it is finished. I guess there's no reason not to upload the coefficients that I already calculated (since it's the same every time) and allow MeshArrays to download it. A similar thing is already available for the LLC90 grid to 0.5° lat-lon grid, but the interpolation here is much faster anyway. Plan?

[glwagner]

How long does the test take?

I don't understand a few things:

• what are the "coefficients"?
• can you clarify the role of MeshArrays? I thought MeshArrays was for interpolation, not downloading, but maybe I misunderstand.

[seamanticscience]

@glwagner coefficients -> interpolation weights? Here's the relevant code:

ClimaOcean.jl/src/DataWrangling/ECCO/ECCO_darwin.jl

Lines 132 to 152 in 8def2b6

	# Check if the interpolation coefficients are already calculated
	interp_file = joinpath(dirname(metadata_path(metadata)),"native_interp_coeffs.jld2")
	if !isfile(interp_file)
	# Calculate coefficients to interpolate from native grid to regular lat-lon grid (as in the ECCO netcdf files)
	resolution_X = 360/Nx
	resolution_Y = 180/Ny
	# Regular lat-lon grid
	longitudes = longitude_interfaces(metadata.dataset)
	latitudes = latitude_interfaces(metadata.dataset)
	lon = [i for i = longitudes[1]+resolution_X/2:resolution_X:longitudes[2]-resolution_X/2,
	j = latitudes[1]+resolution_Y/2:resolution_Y:latitudes[2]-resolution_Y/2]
	lat = [j for i = longitudes[1]+resolution_X/2:resolution_X:longitudes[2]-resolution_X/2,
	j = latitudes[1]+resolution_Y/2:resolution_Y:latitudes[2]-resolution_Y/2]
	# Interpolation factors for the native grid (writes out to a file "interp_file" for later use)
	coeffs = interpolation_setup(; Γ=native_grid_coords, lat, lon, filename=interp_file)
	else
	# Read the coefficients from the file that was previously calculated
	coeffs = interpolation_setup(interp_file)
	end

If interp_file does not exist, the weights are computed; otherwise, the weights are loaded from the file, which is fine for regular use where Julia artifacts are preserved. I'm guessing the CI tests start afresh each time and therefore have to recompute everything from scratch. This works fine for LLC90 (ECCO4DarwinMonthly) but is obviously too much of a heavy-lift for LLC270/ECCO2DarwinMonthly testing (CPU metadata tests for ECCO2DarwinMonthly take about an hour).

However, MeshArrays has a neat trick, where if the filename is precisely interp_coeffs_halfdeg.jld2 then it will download a set of precalculated LLC90 -> 0.5° lat lon (i.e. binary_data_grid to native_grid) weights from Zenodo. I am proposing adding the same trick for an LLC270 -> 0.25° lat lon weight file.

[simone-silvestri]

That makes sense! So we can reintroduce the tests