Skip to content

Conversation

@ma595
Copy link
Member

@ma595 ma595 commented Nov 5, 2025

This PR captures two errors in the calculation of the metrics:

  • We had accidentally passed velocity_u into temperature_DWbox_metric, should have been temperature.
  • The second error is more subtle:

Previously, we had renamed dims as so:

    if "x" in ds.dims:
        ds = ds.rename({"x": "nav_lon"})
    if "y" in ds.dims:
        ds = ds.rename({"y": "nav_lat"})

So this made the metrics (as they were given to us by Etienne) work fine.

However, when we select variables from the DataSet, for example,

temperature = ds.tn # giving a DataArray

The pre-existing dimensions (time_counter, nav_lev, nav_lat, nav_lon) are carried across as well as coordinates (time_counter, nav_lev). i.e. the nav_lat and nav_lon variables are not carried over. And note that the dims have been renamed to nav_lon and nav_lat.

To fix this, we need to promote the nav_lat(nav_lat, nav_lon) and nav_lon(nav_lat, nav_lon) to coordinates. The idiomatic way to do this is with the set_coords function:

    for name in ("nav_lat", "nav_lon"):
        if name in ds and name not in ds.coords:
            ds = ds.set_coords(name)  # zero-copy promotion to coordinate

Exploring xarray further, if Nemo had output nav_lat without the redundancy i.e. as 1D arrays. It would have been parameterised as so: nav_lat(nav_lat), which, on reading by xarray would automatically be recognised as a coordinate. The same is true of nav_lon.

Results

Running the following command:
python3 -m spinup_evaluation --sim-path ./tests/data/DINO_subsampled_A/ --mode restart

timestamp,check_density_from_file,check_density_computed,temperature_500m_30NS_metric,temperature_BWbox_metric,temperature_DWbox_metric,ACC_Drake_metric,ACC_Drake_metric_2,NASTG_BSF_max 
576000.0,0.000020,0.000032,3.236497,12.828081,0.007553,188.689106,188.689106,59.839730
576000.0,0.000020,0.000032,11.507896,5.197274,4.468086,188.689106,188.689106,35.515781
Metric Old New Difference (new - old)
temperature_500m_30NS_metric 3.24 11.51 8.27
temperature_BWbox_metric 12.83 5.20 -7.63
temperature_DWbox_metric 0.0076 4.46 4.46
NASTG_BSF_max 59.84 35.52 -24.32

TODOs

  • Parameterise relevant metrics with depth bounds
  • Decide whether we need to rename x and y or enforce correspondence to indices from now on.
  • Correct failing tests.
  • There's also a mesh_mask path resolution fix which should probably be removed from this PR.

This closes #81

ma595 added 4 commits October 15, 2025 18:18
If the mesh mask is a file name, i.e. mesh_mask.nc - assume that it exists in
the sim-dir and ref-dir. If it's absolute, override this behaviour.

Also throw an error if it can't be loaded.
Provide user with more helpful error message if mesh_mask cannot be found.
* Use x and y indices rather than nav_lon and nav_lat in metrics
* If nav_lat or nav_lon exists as dim, promote to coord

This error is exhibited in the DINO restart file.
When using x and y indices in metrics we are not selecting a
window based on degrees, but indices.
@ma595 ma595 self-assigned this Nov 27, 2025
@ma595 ma595 added the bug Something isn't working label Nov 27, 2025
@ma595
Copy link
Member Author

ma595 commented Dec 4, 2025

  • run metrics on climatology of the true data.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Verify diffusion model restart files and provide documentation to create from .npy files

2 participants