Skip to content

Preserve the ERA-5 surface geopotential and start the model-level integration from it - #820

Open
s-sasaki-earthsea-wizard wants to merge 4 commits into
dbekaert:devfrom
s-sasaki-earthsea-wizard:fix/era5-ml-true-surface-geopotential
Open

Preserve the ERA-5 surface geopotential and start the model-level integration from it#820
s-sasaki-earthsea-wizard wants to merge 4 commits into
dbekaert:devfrom
s-sasaki-earthsea-wizard:fix/era5-ml-true-surface-geopotential

Conversation

@s-sasaki-earthsea-wizard

Copy link
Copy Markdown

Description

Before this change, _get_from_cds computed the full-model-level geopotential cube from the downloaded surface geopotential and then discarded that surface field. _load_model_level therefore had to start the hydrostatic integration from the lowest full model level (z[0]), where _calculategeoh's contract is the true surface geopotential, so every reconstructed column sat a near-uniform +9.4 to +10.2 m too high (the height of the lowest full level above ground).

This PR:

  • keeps the downloaded surface geopotential in the fetched file as its own z_surface variable, aligned on the level coordinate exactly like lnsp (fetch side);
  • returns it from _makeDataCubes (None when absent, or when the stored variable does not match the layout the fetch writes) and prefers it in _load_model_level, falling back to the previous z[0] behavior for files fetched before this change — HRES files, which share this loader, continue through the same fallback path (HRES itself was not independently exercised).

Motivation and Context

Fixes #812; follow-up to #811 as discussed there and in #806.

Relation to #805: that rework of the ML fetch also replaces the downloaded surface z with the recomputed cube. If it lands first, the fetch-side change here is a small rebase onto its fetch path (the acceptance criterion is the same: fetched files retain the true surface geopotential as a separate variable); the loader-side change is independent of it.

How Has This Been Tested?

Measured on real archived CDS responses (the raw lnsp/z and t/q files CDS served for a Kanto scene, 2026-03-01 00:00 UTC), replayed through _get_from_cds with a mocked cdsapi.Client so the exact post-processing of this branch runs on the exact bytes CDS delivered. Full report with commit-pinned RAiDER builds and package versions: https://github.com/s-sasaki-earthsea-wizard/raider-arco-era5/blob/8576fbb0585e42ee05f9a206f16b6dd139d61562/reports/ml_offset_regression_812_fix.md

z_surface passed to _calculategeoh (bitwise identity) column offset columns
dev@c80cf1e, pre-fix file lowest full level +9.46 to +10.19 m n=575
fix, pre-fix file lowest full level +9.46 to +10.19 m (identical to the last digit) n=575
fix, file written by the fixed fetch the surface z CDS delivered +0.04 to +0.35 m n=441

(The last row uses the replayed file's own grid, hence the different column count.) The remaining +0.04 to +0.35 m is reproduced quantitatively by a separate pre-existing defect — the fetch stores virtual temperature in t because calcgeoh mutates its argument in place — filed as #819; the surface field itself arrives bitwise intact.

Unit tests (all pass locally, pytest test/test_ecmwf_fetch.py test/test_ecmwf_levels.py):

  • test_ecmwf_fetch.py: the fetched file carries z_surface bitwise in the same layout as lnsp; the loader reproduces the fetch-time lowest-level height (0.02 m tolerance, which the ~10 m pre-fix behavior exceeds with ample margin); a file with z_surface stripped falls back to the pre-fix behavior, sitting higher by the lowest level's height above ground (to within 0.01 m); a z_surface stored in a foreign layout is ignored rather than misread.
  • test_ecmwf_levels.py: the descending-latitude fixture now runs in both layouts (with and without z_surface), guarding the orientation of both the z cube and the stored surface field; a new per-column check pins the reconstruction to the stored cube at 1e-3 m.

Type of change

  • Bug fix (non-breaking change which fixes an issue)

Checklist:

  • I have added an explanation of what your changes do and why you'd like us to include them.
  • I have written new tests for your core changes, as applicable.
  • I have successfully ran tests with your changes locally.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.

The model-level fetch recomputes z at all levels with calcgeoh and
overwrote the downloaded surface geopotential, so _load_model_level had
to start the hydrostatic integration from the lowest full level (z[0])
where _calculategeoh expects the true surface geopotential. Every
reconstructed column carried the height of that level above ground -- a
near-uniform +9.4 to +10.2 m -- into the delay calculation.

Keep the downloaded field as z_surface alongside the recomputed cube,
aligned on the level coordinate like lnsp, and prefer it in
_load_model_level. Files fetched before z_surface was stored fall back
to the previous z[0] behavior, as do HRES files, which share the loader.

Addresses dbekaert#812.
Treat a z_surface that reads back non-finite (a layout that did not land
on the first level) as absent rather than poisoning the integration,
quantify the fetch-test tolerance against the known stored-t defect, and
clarify that the synthetic fixture writes the temperature CDS delivers.
Recover z_surface only when the [0, 0] read yields a field of the same
shape as lnsp; a variable stored in any other layout (native 2D, or one
that did not land on the first level) falls back to the lowest full
level instead of crashing or broadcasting a wrong value. Assert the
on-disk layout in the fetch test and cover the foreign-layout fallback.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ERA-5 model levels: loader passes the lowest full-level z where _calculategeoh expects the true surface geopotential (~+10 m column height bias)

2 participants