Preserve the ERA-5 surface geopotential and start the model-level integration from it - #820
Open
s-sasaki-earthsea-wizard wants to merge 4 commits into
Conversation
The model-level fetch recomputes z at all levels with calcgeoh and overwrote the downloaded surface geopotential, so _load_model_level had to start the hydrostatic integration from the lowest full level (z[0]) where _calculategeoh expects the true surface geopotential. Every reconstructed column carried the height of that level above ground -- a near-uniform +9.4 to +10.2 m -- into the delay calculation. Keep the downloaded field as z_surface alongside the recomputed cube, aligned on the level coordinate like lnsp, and prefer it in _load_model_level. Files fetched before z_surface was stored fall back to the previous z[0] behavior, as do HRES files, which share the loader. Addresses dbekaert#812.
Treat a z_surface that reads back non-finite (a layout that did not land on the first level) as absent rather than poisoning the integration, quantify the fetch-test tolerance against the known stored-t defect, and clarify that the synthetic fixture writes the temperature CDS delivers.
Recover z_surface only when the [0, 0] read yields a field of the same shape as lnsp; a variable stored in any other layout (native 2D, or one that did not land on the first level) falls back to the lowest full level instead of crashing or broadcasting a wrong value. Assert the on-disk layout in the fetch test and cover the foreign-layout fallback.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Before this change,
_get_from_cdscomputed the full-model-level geopotential cube from the downloaded surface geopotential and then discarded that surface field._load_model_leveltherefore had to start the hydrostatic integration from the lowest full model level (z[0]), where_calculategeoh's contract is the true surface geopotential, so every reconstructed column sat a near-uniform +9.4 to +10.2 m too high (the height of the lowest full level above ground).This PR:
z_surfacevariable, aligned on the level coordinate exactly likelnsp(fetch side);_makeDataCubes(Nonewhen absent, or when the stored variable does not match the layout the fetch writes) and prefers it in_load_model_level, falling back to the previousz[0]behavior for files fetched before this change — HRES files, which share this loader, continue through the same fallback path (HRES itself was not independently exercised).Motivation and Context
Fixes #812; follow-up to #811 as discussed there and in #806.
Relation to #805: that rework of the ML fetch also replaces the downloaded surface z with the recomputed cube. If it lands first, the fetch-side change here is a small rebase onto its fetch path (the acceptance criterion is the same: fetched files retain the true surface geopotential as a separate variable); the loader-side change is independent of it.
How Has This Been Tested?
Measured on real archived CDS responses (the raw
lnsp/zandt/qfiles CDS served for a Kanto scene, 2026-03-01 00:00 UTC), replayed through_get_from_cdswith a mockedcdsapi.Clientso the exact post-processing of this branch runs on the exact bytes CDS delivered. Full report with commit-pinned RAiDER builds and package versions: https://github.com/s-sasaki-earthsea-wizard/raider-arco-era5/blob/8576fbb0585e42ee05f9a206f16b6dd139d61562/reports/ml_offset_regression_812_fix.md_calculategeoh(bitwise identity)dev@c80cf1e, pre-fix file(The last row uses the replayed file's own grid, hence the different column count.) The remaining +0.04 to +0.35 m is reproduced quantitatively by a separate pre-existing defect — the fetch stores virtual temperature in
tbecausecalcgeohmutates its argument in place — filed as #819; the surface field itself arrives bitwise intact.Unit tests (all pass locally,
pytest test/test_ecmwf_fetch.py test/test_ecmwf_levels.py):test_ecmwf_fetch.py: the fetched file carriesz_surfacebitwise in the same layout aslnsp; the loader reproduces the fetch-time lowest-level height (0.02 m tolerance, which the ~10 m pre-fix behavior exceeds with ample margin); a file withz_surfacestripped falls back to the pre-fix behavior, sitting higher by the lowest level's height above ground (to within 0.01 m); az_surfacestored in a foreign layout is ignored rather than misread.test_ecmwf_levels.py: the descending-latitude fixture now runs in both layouts (with and withoutz_surface), guarding the orientation of both the z cube and the stored surface field; a new per-column check pins the reconstruction to the stored cube at 1e-3 m.Type of change
Checklist: