Update variables for v3 ELM by chengzhuzhang · Pull Request #334 · E3SM-Project/e3sm_to_cmip

chengzhuzhang · 2025-12-16T18:53:59Z

Description

● Brief Summary of Changes

Variables Added/Modified in handlers.yaml

Land Variables (CMIP6_Lmon.json) - 12 variables

burntFractionAll - Burnt area percentage from fire
cCwd - Coarse woody debris carbon pool
cLeaf - Leaf carbon pool
cRoot - Total root carbon (fine + coarse live + coarse dead)
nppLeaf/nppRoot/nppWood - NPP allocation to plant components
rGrowth/rMaint - Growth and maintenance respiration
tran - Transpiration (modified to exclude soil evaporation)

Land Ice Variables (CMIP6_LImon.json) - 7 variables

hfdsn - Heat flux into snow
lwsnl - Liquid water in snow
snc/snd/snm/snw - Snow coverage, depth, melt, and amount
sootsn - Snow impurities (BC + dust + OC)
Closes [Feature]: add land variables for v3 #292

Checklist

My code follows the style guidelines of this project
I have performed a self-review of my own code
My changes generate no new warnings
Any dependent changes have been merged and published in downstream modules

If applicable:

New and existing unit tests pass with my changes (locally and CI/CD build)
I have added tests that prove my fix is effective or that my feature works
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have noted that this is a breaking change for a major release (fix or feature that would cause existing functionality to not work as expected)

chengzhuzhang · 2025-12-16T22:17:42Z

@tomvothecoder Somethow I got inconsistent mypy results again CI/CD vs my local pre-commit run, which passed. This is not urgent, but when you have a change, can you help to fix the test?

@acme-y9s The proposed new v3 variables are supported by this PR. (I removed several variables only present in CMIP5 but not in CMIP6, e.g. cMisc, cWood, fLuc. burntArea is renamed to burnAreaFractionAll from CMIP5 to CMIP6. I have a example script to process and cmorize these variables as well.

Could you please inspect the sample CMIP files at LCRC under: /gpfs/fs0/globalscratch/ac.zhang40/e2c_tests/CMIP6/CMIP/E3SM-Project/E3SM-1-0/piControl/r1i1p1f1/
Please review if the values (including signs), distribution are reasonable. Thanks!

tomvothecoder

Review of my latest commits. Should be good to go now.

tomvothecoder · 2025-12-23T20:02:51Z

e3sm_to_cmip/mpas.py

+        moc = xarray.concat(mocSlices, dim="lat")  # type: ignore[arg-type]
+        moc = moc.transpose("Time", "nVertLevelsP1", "lat")
+
        # average to bin and level centers
-        moc = 0.25 * (
-            moc[:, 0:-1, 0:-1] + moc[:, 0:-1, 1:] + moc[:, 1:, 0:-1] + moc[:, 1:, 1:]
+        moc_avg = (
+            0.25
+            * (
+                moc[:, 0:-1, 0:-1]  # type: ignore[index]
+                + moc[:, 0:-1, 1:]  # type: ignore[index]
+                + moc[:, 1:, 0:-1]  # type: ignore[index]
+                + moc[:, 1:, 1:]  # type: ignore[index]
+            )
        )
-        moc = moc.rename({"nVertLevelsP1": "depth"})  # type: ignore
-        binCounts = xarray.DataArray(binCounts, dims=("lat"))  # type: ignore
-        moc = moc.where(binCounts > 0)  # type: ignore
+        moc_avg = moc_avg.rename({"nVertLevelsP1": "depth"})
+        binCounts = xarray.DataArray(binCounts, dims=("lat"))  # type: ignore[assignment]
+        moc_masked = moc_avg.where(binCounts > 0)  # type: ignore[operator]

-        _compute_dask(moc, showProgress, "Computing {} MOC".format(regionName))
+        _compute_dask(moc_masked, showProgress, "Computing {} MOC".format(regionName))

-        mocs[regionName] = moc
+        mocs[regionName] = moc_masked


mpas.py changes to make mypy happy

tomvothecoder · 2025-12-23T20:02:56Z

e3sm_to_cmip/runner.py

        for path in paths:
            ds = xr.open_dataset(path, decode_timedelta=True)
-            data_vars = list(ds.data_vars.keys())
+            data_vars = [str(var) for var in ds.data_vars.keys()]


mpas.py changes to make mypy happy

tomvothecoder · 2025-12-23T20:03:18Z

tests/cmor_handlers/test__formulas.py

    with pytest.raises(KeyError):
-        rsuscs(xr.Dataset())
+        rtmt(xr.Dataset())


Fixed incorrect function reference in test.

tomvothecoder · 2025-12-23T20:03:23Z

tests/cmor_handlers/test__formulas.py

    # Test when required variable keys are NOT in the data dictionary.
    with pytest.raises(KeyError):
-        rsuscs(xr.Dataset())
+        tran(xr.Dataset())


Fixed incorrect function reference in test.

tomvothecoder · 2025-12-23T20:03:40Z

tests/cmor_handlers/test__formulas.py

 def test_tran():
-    ds = xr.Dataset(
-        data_vars={"QSOIL": _dummy_dataarray(), "QVEGT": _dummy_dataarray()}
-    )
+    ds = xr.Dataset(data_vars={"QVEGT": _dummy_dataarray()})

    result = tran(ds)
    expected = xr.DataArray(
        dims=["lat", "lon"],
-        data=np.array([[0, 2, 4], [0, 2, 4], [0, 2, 4]]),
+        data=np.array([[0, 1, 2], [0, 1, 2], [0, 1, 2]]),


Updated test_tran() to get it passing for v3 formula code change.

chengzhuzhang · 2026-03-11T22:32:33Z

@acme-y9s Xiaoying, I'm trying to wrapping up this PR in preparation for a new e3sm_to_cmip release. Do you have a chance to review the variables mentioned here. We are aiming to merge PRs by end of March. Thanks!

Copilot

Pull request overview

This PR updates E3SM-to-CMIP CMOR handlers and formulas to support additional ELM v3 land (Lmon) and land-ice (LImon) variables, including adjusting the definition of tran to exclude soil evaporation.

Changes:

Add new CMIP6 land and land-ice variable handler entries (and corresponding formula functions where needed).
Update tran handler/formula and align unit tests with the new definition.
Minor robustness/type-related adjustments in the runner and MPAS MOC computation, plus a new debug run script.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
`e3sm_to_cmip/cmor_handlers/handlers.yaml`	Adds new Lmon/LImon handlers and updates `tran` mapping/formula.
`e3sm_to_cmip/cmor_handlers/_formulas.py`	Updates `tran` formula and adds new formula functions for newly introduced handlers.
`tests/cmor_handlers/test__formulas.py`	Fixes/updates unit tests for `rtmt` KeyError behavior and `tran` output/inputs.
`e3sm_to_cmip/runner.py`	Ensures discovered dataset variable names are collected as `str`.
`e3sm_to_cmip/mpas.py`	Refactors MOC time series computation variables for clearer typing/flow and computes the masked/averaged result.
`scripts/debug/334-new_v3_land_handlers/run_v3_land.sh`	Adds a debug script for running v3 land handler CMORization.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

scripts/debug/334-new_v3_land_handlers/run_v3_land.sh

e3sm_to_cmip/cmor_handlers/handlers.yaml

e3sm_to_cmip/cmor_handlers/_formulas.py

update variables for v3

a7527e9

github-project-automation bot added this to E3SM to CMIP Development Dec 16, 2025

github-project-automation bot moved this to In progress in E3SM to CMIP Development Dec 16, 2025

chengzhuzhang added 3 commits December 16, 2025 13:03

remove sfcWind

a00d1aa

fix handlers and formula

900b83f

add run script example

0c889d4

chengzhuzhang marked this pull request as ready for review December 16, 2025 22:11

chengzhuzhang requested review from acme-y9s and tomvothecoder December 16, 2025 22:11

tomvothecoder added 2 commits December 23, 2025 13:21

Fix tran() test for v3 formula

9017c66

Fix mypy warnings

ef9ec17

tomvothecoder reviewed Dec 23, 2025

View reviewed changes

tomvothecoder approved these changes Dec 23, 2025

View reviewed changes

acme-y9s approved these changes Mar 12, 2026

View reviewed changes

chengzhuzhang requested a review from Copilot March 12, 2026 15:19

Copilot started reviewing on behalf of chengzhuzhang March 12, 2026 15:20 View session

Copilot AI reviewed Mar 12, 2026

View reviewed changes

chengzhuzhang added 2 commits March 23, 2026 19:08

Merge branch 'master' into update_land_var_v3

ae495be

address mypy issue

2d3fc95

chengzhuzhang merged commit da26c3d into master Mar 24, 2026
7 checks passed

github-project-automation bot moved this from In progress to Done in E3SM to CMIP Development Mar 24, 2026

chengzhuzhang deleted the update_land_var_v3 branch March 24, 2026 02:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update variables for v3 ELM#334

Update variables for v3 ELM#334
chengzhuzhang merged 8 commits intomasterfrom
update_land_var_v3

chengzhuzhang commented Dec 16, 2025 •

edited

Loading

Uh oh!

chengzhuzhang commented Dec 16, 2025

Uh oh!

tomvothecoder left a comment

Uh oh!

tomvothecoder Dec 23, 2025

Uh oh!

tomvothecoder Dec 23, 2025

Uh oh!

tomvothecoder Dec 23, 2025

Uh oh!

tomvothecoder Dec 23, 2025

Uh oh!

tomvothecoder Dec 23, 2025

Uh oh!

chengzhuzhang commented Mar 11, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

chengzhuzhang commented Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Uh oh!

chengzhuzhang commented Dec 16, 2025

Uh oh!

tomvothecoder left a comment

Choose a reason for hiding this comment

Uh oh!

tomvothecoder Dec 23, 2025

Choose a reason for hiding this comment

Uh oh!

tomvothecoder Dec 23, 2025

Choose a reason for hiding this comment

Uh oh!

tomvothecoder Dec 23, 2025

Choose a reason for hiding this comment

Uh oh!

tomvothecoder Dec 23, 2025

Choose a reason for hiding this comment

Uh oh!

tomvothecoder Dec 23, 2025

Choose a reason for hiding this comment

Uh oh!

chengzhuzhang commented Mar 11, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

chengzhuzhang commented Dec 16, 2025 •

edited

Loading