add EAMxx support for cmorization

[chengzhuzhang]

Description

Summary

Adds handler entries to map EAMxx output variable names to CMIP6 variables, resolving
#339. EAMxx uses descriptive variable names (e.g., LW_flux_up_at_model_top,
surf_radiative_T) rather than the short names used by EAM (e.g., FLUT, TS). The new entries coexist with existing EAM entries — the handler system selects whichever raw_variables are present in the input data.

31 new handler entries covering:

Category: Precipitation
CMIP variables: pr, prw
EAMxx source variables: precip_liq_surf_mass_flux, precip_ice_surf_mass_flux, VapWaterPath
────────────────────────────────────────
Category: Near-surface state
CMIP variables: ts, tas, huss, sfcWind, psl
EAMxx source variables: surf_radiative_T, T_2m, qv_2m, wind_speed_10m, SeaLevelPressure
────────────────────────────────────────
Category: Surface fluxes
CMIP variables: hfls, hfss, evspsbl, tauu, tauv
EAMxx source variables: surface_upward_latent_heat_flux, surf_sens_flux, surf_evap, surf_mom_flux_U/V
────────────────────────────────────────
Category: TOA radiation
CMIP variables: rsdt, rsut, rsutcs, rlut, rlutcs
EAMxx source variables: SW/LW_flux_at_model_top, clrsky
────────────────────────────────────────
Category: Surface radiation
CMIP variables: rsds, rsus, rsdscs, rsuscs, rlds, rlus, rldscs
EAMxx source variables: SW/LW_flux_at_model_bot, clrsky
────────────────────────────────────────
Category: Cloud water
CMIP variables: clwvi, clivi
EAMxx source variables: LiqWaterPath + IceWaterPath, IceWaterPath
────────────────────────────────────────
Category: Aerosol
CMIP variables: od550aer
EAMxx source variables: AerosolOpticalDepth550nm
────────────────────────────────────────
Category: 3D pressure-level
CMIP variables: ta, hus, hur, wap, zg
EAMxx source variables: T_mid, qv, RelativeHumidity, omega, z_mid

Key implementation notes

• pr formula (_formulas.py): Extended to handle EAMxx variables (precip_liq_surf_mass_flux + precip_ice_surf_mass_flux). EAMxx fluxes are
  already in kg m-2 s-1, so no unit scaling is needed (unlike EAM's PRECC+PRECL which requires ×1000).
• clwvi formula (_formulas.py): EAMxx LiqWaterPath is liquid-only (equivalent to EAM TGCLDLWP). To satisfy the CMIP6 definition ("condensed
  water path = liquid + ice"), both LiqWaterPath and IceWaterPath must be summed. This differs from EAM, where TGCLDCWP already contains the
  total.
• tauu/tauv: EAMxx surf_mom_flux_U/V uses opposite sign convention to CMIP6, consistent with EAM TAUX/TAUY — unit_conversion: "-1" applied.
• Radiation sign conventions: All TOA and surface radiation handlers have positive: up or positive: down set to match the physical
  direction of the EAMxx variable name. For several EAM variables that require derived formulas (e.g., rlus = FLDS + FLNS, rsus = FSDS -
  FSNS), EAMxx provides direct output variables, simplifying the handlers.

Mappings verified against

DERIVED_VARIABLES in https://github.com/E3SM-Project/e3sm_diags/blob/main/e3sm_diags/derivations/derivations.py — all EAMxx entries (marked EAMxx) cross-checked.

• Closes [Feature]: Support EAMxx variable convention #339

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• My changes generate no new warnings
• Any dependent changes have been merged and published in downstream modules

If applicable:

• New and existing unit tests pass with my changes (locally and CI/CD build)
• I have added tests that prove my fix is effective or that my feature works
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have noted that this is a breaking change for a major release (fix or feature that would cause existing functionality to not work as expected)

[chengzhuzhang]

@zhangshixuan1987 and @crterai just to FYI, this is a PR to add EAMxx support to convert variables to format that is supported by community diagnostics package such as PMP and ILAMB. I will test with existing EAMxx run from you and tag reviewers after a test run.

[chengzhuzhang]

@copilot can you try fix the build error?

[Copilot]

@chengzhuzhang I've opened a new pull request, #341, to work on those changes. Once the pull request is ready, I'll request review from you.

[chengzhuzhang]

Both Claude Code and Copilot found the root cause: pd.DataFrame converts YAML null → pd.NA in pandas 3.x; pd.NA is not None is True, incorrectly triggering the ValueError on abs550aer
(the first handler in the file)

However, this solution provided by Claude Code 40c65bf:Iterate directly over the YAML list of dicts — YAML null stays as Python None with no conversion artifacts. It seems to be slightly cleaner and more elegant than what is provided by Copilot here #341.

[chengzhuzhang]

@zhangshixuan1987 I think this PR to add e3sm_to_cmip conversion for EAMxx variables is close to be done.
Based on your data, I created the cmip version of variables here: /pscratch/sd/c/chengzhu/e3sm_to_cmip_test/339-eamxx-handlers/CMIP6/CMIP/E3SM-Project/ using a template metadata file. When you get a chance, could try your pcmdi_diags workflow on these, so that we can cross compare the results?

Below is a list of variables being generated. Some are missing because lack of native output.

EAMxx to CMIP Variable Mapping

2D variables

CMIP name	Native EAMxx name(s)	Missing?
pr	precip_liq_surf_mass_flux, precip_ice_surf_mass_flux	No
prw	VapWaterPath	Yes
ts	surf_radiative_T	No
tas	T_2m	No
huss	qv_2m	No
hfls	surface_upward_latent_heat_flux	No
hfss	surf_sens_flux	No
evspsbl	surf_evap	No
tauu	surf_mom_flux_U	Yes
tauv	surf_mom_flux_V	Yes
sfcWind	wind_speed_10m	No
psl	SeaLevelPressure	No
rsdt	SW_flux_dn_at_model_top	No
rsut	SW_flux_up_at_model_top	No
rsutcs	SW_clrsky_flux_up_at_model_top	No
rlut	LW_flux_up_at_model_top	No
rlutcs	LW_clrsky_flux_up_at_model_top	No
rsds	SW_flux_dn_at_model_bot	No
rsus	SW_flux_up_at_model_bot	No
rsdscs	SW_clrsky_flux_dn_at_model_bot	No
rsuscs	SW_clrsky_flux_up_at_model_bot	No
rlds	LW_flux_dn_at_model_bot	No
rlus	LW_flux_up_at_model_bot	No
rldscs	LW_clrsky_flux_dn_at_model_bot	No
clwvi	LiqWaterPath, IceWaterPath	Yes
clivi	IceWaterPath	No
od550aer	AerosolOpticalDepth550nm	Yes

3D variables

CMIP name	Native EAMxx name	Missing?
ta	T_mid	No
hus	qv	No
hur	RelativeHumidity	No
wap	omega	No
zg	z_mid	No

[chengzhuzhang]

@tomvothecoder This is mostly an FYI, but it would be great if you could review the code to see if something is missing.

[Copilot]

Pull request overview

Adds EAMxx variable-name support to the cmorization handler system so EAMxx
outputs can be mapped to CMIP6 variables (issue #339) alongside existing EAM
mappings.

Changes:

• Added a new block of EAMxx handler entries to map descriptive EAMxx variable
  names to CMIP6 targets.
• Extended cmor handler formula support (notably pr) and added a clwvi
  formula helper for EAMxx condensed water path.
• Refactored YAML handler loading to remove the pandas dependency; updated tests
  and added debug scripts for Perlmutter workflows.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
e3sm_to_cmip/cmor_handlers/handlers.yaml	Adds EAMxx handler entries for CMIP6 variables using EAMxx raw variable names.
e3sm_to_cmip/cmor_handlers/_formulas.py	Extends pr formula for EAMxx precip fluxes and adds clwvi sum for EAMxx water paths.
e3sm_to_cmip/cmor_handlers/utils.py	Removes pandas-based YAML parsing and iterates YAML entries directly.
tests/cmor_handlers/test_utils.py	Updates expected handler list to include the new EAMxx pr handler.
scripts/debug/339-eamxx-handlers/run_eamxx_vars.py	Adds a Perlmutter-only debug driver script for generating/regridding and cmorizing EAMxx variables.
scripts/debug/339-eamxx-handlers/prep_vert_remap.sh	Adds a helper script to build vert_L128.nc by extracting hybrid-coord vars and injecting P0.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[chengzhuzhang]

@zhangshixuan1987 thank you for review the datasets and the suggestions! I added one commit (c28a069) to fix the precipitation handler and another (f56e5f6) for resume the vertical interpolation options to matching zppy's workflow, this will make EAM and EAMxx results more comparable as you suggested.