errh2o in BalanceCheck fails in NorESM-FATES historic simulations

[JessicaNeedham]

Brief summary of bug

Column level water balance checks are failing in an offline NorESM-FATES run.

General bug information

CTSM version you are using:
ctsm5.4.002_noresm_v1
FATES: sci.1.88.6_api.42.0.0_nor_sci1_api1

Configurations affected:
This seems to be triggered when we go from steadystate 1850 to transient although it might be coincident.

Details of bug

I ran 175 years of AD spinup and tried to use the restart file to start some historic simulations in which we don't care about soil carbon but just needed some spun up vegetation (see #182 ). It crashed during the second month with a water balance error.
Case is here: /cluster/work/users/jessica/fates-cal-runs/ppe_transient/noresm-fates_ne16_1850s_default_v2.2026-01-08

I then tried running a 10 year post-AD spinup (steady state 1850 climate, CO2, land use) which completed with no errors.
Using the restart from the post-AD spinup to start the historic simulation again resulted in errh2o fails eight months into the simulation.

405: WARNING: column-level water balance error nstep= 12627 local indexc=
405: 1 global indexc= 320 errh2o= 3.418778315640494E-005
405: CTSM is stopping because errh2o > 1.000000000000000E-005 mm
405: nstep = 12627
405: errh2o_col = 3.418778315640494E-005
405: forc_rain = 3.464186607388611E-003
405: forc_snow = 0.000000000000000E+000
405: endwb_col = 1504.92424297582
405: begwb_col = 1504.93135951385
405: qflx_evap_tot = 1.053306899608913E-002
405: qflx_sfc_irrig = 0.000000000000000E+000
405: qflx_surf = 8.184342741619088E-005
405: qflx_qrgwl = 0.000000000000000E+000
405: qflx_drain = 0.000000000000000E+000
405: qflx_ice_runoff = 0.000000000000000E+000
405: qflx_snwcp_discarded_ice = 0.000000000000000E+000
405: qflx_snwcp_discarded_liq = 0.000000000000000E+000
405: deltawb = -7.116538032960307E-003
405: deltawb/dtime = -3.953632240533503E-006
405: qflx_drain_perched = 0.000000000000000E+000
405: qflx_flood = 0.000000000000000E+000
405: qflx_glcice_dyn_water_flux = 0.000000000000000E+000
405: CTSM is stopping
405: iam = 277: local column index = 1
405: iam = 277: global column index = 320
405: iam = 277: global landunit index = 304
405: iam = 277: global gridcell index = 278
405: iam = 277: gridcell longitude = 30.9375000
405: iam = 277: gridcell latitude = -5.6353461
405: iam = 277: column type = 1
405: iam = 277: landunit type = 1
405: ENDRUN:
405: ERROR in BalanceCheckMod.F90 at line 705

See related issues: #174 and #165

[JessicaNeedham]

Also now happening here NorESMhub/noresm3_dev_simulations#313

[kjetilaas]

Some updates on this problem:

• it seems to mainly (only?) appear in transient land use cases. Our NorESM3 spinup simulations have been running with constant harvest land use timeseries for many centuries without triggering this error. But it is triggered after just a few months when switching to transient LU (historical cases).
• My test with setting really high treshold for triggering the endrun (1e4 times larger than the default) completed 15 years. This showed only 4 problematic grid cells, but the large h2o error was there for many consecutive time step, showing up clearly in the monthly time series (see figure below).
• In my run, all the problematic grid cells have full or close to full coverage of broadleaf evergreen tropical trees (BETTs), and a small but growing area of rangeland.
• Switching from clearcutting when transitioning from forest to rangeland (LU clearing logic 1) to not clearcutting (LU clearing logic 4) seems to reduce the problem (fewer grid cells and timesteps where the errh2o is large), but not make it go away fully.

Here are timeseries of hydrological variables for a non-problematic grid cell (first column), and the four (column 2-5) that have substantial ERRH2O (bottom row):

[kjetilaas]

Rerunning with daily output reveals that the error occurs in periods where the QVEGT (transpiration) is zero (second column, this only shows first 120 days):

[kjetilaas]

Path to relevant cases on betzy:

• nhist case where error is triggered: /cluster/work/users/kjetisaa/nhist.ne16pg3_tn14.nhist_cmip7.20260116
• case where endrun is only riggered with much higher errors (1e4 times higher): i1850.ne16pg3_tn14.ctsm5.4.002_noresm_v1.CPLHIST_histLU_highErrH2Olim.2026-01-19
• same case with daily output: /cluster/work/users/kjetisaa/nhist.ne16pg3_tn14.nhist_cmip7.VHErrH2Olim_daily.2026-01-21
• case with constant land use that does not trigger the error: /cluster/work/users/kjetisaa/nhist.ne16pg3_tn14.nhist_cmip7.constLU.20260118

[rosiealice]

The issue here seems to be that the plants are not responding to shifts in BTRAN, which seems to suggest to me that the minimum conductance is high enough that stomatal closure has little impact on QVEGT. So QVEGT continues to be high until the soil completely dries out, at which point there is no more water available for extraction and it crashes to 0.

To fix this, my suggestion is to use the fates_leaf_stomatal_btran_model set to 1 instead of 0. This will multiply the minimum stomatal conductance by BTRAN and thus make it, and hopefully also QVEGT, responsive to the soil drying.

I made a run with @JessicaNeedham's script that crashes after < 1 month, and another using this parameter file

/cluster/shared/noresm/inputdata/lnd/clm2/paramdata/fates_params_sci.1.88.6_api.42.0.0_14pft_nor_sci3_api1_c260123.nc, which @kjetilaas made, that runs for 20 years.

The changes are

< netcdf fates_params_sci.1.88.6_api.42.0.0_14pft_nor_sci3_api1_c260123 {
> netcdf fates_params_sci.1.88.6_api.42.0.0_14pft_nor_sci3_api1_c260110 {
1416c1416
<  fates_leaf_stomatal_btran_model = 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 3 ;
>  fates_leaf_stomatal_btran_model = 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 3 ;

[rosiealice]

The impact of this change on fluxes (in CPLHIST mode) is in here: https://ns9560k.web.sigma2.no/datalake/diagnostics/noresm/rosief/i2000.ne16pg3_tn14.fatesnocomp.noresm3_0_beta09.CPLHIST.BT3_LSP_MS3.25_BT1EBT.2026-01-24/OBS_comparison/ANN/

LH

GPP

NPP

LAI

n.b. that the GPP axes are set for the difference plot, whereas NPP is not. So that is why the GPP diffs are not clear, I think.

[rosiealice]

In terms of biases, this is the new LH FLUXCOM bias:

https://ns9560k.web.sigma2.no/datalake/diagnostics/noresm/rosief/i2000.ne16pg3_tn14.fatesnocomp.noresm3_0_beta09.CPLHIST.BT3_LSP_MS3.25_BT1EBT.2026-01-24/OBS_comparison/ANN/

compared with the previous/old/control version:
https://ns9560k.web.sigma2.no/datalake/diagnostics/noresm/rosief/i2000.ne16pg3_tn14.fatesnocomp.noresm3_0_beta09.CPLHIST.BT3_LSP_MS3.25.2026-01-20/OBS_comparison/ANN/

For GPP
new

old

LAI
new (don't have old yet)