`dribble_crophrv_xsmrpool_2atm=.true.` crashes SASU runs

[wwieder]

Brief summary of bug

setting dribble_crophrv_xsmrpool_2atm=.true. crashes SASU runs, but AD and pSASU modes seem ok with this default setting

General bug information

CTSM version you are using: [output of git describe]
branch_tags/wetlands.n05.ctsm5.4.054

Does this bug cause significantly incorrect results in the model's science? [Yes / No]
Unknown, but crashes SASU runs

Configurations affected: [Fill this in if known.]
SASU spinup only.
As far as I can tell this is a new namelist setting, but I'm not sure where it came from?

Details of bug

AD and pSASU runs complete fine, but SASU run crashes with a C balance warning after ~18 months. As far as I can tell this is on a crop landunit in the tropics.

Case directory is:
/glade/work/wwieder/clm6_cases/wetlands_CplHist_377_ne30_BgcCrop_178_SASU

The case has failed twice now it the same spot with this error in the cesm.log

dec0496.hsn.de.hpc.ucar.edu 1995:  cbalance warning at c =          18  7.114417033740536E-004
dec0496.hsn.de.hpc.ucar.edu 1995:    5702.21044018915
dec0496.hsn.de.hpc.ucar.edu 1995:  column cbalance error    =   7.114417033740536E-004          18
dec0496.hsn.de.hpc.ucar.edu 1995:  is fates column?         =  F
dec0496.hsn.de.hpc.ucar.edu 1995:  Latdeg,Londeg=  -7.37580627510179        280.500000000000
dec0496.hsn.de.hpc.ucar.edu 1995:  begcb                    =    5702.21718595468
dec0496.hsn.de.hpc.ucar.edu 1995:  endcb                    =    5702.21044018915
dec0496.hsn.de.hpc.ucar.edu 1995:  delta store              =  -6.745765530467907E-003
dec0496.hsn.de.hpc.ucar.edu 1995:  --- Inputs ---
dec0496.hsn.de.hpc.ucar.edu 1995:  gpp                      =   0.000000000000000E+000
dec0496.hsn.de.hpc.ucar.edu 1995:  --- Outputs ---
dec0496.hsn.de.hpc.ucar.edu 1995:  er                       =   1.559041457422894E-002
dec0496.hsn.de.hpc.ucar.edu 1995:  col_fire_closs           =   0.000000000000000E+000
dec0496.hsn.de.hpc.ucar.edu 1995:  col_hrv_xsmrpool_to_atm  =   0.000000000000000E+000
dec0496.hsn.de.hpc.ucar.edu 1995:  col_xsmrpool_to_atm      =  -9.556090747057870E-003
dec0496.hsn.de.hpc.ucar.edu 1995:  wood_harvestc            =   0.000000000000000E+000
dec0496.hsn.de.hpc.ucar.edu 1995:  crop_harvestc_to_cropprodc =   0.000000000000000E+000
dec0496.hsn.de.hpc.ucar.edu 1995:  -1*som_c_leached         =   7.722170963320667E-014
dec0496.hsn.de.hpc.ucar.edu 1995: iam = 1867: local  column   index = 18
dec0496.hsn.de.hpc.ucar.edu 1995: iam = 1867: global column   index = 34362
dec0496.hsn.de.hpc.ucar.edu 1995: iam = 1867: global landunit index = 16070
dec0496.hsn.de.hpc.ucar.edu 1995: iam = 1867: global gridcell index = 8012
dec0496.hsn.de.hpc.ucar.edu 1995: iam = 1867: landunit type         = 2
dec0496.hsn.de.hpc.ucar.edu 1995: iam = 1867: column   type         = 267
dec0496.hsn.de.hpc.ucar.edu 1995: iam = 1867: gridcell longitude    =  280.5000000
dec0496.hsn.de.hpc.ucar.edu 1995: iam = 1867: gridcell latitude     =   -7.3758063
dec0496.hsn.de.hpc.ucar.edu 1995: WARNING: Trouble getting the 2D gridcell indices
dec0496.hsn.de.hpc.ucar.edu 1995:  ENDRUN:
dec0496.hsn.de.hpc.ucar.edu 1995:  ERROR in CNBalanceCheckMod.F90 at line 384

Setting dribble_crophrv_xsmrpool_2atm=.false. fixes this issue in SASU runs

see also discussion on LMWG_dev_#178

[wwieder]

Talking a bit more with @olyson we noticed that dribble_crophrv_xsmrpool_2atm is NOT in this case:
/glade/work/wwieder/wiemip/wetlands.n03.ctsm5.4.029_f09noCrop_flat_164_SASU

But it is a namelist option in this one
~slevis/cases_dev/wetlands.n04.ctsm5.4.044_f09BgcCrop_flat_174_SASU

• and it's set to true!?

Hopefully this narrows down when / how something broke?

See also, seems to have been set to true in 5.1.008

[ekluzek]

@wwieder it sounds like we sometimes have set this to false and sometimes not in SASU spinup cases. Can we tell how far back we've been doing that in spinup cases? I didn't find it explicitly mentioned in other LMWG_dev issues.

I also wonder if you can set it to false, for just a short period of time to get past a problem, and leave it on for most of the simulation?

[ekluzek]

Note, there is explicitly handling for matrixcn in how the code is used in CNCStateUpdate1Mod.F90. And that code goes back to 2020. It's still possible it's wrong for matrixcn, but it's not something where the code neglected matrixcn when it came in. And @olyson fixed a part of the matrixcn code here back in 2020.

We possibly need a bunch of people to look at the small bits of code in CNCStateUpdate1Mod.F90 that govern this and make sure it's correct. In looking at the above error diagnostic it does look like the error has to be in col_xsmrpool_to_atm or in er, as they are the only flux terms of substanstial value.