v12 SCM cfmip2 cases are broken

[mathomp4]

In the recent v12 development, all the cfmip2 cases are failing:

> rg fail slurm-44897258_*out
slurm-44897258_7.out
3:TASK_ID: 7. Experiment: /discover/nobackup/mathomp4/SystemTests/runs/SCM_V12/CURRENT/run/scm_cfmip2-s11/. Result: FAILURE
4:TASK_ID: 7. Experiment: /discover/nobackup/mathomp4/SystemTests/runs/SCM_V12/CURRENT/run/scm_cfmip2-s12/. Result: FAILURE
5:TASK_ID: 7. Experiment: /discover/nobackup/mathomp4/SystemTests/runs/SCM_V12/CURRENT/run/scm_cfmip2-s6/. Result: FAILURE

slurm-44897258_6.out
4:TASK_ID: 6. Experiment: /discover/nobackup/mathomp4/SystemTests/runs/SCM_V12/CURRENT/run/scm_cfmip2-p11/. Result: FAILURE
5:TASK_ID: 6. Experiment: /discover/nobackup/mathomp4/SystemTests/runs/SCM_V12/CURRENT/run/scm_cfmip2-p12/. Result: FAILURE
6:TASK_ID: 6. Experiment: /discover/nobackup/mathomp4/SystemTests/runs/SCM_V12/CURRENT/run/scm_cfmip2-p6/. Result: FAILURE

and they are the only ones failing.

Now, @atrayano and I found that other failures in recent v12 (at c12 and c24) could be "fixed" by lowering the time steps (see GEOS-ESM/GEOSgcm_App#715).

But the cfmip2 cases are running at like 450 seconds, so that doesn't matter.

So, my query for @narnold1 is: Is it possible moist currently has defaults that might not be "right" for SCM runs? For example, I see some code that looks resolution dependent:

           ! include grid cell area scaling and limit RHcrit to > 70%
             minrhcrit = 1.0 - min(0.3,(1.0-minrhcrit)*SQRT(SQRT(AREA(I,J)/1.e10))+0.01)

and, well, I'm not sure what the AREA is for an SCM run. We must fill it with something, but maybe it's just garbage?

[atrayano]

The actual error was due to temperature exceeding temp_ref_max (internally set to 355K) inside Solar, but the largest value in tlay array was 392.4. This reminds me to the increment spikes that we observed in January

[narnold1]

For the SCM, AREA is set to 1e10 in DatmoDyn. I just checked the cfmip2-s11 case and AREA looks fine. The run fails after about 24 h, seemingly due to large T/Q tendencies coming from Moist, and a large UW mass flux. The root cause is likely the initial conditions, which have the initial temperature profile at a constant ~236 K over most of the troposphere. Did the cfmip2 binary files change recently? If not, this case may have been crazy for awhile, and v12 is just less able to handle the stress test. (EDIT: corrected to refer to cfmip2 files, not restarts)

[mathomp4]

@narnold1 I don't think I've touched much of anything. I mean, I did add some ESMF weight files in an attempt to speed things up, but that didn't seem to work.