v12 FV3 Standalone Broken at c360+ in weird way

[mathomp4]

Testing with the v12 FV3 Standalone finds that when running at c360 and higher (well, c360, 720, 1440), it produces garbage output.

For example, at c180 after 1 day (18910302 00z):

> cdo infon -selname,U fv3-v12-c180-2025Aug15-GNU/scratch_c180_L181_T2_20x120_1threads_GEOSgcm-v12.0.0-R4R8/c180_L181_T2_20x120_1threads.inst3_3d_diag.18910302_0000z.nc4
    -1 :       Date     Time   Level Gridsize    Miss :     Minimum        Mean     Maximum : Parameter name
     1 : 1891-03-02 00:00:00    1000   259920  259920 :                     nan             : U
     2 : 1891-03-02 00:00:00     975   259920       0 :    -0.22725      4.7740      9.6737 : U
     3 : 1891-03-02 00:00:00     950   259920       0 :    -0.11787      5.3865      10.895 : U
     4 : 1891-03-02 00:00:00     925   259920       0 :   -0.083330      6.0099      12.163 : U
...
    23 : 1891-03-02 00:00:00     200   259920       0 :   -0.059872      17.365      34.846 : U
    24 : 1891-03-02 00:00:00     150   259920       0 :   -0.035101      17.118      34.358 : U
    25 : 1891-03-02 00:00:00     100   259920       0 :   -0.013363      16.715      33.538 : U
cdo    infon: Processed 6498000 values from 1 variable over 1 timestep [0.17s 32MB]

There is a NaN at 1000 mbar, but I only ran this for 1-day.

Now we run at c360 and after one day (18910302 00z):

> cdo infon -selname,U fv3-v12-c360-2025Aug15-GNU/scratch_c360_L181_T2_30x180_1threads_GEOSgcm-v12.0.0-R4R8/c360_L181_T2_30x180_1threads.inst3_3d_diag.18910302_0000z.nc4
cdo(1) selname: Process started
    -1 :       Date     Time   Level Gridsize    Miss :     Minimum        Mean     Maximum : Parameter name
     1 : 1891-03-02 00:00:00    1000  1038240 1038240 :                     nan             : U
     2 : 1891-03-02 00:00:00     975  1038240 1038240 :                     nan             : U
     3 : 1891-03-02 00:00:00     950  1038240 1038240 :                     nan             : U
     4 : 1891-03-02 00:00:00     925  1038240 1038240 :                     nan             : U
...
    23 : 1891-03-02 00:00:00     200  1038240 1038240 :                     nan             : U
    24 : 1891-03-02 00:00:00     150  1038240 1038240 :                     nan             : U
    25 : 1891-03-02 00:00:00     100  1038240 1038240 :                     nan             : U
cdo    infon: Processed 25956000 values from 1 variable over 1 timestep [0.52s 35MB]

So EVERYTHING is a NaN! Not good.

I put out history a bit more often and it looks like at c360, from 2315z to 2330z, the whole atmosphere disappears.

I'll investigate more with prints.

[mathomp4]

Okay. I added some prints in DynCore_GridCompMod.F90:

      if (DEBUG_DYN) then
         call MAPL_MaxMin('DYN: Q_FILLOUT ', qv)
         call MAPL_MaxMin('DYN: T_FILLOUT ', tempxy)
         call MAPL_MaxMin('DYN: U_FILLOUT ', ur)
         call MAPL_MaxMin('DYN: V_FILLOUT ', vr)
      endif

and turned on DEBUG_DYN and:

 DYN Date: 1891/03/01  Time: 23:00:00  Throughput(days/day)[Avg Tot Run]:        683.8        703.7        724.1  TimeRemaining(Est) 000:00:05   15.4% :  16.5% Mem Comm:Used  Wallclock: 2025/08/18 11:54
 DYN: Q_AF_DYN    max =   0.0000000E+00  min =   0.0000000E+00
 DYN: T_AF_DYN    max =    331.6082      min =    138.4728
 DYN: U_AF_DYN    max =    564.5676      min =   -604.8680
 DYN: V_AF_DYN    max =    538.1857      min =   -604.0833
 DYN: Q_FILLOUT   max =   0.0000000E+00  min =   0.0000000E+00
 DYN: T_FILLOUT   max =    331.6082      min =    138.4728
 DYN: U_FILLOUT   max =    578.3463      min =   -604.8637
 DYN: V_FILLOUT   max =    544.8901      min =   -545.2348
 DYN Date: 1891/03/01  Time: 23:07:30  Throughput(days/day)[Avg Tot Run]:        683.9        720.3        721.0  TimeRemaining(Est) 000:00:04   15.4% :  16.5% Mem Comm:Used  Wallclock: 2025/08/18 11:54
 DYN: Q_AF_DYN    max =   0.0000000E+00  min =   0.0000000E+00
 DYN: T_AF_DYN    max =    332.1838      min =    138.7273
 DYN: U_AF_DYN    max =    743.6734      min =   -719.0574
 DYN: V_AF_DYN    max =    479.3998      min =   -640.3272
 DYN: Q_FILLOUT   max =   0.0000000E+00  min =   0.0000000E+00
 DYN: T_FILLOUT   max =    332.1838      min =    138.7273
 DYN: U_FILLOUT   max =    608.7967      min =   -640.2749
 DYN: V_FILLOUT   max =    718.8397      min =   -744.3303
 DYN Date: 1891/03/01  Time: 23:15:00  Throughput(days/day)[Avg Tot Run]:        684.1        718.8        720.1  TimeRemaining(Est) 000:00:03   15.4% :  16.5% Mem Comm:Used  Wallclock: 2025/08/18 11:54
 DYN: Q_AF_DYN    max =   0.0000000E+00  min =   0.0000000E+00
 DYN: T_AF_DYN    max =  -3.4028235E+38  min =   3.4028235E+38  has NaN
 DYN: U_AF_DYN    max =    555.5521      min =   -572.6341      has NaN
 DYN: V_AF_DYN    max =    609.6443      min =   -528.1160      has NaN
 DYN: Q_FILLOUT   max =   0.0000000E+00  min =   0.0000000E+00
 DYN: T_FILLOUT   max =  -3.4028235E+38  min =   3.4028235E+38  has NaN
 DYN: U_FILLOUT   max =    765.2757      min =   -532.7527      has NaN
 DYN: V_FILLOUT   max =    752.0729      min =   -538.2581      has NaN
 DYN Date: 1891/03/01  Time: 23:22:30  Throughput(days/day)[Avg Tot Run]:        684.2        702.5        705.4  TimeRemaining(Est) 000:00:03   15.4% :  16.5% Mem Comm:Used  Wallclock: 2025/08/18 11:54
 DYN: Q_AF_DYN    max =   0.0000000E+00  min =   0.0000000E+00
 DYN: T_AF_DYN    max =  -3.4028235E+38  min =   3.4028235E+38  has NaN
 DYN: U_AF_DYN    max =  -3.4028235E+38  min =   3.4028235E+38  has NaN
 DYN: V_AF_DYN    max =  -3.4028235E+38  min =   3.4028235E+38  has NaN
 DYN: Q_FILLOUT   max =   0.0000000E+00  min =   0.0000000E+00
 DYN: T_FILLOUT   max =  -3.4028235E+38  min =   3.4028235E+38  has NaN
 DYN: U_FILLOUT   max =  -3.4028235E+38  min =   3.4028235E+38  has NaN
 DYN: V_FILLOUT   max =  -3.4028235E+38  min =   3.4028235E+38  has NaN

So, at 2315z all of the sudden, BAM, the world disappears!

[mathomp4]

Note: weirdly, the standalone keeps running just fine after this. Even though everything seems to be NaN? It runs at least 2 days past this .

[mathomp4]

Another update: I ran C48 for 1 month and I never saw any NaNs appear. I'll try a long C180 run as well to see if maybe this happens there? Baffled...

[mathomp4]

Okay. I think the issue might just be "C360 is unstable". First, here is U from C48 running a month:

Now here is C360 before the NaNs appear:

[mathomp4]

And here is the odd proof that FV3 is just fine running with NaN winds?

[wmputman]

Well, those winds are ridiculous. Must be a setup issue with the changed DT

[mathomp4]

@wmputman Should the standalone use slower DT? At the moment, I have fv3_setup mirroring gcm_setup. So C360, for example, runs with a DT of 450 s. Should I drop that down to like 300?