Add coastal_sandy_duck_atm2sch2ww3 regression test

[mansurjisan]

Commit Queue Requirements:

• Fill out all sections of this template.
• All sub component pull requests have been reviewed by their code managers.
• Run the full Intel+GNU RT suite (compared to current baselines) on either Hera/Derecho/Hercules
• Commit 'test_changes.list' from previous step

---

Description:

This PR adds a new coastal regression test for Hurricane Sandy at Duck, NC with ATM-SCHISM-WW3 coupling configuration. The test includes:

• 6-hour simulation using Data ATM, SCHISM, and WW3 models
• 80 CPU configuration with overlapping MED/ATM PE layout
• Fixes to enable WW3 compilation for CSTLSW application
• Updates to ensure all required WW3 input files are copied during test setup

This PR also fixes an issue where coastal_ike_shinnecock_atm2sch2ww3 was failing in the feature/new_rt branch due to missing CSTLSW in the WW3 enable regex and incompatible build flags.

Commit Message:

• UFSWM - Add coastal_sandy_duck_atm2sch2ww3 regression test
  • Fix CSTLSW WW3 component enabling in configure_apps.cmake
  • Remove -DBUILD_TOOLS=ON from atm2sch2ww3 compile configurations
  • Add nest.ww3 and namelists.nml copy to coastal_ww3_run.IN
  • Configure 6-hour run with 80 CPUs
  • Update LIST_FILES to include WW3 field outputs and restart files

* UFSWM - 
  * AQM - 
  * CDEPS - 
  * CICE - 
  * CMEPS - 
  * CMakeModules - 
  * FV3 - 
    * ccpp-physics - 
    * atmos_cubed_sphere - 
  * GOCART - 
  * HYCOM - 
  * MOM6 - 
  * NOAHMP - 
  * WW3 - 
  * fire_behavior
  * stochastic_physics - 
  * ADCIRC -
  * FVCOM -
  * PAHM -
  * ROMS -
  * SCHISM -
  * SCHISM-ESMF -
  * GEOGATE -

Priority:

• Normal

Git Tracking

UFSWM:

• None

Sub component Pull Requests:

• None
• AQM:
• CDEPS:
• CICE:
• CMEPS:
• CMakeModules:
• FV3:
  • ccpp-physics:
  • atmos_cubed_sphere:
• GOCART:
• HYCOM:
• MOM6:
• NOAHMP:
• WW3:
• fire_behavior:
• stochastic_physics:
• ADCIRC:
• FVCOM:
• PAHM:
• ROMS:
• SCHISM:
• SCHISM-ESMF:
• GEOGATE:
• None

UFSWM Blocking Dependencies:

• None

• Blocked by #

• None

Documentation:

• No documentation update is required for this PR (new regression test added to existing framework).

---

Changes

Regression Test Changes (Please commit test_changes.list):

• PR Adds New Tests/Baselines.

Input data Changes:

• New input data.
  • Hercules location: /work2/noaa/nos-surge/mjisan/RT_INPUT/
  • Components:
    • DATM: /work2/noaa/nos-surge/mjisan/RT_INPUT/DATM_input_data/Sandy/ (mesh_datm.nc, wind.nc)
    • SCHISM: /work2/noaa/nos-surge/mjisan/RT_INPUT/SCHISM/Sandy/atm2sch2wav/ (hgrid.gr3, vgrid.in, param.nml, bctides.in, etc.)
    • WW3: /work2/noaa/nos-surge/mjisan/RT_INPUT/WW3/Sandy/atm2sch2wav/ (mesh_duck.nc, mod_def.ww3, nest.ww3, ww3_shel.nml, namelists.nml, points.list)

Baseline Location:

• Hercules: /work2/noaa/nos-surge/mjisan/ufs-weather-model/tests/stmp/mjisan/FV3_RT/REGRESSION_TEST/coastal_sandy_duck_atm2sch2ww3_intel/

Library Changes/Upgrades:

• Required
  • Library names w/versions:
  • Git Stack Issue (JCSDA/spack-stack#)
• No Updates

---

Testing Log:

• RDHPCS
  • Hera
  • Orion
  • Hercules
  • GaeaC6
  • Derecho
  • Ursa
• WCOSS2
  • Dogwood/Cactus
  • Acorn
• CI
• opnReqTest (complete task if unnecessary)

[mansurjisan]

Hi @uturuncoglu ,

I added the Duck NC regression test for the DATM+SCHISM+WW3 configuration. Please take a look when you get a chance.

The input data location on Hercules HPC is: /work2/noaa/nos-surge/mjisan/RT_INPUT/

And the baseline directory is:
/work2/noaa/nos-surge/mjisan/ufs-weather-model/tests/stmp/mjisan/FV3_RT/REGRESSION_TEST/coastal_sandy_duck_atm2sch2ww3_intel/

The regression tests show a failing status when compared to the baseline results. I double checked the results and the differences in water elevation and wave height between baseline results and RT are zero at the first time step, but differences increase with each time step. This is something we also observed in Ike Shinnecock DATM+SCHISM+WW3 run.

[uturuncoglu]

@mansurjisan I checked your configuration. It seems you are using 300 s as coupling interval. So, I modified the case to write mediator history file in every Couling step to see when or which component is starting to change the answer. The first coupling time step (t=300) the files were identical (I am checking with NCAR's cprnc tool) between two different runs but then at 600, it started to change. Here are the variables that are changing,

 wavImp_Sw_hs   (wavImp_nx,wavImp_ny,time)  t_index =      1     1
          2    23018  (   924,     1,     1) (     1,     1,     1) (  3342,     1,     1) (  3342,     1,     1)
               23018   1.727665305137634E+00   0.000000000000000E+00 2.2E-08  1.147124618291855E-01 1.2E-11  1.147124618291855E-01
               23018   1.727665305137634E+00   0.000000000000000E+00          1.147124841809273E-01          1.147124841809273E-01
               23018  (   924,     1,     1) (     1,     1,     1)
          avg abs field values:    8.082532221122861E-02    rms diff: 1.6E-10   avg rel diff(npos):  1.2E-11
                                   8.082532221252335E-02                        avg decimal digits(ndif):  6.9 worst:  6.7
 RMS wavImp_Sw_hs                     1.5529E-10            NORMALIZED  1.9214E-09

 wavImp_Sw_t0m1   (wavImp_nx,wavImp_ny,time)  t_index =      1     1
          3    23018  (    24,     1,     1) (     1,     1,     1) (  3342,     1,     1) (  3342,     1,     1)
               23018   8.573410034179688E+00   0.000000000000000E+00 3.8E-06  7.374127388000488E+00 3.1E-11  7.374127388000488E+00
               23018   8.573410034179688E+00   0.000000000000000E+00          7.374123573303223E+00          7.374123573303223E+00
               23018  (    24,     1,     1) (     1,     1,     1)
          avg abs field values:    2.623224275070617E+00    rms diff: 2.6E-08   avg rel diff(npos):  3.1E-11
                                   2.623224274842743E+00                        avg decimal digits(ndif):  6.8 worst:  6.3
 RMS wavImp_Sw_t0m1                   2.6107E-08            NORMALIZED  9.9523E-09

 wavImp_Sw_tauox   (wavImp_nx,wavImp_ny,time)  t_index =      1     1
          7    23018  (     1,     1,     1) ( 22853,     1,     1) (  3341,     1,     1) (  3341,     1,     1)
               23018   0.000000000000000E+00  -2.491105442459229E-05 2.7E-12 -8.338663064932916E-06 4.1E-11 -8.338663064932916E-06
               23018   0.000000000000000E+00  -2.491105442459229E-05         -8.338665793417022E-06         -8.338665793417022E-06
               23018  (     1,     1,     1) ( 22853,     1,     1)
          avg abs field values:    1.773267464934847E-05    rms diff: 2.3E-14   avg rel diff(npos):  4.1E-11
                                   1.773267464958554E-05                        avg decimal digits(ndif):  7.0 worst:  6.5
 RMS wavImp_Sw_tauox                  2.3217E-14            NORMALIZED  1.3093E-09

 wavImp_Sw_tauoy   (wavImp_nx,wavImp_ny,time)  t_index =      1     1
          5    23018  (     1,     1,     1) ( 19024,     1,     1) (  3341,     1,     1) (  3341,     1,     1)
               23018   0.000000000000000E+00  -1.926619916048367E-05 1.8E-12 -6.381183538906043E-06 4.4E-11 -6.381183538906043E-06
               23018   0.000000000000000E+00  -1.926619916048367E-05         -6.381185357895447E-06         -6.381185357895447E-06
               23018  (     1,     1,     1) ( 19024,     1,     1)
          avg abs field values:    1.356660737590916E-05    rms diff: 2.3E-14   avg rel diff(npos):  4.4E-11
                                   1.356660737622526E-05                        avg decimal digits(ndif):  6.7 worst:  6.5
 RMS wavImp_Sw_tauoy                  2.2827E-14            NORMALIZED  1.6826E-09

 wavImp_Sw_thm   (wavImp_nx,wavImp_ny,time)  t_index =      1     1
          2    23018  (    33,     1,     1) (     1,     1,     1) (  3342,     1,     1) (  4134,     1,     1)
               23018   1.018731679432967E+02   0.000000000000000E+00 1.4E-05  7.589219918864546E+01 1.6E-11  7.524442408332834E+01
               23018   1.018731679432967E+02   0.000000000000000E+00          7.589218552826787E+01          7.524441042295075E+01
               23018  (    33,     1,     1) (     1,     1,     1)
          avg abs field values:    5.515447084941805E+01    rms diff: 1.3E-07   avg rel diff(npos):  1.6E-11
                                   5.515447084823114E+01                        avg decimal digits(ndif):  6.7 worst:  6.7
 RMS wavImp_Sw_thm                    1.2733E-07            NORMALIZED  2.3087E-09

wavImp_Sw_ustokes   (wavImp_nx,wavImp_ny,time)  t_index =      1     1
          6    23018  (     1,     1,     1) (  1637,     1,     1) (  3342,     1,     1) (  3342,     1,     1)
               23018   0.000000000000000E+00  -1.459205001592636E-01 4.9E-09 -3.351771971210837E-03 1.1E-10 -3.351771971210837E-03
               23018   0.000000000000000E+00  -1.459205001592636E-01         -3.351776860654354E-03         -3.351776860654354E-03
               23018  (     1,     1,     1) (  1637,     1,     1)
          avg abs field values:    2.266893407731473E-03    rms diff: 3.3E-11   avg rel diff(npos):  1.1E-10
                                   2.266893408055159E-03                        avg decimal digits(ndif):  6.6 worst:  5.8
 RMS wavImp_Sw_ustokes                3.3270E-11            NORMALIZED  1.4677E-08

 wavImp_Sw_vstokes   (wavImp_nx,wavImp_ny,time)  t_index =      1     1
          6    23018  (     1,     1,     1) (  1637,     1,     1) (  3342,     1,     1) (  3342,     1,     1)
               23018   0.000000000000000E+00  -1.116150170564651E-01 4.0E-09 -2.498492598533630E-03 1.1E-10 -2.498492598533630E-03
               23018   0.000000000000000E+00  -1.116150170564651E-01         -2.498496556654572E-03         -2.498496556654572E-03
               23018  (     1,     1,     1) (  1637,     1,     1)
          avg abs field values:    1.640613026644347E-03    rms diff: 2.7E-11   avg rel diff(npos):  1.1E-10
                                   1.640613026907341E-03                        avg decimal digits(ndif):  6.7 worst:  5.8
 RMS wavImp_Sw_vstokes                2.7259E-11            NORMALIZED  1.6615E-08

 wavImp_Sw_wavsuu   (wavImp_nx,wavImp_ny,time)  t_index =      1     1
          1    23018  (   902,     1,     1) (     1,     1,     1) (  3342,     1,     1) (  3342,     1,     1)
               23018   9.622412719726562E+02   0.000000000000000E+00 4.8E-07  6.471410751342773E+00 3.2E-12  6.471410751342773E+00
               23018   9.622412719726562E+02   0.000000000000000E+00          6.471411228179932E+00          6.471411228179932E+00
               23018  (   902,     1,     1) (     1,     1,     1)
          avg abs field values:    6.322761208439165E+00    rms diff: 3.1E-09   avg rel diff(npos):  3.2E-12
                                   6.322761208459881E+00                        avg decimal digits(ndif):  7.1 worst:  7.1
 RMS wavImp_Sw_wavsuu                 3.1429E-09            NORMALIZED  4.9708E-10

 wavImp_Sw_wavsuv   (wavImp_nx,wavImp_ny,time)  t_index =      1     1
          5    23018  (    57,     1,     1) (    33,     1,     1) (  3342,     1,     1) (  3342,     1,     1)
               23018   8.336565399169922E+01  -1.347398071289062E+02 4.8E-07  7.341479063034058E-01 4.4E-11  7.341479063034058E-01
               23018   8.336565399169922E+01  -1.347398071289062E+02          7.341483831405640E-01          7.341483831405640E-01
               23018  (    57,     1,     1) (    33,     1,     1)
          avg abs field values:    6.254981245052286E-01    rms diff: 3.2E-09   avg rel diff(npos):  4.4E-11
                                   6.254981245350075E-01                        avg decimal digits(ndif):  6.9 worst:  6.2
 RMS wavImp_Sw_wavsuv                 3.2218E-09            NORMALIZED  5.1507E-09

 wavImp_Sw_wavsvv   (wavImp_nx,wavImp_ny,time)  t_index =      1     1
          3    23018  (   902,     1,     1) (     1,     1,     1) (  3342,     1,     1) (  3342,     1,     1)
               23018   4.894202880859375E+02   0.000000000000000E+00 4.8E-07  2.434401988983154E+00 2.0E-11  2.434401988983154E+00
               23018   4.894202880859375E+02   0.000000000000000E+00          2.434402465820312E+00          2.434402465820312E+00
               23018  (   902,     1,     1) (     1,     1,     1)
          avg abs field values:    2.563598675203335E+00    rms diff: 4.0E-09   avg rel diff(npos):  2.0E-11
                                   2.563598675244767E+00                        avg decimal digits(ndif):  6.9 worst:  6.7
 RMS wavImp_Sw_wavsvv                 4.0065E-09            NORMALIZED  1.5628E-09

 wavImp_Sw_wnmean   (wavImp_nx,wavImp_ny,time)  t_index =      1     1
          5    23018  (  1016,     1,     1) (     1,     1,     1) (  3342,     1,     1) (  3342,     1,     1)
               23018   7.396649360656738E+00   0.000000000000000E+00 6.6E-07  4.718177914619446E-01 1.1E-10  4.718177914619446E-01
               23018   7.396649360656738E+00   0.000000000000000E+00          4.718184471130371E-01          4.718184471130371E-01
               23018  (  1016,     1,     1) (     1,     1,     1)
          avg abs field values:    4.496921698925085E+00    rms diff: 4.8E-09   avg rel diff(npos):  1.1E-10
                                   4.496921698978170E+00                        avg decimal digits(ndif):  6.5 worst:  5.9
 RMS wavImp_Sw_wnmean                 4.7835E-09            NORMALIZED  1.0637E-09

As you could see all the fields have wavImp_* prefix. So, those are exported from the wave component. There might be some issue in the wave model configuration that leads to the difference. I am not seeing any issue in ocean side. To be sure, it would be nice to

(a) run the model without any feedback from ocean to wave. If you still see some difference, that definitely there is something wrong with wave setup.

(b) run the same case without ocean like DATM-WAV. If WAV results are still changing, there again there is something wrong with wave.

Another question is that this configuration uses different atmospheric forcing than atm2sch. Why? We are trying to bring different Duck configurations but I am expecting to have them consistent in terms of forcing and just adding component or removing. So, we could clearly see the affect of single component or catch the issues. Anyway, let's discuss those in the meeting.