Issue with the initial timestep in ww3_multi

[CarstenHansen]

I am trying a simple setup with ww3_multi. This is just for testing it it works with our compilers - we are going to select a proper compiler (Gnu gfortran-13 or Intel ifx-2023) for our new cluster, but at now testing on our old (year 2019) cluster. We don't use ww3_multi for our forecast production, but we may do so in the future. I have managed to make it work with the main branch (version 6.06) where I had to select compile options with gfortran-13 ( -finit-local-zero -finit-derived ) to force initialization of arrays to zero.

With the develop branch I run into a new issue. After putting some extra MPI_BARRIER calls into wmwavemd.F90 I got a backtrace output to stderr tracing back to this line in w3wavemd.F90

WW3/model/src/w3wavemd.F90

Line 2413 in 3eb8161

( (DSEC21(TIME,TONEXT(:,7)).EQ.0.) .AND. FLOUT(7) .AND. SBSED ) ) THEN

This seems to be logically wrong, as the variable TONEXT(:,7) is defined (or initialised) only for coupled setup (with ocean, atmosphere model), which I am not testing. If I am wrong in this assumption, I will look further into my setup.

UPDATE: I will elaborate the question a little further.

First, I find from looking through recent Issues, that the identification by @ukmo-ccbunney of uninitialized local variables has helped me identify this issue. I think it is a bug, but I don't understand that it hasn't been caught in the regression tests.

It is a bug because there remains no issue with my setup after changing the condition

        IF ( ( (DSEC21(TIME,TONEXT(:,1)).EQ.0.) .AND. FLOUT(1) ) .OR. &
        (  (DSEC21(TIME,TONEXT(:,7)).EQ.0.) .AND. FLOUT(7) .AND. SBSED ) ) THEN

to:

        IF ( FLOUTG .OR. ( FLOUTG2 .AND. SBSED ) ) THEN

FLOUTG and FLOUTG2 are local logicals defined exactly so shortly before in the same code block.

My compile and link commands are as follows with openmpi-4.1.5/gfortran-13.1.0-debug:
Compile: mpif90 -c -J/home/cha/Projects/WAVEWATCH_v7_devel/BUILD/mod -g -fno-second-underscore -ffree-line-length-none -O0 -Wall -Wno-unused -fcheck=all -ffpe-trap=invalid,zero,overflow -frecursive -fbacktrace -fallow-argument-mismatch -finit-local-zero -finit-derived -I/opt/mpi/openmpi-4.1.5/gfortran-13.1.0-debug/include -I/home/cha/Projects/WAVEWATCH_v7_devel/BUILD/src wmwavemd.F90 (for each source file)
Link: mpif90 -o ww3_multi -g -O0 -Wall -fcheck=all -ffpe-trap=invalid,zero,overflow -frecursive -fbacktrace -finit-local-zero -finit-derived $objects -Wl,-R/opt/gcc-13.1.0/lib64 -Wl,-R/opt/isl-0.24/lib -Wl,-R/opt/mpc-1.3.1/lib -Wl,-R/opt/mpi/openmpi-4.1.5/gfortran-13.1.0-debug/lib -L/opt/mpi/openmpi-4.1.5/gfortran-13.1.0-debug/lib -lmpi -Wl,-rpath,/opt/mpi/openmpi-4.1.5/gfortran-13.1.0-debug/lib (where $objects are all the compiled object files)

I test a simplified setup with two grids corresponding to our grids for the North Sea-Baltic region, but omitting two-way boundary transfer, and running in spin-up mode for two hours. See the attached ww3_multi.nml file.

Here are the backtrace lines written to standard error:

 Error termination. Backtrace:
 At line 559 of file /home/cha/Projects/WAVEWATCH_v7_devel/BUILD/src/w3timemd.F90
 Fortran runtime error:
 Index '43' of dimension 1 of array 'ndpm' above upper bound of 12
 
 Error termination. Backtrace:
 #0  0x8d7cfc in mymd21
         at /home/cha/Projects/WAVEWATCH_v7_devel/BUILD/src/w3timemd.F90:559
 #1  0x8d7e72 in __w3timemd_MOD_dsec21
         at /home/cha/Projects/WAVEWATCH_v7_devel/BUILD/src/w3timemd.F90:416
 (... same lines from other MPI processes)
 #2  0x6e7006 in __w3wavemd_MOD_w3wave
         at /home/cha/Projects/WAVEWATCH_v7_devel/BUILD/src/w3wavemd.F90:2412
 (... same lines from other MPI processes)
 #3  0x58d03e in __wmwavemd_MOD_wmwave
         at /home/cha/Projects/WAVEWATCH_v7_devel/BUILD/src/wmwavemd.F90:1331
 (... same lines from other MPI processes)

ww3_multi.nml.txt
ww3_grid_NSB9.nml.txt
ww3_grid_BAL3.nml.txt

[CarstenHansen]

Here are some final results that form an answer

1. To reproduce the error, run e.g. the regtest mww3_test08, where ww3_multi is compiled using the bin/comp, bin/link scripts resulting from w3_setup -c gnu_debug .... Use the latest version gfortran-13. The MPI program and library is openmpi-4.1.5 compiled with gfortran-13 and debug options. In bin/comp I had to add the compiler runtime option -fallow-argument-mismatch (which moderates -fcheck=all) to overcome type mismatch errors (REAL(4)/INTEGER(4)) for several calls MPI_SEND, MPI_RECV, etc.

2. One other failure is found (which is shielded when the compile option -fcheck=all is moderated with -finit-local-zero): In wmwavemd.F90,

   WW3/model/src/wmwavemd.F90

   Lines 412 to 415 in 1cacc43

   	LOOP_OUTER: DO
   	!
   	IF ( MDSS.NE.MDSO .AND. NMPSCR.EQ.IMPROC .AND. &
   	DSEC21(TPRNT,TSYNC(:,0)).NE.0. ) THEN

   
   Here, the value of DSEC21(TPRNT,TSYNC(:,0)) is evaluated even though the first two conditions are false and TPRNT has not been set. This is an example of an uninitialised array ( Un-initialised variables #1017 ). It is evaluated because the Fortran standard does not have “short-circuit” evaluation as explained by Steve Lionel in https://stevelionel.com/drfortran/2021/04/03/doctor-fortran-in-order-order/. A solution may be to set its value after the initialization of TSYNC(:,0). In the code after

    !
    !     Consistency of times within groups, set global TSYNC(:,0)
    !
    DO J=1, NRGRP
      [ ... ]
      IF ( GRANK(INGRP(J,1)).EQ.1 .AND. TSYNC(1,0).EQ.-1 )          &
           TSYNC(:,0) = TSYNC(:,INGRP(J,1))
    END DO

add:

    !
    ! TPRNT has to be initialised as its value is evaluated in the beginning of
    ! each step of LOOP_OUTER
    TPRNT  = TSYNC(:,0)

For the first error in this discussion, it makes no sense to try to initialise the date-timeTONEXT(:,7).

3. I find no failure when applying optimisation (-O3) in compilation with gfortran-13, gfortran-10, intel ifort-19 and ifx-2023, where comp, link scripts are generated from w3_setup -c gnu and and w3_setup -c intel optimised cmplr.env keywords gnu and intel (for ifx-2023 "comp_seq=ifort" must be changed to "comp_seq=ifx"). This means I did something wrong in my first tests. I think I accidentally applied a wrong combination of compiler and libraries.

[CarstenHansen]

The first bug mentioned here was discovered and explained independently by @JessicaMeixner-NOAA, then solved in #1114 and merged into develop. I will work the other bug into an issue:

WW3/model/src/wmwavemd.F90

Lines 412 to 415 in 1cacc43

	LOOP_OUTER: DO
	!
	IF ( MDSS.NE.MDSO .AND. NMPSCR.EQ.IMPROC .AND. &
	DSEC21(TPRNT,TSYNC(:,0)).NE.0. ) THEN

where TPRNT has not been set.

[CarstenHansen]

Closing because the second bug is now addressed in issue #1132