Segmentation Fault when running example case

[AkshayPatil1994]

Hi,

After compiling FastEddy on my local workstation (Ubuntu 20.04 + Nvidia RTX 3060) with CUDA version details below, when I run the tutorial case Example01_NBL.in. Error Message listed below, could you please provide some tips/hints to fix this issue? Perhaps I am doing something silly. Thanks in advance!

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Fri_Jan__6_16:45:21_PST_2023
Cuda compilation tools, release 12.0, V12.0.140
Build cuda_12.0.r12.0/compiler.32267302_0

Error log

Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
Obtaining parameters from Example01_NBL.in
Successfully opened Example01_NBL.in and read 129 lines.
Warning more than 10000 entries were requested for the parameters hash table.
Consider changing the requested table size in the PARAMETERS module.
WARNING: optional parameter gridFile defaulted to NULL.
WARNING: optional parameter pgfSelector defaulted to 1.
WARNING: optional parameter buoyancySelector defaulted to 1.
WARNING: optional parameter surflayer_stab defaulted to 0.
WARNING: optional parameter surflayer_z0tdyn defaulted to 1.
WARNING: optional parameter surflayer_offshore_visc defaulted to 1.
WARNING: optional parameter canopySelector defaulted to 0.
WARNING: optional parameter filter_6thdiff_hori defaulted to 0.
WARNING: optional parameter filter_divdamp defaulted to 0.
WARNING: optional parameter NhydroAuxScalars defaulted to 0.
WARNING: optional parameter physics_oneRKonly defaulted to 1.
Rank 0/4:  [rankXid, rankYid] = [0,0]
mpi_rank_world--0/4 inFile = (null) !!
mpi_rank_world--0/4 Obtaining devices...
Rank 1/4:  [rankXid, rankYid] = [1,0]
mpi_rank_world--1/4 inFile = (null) !!
mpi_rank_world--1/4 Obtaining devices...
Rank 2/4:  [rankXid, rankYid] = [2,0]
mpi_rank_world--2/4 inFile = (null) !!
mpi_rank_world--2/4 Obtaining devices...
Rank 3/4:  [rankXid, rankYid] = [3,0]
mpi_rank_world--3/4 inFile = (null) !!
mpi_rank_world--3/4 Obtaining devices...
Obtaining properties for 1 devices...

Obtaining properties for 1 devices...

Obtaining properties for 1 devices...

Obtaining properties for 1 devices...

mpi_rank_world--3/4 assigned accelerator device 0/1...
Device Number:          0
---------------------------------------------------------------------

__General__
   Name                               : NVIDIA GeForce RTX 3060
   Compute Capability                 : 8.6
   Clock Rate                         : 1807000
   Number of multiprocessors          : 28
   Max Threads/multiprocessor         : 1536
   Concurrent Copy/Execution          : Yes
   Concurrent Kernels                 : 1
   Compute Mode                       : 0
   Kernel Timeout Enabled             : Yes
   Integrated                         : No
   Host Memory Mapping                : Yes

__Configuration__
   Registers/Block                    : 65536
   Warp Size                          : 32
   Max Threads/Block                  : 1024
   Max Block-extent / Dimension       :         1024         1024           64
   Max Grid-extent / Dimension        :   2147483647        65535        65535

__Memory__
mpi_rank_world--2/4 assigned accelerator device 0/1...
   Tot. Global Memory (GB) / (B)      :   11.75946 / 4036689920
   Shared Mem/Block   (GB) / (B)      :    0.00005 / 49152
   Constant Memory    (GB) / (B)      :    0.00006 / 65536
   Max 1-D Texture                    :       131072
   Max 2-D Texture                    :       131072        65536
   Max 3-D Texture                    :        16384        16384        16384
   Texture Alignment  (B)             : 512
   Max Memory pitch                   : 2147483647
mpi_rank_world--1/4 assigned accelerator device 0/1...
Device Number:          0
---------------------------------------------------------------------

__General__
   Name                               : NVIDIA GeForce RTX 3060
   Compute Capability                 : 8.6
   Clock Rate                         : 1807000
   Number of multiprocessors          : 28
   Max Threads/multiprocessor         : 1536
   Concurrent Copy/Execution          : Yes
   Concurrent Kernels                 : 1
   Compute Mode                       : 0
   Kernel Timeout Enabled             : Yes
   Integrated                         : No
   Host Memory Mapping                : Yes

__Configuration__
   Registers/Block                    : 65536
   Warp Size                          : 32
   Max Threads/Block                  : 1024
   Max Block-extent / Dimension       :         1024         1024           64
Device Number:          0
---------------------------------------------------------------------

__General__
   Name                               : NVIDIA GeForce RTX 3060
   Compute Capability                 : 8.6
   Clock Rate                         : 1807000
   Number of multiprocessors          : 28
   Max Threads/multiprocessor         : 1536
   Concurrent Copy/Execution          : Yes
   Concurrent Kernels                 : 1
   Compute Mode                       : 0
   Kernel Timeout Enabled             : Yes
   Integrated                         : No
   Host Memory Mapping                : Yes

__Configuration__
   Registers/Block                    : 65536
   Warp Size                          : 32
   Max Threads/Block                  : 1024
   Max Block-extent / Dimension       :         1024         1024           64
   Max Grid-extent / Dimension        :   2147483647        65535        65535

__Memory__
   Tot. Global Memory (GB) / (B)      :   11.75946 / 4036689920
   Shared Mem/Block   (GB) / (B)      :    0.00005 / 49152
   Constant Memory    (GB) / (B)      :    0.00006 / 65536
   Max 1-D Texture                    :       131072
   Max 2-D Texture                    :       131072        65536
   Max 3-D Texture                    :        16384        16384        16384
   Texture Alignment  (B)             : 512
   Max Memory pitch                   : 2147483647
   Max Grid-extent / Dimension        :   2147483647        65535        65535

__Memory__
   Tot. Global Memory (GB) / (B)      :   11.75946 / 4036689920
   Shared Mem/Block   (GB) / (B)      :    0.00005 / 49152
   Constant Memory    (GB) / (B)      :    0.00006 / 65536
   Max 1-D Texture                    :       131072
   Max 2-D Texture                    :       131072        65536
   Max 3-D Texture                    :        16384        16384        16384
   Texture Alignment  (B)             : 512
   Max Memory pitch                   : 2147483647
mpi_rank_world--0/4 assigned accelerator device 0/1...

mpi_rank_world--3/4 done!
mpi_rank_world--3/4 Assigning threadPerBlock.[xyz]
mpi_rank_world--3/4 done!

mpi_rank_world--2/4 done!
mpi_rank_world--2/4 Assigning threadPerBlock.[xyz]
mpi_rank_world--2/4 done!

mpi_rank_world--1/4 done!
mpi_rank_world--1/4 Assigning threadPerBlock.[xyz]
mpi_rank_world--1/4 done!
Device Number:          0
---------------------------------------------------------------------

__General__
   Name                               : NVIDIA GeForce RTX 3060
   Compute Capability                 : 8.6
   Clock Rate                         : 1807000
   Number of multiprocessors          : 28
   Max Threads/multiprocessor         : 1536
   Concurrent Copy/Execution          : Yes
   Concurrent Kernels                 : 1
   Compute Mode                       : 0
   Kernel Timeout Enabled             : Yes
   Integrated                         : No
   Host Memory Mapping                : Yes

__Configuration__
   Registers/Block                    : 65536
   Warp Size                          : 32
   Max Threads/Block                  : 1024
   Max Block-extent / Dimension       :         1024         1024           64
   Max Grid-extent / Dimension        :   2147483647        65535        65535

__Memory__
   Tot. Global Memory (GB) / (B)      :   11.75946 / 4036689920
   Shared Mem/Block   (GB) / (B)      :    0.00005 / 49152
   Constant Memory    (GB) / (B)      :    0.00006 / 65536
   Max 1-D Texture                    :       131072
   Max 2-D Texture                    :       131072        65536
   Max 3-D Texture                    :        16384        16384        16384
   Texture Alignment  (B)             : 512
   Max Memory pitch                   : 2147483647

mpi_rank_world--0/4 done!
mpi_rank_world--0/4 Assigning threadPerBlock.[xyz]
mpi_rank_world--0/4 done!
Nxp = 60, Nyp = 234, Nzp = 58
****Spatial loop-index Mins & Maxs ****  (i,j,k)-min = (3,3,3) & (i,j,k)-max = (63,237,61)
Nxp = 60, Nyp = 234, Nzp = 58
****Spatial loop-index Mins & Maxs ****  (i,j,k)-min = (3,3,3) & (i,j,k)-max = (63,237,61)
Nxp = 60, Nyp = 234, Nzp = 58
****Spatial loop-index Mins & Maxs ****  (i,j,k)-min = (3,3,3) & (i,j,k)-max = (63,237,61)
Nxp = 60, Nyp = 234, Nzp = 58
****Spatial loop-index Mins & Maxs ****  (i,j,k)-min = (3,3,3) & (i,j,k)-max = (63,237,61)
gridInit:topoPos stored at 0x5b03ec9076c0, has been registered with IO.
gridInit:topoPos stored at 0x5626f03cf6c0, has been registered with IO.
gridInit:topoPos stored at 0x5e0101b9a6c0, has been registered with IO.
Entering hydro_coreInit:------
WARNING: parameter surflayer_qskin_input was not found but was requested for printing.
WARNING: parameter surflayer_offshore_opt was not found but was requested for printing.
WARNING: parameter filter_6thdiff_hori_coeff was not found but was requested for printing.
hydro_coreInit: allocating/registering arrays/fields with IO
Entering hydro_coreInit:------
hydro_coreInit: allocating/registering arrays/fields with IO
gridInit:topoPos stored at 0x5ddd5c9b56c0, has been registered with IO.
Entering hydro_coreInit:------
hydro_coreInit: allocating/registering arrays/fields with IO
Entering hydro_coreInit:------
hydro_coreInit: allocating/registering arrays/fields with IO
hydro_coreInit:hydroFlds[0] = rho stored at 0x76b7d9000080, has been registered with IO.
hydro_coreInit:hydroFlds[1] = u stored at 0x76b7d93de080, has been registered with IO.
hydro_coreInit:hydroFlds[2] = v stored at 0x76b7d97bc080, has been registered with IO.
hydro_coreInit:hydroFlds[3] = w stored at 0x76b7d9b9a080, has been registered with IO.
hydro_coreInit:hydroFlds[4] = theta stored at 0x76b7d9f78080, has been registered with IO.
hydro_coreInit:hydroFlds[0] = rho stored at 0x7a3bb2c00080, has been registered with IO.
hydro_coreInit:hydroFlds[1] = u stored at 0x7a3bb2fde080, has been registered with IO.
hydro_coreInit:hydroFlds[2] = v stored at 0x7a3bb33bc080, has been registered with IO.
hydro_coreInit:hydroFlds[3] = w stored at 0x7a3bb379a080, has been registered with IO.
hydro_coreInit:hydroFlds[4] = theta stored at 0x7a3bb3b78080, has been registered with IO.
hydro_coreInit:hydroFlds[0] = rho stored at 0x7b6e42c00080, has been registered with IO.
hydro_coreInit:hydroFlds[1] = u stored at 0x7b6e42fde080, has been registered with IO.
hydro_coreInit:hydroFlds[2] = v stored at 0x7b6e433bc080, has been registered with IO.
hydro_coreInit:hydroFlds[3] = w stored at 0x7b6e4379a080, has been registered with IO.
hydro_coreInit:hydroFlds[4] = theta stored at 0x7b6e43b78080, has been registered with IO.
hydro_coreInit:hydroFlds[0] = rho stored at 0x75f8f0200080, has been registered with IO.
hydro_coreInit:hydroFlds[1] = u stored at 0x75f8f05de080, has been registered with IO.
hydro_coreInit:hydroFlds[2] = v stored at 0x75f8f09bc080, has been registered with IO.
hydro_coreInit:hydroFlds[3] = w stored at 0x75f8f0d9a080, has been registered with IO.
hydro_coreInit:hydroFlds[4] = theta stored at 0x75f8f1178080, has been registered with IO.
hydro_coreInit:Field = pressure stored at 0x5ddd61556400, has been registered with IO.
hydro_coreInit:Field = pressure stored at 0x5626f3f33a00, has been registered with IO.
hydro_coreInit:Field = pressure stored at 0x5b03f0812e80, has been registered with IO.
hydro_coreInit:Field = pressure stored at 0x5e010527d000, has been registered with IO.
hydro_coreInit:sgstkeScalars[0] = TKE_0 stored at 0x5e010565b080, has been registered with IO.
hydro_coreInit: allocating internal arrays
hydro_coreInit:sgstkeScalars[0] = TKE_0 stored at 0x5626f4311b00, has been registered with IO.
hydro_coreInit: allocating internal arrays
hydro_coreInit:sgstkeScalars[0] = TKE_0 stored at 0x5b03f0bf0f80, has been registered with IO.
hydro_coreInit: allocating internal arrays
hydro_coreInit:sgstkeScalars[0] = TKE_0 stored at 0x5ddd61934480, has been registered with IO.
hydro_coreInit: allocating internal arrays
hydro_coreInit:hydroTauFlds[0] = Tau11 stored at 0x75f8c1c00080, has been registered with IO.
hydro_coreInit:hydroTauFlds[1] = Tau21 stored at 0x75f8c1fde080, has been registered with IO.
hydro_coreInit:hydroTauFlds[2] = Tau31 stored at 0x75f8c23bc080, has been registered with IO.
hydro_coreInit:hydroTauFlds[3] = Tau32 stored at 0x75f8c279a080, has been registered with IO.
hydro_coreInit:hydroTauFlds[4] = Tau22 stored at 0x75f8c2b78080, has been registered with IO.
hydro_coreInit:hydroTauFlds[5] = Tau33 stored at 0x75f8c2f56080, has been registered with IO.
hydro_coreInit:hydroTauFlds[6] = TauTH1 stored at 0x75f8c3334080, has been registered with IO.
hydro_coreInit:hydroTauFlds[7] = TauTH2 stored at 0x75f8c3712080, has been registered with IO.
hydro_coreInit:hydroTauFlds[8] = TauTH3 stored at 0x75f8c3af0080, has been registered with IO.
hydro_coreInit:hydroTauFlds[0] = Tau11 stored at 0x7b6e2dc00080, has been registered with IO.
hydro_coreInit:hydroTauFlds[1] = Tau21 stored at 0x7b6e2dfde080, has been registered with IO.
hydro_coreInit:hydroTauFlds[2] = Tau31 stored at 0x7b6e2e3bc080, has been registered with IO.
hydro_coreInit:hydroTauFlds[3] = Tau32 stored at 0x7b6e2e79a080, has been registered with IO.
hydro_coreInit:hydroTauFlds[4] = Tau22 stored at 0x7b6e2eb78080, has been registered with IO.
hydro_coreInit:hydroTauFlds[5] = Tau33 stored at 0x7b6e2ef56080, has been registered with IO.
hydro_coreInit:hydroTauFlds[6] = TauTH1 stored at 0x7b6e2f334080, has been registered with IO.
hydro_coreInit:hydroTauFlds[7] = TauTH2 stored at 0x7b6e2f712080, has been registered with IO.
hydro_coreInit:hydroTauFlds[8] = TauTH3 stored at 0x7b6e2faf0080, has been registered with IO.
hydro_coreInit:Field = tskin stored at 0x5ddd5f947c00, has been registered with IO.
hydro_coreInit:Field = fricVel stored at 0x5ddd5f928c00, has been registered with IO.
hydro_coreInit:Field = htFlux stored at 0x5ddd5f938400, has been registered with IO.
hydro_coreInit:Field = invOblen stored at 0x5ddd5f957400, has been registered with IO.
hydro_coreInit:Field = z0m stored at 0x5ddd5f966c00, has been registered with IO.
hydro_coreInit:Field = z0t stored at 0x5ddd5f976400, has been registered with IO.
hydro_coreInit:hydroTauFlds[0] = Tau11 stored at 0x7a3b9dc00080, has been registered with IO.
hydro_coreInit:hydroTauFlds[1] = Tau21 stored at 0x7a3b9dfde080, has been registered with IO.
hydro_coreInit:hydroTauFlds[2] = Tau31 stored at 0x7a3b9e3bc080, has been registered with IO.
hydro_coreInit:hydroTauFlds[3] = Tau32 stored at 0x7a3b9e79a080, has been registered with IO.
hydro_coreInit:hydroTauFlds[4] = Tau22 stored at 0x7a3b9eb78080, has been registered with IO.
hydro_coreInit:hydroTauFlds[5] = Tau33 stored at 0x7a3b9ef56080, has been registered with IO.
hydro_coreInit:hydroTauFlds[6] = TauTH1 stored at 0x7a3b9f334080, has been registered with IO.
hydro_coreInit:hydroTauFlds[7] = TauTH2 stored at 0x7a3b9f712080, has been registered with IO.
hydro_coreInit:hydroTauFlds[8] = TauTH3 stored at 0x7a3b9faf0080, has been registered with IO.
hydro_coreInit:Field = tskin stored at 0x5626f5667e00, has been registered with IO.
hydro_coreInit:Field = fricVel stored at 0x5626f23bcd80, has been registered with IO.
hydro_coreInit:Field = htFlux stored at 0x5626f23cc580, has been registered with IO.
hydro_coreInit:Field = invOblen stored at 0x5626f5677600, has been registered with IO.
hydro_coreInit:Field = z0m stored at 0x5626f5686e00, has been registered with IO.
hydro_coreInit:Field = z0t stored at 0x5626f5696600, has been registered with IO.
hydro_coreInit:hydroTauFlds[0] = Tau11 stored at 0x76b7c1c00080, has been registered with IO.
hydro_coreInit:hydroTauFlds[1] = Tau21 stored at 0x76b7c1fde080, has been registered with IO.
hydro_coreInit:hydroTauFlds[2] = Tau31 stored at 0x76b7c23bc080, has been registered with IO.
hydro_coreInit:hydroTauFlds[3] = Tau32 stored at 0x76b7c279a080, has been registered with IO.
hydro_coreInit:hydroTauFlds[4] = Tau22 stored at 0x76b7c2b78080, has been registered with IO.
hydro_coreInit:hydroTauFlds[5] = Tau33 stored at 0x76b7c2f56080, has been registered with IO.
hydro_coreInit:hydroTauFlds[6] = TauTH1 stored at 0x76b7c3334080, has been registered with IO.
hydro_coreInit:hydroTauFlds[7] = TauTH2 stored at 0x76b7c3712080, has been registered with IO.
hydro_coreInit:hydroTauFlds[8] = TauTH3 stored at 0x76b7c3af0080, has been registered with IO.
hydro_coreInit:Field = tskin stored at 0x5b03f1f47300, has been registered with IO.
hydro_coreInit:Field = fricVel stored at 0x5b03eeca4580, has been registered with IO.
hydro_coreInit:Field = htFlux stored at 0x5b03eecb3d80, has been registered with IO.
hydro_coreInit:Field = invOblen stored at 0x5b03f1f56b80, has been registered with IO.
hydro_coreInit:Field = z0m stored at 0x5b03f1f66380, has been registered with IO.
hydro_coreInit:Field = z0t stored at 0x5b03f1f75b80, has been registered with IO.
hydro_coreInit:Field = tskin stored at 0x5e010371f880, has been registered with IO.
hydro_coreInit:Field = fricVel stored at 0x5e0103700800, has been registered with IO.
hydro_coreInit:Field = htFlux stored at 0x5e0103710000, has been registered with IO.
hydro_coreInit:Field = invOblen stored at 0x5e01069b1480, has been registered with IO.
hydro_coreInit:Field = z0m stored at 0x5e01069c0c80, has been registered with IO.
hydro_coreInit:Field = z0t stored at 0x5e01069d0480, has been registered with IO.
mpi_rank_world--3/4: in timeInit(), simTime_it = 0 and simTime =         0.000000 !
mpi_rank_world--1/4: in timeInit(), simTime_it = 0 and simTime =         0.000000 !
mpi_rank_world--2/4: in timeInit(), simTime_it = 0 and simTime =         0.000000 !
mpi_rank_world--0/4: in timeInit(), simTime_it = 0 and simTime =         0.000000 !

## time_test
Description=This is an idealized Neutral Boundary Layer V&V scenario. # ## A description of this input file.
## FEMPI parameters---
numProcsX=4        # ## Number of cores to be used for horizontal domain decomposition in X
numProcsY=1        # ## Number of cores to be used for horizontal domain decomposition in Y
## IO parameters---
ioOutputMode=1     # ## 0: N-to-1 gather and write to a netcdf file, 1:N-to-N writes of FastEddy binary files
inPath=            # ## Path where initial/restart file is read in from
inFile=            # ## name of the input file for coordinate system and initial or restart conditions
outPath=output/    # ## Path where output files are to be written
outFileBase=FE_NBL # ## Base name of the output file series as in (outFileBase).element-in-series
frqOutput=7500     # ## frequency (in timesteps) at which to produce output
## FECUDA parameters---
tBx=1              # ## Number of threads in x-dimension.
tBy=4              # ## Number of threads in y-dimension.
tBz=64             # ## Number of threads in z-dimension.
## GRID parameters---
gridFile=          # (DEFAULTED) ## A file containing a complete grid specification
topoFile=          # ## A file containing topography (surface elevation in meters ASL)
Nx=240             # ## Number of discretised domain elements in the x (zonal) direction.
Ny=234             # ## Number of discretised domain elements in the y (meridional) direction.
Nz=58              # ## Number of discretised domain elements in the z (vertical) direction.
Nh=3               # ## Number of halo cells to be used (dependent on largest stencil extent).
d_xi=15            # ## Computational domain fixed resolution in the 'i' direction.
d_eta=15           # ## Computational domain fixed resolution in the 'j' direction.
d_zeta=20          # ## Computational domain fixed resolution in the 'k' direction.
coordHorizHalos=1  # ## switch to setup coordiante halos as periodic=1 or gradient-following=0.
verticalDeformSwitch=1 # ## switch to use vertical coordinate deformation 0=off, 1=on
verticalDeformFactor=0.75 # ## deformation factor (0.0=max compression,  1.0=no compression)
verticalDeformQuadCoeff=0 # ## deformation factor (0.0=max compression,  1.0=no compression)
## HYDRO_CORE parameters---
## ----------: HYDRO_CORE Submodule Selectors ---
## ----------: Boundary Conditions Set ---
hydroBCs=2         # ## Selector for hydro BC set. 2= periodicHorizVerticalAbl
hydroForcingWrite=0 # ## Switch for dumping hydroFldsFrhs for prognositic fields. 0 = off, 1=on
hydroSubGridWrite=1 # ## Switch for dumping Tauij fields. 0 = off, 1=on
hydroForcingLog=0  # ## Switch for logging Frhs summary metrics. 0 = off, 1=on
## ----------: PRESSURE GRADIENT FORCE ---
pgfSelector=1      # (DEFAULTED) ## Pressure Gradient Force (pgf) selector: 0=off, 1=on
## ----------: BUOYANCY ---
buoyancySelector=1 # (DEFAULTED) ## Buoyancy force  selector: 0=off, 1=on
## ----------: CORIOLIS ---
coriolisSelector=1 # ## Corilis force selector: 0= none, 1= horiz. terms, 2= horiz. & vert. terms
coriolisLatitude=54 # ## Charactersitc latitude in degrees from equator of the LES domain
## ----------: TURBULENCE ---
turbulenceSelector=1 # ## turbulence scheme selector: 0= none, 1= Lilly/Smagorinsky 
TKESelector=1      # ## Prognostic TKE selector: 0= none, 1= Prognostic
TKEAdvSelector=3   # ## advection scheme for SGSTKE equation
TKEAdvSelector_b_hyb=0 # ## hybrid advection scheme parameter
c_s=0.18           # ## Smagorinsky model constant used for turbulenceSelector = 1 and TKESelector = 0
c_k=0.1            # ## Lilly model constant used for turbulenceSelector = 1 and TKESelector > 0
## ----------: ADVECTION ---
advectionSelector=3 # ## advection scheme selector: 0= 1st-order upwind, 1= 3rd-order QUICK, 2= hybrid 3rd-4th order, 3= hybrid 5th-6th order
b_hyb=0            # ## hybrid advection scheme parameter: 0.0= lower-order upwind, 1.0=higher-order cetered, 0.0 < b_hyb < 1.0 = hybrid
## ----------: DIFFUSION ---
diffusionSelector=0 # ## diffusivity selector: 0= none, 1= const.
nu_0=0             # ## constant diffusivity used when diffusionSelector = 1
## ----------: SURFACE LAYER ---
surflayerSelector=1 # ## surfacelayer selector: 0= off, 1,2= on
surflayer_z0=0.1   # ## roughness length (momentum) when surflayerSelector > 0
surflayer_z0t=0.01 # ## roughness length (temperature) when surflayerSelector > 0
surflayer_wth=0    # ## kinematic sensible heat flux at the surface when surflayerSelector = 1
surflayer_tr=-0    # ## temperature rate at the surface when surflayerSelector = 2 (>0 for warming; <0 for cooling)
surflayer_idealsine=0 # ## selector for idealized sinusoidal surface heat flux or skin temperature forcing: 0= off, 1= on
surflayer_stab=0   # (DEFAULTED) ## exchange coeffcient stability correction selector: 0= on, 1= off
surflayer_z0tdyn=1 # (DEFAULTED) ## dynamic z0t calculation following Zilitinkevich (1995) approach: 0= off, 1= constant Zilitinkevich coeff, 2= variable Zilitinkevich coeff
## ----------: OFFSHORE ROUGHNESS ---
surflayer_offshore=0 # ## offshore selector: 0=off, 1=on
## ----------: CANOPY MODEL ---
canopySelector=0   # (DEFAULTED) ## canopy selector: 0= off, 1= on
## ----------: LARGE-SCALE FORCINGS MODEL ---
lsfSelector=0      # ## large-scale forcings selector: 0= off, 1= on
## ----------: MOISTURE ---
moistureSelector=0 # ## moisture selector: 0= off, 1= on
## ----------: EXPLICIT FILTERS ---
filterSelector=1   # ## explicit filter selector: 0=off, 1=on
filter_6thdiff_vert=1 # ## vertical 6th-order filter on w selector: 0=off, 1=on
filter_6thdiff_vert_coeff=0.03 # ## vertical 6th-order filter factor: 0.0=off, 1.0=full
filter_6thdiff_hori=0 # (DEFAULTED) ## horizontal 6th-order filter on rho,theta,qv selector: 0=off, 1=on
filter_divdamp=0   # (DEFAULTED) ## divergence damping selector: 0=off, 1=on
## ----------: RAYLEIGH DAMPING LAYER ---
dampingLayerSelector=1 # ## Rayleigh damping layer selector: 0= off, 1= on.
dampingLayerDepth=200 # ## Rayleigh damping layer depth in meters
## ----------: AUXILIARY SCALARS ---
NhydroAuxScalars=0 # (DEFAULTED) ## Number of prognostic auxiliary scalar fields
## ----------: BASE-STATE ---
stabilityScheme=2  # ## Scheme used to set hydrostatic, stability-dependent Base-State EOS fields
temp_grnd=300      # ## Air Temperature (K) at the ground used to set hydrostatic Base-State EOS fields
pres_grnd=100000   # ## Pressure (Pa) at the ground used to set hydrostatic Base-State EOS fields
zStableBottom=500  # ## Height (m) of the first stable upper-layer when stabilityScheme = 1 or 2
stableGradient=0.08 # ## Vertical gradient (K/m) of the first stable upper-layer when stabilityScheme = 1 or 2
zStableBottom2=650 # ## Height (m) of the second stable upper-layer when stabilityScheme = 2
zStableBottom3=50000 # ## Height (m) of the third stable upper-layer when stabilityScheme = 2
stableGradient2=0.003 # ## Vertical gradient (K/m) of the second stable upper-layer when stabilityScheme = 2
stableGradient3=0.003 # ## Vertical gradient (K/m) of the third stable upper-layer when stabilityScheme = 2
U_g=10             # ## Zonal (West-East) component of the geostrophic wind (m/s).
V_g=0              # ## Meridional (South-North) component of the geostrophic wind (m/s).
z_Ug=10000         # ## Height (m) above ground for linear geostrophic wind gradient (zonal component).
z_Vg=10000         # ## Height (m) above ground for linear geostrophic wind gradient (meridional component).
Ug_grad=0          # ## U_g gradient above z_Ug (ms-1/m).
Vg_grad=0          # ## V_g gradient above z_Vg (ms-1/m).
thetaPerturbationSwitch=1 # ## Switch to include initial theta perturbations: 0=off, 1=on
thetaHeight=375    # ## Height below which to include initial theta perturbations: (meters)
thetaAmplitude=0.25 # ## Maximum amplitude for theta perturbations: thetaAmplitude*[-1,+1] K
physics_oneRKonly=1 # (DEFAULTED) ## selector to apply physics RHS forcing only at the latest RK stage: 0= off, 1= on
## TIME_INTEGRATION parameters---
timeMethod=0       # ## Selector for time integration method. [0=RK3-WS2002 (default)]
Nt=630000          # ## Number of timesteps to perform.
dt=0.04            # ## timestep resolution in seconds.
NtBatch=7500       # ## Number of timesteps to compute in batch launch, must have NtBatch <= Nt.

moistureMPcallTscale = 1.0   # NOT VALIDATED
moistureCond = 3   # NOT VALIDATED
moistureAdvSelectorQi = 2   # NOT VALIDATED
moistureAdvSelectorQv = 2   # NOT VALIDATED
surflayer_qr = 0.0   # NOT VALIDATED
surflayer_wq = 0.0   # NOT VALIDATED
lsf_freq = 1.0   # NOT VALIDATED
moistureNvars = 2   # NOT VALIDATED
lsf_th_zlev1 = 1500.0   # NOT VALIDATED
lsf_th_zlev2 = 3000.0   # NOT VALIDATED
lsf_qv_zlev1 = 300.0   # NOT VALIDATED
lsf_qv_zlev2 = 500.0   # NOT VALIDATED
lsf_horMnSubTerms = 0   # NOT VALIDATED
moistureCondTscale = 1.0   # NOT VALIDATED
moistureAdvSelectorQv_b = 0.0   # NOT VALIDATED
lsf_th_lev1 = -0.0833333   # NOT VALIDATED
lsf_th_lev2 = 0.0   # NOT VALIDATED
lsf_th_surf = -0.0833333   # NOT VALIDATED
lsf_qv_lev1 = -0.0432   # NOT VALIDATED
lsf_qv_lev2 = 0.0   # NOT VALIDATED
lsf_qv_surf = -0.0432   # NOT VALIDATED
lsf_w_zlev1 = 1500.0   # NOT VALIDATED
lsf_w_zlev2 = 2100.0   # NOT VALIDATED
surflayer_ideal_qamp = 2.0   # NOT VALIDATED
lsf_w_lev1 = -23.4   # NOT VALIDATED
lsf_w_lev2 = 0.0   # NOT VALIDATED
lsf_w_surf = 0.0   # NOT VALIDATED
surflayer_ideal_amp = 4.0   # NOT VALIDATED
surflayer_ideal_qte = 10.0   # NOT VALIDATED
surflayer_ideal_qts = 1.75   # NOT VALIDATED
surflayer_ideal_te = 43200   # NOT VALIDATED
surflayer_ideal_ts = 0.0   # NOT VALIDATED
moistureCondBasePres = 1   # NOT VALIDATED
moistureSGSturb = 1   # NOT VALIDATED
Input parameters passed verification.
No input file has been specified, no gridFile has been specified.
 Using default grid and initial condition schemes.
No input file has been specified, no gridFile has been specified.
 Using default grid and initial condition schemes.
No input file has been specified, no gridFile has been specified.
 Using default grid and initial condition schemes.
**** Coordinate generation ****  (i,j,k)-extents = (240,234,58) & (i,j,k)-deltas = (15.000000,15.000000,20.000000)
**** Coordinate generation ****  (i,j,k)-extents = (240,234,58) & (i,j,k)-deltas = (15.000000,15.000000,20.000000)
**** Coordinate generation ****  (i,j,k)-extents = (240,234,58) & (i,j,k)-deltas = (15.000000,15.000000,20.000000)
No input file has been specified, no gridFile has been specified.
 Using default grid and initial condition schemes.
**** Coordinate generation ****  (i,j,k)-extents = (240,234,58) & (i,j,k)-deltas = (15.000000,15.000000,20.000000)
mpi_rank_world--1/4 grid-Coord-gen(), else-coordHorizHalos = 1!
mpi_rank_world--3/4 grid-Coord-gen(), else-coordHorizHalos = 1!
mpi_rank_world--2/4 grid-Coord-gen(), else-coordHorizHalos = 1!
mpi_rank_world--0/4 grid-Coord-gen(), else-coordHorizHalos = 1!
stabilityScheme == 2: Base State setup complete.
stabilityScheme == 2: Base State setup complete.
stabilityScheme == 2: Base State setup complete.
stabilityScheme == 2: Base State setup complete.
cuda_gridDeviceSetup:-- (i,j,k)Min, ()Max = (3,3,3), (63,237,61)
cuda_gridDeviceSetup:-- (i,j,k)Min, ()Max = (3,3,3), (63,237,61)
cuda_gridDeviceSetup:-- (i,j,k)Min, ()Max = (3,3,3), (63,237,61)
cuda_gridDeviceSetup:-- (i,j,k)Min, ()Max = (3,3,3), (63,237,61)
fecuda_SetBlocksPerGrid: tBlock = {1, 4, 64}
fecuda_SetBlocksPerGrid: grid = {66, 60, 1}
fecuda_SetBlocksPerGrid: tBlock = {1, 4, 64}
fecuda_SetBlocksPerGrid: grid = {66, 60, 1}
fecuda_SetBlocksPerGrid: tBlock = {1, 4, 64}
fecuda_SetBlocksPerGrid: grid = {66, 60, 1}
fecuda_SetBlocksPerGrid: tBlock = {1, 4, 64}
fecuda_SetBlocksPerGrid: grid = {66, 60, 1}
cuda_hydroCoreDeviceSetup() complete.
cuda_hydroCoreDeviceSetup() complete.
cuda_hydroCoreDeviceSetup() complete.
cuda_hydroCoreDeviceSetup() complete.
Initializations complete after   0.3102 (s).
Main Time: Advancing simulation timestep = 0... and time =    0.00000

_____________________#######_________  STATE-SUMMARY @ it = 0 _________#######____________________ 

			!!!!!	  HYDRO_CORE 	 !!!!! 
Field 	|	 max	     --(i,j,k)		 |	  min	     --(i,j,k)		 |	 mean
---------------------------------------------------------------------------------------------------------
********-----
 Model Fields -
 ********-----
rho	Rank 0/4:       1.16056967 	 (3,3,3)	 |        1.01215839 	 (3,3,60)	 |       1.09533846 
-----
rho	Rank 3/4:       1.16056967 	 (3,3,3)	 |        1.01215839 	 (3,3,60)	 |       1.09533846 
rho	Rank 1/4:       1.16056967 	 (3,3,3)	 |        1.01215839 	 (3,3,60)	 |       1.09533846 
rho	Rank 2/4:       1.16056967 	 (3,3,3)	 |        1.01215839 	 (3,3,60)	 |       1.09533846 
u	Rank 1/4:      10.00000095 	 (3,3,22)	 |        9.99999905 	 (3,3,52)	 |      10.00000000 
u	Rank 2/4:      10.00000095 	 (3,3,22)	 |        9.99999905 	 (3,3,52)	 |      10.00000000 
u	Rank 0/4:      10.00000095 	 (3,3,22)	 |        9.99999905 	 (3,3,52)	 |      10.00000000 
-----
u	Rank 3/4:      10.00000095 	 (3,3,22)	 |        9.99999905 	 (3,3,52)	 |      10.00000000 
v	Rank 0/4:       0.00000000 	 (3,3,3)	 |        0.00000000 	 (3,3,3)	 |       0.00000000 
v	Rank 1/4:       0.00000000 	 (3,3,3)	 |        0.00000000 	 (3,3,3)	 |       0.00000000 
v	Rank 2/4:       0.00000000 	 (3,3,3)	 |        0.00000000 	 (3,3,3)	 |       0.00000000 
-----
v	Rank 3/4:       0.00000000 	 (3,3,3)	 |        0.00000000 	 (3,3,3)	 |       0.00000000 
w	Rank 1/4:       0.00000000 	 (3,3,3)	 |        0.00000000 	 (3,3,3)	 |       0.00000000 
w	Rank 0/4:       0.00000000 	 (3,3,3)	 |        0.00000000 	 (3,3,3)	 |       0.00000000 
-----
w	Rank 2/4:       0.00000000 	 (3,3,3)	 |        0.00000000 	 (3,3,3)	 |       0.00000000 
w	Rank 3/4:       0.00000000 	 (3,3,3)	 |        0.00000000 	 (3,3,3)	 |       0.00000000 
theta	Rank 0/4:     313.49996948 	 (3,3,60)	 |      299.75000000 	 (10,166,17)	 |     307.20440674 
theta	Rank 1/4:     313.49996948 	 (3,3,60)	 |      299.75000000 	 (7,168,24)	 |     307.20693970 
theta	Rank 2/4:     313.49996948 	 (3,3,60)	 |      299.75000000 	 (4,9,4)	 |     307.20468140 
theta	Rank 3/4:     313.49996948 	 (3,3,60)	 |      299.75000000 	 (11,181,24)	 |     307.20712280 
-----
press	Rank 0/4:       0.00000000 	 (3,3,3)	 |        0.00000000 	 (3,3,3)	 |       0.00000000 
press	Rank 1/4:       0.00000000 	 (3,3,3)	 |        0.00000000 	 (3,3,3)	 |       0.00000000 
press	Rank 2/4:       0.00000000 	 (3,3,3)	 |        0.00000000 	 (3,3,3)	 |       0.00000000 
-----
press	Rank 3/4:       0.00000000 	 (3,3,3)	 |        0.00000000 	 (3,3,3)	 |       0.00000000 
TKE_0	Rank 2/4:       0.00000000 	 (3,3,3)	 |        0.00000000 	 (3,3,3)	 |       0.00000000 
TKE_0	Rank 0/4:       0.00000000 	 (3,3,3)	 |        0.00000000 	 (3,3,3)	 |       0.00000000 
Dumping state at timestep = 0...
TKE_0	Rank 1/4:       0.00000000 	 (3,3,3)	 |        0.00000000 	 (3,3,3)	 |       0.00000000 
TKE_0	Rank 3/4:       0.00000000 	 (3,3,3)	 |        0.00000000 	 (3,3,3)	 |       0.00000000 
Dumped state at timestep = 0...
[noether:92023] Read -1, expected 184320, errno = 14
[noether:92020] *** Process received signal ***
[noether:92020] Signal: Segmentation fault (11)
[noether:92020] Signal code: Invalid permissions (2)
[noether:92020] Failing at address: 0x76b7aa02d000
[noether:92020] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x76b807242520]
[noether:92020] [ 1] /lib/x86_64-linux-gnu/libc.so.6(+0x1a07cd)[0x76b8073a07cd]
[noether:92020] [ 2] /usr/lib/x86_64-linux-gnu/openmpi/lib/openmpi3/mca_btl_vader.so(+0x3244)[0x76b80562f244]
[noether:92020] [ 3] /usr/lib/x86_64-linux-gnu/openmpi/lib/openmpi3/mca_pml_ob1.so(mca_pml_ob1_send_request_schedule_once+0x1b6)[0x76b804073556]
[noether:92020] [ 4] /usr/lib/x86_64-linux-gnu/openmpi/lib/openmpi3/mca_pml_ob1.so(mca_pml_ob1_recv_frag_callback_ack+0x201)[0x76b804071811]
[noether:92020] [ 5] /usr/lib/x86_64-linux-gnu/openmpi/lib/openmpi3/mca_btl_vader.so(mca_btl_vader_poll_handle_frag+0x95)[0x76b805633ae5]
[noether:92020] [ 6] /usr/lib/x86_64-linux-gnu/openmpi/lib/openmpi3/mca_btl_vader.so(+0x7f91)[0x76b805633f91]
[noether:92020] [ 7] /lib/x86_64-linux-gnu/libopen-pal.so.40(opal_progress+0x34)[0x76b807ab9714]
[noether:92020] [ 8] /lib/x86_64-linux-gnu/libmpi.so.40(ompi_request_default_wait+0x55)[0x76b807ff80e5]
[noether:92020] [ 9] /lib/x86_64-linux-gnu/libmpi.so.40(PMPI_Wait+0x58)[0x76b8080391d8]
[noether:92020] [10] FastEddy(+0x1f5ec)[0x5e010173f5ec]
[noether:92020] [11] FastEddy(+0x1447c)[0x5e010173447c]
[noether:92020] [12] FastEddy(+0x532f)[0x5e010172532f]
[noether:92020] [13] FastEddy(+0x4f24)[0x5e0101724f24]
[noether:92020] [14] FastEddy(+0x4664)[0x5e0101724664]
[noether:92020] [15] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x76b807229d90]
[noether:92020] [16] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x76b807229e40]
[noether:92020] [17] FastEddy(+0x3c85)[0x5e0101723c85]
[noether:92020] *** End of error message ***
[noether:92021] Read -1, expected 184320, errno = 14
[noether:92022] Read -1, expected 184320, errno = 14
[noether:92022] *** Process received signal ***
[noether:92022] Signal: Segmentation fault (11)
[noether:92022] Signal code: Invalid permissions (2)
[noether:92022] Failing at address: 0x75f8aa02d000
[noether:92022] [ 0] [noether:92023] *** Process received signal ***
[noether:92023] Signal: Segmentation fault (11)
[noether:92023] Signal code: Invalid permissions (2)
[noether:92023] Failing at address: 0x7a3b7402d000
[noether:92023] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x75f908642520]
[noether:92022] [ 1] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7a3bd2842520]
[noether:92023] [ 1] /lib/x86_64-linux-gnu/libc.so.6(+0x1a07cd)[0x75f9087a07cd]
[noether:92022] [ 2] /usr/lib/x86_64-linux-gnu/openmpi/lib/openmpi3/mca_btl_vader.so(+0x3244)[0x75f907419244]
[noether:92022] [ 3] /usr/lib/x86_64-linux-gnu/openmpi/lib/openmpi3/mca_pml_ob1.so(mca_pml_ob1_send_request_schedule_once+0x1b6)[0x75f90545d556]
[noether:92022] [ 4] /usr/lib/x86_64-linux-gnu/openmpi/lib/openmpi3/mca_pml_ob1.so(mca_pml_ob1_recv_frag_callback_ack+0x201)[0x75f90545b811]
[noether:92022] [ 5] /usr/lib/x86_64-linux-gnu/openmpi/lib/openmpi3/mca_btl_vader.so(mca_btl_vader_poll_handle_frag+0x95)[0x75f90741dae5]
[noether:92022] /lib/x86_64-linux-gnu/libc.so.6(+0x1a07cd)[0x7a3bd29a07cd]
[noether:92023] [ 2] /usr/lib/x86_64-linux-gnu/openmpi/lib/openmpi3/mca_btl_vader.so(+0x3244)[0x7a3bd1486244]
[noether:92023] [ 3] [ 6] /usr/lib/x86_64-linux-gnu/openmpi/lib/openmpi3/mca_btl_vader.so(+0x7db1)[0x75f90741ddb1]
[noether:92022] [ 7] /usr/lib/x86_64-linux-gnu/openmpi/lib/openmpi3/mca_pml_ob1.so(mca_pml_ob1_send_request_schedule_once+0x1b6)[0x7a3bd1457556]
[noether:92023] [ 4] /usr/lib/x86_64-linux-gnu/openmpi/lib/openmpi3/mca_pml_ob1.so(mca_pml_ob1_recv_frag_callback_ack+0x201)[0x7a3bd1455811]
[noether:92023] [ 5] /usr/lib/x86_64-linux-gnu/openmpi/lib/openmpi3/mca_btl_vader.so(mca_btl_vader_poll_handle_frag+0x95)[0x7a3bd148aae5]
[noether:92023] [ 6] /usr/lib/x86_64-linux-gnu/openmpi/lib/openmpi3/mca_btl_vader.so(+0x7db1)[0x7a3bd148adb1]
[noether:92023] [ 7] /lib/x86_64-linux-gnu/libopen-pal.so.40(opal_progress+0x34)[0x7a3bd2f62714]
[noether:92023] [ 8] /lib/x86_64-linux-gnu/libopen-pal.so.40(opal_progress+0x34)[0x75f908eb9714]
[noether:92022] [ 8] /lib/x86_64-linux-gnu/libmpi.so.40(ompi_request_default_wait+0x55)[0x75f9093ec0e5]
[noether:92022] [ 9] /lib/x86_64-linux-gnu/libmpi.so.40(ompi_request_default_wait+0x55)[0x7a3bd348f0e5]
[noether:92023] [ 9] /lib/x86_64-linux-gnu/libmpi.so.40(PMPI_Wait+0x58)[0x75f90942d1d8]
[noether:92022] [10] FastEddy(+0x1f5ec)[0x5b03ec4ac5ec]
[noether:92022] [11] FastEddy(+0x1447c)[0x5b03ec4a147c]
[noether:92022] [12] FastEddy(+0x532f)[0x5b03ec49232f]
[noether:92022] [13] FastEddy(+0x4f24)[0x5b03ec491f24]
[noether:92022] [14] FastEddy(+0x4664)[0x5b03ec491664]
[noether:92022] [15] /lib/x86_64-linux-gnu/libmpi.so.40(PMPI_Wait+0x58)[0x7a3bd34d01d8]
[noether:92023] [10] FastEddy(+0x1f5ec)[0x5626eff745ec]
[noether:92023] [11] FastEddy(+0x1447c)[0x5626eff6947c]
/lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x75f908629d90]
[noether:92022] [16] [noether:92023] [12] FastEddy(+0x532f)[0x5626eff5a32f]
[noether:92023] [13] FastEddy(+0x4f24)[0x5626eff59f24]
[noether:92023] [14] FastEddy(+0x4664)[0x5626eff59664]
[noether:92023] [15] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x75f908629e40]
[noether:92022] [17] FastEddy(+0x3c85)[0x5b03ec490c85]
[noether:92022] *** End of error message ***
/lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7a3bd2829d90]
[noether:92023] [16] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7a3bd2829e40]
[noether:92023] [17] FastEddy(+0x3c85)[0x5626eff58c85]
[noether:92023] *** End of error message ***

[jprestop]

@AkshayPatil1994 Thank you for your feedback. We will review it as time permits.

[mhammoud115]

Hi @AkshayPatil1994, were you able to figure this error out?
I tried to run the example as is on a cluster using 4 NVIDIA A100-SXM4-40GB, but ran into a similar error
Something along the lines:
cma_ep.c:88 process_vm_readv(pid=2162282 {0x151a94d20dbe,524288}-->{0x14887612edbe,524288}) returned -1: Bad address
==== backtrace (tid:2162281) ====

[CaThSt]

I previously had a the same issue with segmentation fault during Dumping state at timestep = 0 ... originating from mpi/cuda error. It only occurred when running more than one mpi task and seemed to be caused by failed MPI communication.
We changed to a CUDA-aware OpenMPI installation (from NVIDIA HPC SDK) and the problem was fixed. Machine was a workstation with 2 NVIDIA RTX A6000.

[jsauer-NCAR]

It is correct that FastEddy requires a CUDA-aware MPI installation (and/or ROCm-aware MPI for AMD GPUs). This is something a user must ensure is in place and functioning properly before attempting to compile and execute FastEddy in their specific hardware environment.