Running GEOS CTM

[JulesKouatchou]

I cloned GEOS CTM and was able to compile it. The ctm_setup script did not properly create the experiment directory because it was still referring to the old configuration (Linux/ instead of install/). I fix the ctm_setup file. The code is crashing during the initialization steps because it cannot create the grid. The code is failing on Line 9193 of MAPL_Generic.F90:

call ESMF_ConfigGetAttribute(state%cf,gridname,label=trim(comp_name)//CF_COMPONENT_SEPARATOR//'GRIDNAME:',rc=status)
VERIFY_(status)

I can quickly understand why there is a problem: the label should only be 'GRIDNAME:'.

I checked a couple of CVS tags I have and could not locate any MAPL version similar to the one in the git repository. I am wondering if MAPL has to be updated before GEOS CTM can run.

[kgerheiser]

I have the CTM running with this version of MAPL. You just need to add something like this into GEOSCTM.rc:

GEOSctm.GRID_TYPE: Cubed-Sphere
GEOSctm.GRIDNAME: PE360x2160-CF
GEOSctm.NF: 6
GEOSctm.LM: 72
GEOSctm.IM_WORLD: 360

And for Dynamics:

  DYNAMICS.GRID_TYPE: Cubed-Sphere
  DYNAMICS.GRIDNAME: PE360x2160-CF
  DYNAMICS.NF: 6
  DYNAMICS.LM: 72
  DYNAMICS.IM_WORLD: 360

Though, there is some duplication and it could probably be cleaned up and added to ctm_setup.

[JulesKouatchou]

Thank you. Very interesting. If it works, I will change the set up script...

[JulesKouatchou]

Do you have by chance an experiment directory on discover I can look at? My code is still crashing. Thanks.

[kgerheiser]

I do have an experiment located at: /discover/nobackup/kgerheis/experiments/ctm_test_experiment

I think there might be a few other small things you need to change to get it to run. If you run into any errors I can probably quickly diagnose them as I've gone through this process several times.

[JulesKouatchou]

Kyle: Thank you. I will get back to you if I need more assistance.

[JulesKouatchou]

Kyle:

It seems that I am still missing something as my code continues to crash. My working directory is:

/gpfsm/dnb32/jkouatch/GEOS_CTM/GitRepos/testTR

I believe that I have a rc setting issue that I cannot identify. I noticed that in my standard output file I have:

In MAPL_Shmem:
NumCores per Node = 96
NumNodes in use = 1
Total PEs = 96

In MAPL_InitializeShmem (NodeRootsComm):
NumNodes in use = 1

That is not correct as there should be 4 nodes in use.

[kgerheiser]

You need to register the grid for the GridManager before you can create the grid by adding this to AdvCore_GridComp.F90.

subroutine register_grid_and_regridders()
    use MAPL_GridManagerMod, only: grid_manager
    use CubedSphereGridFactoryMod, only: CubedSphereGridFactory
    use MAPL_RegridderManagerMod, only: regridder_manager
    use MAPL_RegridderSpecMod, only: REGRID_METHOD_BILINEAR
    use LatLonToCubeRegridderMod
    use CubeToLatLonRegridderMod
    use CubeToCubeRegridderMod

    type (CubedSphereGridFactory) :: factory

    type (CubeToLatLonRegridder) :: cube_to_latlon_prototype
    type (LatLonToCubeRegridder) :: latlon_to_cube_prototype
    type (CubeToCubeRegridder) :: cube_to_cube_prototype

    call grid_manager%add_prototype('Cubed-Sphere',factory)
    associate (method => REGRID_METHOD_BILINEAR, mgr => regridder_manager)
      call mgr%add_prototype('Cubed-Sphere', 'LatLon', method, cube_to_latlon_prototype)
      call mgr%add_prototype('LatLon', 'Cubed-Sphere', method, latlon_to_cube_prototype)
      call mgr%add_prototype('Cubed-Sphere', 'Cubed-Sphere', method, cube_to_cube_prototype)
    end associate

  end subroutine register_grid_and_regridders

and calling it in AdvCore SetServices (line 318)

if (.NOT. FV3_DynCoreIsRunning) then
         call fv_init2(FV_Atm, dt, grids_on_my_pe, p_split)
         call register_grid_and_regridders() ! add this line
end if

This register_grid_and_regridders routine duplicates code in DynCore and should probably be added to its own module that AdvCore and DynCore can call.

[JulesKouatchou]

Kyle: That helps to go further. Thanks. The code still crashes. I am now running in a debugging mode...

[lizziel]

Jules, keep reporting the problems you are running into since I had to go through these issues as well when getting GCHP to work with the new MAPL. I may be able to help.

[lizziel]

One note of caution so that you do not make the same mistake I did. Regarding the earlier comment to add this to your config file:

GEOSctm.GRID_TYPE: Cubed-Sphere
GEOSctm.GRIDNAME: PE360x2160-CF
GEOSctm.NF: 6
GEOSctm.LM: 72
GEOSctm.IM_WORLD: 360

I made the mistake of adding the prefix (in my case GCHP. rather than GEOSctm.) to the existing lines rather than adding them. This caused a silent bug in AdvCore_GridCompMod.F90 since it expects entries for IM, JM, and LM in the file, without prefixes. If those lines are not found then the calculated areas get messed up since it uses default values.

My fix was to change AdvCore_GridCompMod.F90 to expect lines with the prefix rather than without.

-      call MAPL_GetResource( MAPL, FV_Atm(1)%flagstruct%npx, 'IM:', default= 32, RC=STATUS )
+      ! Customize for GCHP (ewl, 4/9/2019)
+!      call MAPL_GetResource( MAPL, FV_Atm(1)%flagstruct%npx, 'IM:', default= 32, RC=STATUS )
+      call MAPL_GetResource( MAPL, FV_Atm(1)%flagstruct%npx, 'GCHP.IM:', default= 32, RC=STATUS )
       _VERIFY(STATUS)
-      call MAPL_GetResource( MAPL, FV_Atm(1)%flagstruct%npy, 'JM:', default=192, RC=STATUS )
+!      call MAPL_GetResource( MAPL, FV_Atm(1)%flagstruct%npy, 'JM:', default=192, RC=STATUS )
+      call MAPL_GetResource( MAPL, FV_Atm(1)%flagstruct%npy, 'GCHP.JM:', default=192, RC=STATUS )
       _VERIFY(STATUS)
-      call MAPL_GetResource( MAPL, FV_Atm(1)%flagstruct%npz, 'LM:', default= 72, RC=STATUS )
+!      call MAPL_GetResource( MAPL, FV_Atm(1)%flagstruct%npz, 'LM:', default= 72, RC=STATUS )
+      call MAPL_GetResource( MAPL, FV_Atm(1)%flagstruct%npz, 'GCHP.LM:', default= 72, RC=STATUS )

[JulesKouatchou]

Lizzie,

Thank you for your comments. I am wondering if it was necessary to make any change to my GEOSCTM.rc file as the AdvCore source coide has:

     call MAPL_GetResource( MAPL, FV_Atm(1)%flagstruct%npx, 'IM:', default= 32, RC=STATUS )
     call MAPL_GetResource( MAPL, FV_Atm(1)%flagstruct%npy, 'JM:', default=192, RC=STATUS )
     call MAPL_GetResource( MAPL, FV_Atm(1)%flagstruct%npz, 'LM:', default= 72, RC=STATUS )

In any case, my code is still crashing and I still trying to figure out why.

[JulesKouatchou]

Something unusual that I mentioned before is the following print out at the beginning of the code:

In MAPL_Shmem:
NumCores per Node = 84
NumNodes in use = 1
Total PEs = 84

In MAPL_InitializeShmem (NodeRootsComm):
NumNodes in use = 1

I went ahead in MAPL_ShmemMod.F90 and printed the name of all the processors (nodes) after the call:
call MPI_AllGather(name ,MPI_MAX_PROCESSOR_NAME,MPI_CHARACTER,&
names,MPI_MAX_PROCESSOR_NAME,MPI_CHARACTER,Comm,status)

All the entries of the variable "names" only had the name of the head node. Something is wrong but I do not know what. I am sure that that there is a new rc setting I need to add.

[mathomp4]

@JulesKouatchou This sounds suspiciously like calling MPT with mpirun. How are you running your executable? If you're using MPT (which you probably are), you need to use either mpiexec_mpt or use esma_mpirun (which autodetects MPI stack and will run mpiexec_mpt)

ETA: For everyone, MPT does have an mpirun command but it is weird. It can do some interesting things, but it doesn't perform like you'd expect without a lot of extra work. mpiexec_mpt understands SLURM (ish) and does what is expected.

[mathomp4]

@JulesKouatchou Actually, I might need to work with you on the ctm_setup script. There are some changes we had to make to gcm_setup for the move to CMake that aren't reflected in that script as I see it on this repo. It's possible it's getting confused and picking the wrong MPI stack or the like.

[kgerheiser]

@mathomp4 That is something I had to change in ctm_run.j, and why I asked you about MPT's mpirun. I changed RUN_CMD to mpiexec