cam6_3_016: Refactor coupling with WACCMX ionosphere #264

fvitt · 2020-11-02T17:56:15Z

This PR is a refactor of how WACCMX ionosphere is coupled to CAM to be independent of dynamical core.

Previously, the FV dycore grid was used as the ion transport grid. Thus, WACCMX could only be run with FV dycore.

Here, ESMF regridding utilities are used to regrid fields between independent ion transport lat/lon grid, electro-dynamo geomagnetic lat/lon grid, and a generalized cam physics grid mesh. The use of an ESMF gridded component to contain the ion transport and electro-dynamo ionosphere components allows the ionosphere to be executed on a subset of CAM's MPI tasks and for multi-instance WACCMX configurations (a requirement for WACCMX-DART).

closes #84
closes #223
closes #151

deleted: cime_config/testdefs/testmods_dirs/cam/outfrq3s_WX81/shell_commands deleted: cime_config/testdefs/testmods_dirs/cam/outfrq3s_WX81/user_nl_cam deleted: cime_config/testdefs/testmods_dirs/cam/outfrq3s_WX81/user_nl_clm modified: cime_config/testdefs/testmods_dirs/cam/outfrq9s_576tsks/user_nl_clm modified: cime_config/config_pes.xml modified: cime_config/testdefs/testlist_cam.xml modified: src/dynamics/se/dp_coupling.F90 modified: src/dynamics/se/dycore/dimensions_mod.F90 modified: src/dynamics/se/dycore/global_norms_mod.F90 modified: src/dynamics/se/dycore/prim_advance_mod.F90 modified: src/dynamics/se/dyn_comp.F90 modified: src/dynamics/se/stepon.F90

…ning of each time step; misc cleanup deleted: src/unit_drivers/wxie/unit_driver.F90 modified: bld/config_files/definition.xml modified: bld/configure modified: cime_config/testdefs/testlist_cam.xml modified: src/physics/cam/physpkg.F90 modified: src/physics/cam/rk_stratiform.F90 modified: src/physics/cam/waccmx_phys_intr.F90 modified: src/physics/waccmx/ion_electron_temp.F90 modified: src/unit_drivers/drv_input_data.F90

modified: src/chemistry/mozart/mo_apex.F90 modified: src/utils/physconst.F90

modified: bld/namelist_files/namelist_defaults_cam.xml modified: cime_config/testdefs/testlist_cam.xml

modified: bld/namelist_files/namelist_defaults_cam.xml

modified: bld/build-namelist modified: bld/namelist_files/namelist_defaults_cam.xml modified: cime_config/testdefs/testlist_cam.xml modified: doc/ChangeLog modified: src/ionosphere/waccmx/ionosphere_interface.F90

modified: cime_config/testdefs/testlist_cam.xml deleted: cime_config/testdefs/testmods_dirs/cam/outfrq3s_waccm_ma/shell_commands deleted: cime_config/testdefs/testmods_dirs/cam/outfrq3s_waccm_ma/user_nl_cam new file: cime_config/testdefs/testmods_dirs/cam/outfrq9s_multi/shell_commands new file: cime_config/testdefs/testmods_dirs/cam/outfrq9s_multi/user_nl_cam renamed: cime_config/testdefs/testmods_dirs/cam/outfrq3s_waccm_ma/user_nl_clm -> cime_config/testdefs/testmods_dirs/cam/outfrq9s_multi/user_nl_clm

modified: bld/configure

into phys-ionos-cpl_cam6_3_002

modified: src/ionosphere/waccmx/edyn_geogrid.F90 modified: src/ionosphere/waccmx/edyn_grid_comp.F90 modified: src/ionosphere/waccmx/regridder.F90

nusbaume

Mostly looks ok to me! I just have some clean-up and clarification requests.

Also, one other issue I have that is not listed below is that in some of the source files the WACCMX_EDYN_ESMF def is so prevalent that it impacts readability (at least for me). I assume this pre-proc def is necessary in order to prevent users from having to build ESMF?

If so then it might be good to create a Github issue stating that this WACCMX_EDYN_ESMF def can be replaced with a fortran run-time option once NUOPC/ESMF becomes the default CAM configuration (which I can do, I just want to make sure I am understanding the purpose of WACCMX_EDYN_ESMF correctly). Thus any info you can provide would certainly be appreciated.

bld/configure

bld/namelist_files/use_cases/waccmx_ma_2000_cam4.xml

doc/ChangeLog

src/ionosphere/waccmx/adotv_mod.F90

src/ionosphere/waccmx/edyn_esmf.F90

fvitt · 2020-12-30T16:12:13Z

@nusbaume,
The WACCMX_EDYN_ESMF cpp variable is intended to allow ion transport without the need for the ESMF library and without an active electro-dynamo. This was the "-ionosphere wxi" configure option which was unsupported and was removed in this PR. We may be able to remove the WACCMX_EDYN_ESMF cpp defs from the code. I will explore this possibility.

modified: bld/namelist_files/namelist_defaults_cam.xml modified: cime_config/testdefs/testmods_dirs/cam/outfrq9s_576tsks/shell_commands modified: src/ionosphere/waccmx/edyn_grid_comp.F90

modified: bld/configure modified: src/ionosphere/waccmx/amie_module.F90 modified: src/ionosphere/waccmx/edyn_esmf.F90 modified: src/ionosphere/waccmx/edyn_init.F90 modified: src/ionosphere/waccmx/edynamo.F90 modified: src/ionosphere/waccmx/ionosphere_interface.F90 modified: src/ionosphere/waccmx/ltr_module.F90 modified: src/ionosphere/waccmx/utils_mod.F90

modified: bld/configure modified: src/ionosphere/waccmx/adotv_mod.F90 modified: src/ionosphere/waccmx/amie_module.F90 modified: src/ionosphere/waccmx/dpie_coupling.F90 modified: src/ionosphere/waccmx/edyn_init.F90 modified: src/ionosphere/waccmx/edyn_params.F90 modified: src/ionosphere/waccmx/ionosphere_interface.F90 modified: src/ionosphere/waccmx/ltr_module.F90 modified: src/ionosphere/waccmx/utils_mod.F90 modified: src/physics/cam/cam_diagnostics.F90 modified: src/physics/cam/phys_control.F90 These are reverted to earlier version: modified: bld/namelist_files/namelist_defaults_cam.xml modified: src/ionosphere/waccmx/edyn_grid_comp.F90

modified: src/ionosphere/waccmx/edyn_grid_comp.F90 modified: src/ionosphere/waccmx/edyn_init.F90

modified: src/ionosphere/waccmx/edyn_esmf.F90 modified: src/ionosphere/waccmx/edyn_geogrid.F90 modified: src/ionosphere/waccmx/edyn_grid_comp.F90 modified: src/ionosphere/waccmx/edyn_mpi.F90 modified: src/ionosphere/waccmx/edynamo.F90 modified: src/ionosphere/waccmx/wei05sc.F90

modified: cime_config/testdefs/testlist_cam.xml modified: cime_config/testdefs/testmods_dirs/cam/outfrq9s_ltr/user_nl_cam modified: src/ionosphere/waccmx/amie_module.F90 modified: src/ionosphere/waccmx/dpie_coupling.F90 modified: src/ionosphere/waccmx/edyn_esmf.F90 modified: src/ionosphere/waccmx/edyn_grid_comp.F90 modified: src/ionosphere/waccmx/edyn_mpi.F90 modified: src/ionosphere/waccmx/edyn_mudmod.F90 modified: src/ionosphere/waccmx/edynamo.F90 modified: src/ionosphere/waccmx/ionosphere_interface.F90 modified: src/ionosphere/waccmx/ltr_module.F90 modified: src/ionosphere/waccmx/utils_mod.F90 modified: src/ionosphere/waccmx/wei05sc.F90

nusbaume

Looks great to me now. Thanks for getting these major modifications ready!

bld/namelist_files/namelist_defaults_cam.xml

cime_config/testdefs/testlist_cam.xml

cacraigucar · 2021-03-05T17:59:27Z

cime_config/testdefs/testmods_dirs/cam/outfrq9s_576tsks/user_nl_clm

 hist_mfilt  = 1
 hist_ndens  = 1

+fsurdat = '/glade/p/cesmdata/cseg/inputdata/lnd/clm2/surfdata_map/surfdata_0.47x0.63_78pfts_CMIP6_simyr2000_c180508.nc'


I verified this file exists in the svn repo, so it should be using a relative path in case the test is used anywhere other than cheyenne in the future.

The file path is now relative to $DIN_LOC_ROOT

src/ionosphere/waccmx/edyn_geogrid.F90

src/ionosphere/waccmx/edyn_grid_comp.F90

bld/configure

src/ionosphere/waccmx/wei05sc.F90

…r f19 res; correct path for fsurdat file in test modified: bld/namelist_files/namelist_defaults_cam.xml modified: cime_config/config_pes.xml modified: cime_config/testdefs/testmods_dirs/cam/outfrq9s_576tsks/user_nl_clm

modified: src/ionosphere/waccmx/edyn_geogrid.F90 modified: src/ionosphere/waccmx/edyn_grid_comp.F90 modified: src/ionosphere/waccmx/oplus.F90 modified: src/ionosphere/waccmx/wei05sc.F90

modified: src/ionosphere/waccmx/edyn_mpi.F90

modified: bld/build-namelist modified: bld/config_files/definition.xml modified: bld/configure modified: bld/namelist_files/namelist_defaults_cam.xml modified: bld/namelist_files/namelist_definition.xml modified: src/chemistry/mozart/short_lived_species.F90 modified: src/ionosphere/waccmx/edyn_init.F90 modified: src/ionosphere/waccmx/edyn_maggrid.F90 modified: src/ionosphere/waccmx/edyn_mud.F90 modified: src/ionosphere/waccmx/edyn_mudmod.F90 modified: src/ionosphere/waccmx/edyn_muh2cr.F90 modified: src/ionosphere/waccmx/edyn_solve.F90 modified: src/ionosphere/waccmx/edynamo.F90 modified: src/ionosphere/waccmx/getapex.F90 modified: src/ionosphere/waccmx/ionosphere_interface.F90 modified: src/ionosphere/waccmx/wei05sc.F90

gold2718

I have a couple of questions and a few suggestions but I think it is basically ready to go. It still needs some work to get up to standard but it is a whole lot better.

gold2718 · 2021-03-26T18:21:49Z

src/ionosphere/waccmx/edyn_mpi.F90

+        nlon_geo,        & ! size of geo lon dimension
+        nlat_geo,        & ! size of geo lat dimension


These two variables are private so the protected attribute has no effect. Are they supposed to be public? The same question goes for some of the other variables below.

I removed the protected attribute from the private variables in this module.

gold2718 · 2021-03-26T20:23:00Z

src/ionosphere/waccmx/edyn_mpi.F90

+         write(iulog,"('mp_distribute_geo: mytid=',i4,' ntaski,j=',2i4,' mytidi,j=',2i4,&
+              ' lon0,1=',2i4,' lat0,1=',2i4,' lev0,1=',2i4)") &
+              mytid,ntaski,ntaskj,mytidi,mytidj,lon0,lon1,lat0,lat1,lev0,lev1


It looks like these write statements (here and below) are intended to come from each active task. For those, I suggest writing to unit 6 so that all tasks write to the same file.

Now writing to unit 6.

gold2718 · 2021-03-26T20:44:28Z

src/ionosphere/waccmx/edyn_mpi.F90

+      ! Report my stats to stdout:
+      !     write(iulog,"(/,'mytid=',i3,' mytidi,j=',2i3,' lat0,1=',2i3,' (',i2,') lon0,1=',2i3,' (',i2,') ncells=',i4)") &
+      !       mytid,mytidi,mytidj,lat0,lat1,nj,lon0,lon1,ni


I suggest either deleting this comment or un-commenting and wrapping in debug.

I put the write statement inside a if debug block.

gold2718 · 2021-03-26T20:45:15Z

src/ionosphere/waccmx/edyn_mpi.F90

+            write(iulog,"(/,a,i3,a,2i3,a,2i3,a,i2,2a,2i3,a,i2,a,i4)")            &
+                 'mytid = ',mytid, ', magtidi,j = ', magtidi, magtidj,           &
+                 ', mlat0,1 = ', mlat0, mlat1, ' (', nj, ')',                    &
+                 ', mlon0,1 = ', mlon0, mlon1, ' (', ni, ') ncells = ', ncells


Another place where writing to 6 might help.

Now writing to unit 6.

gold2718 · 2021-03-26T20:56:50Z

src/ionosphere/waccmx/edyn_mudmod.F90

@@ -1,26 +1,24 @@
 !-----------------------------------------------------------------------
-      subroutine mudmod(pe,phi_out,jntl,isolve,ier)
+      subroutine mudmod(pe,phi_out,jntl,isolve,nlev,ier)


Still not a module?

Now inside a module.

gold2718 · 2021-03-26T20:59:17Z

src/ionosphere/waccmx/edyn_solve.F90

    ier = 0
    isolve = 2
-    call mudmod(rim_glb,phisolv,jntl,isolve,ier)! solver in mudmod.F
+    call mudmod(rim_glb,phisolv,jntl,isolve,res_nlev,ier)! solver in mudmod.F


This comment is incorrect. If edyn_mudmod.F90 became a module, it would be clear where this came from.

Removed the comment.

gold2718 · 2021-03-26T21:00:16Z

src/ionosphere/waccmx/edyn_mud.F90

      use shr_kind_mod ,only: r8 => shr_kind_r8
      use cam_abortutils   ,only: endrun
-      use edyn_solve,only: nc,ncee,cee
+      use edyn_solve,only: nc,cee


Still not a module? Is this routine (mud) still used anywhere?

The subroutine mud was not used. I removed it. The remaining routines are inside a module.

gold2718 · 2021-03-26T21:23:58Z

src/ionosphere/waccmx/edyn_grid_comp.F90

+   public :: edyn_grid_comp_run2
+   public :: edyn_grid_comp_final
+
+   ! Private data and interfaces


There is no private statement so aren't all these symbols really public? Maybe put a private statement in.

Added a private statement.

gold2718 · 2021-03-27T02:28:50Z

src/ionosphere/waccmx/edyn_grid_comp.F90

+      call ESMF_GridCompSetServices(phys_comp, edyn_gcomp_SetServices, rc=rc)
+      call edyn_esmf_chkerr(subname, 'ESMF_GridCompSetServices '//trim(inst_name), rc)
+      ! Initialize the required component arguments
+      call ESMF_GridCompInitialize(phys_comp, rc=rc)


Can you tell me why you did not need import and export states? It is much simpler! (also for run and finalize)

The import and export states are optional arguments. It works as it is. If you would like a better answer, I will need some time to study the code and think about it.

gold2718 · 2021-03-27T05:17:36Z

doc/ChangeLog

@@ -1,3 +1,166 @@
+


When you finish this file, please be sure to separate the 'A' (adds) and 'D' (deletes) into their correct locations.

modified: src/chemistry/mozart/short_lived_species.F90 modified: src/ionosphere/waccmx/edyn_geogrid.F90 modified: src/ionosphere/waccmx/edyn_init.F90 modified: src/ionosphere/waccmx/edyn_mud.F90 modified: src/ionosphere/waccmx/edyn_mudmod.F90 modified: src/ionosphere/waccmx/edyn_muh2cr.F90 modified: src/ionosphere/waccmx/edyn_solve.F90 modified: src/ionosphere/waccmx/getapex.F90 modified: src/ionosphere/waccmx/ionosphere_interface.F90

modified: src/ionosphere/waccmx/edyn_grid_comp.F90 modified: src/ionosphere/waccmx/edyn_mpi.F90 modified: src/ionosphere/waccmx/edyn_mud.F90 modified: src/ionosphere/waccmx/edyn_mudcom.F90 modified: src/ionosphere/waccmx/edyn_mudmod.F90 modified: src/ionosphere/waccmx/edyn_muh2cr.F90 modified: src/ionosphere/waccmx/edyn_solve.F90 new file: src/ionosphere/waccmx/edyn_solver_coefs.F90

modified: src/ionosphere/waccmx/edyn_mudcom.F90

…onos-cpl_cam6_3_002

fvitt added 8 commits October 27, 2020 15:20

merge origin/phys-ionos-cpl

7936f44

revert bug fixes

495715e

modified: src/chemistry/mozart/mo_apex.F90 modified: src/utils/physconst.F90

revert waccm namelist setteings; tweak waccmx nag test

d5c048c

modified: bld/namelist_files/namelist_defaults_cam.xml modified: cime_config/testdefs/testlist_cam.xml

set se_molecular_diff=0.0 for waccmx

75cb5a2

modified: bld/namelist_files/namelist_defaults_cam.xml

remove unused namelist options

9171bbd

modified: bld/build-namelist modified: bld/namelist_files/namelist_defaults_cam.xml modified: cime_config/testdefs/testlist_cam.xml modified: doc/ChangeLog modified: src/ionosphere/waccmx/ionosphere_interface.F90

fvitt added enhancement New feature or request answer changing answer changing tag labels Nov 2, 2020

fvitt added this to the CESM2.3 milestone Nov 2, 2020

fvitt requested review from brian-eaton, cacraigucar, gold2718, joemci and nusbaume November 2, 2020 17:56

fvitt self-assigned this Nov 2, 2020

set default mag grid resolution according to physics grid resolution

16f46b1

modified: bld/configure

gold2718 mentioned this pull request Dec 23, 2020

cam6_3_007: Initial implementation of weak scaling physics infrastructure #287

Merged

fvitt added 3 commits December 28, 2020 14:24

remove wxi ionosphere match from build-namelist

3593107

Merge branch 'phys-ionos-cpl_cam6_3_002' of https://github.com/fvitt/CAM

fd89794

into phys-ionos-cpl_cam6_3_002

misc clean up

806ea4e

modified: src/ionosphere/waccmx/edyn_geogrid.F90 modified: src/ionosphere/waccmx/edyn_grid_comp.F90 modified: src/ionosphere/waccmx/regridder.F90

nusbaume requested changes Dec 29, 2020

View reviewed changes

fvitt added 6 commits December 30, 2020 12:53

use accurate (at poles) mesh files

0985024

modified: bld/namelist_files/namelist_defaults_cam.xml modified: cime_config/testdefs/testmods_dirs/cam/outfrq9s_576tsks/shell_commands modified: src/ionosphere/waccmx/edyn_grid_comp.F90

correct mgrid config opt in 576tsks test

4415d6e

use correct grid for mag field diags; write to iulog

62b4757

modified: src/ionosphere/waccmx/edyn_grid_comp.F90 modified: src/ionosphere/waccmx/edyn_init.F90

fvitt added 3 commits March 2, 2021 16:03

merge ESCOMP/cam_development

f79929e

add _r8 to constants in oplus.F90

898695a

fvitt requested review from gold2718 and nusbaume March 3, 2021 15:47

nusbaume approved these changes Mar 3, 2021

View reviewed changes

nusbaume requested a review from cacraigucar March 3, 2021 18:18

nusbaume marked this pull request as ready for review March 4, 2021 15:59

nag compiler fixes

bf509c7

cacraigucar requested changes Mar 5, 2021

View reviewed changes

fvitt added 4 commits March 8, 2021 11:36

changes per code review by cacraigucar

fad5370

modified: src/ionosphere/waccmx/edyn_geogrid.F90 modified: src/ionosphere/waccmx/edyn_grid_comp.F90 modified: src/ionosphere/waccmx/oplus.F90 modified: src/ionosphere/waccmx/wei05sc.F90

Increase MPI buffer parameter

47f41d4

modified: src/ionosphere/waccmx/edyn_mpi.F90

gold2718 approved these changes Mar 27, 2021

View reviewed changes

fvitt and others added 5 commits March 27, 2021 13:31

support for nuopc threading

000f19e

clean up

0dd4987

more nag changes

98594ca

modified: src/ionosphere/waccmx/edyn_mudcom.F90

cacraigucar approved these changes Mar 30, 2021

View reviewed changes

move multi-inst test to prealpha

45515f1

cacraigucar changed the title ~~Refactor coupling with WACCMX ionosphere~~ cam6_3_016: Refactor coupling with WACCMX ionosphere Mar 30, 2021

fvitt added 6 commits March 30, 2021 18:27

merge to head of cam_development

dd6fdfc

Merge remote-tracking branch 'jedwards4b/nuopc_threading' into phys-i…

5a5e2b7

…onos-cpl_cam6_3_002

remove erroneous mgrid configure option from outfrq9s_576tsks test mod

0dff441

remvove extra slow aux_cam test ERC_Ln9.f45_f45_mg37.QPC4X.izumi_pgi

e70b159

ChangeLog draft

a68412e

update ChangeLog

765d0d3

fvitt merged commit 58e9d7c into ESCOMP:cam_development Apr 1, 2021

		nlon_geo, & ! size of geo lon dimension
		nlat_geo, & ! size of geo lat dimension

cam6_3_016: Refactor coupling with WACCMX ionosphere #264

cam6_3_016: Refactor coupling with WACCMX ionosphere #264

Uh oh!

Conversation

fvitt commented Nov 2, 2020

Uh oh!

nusbaume left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fvitt commented Dec 30, 2020

Uh oh!

nusbaume left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gold2718 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects