Allow streams to operate on reduced number of PETs

[ekluzek]

I think this maybe something that's really done in the host code that calls streams. But, I'm adding it here in case there are changes required in CDEPS. And because the errors show up in the stream code.

If you have more processors than mesh elements in the stream file -- ESMF dies with an error because of this. AT first I thought I just needed to get an ESMF_DistGrid for the mesh file that was no more than the elements in the file. However, after trying this I see the error below that tells us that the VM for the

Errors in the PET file look like this:

20251116 014306.030 ERROR PET127 /glade/derecho/scratch/csgteam/temp/spack/derecho/24.12/builds/spack-stage-esmf-8.8.1-oigqhlcifmbqcxchruwcjowzoxz5sa4j/spack-src/src/Infrastructure/Mesh/src/ESMCI_Mesh_FileIO.C:456 ESMCI_mesh_create_from_ESMFMesh_ Value unrecognized or out of range - Can't create a Mesh from a file in a VM when that VM contains more PETs than elements in the file.

cesm.log shows it failing in the streams initialization:

dec0745.hsn.de.hpc.ucar.edu 87:   Streams initialization failing for Delta14co2_in_air stream file =
dec0745.hsn.de.hpc.ucar.edu 87:  /glade/campaign/cesm/cesmdata/inputdata/lnd/clm2/isotopes/ctsmforc.Graven.atm_d
dec0745.hsn.de.hpc.ucar.edu 87:  elta_C14_CMIP7_4x1_global_1700-2023_yearly_v3.0_c251013.nc
dec0745.hsn.de.hpc.ucar.edu 87:  ENDRUN:
dec0745.hsn.de.hpc.ucar.edu 87:  ERROR in CTSMForce2DStreamBaseType.F90 at line 197
.
.
.
dec0745.hsn.de.hpc.ucar.edu 107: cesm.exe           00000000011D80E0  shr_abort_mod_mp_          67  shr_abort_mod.F90
dec0745.hsn.de.hpc.ucar.edu 107: cesm.exe           00000000005EBAB0  abortutils_mp_end          63  abortutils.F90
dec0745.hsn.de.hpc.ucar.edu 107: cesm.exe           000000000094CC14  ctsmforce2dstream         197  CTSMForce2DStreamBaseType.F90
dec0745.hsn.de.hpc.ucar.edu 107: cesm.exe           0000000000FAAA08  atmcarbonisotopes         171  AtmCarbonIsotopeStreamType.F90
dec0745.hsn.de.hpc.ucar.edu 107: cesm.exe           00000000007E780A  cisoatmtimeseries         563  CNCIsoAtmTimeSeriesReadMod.F90
dec0745.hsn.de.hpc.ucar.edu 107: cesm.exe           0000000000603196  clm_initializemod         481  clm_initializeMod.F90
dec0745.hsn.de.hpc.ucar.edu 107: cesm.exe           00000000005A691B  lnd_comp_nuopc_mp         677  lnd_comp_nuopc.F90

The lnd.log shows the error as well.

[ekluzek]

The way to get this to work, would be to create a new VM that has a reduced set of PET's. There will need to be a new model mesh created for this VM as well. The regridding can be done on the reduced VM, and then a redist done to the full model VM.

I think this means that the streams code will need to know about both VM's and know when it needs to do the redist between them. This should just need to be done when new data is read in. The time interpolation should be able to be done on the full VM.

In principle this should be fairly straightforward to do, but in practice I suspect it will take some effort to get it done correctly.

[ekluzek]

The two use cases for this that I know about:

1.) Carbon isotopes to be independent of grid details (allowing both CMIP6 and CMIP7 without code changes

I wanted to get this working without the need to have the code know the details of the grid. But, that will require this multiple VM approach.

2.) Ultra high resolution streams

Streams for ultra high resolutions are likewise limited in the number of processors allowed -- because of this limitation. Doing this would allow the PET size to be reduced for regridding and then the redist to the full grid. This would allow ultra high resolutions to not be bounded by the number of gridcells on input datasets.

@briandobbins

[ekluzek]

I'm pretty sure in terms of CDEPS code it's failing in streams/dshr_strdata_mod.F90

at the FieldRegridStore store step like this one:

          else if (trim(sdat%stream(ns)%mapalgo) == 'nn') then
             call ESMF_FieldReGridStore(sdat%pstrm(ns)%field_stream, lfield_dst, &
                  routehandle=sdat%pstrm(ns)%routehandle, &
                  regridmethod=ESMF_REGRIDMETHOD_NEAREST_STOD, &
                  dstMaskValues=(/sdat%stream(ns)%dst_mask_val/), &
                  srcMaskValues=(/sdat%stream(ns)%src_mask_val/), &
                  srcTermProcessing=srcTermProcessing_Value, ignoreDegenerate=.true., &
                  unmappedaction=ESMF_UNMAPPEDACTION_IGNORE, rc=rc)
          else if (trim(sdat%stream(ns)%mapalgo) == 'consf') then

[ekluzek]

In the CSEG meeting today Bob Oehmke thought we could get around this by assigning some PET's to empty DE's. So it would still use all processors, but some would be idle as they wouldn't have any data on them.

[ekluzek]

OK, it looks like the issue is really in the creation of the stream_mesh which is in dshr_strdata_mod.F90 as follows:

          stream_mesh = ESMF_MeshCreate(trim(filename), fileformat=ESMF_FILEFORMAT_ESMFMESH, rc=rc)
          if (ChkErr(rc,__LINE__,u_FILE_u)) return

Since the optional argument elementDistGrid isn't provided it's going to assume one DE per PET and carry that to the next step of creating the Field and then the Regrid route handle shown above.

                lfield = ESMF_FieldCreate(sdat%model_mesh, ESMF_TYPEKIND_r8, &
                     name=trim(sdat%pstrm(ns)%fldlist_model(nfld)), &
                     meshloc=ESMF_MESHLOC_ELEMENT, rc=rc)

So I think the stream_mesh must need to be defined in such a way that each DE corresponds to a PET, and then there are empty DE's for PET's that overflow the number of elements.