Branch run fails on maint-2.1 when reading restart (Error reading rst_calendar) #7847
Description
I am currently trying to do some time slice experiments with E3SM v2.1, with runs branching from each month of the year of a control run (Jan 1, Feb 1, etc.). In the past, I have been successful with my branch runs, but this time I face this error when the restart files are read in:
128: PIO: FATAL ERROR: Aborting... An error occured, Reading variable (rst_calendar, varid=0) from file (no_tags2.eam.r.0001-02-01-00000.nc, ncid=17) failed with PIO_IOTYPE_PNETCDF iotype. The low level (PnetCDF) I/O library call failed to read the variable (Number of regions = 1, iodesc id = 513, Bytes to read on this process = 1555200). NetCDF: Start+count exceeds dimension bound (err=-57). Aborting since the error handler was set to PIO_INTERNAL_ERROR... (/global/u2/a/aaudette/E3SM/externals/scorpio/src/clib/pio_darray_int.c: 1223)
I have checked that the rst_calendar variable is in the restart file and everything seems to be in order. Any help or guidance would be much appreciated.