netcdf files not being written completely due to system issue

[jiandewang]

Two possible (though will happen very rare) scenarios for future operational consideration:

(1) model died due to system issue, at the final moment right before system's death intermediate restart files were written complete but not the history file. In this case post job will fail. We just had one of this case in V17 cycled test run.

(2) model died due to system issue at the time restart files not being fully written yet but restart files exist on disk already. In this case the model will read in this incomplete restart file and fail, complaining that it can't read in the restart file correctly.