If a training job is started but doesn't finish it should be able to pick up from where it left off. Checkpoints are created in both the checkpoints and log folders. These can be used to restore the model from the most recent point.