Add retry capabilities in when launching jobs

[fmbenhassine]

There are several cases where launching a job might fail because of a transient error (like a database that is not ready yet, or a required service that is temporarily not available). I particularly think of cloud environments where services are deployed independently in no particular order so the batch job could be started before the database or other required resources are not ready yet. In these cases, it is unfortunate to not be able to retry launching the job (and therefore miss/waste a scheduled job launch cycle for example).

This feature is about adding the ability to retry the job launching sequence to the Job Operator API with a configurable retry template, something like:

@Configuration
@EnableBatchProcessing(retryTemplateRef = "batchRetryTemplate")
public class JobConfiguration {

    @Bean
    public Job job(JobRepository jobRepository) {
        return new JobBuilder("job", jobRepository)
                           // define job execution flow
                           .build();
    }

    @Bean
    public RetryTemplate batchRetryTemplate() {
        RetryPolicy retryPolicy = RetryPolicy.builder()
                .maxAttempts(3)
                .delay(Duration.ofSeconds(1))
                .includes(DataSourceLookupFailureException.class)
                .build();
        return new RetryTemplate(retryPolicy);
    }

}

This would retry to launch the job according to the defined retry policy when a DataSourceLookupFailureException.class is thrown at startup.

References:

• https://stackoverflow.com/questions/55128067/spring-batch-how-to-configure-retry-period-for-failed-jobs/55137314#55137314

[fmbenhassine]

I am not sure yet about the best way to address this, so feedback is welcome. While it's possible to add a retry template to the JobOperator (see 5582537), or introduce a decorator like RetryableJobOperator, my concern is that we can't be sure that the exception that might be thrown from the start method is due to a resource that is not available (ie the case we are trying to address) or because of an exception is thrown from the job itself. This issue is somehow related to #4865 as well.

Another, more fine-grained option, has been proposed in Boot as well: spring-projects/spring-boot#4779.

I need a second pair of eyes on this, so feedback is welcome.