[Issue]: Quartz with Jdbc JobStore doesn't work. NotSerializableException: Unable to serialize JobDataMap for insertion into database because the value of property 'batchJob' is not serializable: com.paypal.invoices.extractioncreditnotes.batchjobs.CreditNotesRetryBatchJob

[smaxx]

Describe the Issue

With the default configuration Quartz uses RAMJobStore, so all of the job-related data and "locking" happens only in memory. The point is that Quartz is capable of doing the actual persistence using JDBC DataSource. It will mean that jobs could be restarted even after major application failures and subsequent restart, and the following markup @PersistJobDataAfterExecution and @DisallowConcurrentExecution which is currently used on Job level will work in multi-instance(horizontally scaled) setup as well, so concurrent jobs execution will be prevented by acquiring DB-level locks by Quartz.
The issue itself is that the application is NOT ready to use JDBC based persistence.
com.paypal.jobsystem.quartzadapter.job.QuartzBatchJobBuilder defines JobDataMap of the Job and puts the actual job instance to this map:

jobDataMap.put(QuartzBatchJobBean.KEY_BATCH_JOB_BEAN, batchJob);

JobDataMap is stored in the JobStore and allows the job to be restarted in case of failures. Problem is that it can only work with RAMJobStore as with the JDBC based JobStore this JobDataMap is persisted to the database, so all of its content has to be Serializable. In our case with enabled Quartz persistence it is causing the following:
```
Caused by: org.quartz.JobPersistenceException: Couldn't store job: Unable to serialize JobDataMap for insertion into database because the value of property 'batchJob' is not serializable: com.paypal.invoices.extractioncreditnotes.batchjobs.CreditNotesRetryBatchJob
at org.quartz.impl.jdbcjobstore.JobStoreSupport.storeJob(JobStoreSupport.java:1120)
at org.quartz.impl.jdbcjobstore.JobStoreSupport$3.executeVoid(JobStoreSupport.java:1095)
at org.quartz.impl.jdbcjobstore.JobStoreSupport$VoidTransactionCallback.execute(JobStoreSupport.java:3780)
at org.quartz.impl.jdbcjobstore.JobStoreSupport$VoidTransactionCallback.execute(JobStoreSupport.java:3778)
at org.quartz.impl.jdbcjobstore.JobStoreCMT.executeInLock(JobStoreCMT.java:245)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.storeJob(JobStoreSupport.java:1091)
at org.quartz.core.QuartzScheduler.addJob(QuartzScheduler.java:938)
at org.quartz.core.QuartzScheduler.addJob(QuartzScheduler.java:927)
at org.quartz.impl.StdScheduler.addJob(StdScheduler.java:268)
at org.springframework.scheduling.quartz.SchedulerAccessor.addJobToScheduler(SchedulerAccessor.java:283)
at org.springframework.scheduling.quartz.SchedulerAccessor.registerJobsAndTriggers(SchedulerAccessor.java:225)
at org.springframework.scheduling.quartz.SchedulerFactoryBean.afterPropertiesSet(SchedulerFactoryBean.java:507)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1817)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1766)
... 44 common frames omitted
Caused by: java.io.NotSerializableException: Unable to serialize JobDataMap for insertion into database because the value of property 'batchJob' is not serializable: com.paypal.invoices.extractioncreditnotes.batchjobs.CreditNotesRetryBatchJob
at org.quartz.impl.jdbcjobstore.StdJDBCDelegate.serializeJobData(StdJDBCDelegate.java:3083)
at org.quartz.impl.jdbcjobstore.StdJDBCDelegate.updateJobDetail(StdJDBCDelegate.java:647)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.storeJob(JobStoreSupport.java:1115)

What are the alternatives? As opposed to storing the object graph with all of its dependencies(instance of the CreditNotesRetryBatchJob for this case) which doesn't make much sense as this bean and all of its dependencies are stateless Spring services we can only store the batch job class name:

jobDataMap.put(QuartzBatchJobBean.KEY_BATCH_JOB_BEAN, batchJob.getClass());

when the job instance has to be created/recreated we can get the actual instance from the Spring context by class(see how it is done via injected BeanFactory below):

public class QuartzBatchJobBean extends QuartzJobBean {

public static final String KEY_BATCH_JOB_BEAN = "batchJob";

@Autowired
private BeanFactory beanFactory;

@Autowired
private QuartzBatchJobAdapterFactory quartzBatchJobAdapterFactory;

private BatchJob<? extends BatchJobContext, ? extends BatchJobItem<?>> batchJob;

/**
 * {@inheritDoc}
 */
@Override
public void executeInternal(final JobExecutionContext context) throws JobExecutionException {
	quartzBatchJobAdapterFactory.getQuartzJob(batchJob).execute(context);
}

public void setBatchJob(final Class<? extends BatchJob<BatchJobContext, BatchJobItem<?>>> batchJobClass) {
	this.batchJob = beanFactory.getBean(batchJobClass);
}

public static Class<?> getBatchJobClass(final JobExecutionContext context) {
	return (Class<?>) context.getJobDetail().getJobDataMap().get(KEY_BATCH_JOB_BEAN);
}

}

This approach is universal and covers all of the possible cases, both RAMJobStore and persistence-based. The approach has been tested with Postges DB and following configuration:

spring:
quartz:
job-store-type: jdbc
overwrite-existing-jobs: true
jdbc:
initialize-schema: always
properties:
org.quartz.jobStore.driverDelegateClass: org.quartz.impl.jdbcjobstore.PostgreSQLDelegate

### Environment

Development

### Version

5.1.0

### Expected Behavior

JobDataMap has to be successfully persisted to the database table 'qrtz_job_details', column 'job_data'.
In case of failures the job has to be successfully recreated from the persisted state.

### Actual Behavior

Exception is always thrown on an attempt to serialize each instance of the BatchJob inheritors during application startup:

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'invoiceExtractJobController': Injection of resource dependencies failed
at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.postProcessProperties(CommonAnnotationBeanPostProcessor.java:323)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1416)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:597)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:520)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:326)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:324)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:973)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:942)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:608)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:146)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:734)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:436)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:312)
at com.paypal.HyperwalletMiraklConnectorApplication.main(HyperwalletMiraklConnectorApplication.java:43)
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'jobService': Injection of resource dependencies failed
at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.postProcessProperties(CommonAnnotationBeanPostProcessor.java:323)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1416)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:597)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:520)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:326)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:324)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:205)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.resolveBeanByName(AbstractAutowireCapableBeanFactory.java:457)
at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.autowireResource(CommonAnnotationBeanPostProcessor.java:537)
at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.getResource(CommonAnnotationBeanPostProcessor.java:508)
at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor$ResourceElement.getResourceToInject(CommonAnnotationBeanPostProcessor.java:659)
at org.springframework.beans.factory.annotation.InjectionMetadata$InjectedElement.inject(InjectionMetadata.java:270)
at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:145)
at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.postProcessProperties(CommonAnnotationBeanPostProcessor.java:320)
... 15 common frames omitted
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'quartzScheduler' defined in class path resource [org/springframework/boot/autoconfigure/quartz/QuartzAutoConfiguration.class]: Couldn't store job: Unable to serialize JobDataMap for insertion into database because the value of property 'batchJob' is not serializable: com.paypal.invoices.extractioncreditnotes.batchjobs.CreditNotesRetryBatchJob
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1770)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:598)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:520)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:326)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:324)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200)
at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:254)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1417)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1337)
at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.autowireResource(CommonAnnotationBeanPostProcessor.java:531)
at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.getResource(CommonAnnotationBeanPostProcessor.java:508)
at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor$ResourceElement.getResourceToInject(CommonAnnotationBeanPostProcessor.java:659)
at org.springframework.beans.factory.annotation.InjectionMetadata$InjectedElement.inject(InjectionMetadata.java:270)
at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:145)
at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.postProcessProperties(CommonAnnotationBeanPostProcessor.java:320)
... 29 common frames omitted
Caused by: org.quartz.JobPersistenceException: Couldn't store job: Unable to serialize JobDataMap for insertion into database because the value of property 'batchJob' is not serializable: com.paypal.invoices.extractioncreditnotes.batchjobs.CreditNotesRetryBatchJob
at org.quartz.impl.jdbcjobstore.JobStoreSupport.storeJob(JobStoreSupport.java:1120)
at org.quartz.impl.jdbcjobstore.JobStoreSupport$3.executeVoid(JobStoreSupport.java:1095)
at org.quartz.impl.jdbcjobstore.JobStoreSupport$VoidTransactionCallback.execute(JobStoreSupport.java:3780)
at org.quartz.impl.jdbcjobstore.JobStoreSupport$VoidTransactionCallback.execute(JobStoreSupport.java:3778)
at org.quartz.impl.jdbcjobstore.JobStoreCMT.executeInLock(JobStoreCMT.java:245)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.storeJob(JobStoreSupport.java:1091)
at org.quartz.core.QuartzScheduler.addJob(QuartzScheduler.java:938)
at org.quartz.core.QuartzScheduler.addJob(QuartzScheduler.java:927)
at org.quartz.impl.StdScheduler.addJob(StdScheduler.java:268)
at org.springframework.scheduling.quartz.SchedulerAccessor.addJobToScheduler(SchedulerAccessor.java:283)
at org.springframework.scheduling.quartz.SchedulerAccessor.registerJobsAndTriggers(SchedulerAccessor.java:225)
at org.springframework.scheduling.quartz.SchedulerFactoryBean.afterPropertiesSet(SchedulerFactoryBean.java:507)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1817)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1766)
... 44 common frames omitted
Caused by: java.io.NotSerializableException: Unable to serialize JobDataMap for insertion into database because the value of property 'batchJob' is not serializable: com.paypal.invoices.extractioncreditnotes.batchjobs.CreditNotesRetryBatchJob
at org.quartz.impl.jdbcjobstore.StdJDBCDelegate.serializeJobData(StdJDBCDelegate.java:3083)
at org.quartz.impl.jdbcjobstore.StdJDBCDelegate.updateJobDetail(StdJDBCDelegate.java:647)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.storeJob(JobStoreSupport.java:1115)
... 57 common frames omitted

### Steps to Reproduce

simply start the application with Quartz persistence being enabled:

spring:
quartz:
job-store-type: jdbc
overwrite-existing-jobs: true
jdbc:
initialize-schema: always
properties:
org.quartz.jobStore.driverDelegateClass: org.quartz.impl.jdbcjobstore.PostgreSQLDelegate

### Pre-conditions

_No response_

### Relevant log output

_No response_

[max6001]

Hi @smaxx
What you'd like to achieve would require adjustments to be made in the way our scheduler is currently operating.
As it is a sensitive change on our side, would you mind describing the business/functional reason that requires such ability on your side?

Thanks in advance.

[smaxx]

hi @max6001,

1. Scenario is quite simple. We want to run our HyperwalletMiraklConnector being horizontally scaled. With the existing setup(in-memory storage) it seems to be incorrectly working in some cases. We want be able to ensure that scheduled jobs annotated with @DisallowConcurrentExecution are invoked only once on a single pod while the remaining ones will potentially process some other jobs. This will guarantee that we won't send some payments twice or update configuration concurrently. And relying on actual persistence used by Quartz is exactly what ensures us in having this handled in a proper way - by obtaining DB-level locks, keeping the persistent state in DB
2. FYI we have implemented the suggested approach in our project and it works well. The only addition to the previously proposed solution is persisting not a job class, but only class name. What we faced in our env during testing was that sometimes(due to active development process) job classes can be not found by JVM(class is removed from the project or not yet "published" but the "strong reference" to it is stored in job data), as a result we get ClassNotFoundException. Alternatively we store only full class name, and then load class by name and get the job instance from the context while using failsafe approach. We can't fully avoid the loss of a job(custom job belonging to our project in addition to the ones in this repository) as a result of future maintenance, but at least we catch/log/suppress such scenario without unwanted retries and exception handling...