Dead executions after app shutdown

[artem-mindrov]

Expected Behavior

Executions are marked for rescheduling.

Current Behavior

The scheduler is set to pollUsingLockAndFetch(0.5, 3.0).
It fails to mark executions free due to a thread interrupt, see example stack trace below.

Steps to Reproduce the bug

It's a Play framework app with this lifecycle hook:

      coordinatedShutdown.addTask(
          CoordinatedShutdown.PhaseBeforeServiceUnbind(),
          "shut-down-scheduler",
          this::coordinatedSchedulerShutdown);

The shutdown code is

      ExecutorService executorService = Executors.newSingleThreadExecutor();
      executorService.submit(scheduler::stop);

      for (var taskEntry : runningTasks.entrySet()) {
        taskEntry.getValue().interrupt(); // gracefully shutdown currently executing tasks here
      }

      executorService.shutdown();

Context

• DB-Scheduler Version : 14.1.0
• Java Version : OpenJDK 17.0.12
• Spring Boot (check for Yes) : [ ]
• Database and Version : Google Cloud SQL but also happens on PG 13.4.

Logs

Failed while completing execution Execution: task=cleanup-user-task, id=recurring, executionTime=2024-11-14T13:29:12.203933Z, picked=true, pickedBy=apiserver-595664b4fb-84jj4, lastHeartbeat=2024-11-14T13:29:16.503595Z, version=61938, because SQLRuntimeException: 'Unable to open connection'. Execution will likely remain scheduled and locked/picked. The execution should be detected as dead after a while, and handled according to the tasks DeadExecutionHandler.
com.github.kagkarlsson.shaded.jdbc.SQLRuntimeException: Unable to open connection
	at com.github.kagkarlsson.shaded.jdbc.JdbcRunner.withConnection(JdbcRunner.java:228)
	at com.github.kagkarlsson.shaded.jdbc.JdbcRunner.execute(JdbcRunner.java:131)
	at com.github.kagkarlsson.shaded.jdbc.JdbcRunner.execute(JdbcRunner.java:76)
	at com.github.kagkarlsson.scheduler.jdbc.JdbcTaskRepository.rescheduleInternal(JdbcTaskRepository.java:461)
	at com.github.kagkarlsson.scheduler.jdbc.JdbcTaskRepository.reschedule(JdbcTaskRepository.java:432)
	at com.github.kagkarlsson.scheduler.task.ExecutionOperations.reschedule(ExecutionOperations.java:48)
	at com.github.kagkarlsson.scheduler.task.CompletionHandler$OnCompleteReschedule.complete(CompletionHandler.java:63)
	at com.github.kagkarlsson.scheduler.ExecutePicked.complete(ExecutePicked.java:131)
	at com.github.kagkarlsson.scheduler.ExecutePicked.executePickedExecution(ExecutePicked.java:116)
	at com.github.kagkarlsson.scheduler.ExecutePicked.run(ExecutePicked.java:89)
	at com.github.kagkarlsson.scheduler.Executor.lambda$addToQueue$0(Executor.java:53)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.sql.SQLException: Interrupted getting connection from pool
	at io.ebean.datasource.pool.PooledConnectionQueue.obtainConnection(PooledConnectionQueue.java:202)
	at io.ebean.datasource.pool.ConnectionPool.getPooledConnection(ConnectionPool.java:676)
	at io.ebean.datasource.pool.ConnectionPool.getConnection(ConnectionPool.java:666)
	at com.github.kagkarlsson.shaded.jdbc.DataSourceConnectionSupplier.getConnection(DataSourceConnectionSupplier.java:31)
	at com.github.kagkarlsson.shaded.jdbc.JdbcRunner.withConnection(JdbcRunner.java:226)
	... 13 common frames omitted
Caused by: java.lang.InterruptedException: null
	at java.base/java.util.concurrent.locks.ReentrantLock$Sync.lockInterruptibly(ReentrantLock.java:159)
	at java.base/java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:372)
	at io.ebean.datasource.pool.PooledConnectionQueue._obtainConnection(PooledConnectionQueue.java:218)
	at io.ebean.datasource.pool.PooledConnectionQueue.obtainConnection(PooledConnectionQueue.java:195)
	... 17 common frames omitted

[kagkarlsson]

Must be an issue with the shutdown-sequence. It seems you are interrupting the obtainConnection call?