Advice on debugging/diagnosing and resolving GoodJob::InterruptError errors

[ronb54]

Hi everyone,

I've been trying to figure out why some of our jobs are being interrupted then retried over and over again on production. I've read through the READMEs and looked through issues/discussions here, and have implemented some config that I thought might help but we're still seeing the interrupts & retries. I'm currently out of ideas and stuck...

Would you have any advice on how I would go about understanding why the interrupts might be happening?

Here's an example of one job with three attempts.

Here's another for a TurboStreams::ActionBroadcastJob:

And the number of attempts isn't consistent either. All the jobs do eventually succeed.

For context, here are some relevant config I've set up:

    WEB_CONCURRENCY: 2
    RAILS_MAX_THREADS: 4
    GOOD_JOB_MAX_THREADS: 5
    GOOD_JOB_QUEUES: "latency_1s:1;latency_5s,latency_10s:3;latency_5m:5;-latency_1s,latency_5s,latency_10s,latency_5m:2;*"
    GOOD_JOB_SHUTDOWN_TIMEOUT: 30

We've mostly stayed fairly close to default:

Rails.application.configure do
  config.good_job = {
    enable_cron: true,
    cron: {
      clear_expired_appointments: {
        cron: "0 0 * * *",
        class: "ClearExpiredAppointmentsJob",
        description: "Clear expired appointments every day at midnight.",
        enabled_by_default: -> { Rails.env.production? }
      }
    },
    on_thread_error: ->(exception) { Bugsnag.notify(exception) }
  }
end

class ApplicationJob < ActiveJob::Base
  JobTimeoutError = Class.new(StandardError)

  # Automatically retry jobs that encountered a deadlock
  retry_on ActiveRecord::Deadlocked

  # Most jobs are safe to ignore if the underlying records are no longer available
  discard_on ActiveJob::DeserializationError

  queue_as :default

  around_perform do |_job, block|
    Timeout.timeout(5.minutes, JobTimeoutError) do
      block.call
    end
  end

  def log
    @log ||= Rails.logger
  end
end

The current infrastructure is fairly simple:

• 1 worker instance: 8GB RAM / 4 CPUs
• 1 web instance: 8GB RAM / 4 CPUs

Let me know if you need more context than that. I'm still trying to wrap my head around processes & threads, so I'm totally open to any feedback on our config above. It's also possible I'm totally missing something super obvious. 😭

Thanks in advance! And thank you for this gem! 🙌🏽

[bensheldon]

InterruptErrors happen because your process is being SIGKILLed. You'll always see two of them back-to-back because the older execution is the execution that was interrupted, and then the later one is the execution that detected that a previous execution had been interrupted and cleans it up by raising a handleable exception. (btw I have an issue to make this better: #1382)

If you're running a single worker instance, I imagine that there is something incorrectly configured with shutdown signals: https://github.com/bensheldon/good_job?tab=readme-ov-file#Interrupts-graceful-shutdown-and-SIGKILL

You'll want to send a SIGTERM, wait shutdown_timeout to give GoodJob the opportunity to gracefully shutdown and exit, and only then send a SIGKILL if it doesn't.

How are you managing the GoodJob process? In docker or systemd or something else?

[ronb54]

Oh man..it took me a while but I finally realized that the DB connection good_job was using with PgBouncer was in transaction mode, instead of a direct connection. This type of connection is what was causing the interrupts. Basically misconfigured it after we had migrated database providers. 🤦🏽

For future travellers, I took the advice given here in creating a separate parent class for the active record for good_job.