Add recurring scheduler to async-job server (workers + scheduler), with logs, aliases, and tests

[darkamenosa]

Hi Samuel,

Since recurring jobs are important, but this doesn't have that. So I implement it. So what I want is API, config as close as Solid Queue as possible since it is the default of Rails 8.

I test your async-cron, but I feel the style of solid queue make more sense. Just define a recurring.yml, then done. So now, I'm not sure how we handle workers and schedulers better. Just implement initial concept, by default we will have 1 scheduler, and other workers behave as same as the old way.

My concern is now we have a lot of ENV to remember, should we just read the queue.yml config of Solid Queue. Also should we handle scheduler redis here or in async-job-processor-redis.

So please review, and give me some direction.

Summary:

• Introduces a Fugit-based recurring scheduler that runs alongside Active Job workers when you launch the provided server.
• Keeps defaults simple: workers only unless a valid config/recurring.yml exists; one scheduler process if it does.
• Clear logs on startup and per task; supports Redis cross-host dedup and “last enqueued” timestamps (Rails.cache fallback when Redis is absent).
• Adds tests for loader + scheduler, plus quality-of-life improvements (STDOUT sync, fewer dev workers).

Why:

• Provide a single production-style entry point like bin/jobs (Solid Queue).
• Enable apps to run workers and recurring tasks together without extra tooling.
• Emit minimal, dashboard-friendly metadata (last enqueued) without requiring SQL.

Key changes:

• Server composition (unchanged UX; richer behavior when schedule exists)
  • bin/async-job-adapter-active_job-server
    • Workers service: unchanged.
    • Scheduler service: added only if config/recurring.yml has tasks for current env and not skipped.
    • Output flush: STDOUT.sync = true, STDERR.sync = true to avoid buffered job output at shutdown.
• Recurring scheduler (new)
  • lib/async/job/adapter/active_job/recurring/task.rb
    • Task struct; “every N seconds/minutes/hours” → cron normalization.
  • lib/async/job/adapter/active_job/recurring/loader.rb
    • Loads config/recurring.yml per env.
    • Logs: missing file/empty env, unknown job class, unsupported schedules.
    • Accepts path aliases: ASYNC_JOB_RECURRING_SCHEDULE, SOLID_QUEUE_RECURRING_SCHEDULE, RECURRING_SCHEDULE_FILE.
  • lib/async/job/adapter/active_job/recurring/scheduler.rb
    • Per-task async loops; enqueue perform_later with queue/priority/args; optional command.
    • Dedup backends: auto|redis|memory (Redis if REDIS_URL).
    • Last-run backends: redis|cache (Rails.cache fallback if no Redis).
    • Env aliases honored: JOBS_DEDUP_BACKEND, JOBS_LAST_BACKEND, JOBS_SCHEDULER_DEDUP_TTL, JOBS_REDIS_PREFIX.
  • lib/async/job/adapter/active_job/recurring/service.rb
    • Runs the scheduler inside an Async reactor.
    • Startup banner: tasks, env, path, dedup=, last=, prefix=.
    • Per-task schedule lines: key, schedule, queue, priority.
    • Skip flags: ASYNC_JOB_SKIP_RECURRING, SOLID_QUEUE_SKIP_RECURRING, JOBS_SKIP_RECURRING.
• Worker process count (simple default)
  • lib/async/job/adapter/active_job/environment.rb
    • Dev default: 1 worker.
    • Override anywhere with ASYNC_JOB_WORKERS (or JOBS_COUNT).
    • Otherwise: default remains nprocessors.
• Docs
  • guides/recurring/readme.md: format, env vars, Redis keys, examples.
  • Getting started guides mention scheduler.
• Tests (Sus)
  • test/async/job/adapter/active_job/recurring/loader.rb: normalization + parsing.
  • test/async/job/adapter/active_job/recurring/scheduler.rb: enqueue smoke test, last-run cache fallback, backend alias selection.

Defaults and behavior:

• No schedule → workers only (exactly like before).
• With schedule:
  • Production-ish: jobs = nprocessors, scheduler = 1.
  • Development: jobs = 1, scheduler = 1.
• One env to change worker count: ASYNC_JOB_WORKERS=N.

Configuration (env)

• Schedule path: ASYNC_JOB_RECURRING_SCHEDULE | SOLID_QUEUE_RECURRING_SCHEDULE | RECURRING_SCHEDULE_FILE
• Skip scheduler: ASYNC_JOB_SKIP_RECURRING=true | SOLID_QUEUE_SKIP_RECURRING=true | JOBS_SKIP_RECURRING=true
• Redis prefix: ASYNC_JOB_REDIS_PREFIX | JOBS_REDIS_PREFIX (default async-job)
• Dedup backend: ASYNC_JOB_RECURRING_DEDUP | JOBS_DEDUP_BACKEND (default auto)
• Last-run backend: ASYNC_JOB_RECURRING_LAST | JOBS_LAST_BACKEND (default auto)
• Dedup TTL: ASYNC_JOB_RECURRING_DEDUP_TTL | JOBS_SCHEDULER_DEDUP_TTL (default 600)
• Workers count: ASYNC_JOB_WORKERS | JOBS_COUNT
• Redis enable: REDIS_URL

Redis keys (if Redis enabled)

• Dedup lock: :recurring:exec::<run_at_epoch>
• Last-run: :recurring:last (H) key → epoch

Example schedule

config/recurring.yml:

  development:
  my_task:
  class: MyJob
  queue: default
  priority: 0
  args: [42, { ping: "pong" }]
  schedule: every 5 seconds

Backwards compatibility

• No breaking changes. If no schedule is present or scheduler is skipped, server behaves exactly as before.

Test plan

• Unit: bundle exec sus
• Manual:
  • Start server: bundle exec async-job-adapter-active_job-server
  • Watch logs: startup banner + [scheduler] task lines + Enqueued recurring task.
  • Redis (optional): HGETALL async-job:recurring:last and SCAN MATCH async-job:recurring:exec:*

Files worth a look

• bin/async-job-adapter-active_job-server
• lib/async/job/adapter/active_job/recurring/*
• lib/async/job/adapter/active_job/environment.rb
• guides/recurring/readme.md
• tests under test/async/job/adapter/active_job/recurring/*

[ioquatix]

Thanks, I'll take a look later this week.

[trevorturk]

Just wanted to point out that Rails seems to be open to adding to Active Job lately (rails/solid_queue#142 (comment)) so it may be worth considering trying to standardize...

[darkamenosa]

@ioquatix I just checked again. I forgot to look at .github workflow. Give me a little time to check and fix before the review. But the direction is the same.

@trevorturk Yes. That's good news. We need some standard to follow