Skip to content

Possible race condition between startup and shutdownΒ #373

@hmol

Description

@hmol

I'm Im getting this error often:
JobLookupError 'No job by the id of 5b5a4fcd3cf742b0a97cc9dfcbc47d1d was found'
Could it be a race condition between startup and shutdown? See stack trace below:

Traceback (most recent call last):
  File "/app/pp/queues/__main__.py", line 47, in main
    asyncio.run(run_postgres_worker())
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/asyncio/runners.py", line 195, in run
    return runner.run(main)
           ~~~~~~~~~~^^^^^^
  File "/usr/local/lib/python3.13/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/usr/local/lib/python3.13/asyncio/base_events.py", line 725, in run_until_complete
    return future.result()
           ~~~~~~~~~~~~~^^
  File "/app/pp/queues/__main__.py", line 33, in run_postgres_worker
    await postgres_worker.run(settings.queue_worker.null_mode)
  File "/app/pp/queues/postgres_worker.py", line 363, in run
    async with self.worker_lifespan():
               ~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/local/lib/python3.13/contextlib.py", line 221, in __aexit__
    await anext(self.gen)
  File "/app/pp/queues/default.py", line 434, in worker_lifespan
    worker_shut_down_unleash_client(),
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/local/lib/python3.13/contextlib.py", line 221, in __aexit__
    await anext(self.gen)
  File "/app/pp/queues/default.py", line 214, in worker_shut_down_unleash_client
    await unleash_client.shut_down()
  File "/app/shared/clients/unleash/client.py", line 94, in shut_down
    client.destroy()
    ~~~~~~~~~~~~~~^^
  File "/app/shared/clients/unleash/client.py", line 66, in destroy
    self.client.destroy()
    ~~~~~~~~~~~~~~~~~~~^^
  File "/srv/venv/apiserver/pp-9TtSrW0h-py3.13/lib/python3.13/site-packages/UnleashClient/__init__.py", line 399, in destroy
    self.fl_job.remove()
    ~~~~~~~~~~~~~~~~~~^^
  File "/srv/venv/apiserver/pp-9TtSrW0h-py3.13/lib/python3.13/site-packages/apscheduler/job.py", line 123, in remove
    self._scheduler.remove_job(self.id, self._jobstore_alias)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/venv/apiserver/pp-9TtSrW0h-py3.13/lib/python3.13/site-packages/apscheduler/schedulers/base.py", line 722, in remove_job
    raise JobLookupError(job_id)
apscheduler.jobstores.base.JobLookupError: 'No job by the id of f3ac39ab30f14e3db019495df6ac3da7 was found'

(Originallly posted in slack: https://unleash-community.slack.com/archives/C03GWTN7XMG/p1757669139855849)

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

Status

Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions