[Bug]: Some function jobs / celery tasks are stuck in a JOB_TASK_RUN_STATUS_STARTED

[wvangeit]

Is there an existing issue for this?

• I have searched the existing issues

Which deploy/s?

osparc.io

Current Behavior

When running a function map on 2025-10-03 12:36:52.844511+00 some function jobs got stuck in the OB_TASK_RUN_STATUS_STARTED state

function job collection id: 8d823f06-887c-4c68-ac8b-b1f466ccec80
example function job (there are more in that collection): ee371684-f5e3-48cd-87b5-fa278b890435
example task id: e61125ef-6d23-4a33-8faa-57a23682dbf6

After investigation @bisgaard-itis and me noticed that the task returns the following status:

Expected Behavior

No response

Steps To Reproduce

No response

Anything else?

No response

[wvangeit]

When @giancarloromeo and me looked at redis, we couldn't see this tasks mentioned above.
However, we saw there are still 2 tasks in a STARTED state in my task list, although I didn't run a map since at least yesterday:

Task id: 47018b68-71b0-4c45-8bd4-7e70980ea6ab

[wvangeit]

So @bisgaard-itis @giancarloromeo and me looked at the logs.
In graylog the last messages related to this task show that the celery worker has picked up the task.
When looking at the logs of the worker, it seemed to have lived around another 10 minutes, and then there are no logs anymore, possibly the worker had died.
This seems to involve related 'problems':

• the code inside the task involves a loop that is retried for 10 minutes, but with a .5 second polling speed, which causes a strain on the system. An exponential backoff strategy, with a max of 10 minutes would be good here.
• it's not clear why the worker died after these 10 minutes. Is it caused by an exception being raised after polling?
• celery tasks are not restarted if the workers die. In essence this is a good thing, since if a worker dies in the middle of a task, we might not want to retry since some side-effects could have been executed only halfway.

In the MM use case, it would actually make sense to rerun tasks when a worker dies. (with the risk of spending too much credits). There are two options when enabled together (this can apparently also be done on are per-task-basis) would restart the job even if the worker dies:
https://docs.celeryq.dev/en/latest/userguide/configuration.html#task-acks-late
https://docs.celeryq.dev/en/latest/userguide/configuration.html#task-reject-on-worker-lost
The only risk in that case would be two celery workers starting the run of the same function/parameters and overwriting each other's db entries.
The latter could be minimized by:

• using celery task stages ?
• before the celery tasks calls the run of a function, it should check in the db if the function job doesnt have a project id yet, if it has, it should stop
• at the end of the celery task, check the db if the function job has a project id, if it has, don't register the project id. If it hasn't, write your project id. The read/write should be done with an endpoint that atomically does the read/write in the db.