Status for aborted tasks

[stefan6419846]

When using the database backend and stopping the worker, the worker does a graceful shutdown, but keeps the current task in the status "Running". As we are already doing a graceful shutdown, setting the task result to a new status like "Aborted" would make it more clear that the corresponding task is indeed not running any more.

[RealOrangeOne]

If the shutdown is graceful, the task will be finished before terminating. Are you talking about if you try and kill the process again in a less-than-graceful way?

[stefan6419846]

Apparently I enforced an actual kill with a second Ctrl + C, although another testing showed that the state would go into "Failed". So I guess this is more of a feature request or optional behavior: Instead of waiting for the task to finish, allow it to be aborted and the status set to aborted.

As some background: We have some heavier tasks (like license scans or aggregating statistics) which might take 10 minutes or sometimes two hours. Being able to abort them and have a suitable state would make identification/monitoring easier, while allowing for timely worker reloads.

[janheini]

I think there are two issues here:

1. Being able to abort a running task. There is currently no way to do that other than stopping the worker which leads to 2.

2. When the worker is stopped non-gracefully the task it was executing is stuck in RUNNING. Or rather there is a DBTaskResult with state RUNNING that is not executed anymore.

How do you stop (and restart) the worker in production where a user can't press Ctr+C?
(I might need to do that, too.)

[stefan6419846]

How do you stop (and restart) the worker in production where a user can't press Ctr+C?

At the moment, we are using a custom task system which we want to replace with some more "standard one". It basically uses systemd services running the workers. In theory, services can have something like TimeoutStopSec configured. To save the correct state, the current worker is implemented to have a stoppable background thread which we can ask to stop and might interrupt forcefully after some declared timeout. The actual implementation is more complex and might only work on Linux.

[p-r-a-v-i-n]

adding trigger on admin panel , where user can go and check the tasks which are in running state and choose task and terminate it .

any thoughts on about this approach ?

[janheini]

adding trigger on admin panel , where user can go and check the tasks which are in running state and choose task and terminate it .

any thoughts on about this approach ?

You could allow changing the state through the admin panel (or change it any other way) but that does not affect the worker. The worker will continue to execute the task as if nothing happened.

[p-r-a-v-i-n]

The worker will continue to execute the task as if nothing happened.

It will only execute if that task was triggered again. for example i have enpoint get/reports on which i have a task attached which will generate reports in background.
lets say i hit the endpoint get/reports and as per the use cases it taking more time i want to abort it. and i have aborted it(if implemented)

in that scenario worker will only execute if that enpoint hit again, i believe ?