Skip to content

Result handler thread terminates before processing results from pool workers during shutdown #434

@weetster

Description

@weetster

In the Pool class, when the pool is terminating in the _terminate_pool() function, there is a call to result_handler.terminate() that occurs before terminating each worker in the pool. This causes the ResultHandler thread to exit the while loop in the finish_at_shutdown() function, because self._state will be set to TERMINATE. This means that on_state_change() does not get called for any tasks that have added results to the queue during shutdown.

I found this bug while implementing a fix for celery/celery#5998 and a fix for this issue is required in order to allow Celery tasks to complete gracefully during Celery shutdown. I have verified a fix in billiard locally and I will open a pull request.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions