Improve continuous delivery by handling signals more gracefully #264
Description
From @Don-vip
Right now to deploy a new version of v2c, I have to manually stop/upgrade/restart services on two different toolforge projects and six encoder Cloud VPS virtual machines, by taking care of avoiding to abort the running encoding processes... It's a nightmare, exhausting, we should have something more automatic/easy.
I believe the proper fix for this issue is to make the V2C worker handle SIGTERM signals differently than it currently does. By default Python will stop execution when one is received, and this is the signal that Celery sends when shutting down workers.
Although the Celery docs state that receiving SIGTERM will cause a warm (graceful) shutdown, this is not what we have witnessed in practice: https://docs.celeryq.dev/en/stable/userguide/workers.html
Phabricator ticket: T410730