Observability of one-off tasks

[browniebroke]

We use one-off tasks to run some operational scripts that do some data migrations. These one-off tend to be a bit more sensitive than others periodic tasks, and we're looking at a way to add more observability around them.

I was wondering if there is anything different about the task request when it's triggered as "one-off" vs not? Has anyone done anything like that before? Something that we could inspect when the task is starts/finishes, and log it to notify the team that a one-off script was run.

I looked at the code to try to understand whether some attributes of the PeriodicTask model would be alongside the task request (e.g. as request headers), but I didn't see anything.

If not, would that be a welcome addition to the library? Would that live better here or in Celery itself? I assume I could probably extend the scheduler to do that on my end...

[cclauss]

Does your project use Sentry or a similar application monitoring tool? Have you tried putting logging statements in your one-off-tasks?

[browniebroke]

We do, yes. We have of course logging setup as well. Sure I could add explicit logging to each task, but some tasks may be run periodically as well as manually as one off task.

We also have a number of tasks and ensuring we have such log everywhere would be time consuming. What I'm after is a more of generic way to log the "one-off" attribute of the task request, which depends on how it was triggered.

I was thinking to perhaps hook into the prerun and postrun signals to add such log, but I haven't tried yet.