Additional metrics exported from Celery workers

[rbagd]

Description

This PR includes an enhancement regarding metrics exported from Celery workers and implements these measurements

• Prefetched task up-down counter (number of tasks queued locally in the worker) and time spent in prefetch mode
• Active task up-down counter and task processing duration

Currently, there's only one metric being exported - flower.task.runtime.seconds, and it was renamed to follow semantic conventions (reported in the changelog).

Changes in this PR also fix a memory leak present in current instrumentation.

Fixes #3458

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)

How Has This Been Tested?

• Unit tests added for all newly included and updated metrics

Does This PR Require a Core Repo Change?

• No.

Checklist:

• Followed the style guidelines of this project
• Changelogs have been updated
• Unit tests have been added

[rbagd]

@xrmx @emdneto May I ask one of you to give a first look at this? I understand there's nobody who owns Celery component, so I'm afraid this will drift into oblivion.

[bschoenmaeckers]

It would also be nice to have task result counters, like the following

• total finished tasks
• total sucessfull tasks
• total failed tasks
• total retried tasks

[rbagd]

It would also be nice to have task result counters, like the following

* total finished tasks

* total sucessfull tasks

* total failed tasks

* total retried tasks

Agreed that it'd be nice to have. I don't want to increase the scope of this MR as it's already quite large but I can look into this in the near future.

[xrmx]

These 3 new metrics are not in semantic conventions right? I don't think we should add them to the same namespace as the others. If adding unspecified metrics at all.

[rbagd]

Just to summarize my viewpoint

# Already in semconv
* _TASK_PROCESSING_TIME = messaging_metrics.MESSAGING_PROCESS_DURATION

# Probably should be in semconv
* _TASK_COUNT_ACTIVE = "messaging.client.active_tasks"

# Debatable as this is is more linked to Celery
* _TASK_COUNT_PREFETCHED = "messaging.client.prefetched_tasks"
* _TASK_PREFETCH_TIME = "messaging.prefetch.duration"

By different namespace, do you mean replacing messaging by something like celery for the last two (or three) of these?

[xrmx]

Could you please extract this change in another PR so we can get this reviewed and hopefully merged before next release?

[emdneto]

I would say the same. The memory leak seems more feasible to review and merge.

[rbagd]

I opened this small PR to just fix the leak.

[emdneto]

need to discuss if we can keep this and the new metric to avoid breakage.

[rbagd]

It's fine of course to keep this for backwards compatibility, albeit not sure it's worth it in this case. Metric name has nothing to do with OTel or semantic conventions, and references an unrelated project. :/

[rbagd]

Sorry for delays - summer time and all that. I'll go through the review comments.