Skip to content

Do not keep task id reference indefinitely in Celery instrumenation #3690

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
+5 −0
Open

Do not keep task id reference indefinitely in Celery instrumenation #3690

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
ab1247d
Do not keep task id reference indefinitely
rbagd Aug 13, 2025
e1e723b
Updated changelog
rbagd Aug 13, 2025
a4ab037
Merge branch 'main' into fix_celery_memory_leak
xrmx Oct 10, 2025
1a18034
Update CHANGELOG.md
rbagd Oct 10, 2025
c4c5de3
Merge branch 'main' into fix_celery_memory_leak
rbagd Oct 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Fixed

- `opentelemetry-instrumentation-celery`: Fix a memory leak where a reference to a task identifier is kept indefinitely
([#3463](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/3463))
- `opentelemetry-instrumentation-botocore`: migrate off the deprecated events API to use the logs API
([#3624](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/3624))
- `opentelemetry-instrumentation-dbapi`: fix crash retrieving libpq version when enabling commenter with psycopg
Expand Down
1 change: 1 addition & 0 deletions ...opentelemetry-instrumentation-celery/src/opentelemetry/instrumentation/celery/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -362,6 +362,7 @@ def _record_histograms(self, task_id, metric_attributes):
self.task_id_to_start_time.get(task_id),
attributes=metric_attributes,
)
self.task_id_to_start_time.pop(task_id, None)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is that we are leaking? On line 355 we are setting this a time object so don't expect to keep alive something on the celery side?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

task_id_to_start_time is a dictionary initialised with CeleryInstrumentor. Each time a task finishes the dictionary is augmented with a record of time spent for that task id. In the current state the dictionary just keeps growing even though the task finished long time ago (task identifiers are normally unique). There are no references to any other Python objects that prevent GC, it's just this relatively small issue.


def create_celery_metrics(self, meter) -> None:
self.metrics = {
Expand Down
2 changes: 2 additions & 0 deletions instrumentation/opentelemetry-instrumentation-celery/tests/test_metrics.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,8 @@ def test_basic_metric(self):
task_runtime_estimated = (default_timer() - start_time) * 1000

metrics = self.get_metrics()
self.assertEqual(CeleryInstrumentor().task_id_to_start_time, {})
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should another test be added where the task_id does not exist and the pop operation returns None instead?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would that test matter actually? We're not testing return value of the pop here in any case. I may not see your point. Can you maybe explain the case you have in mind?


CeleryInstrumentor().uninstrument()
self.assertEqual(len(metrics), 1)

Expand Down