Issues with OTLP exporter with None values

[aitor0307]

Describe your environment

OS: Ubuntu
Python version: 3.12
SDK version: opentelemetry-sdk==1.36.0
API version: opentelemetry-api==1.36.0
Celery: opentelemetry-instrumentation-celery==0.57b0

What happened?

I am using CeleryInstrumentor() to get logs of celery into Grafana/Loki

First of the issues I get is that I see an incompatibility with the latest opentelemetry-sdk (1.41) version with opentelemetry-api (it does not allow me to install).

Second issue is with None values in the celery results exported to Grafana. I am seeing a warning like:
Invalid type <class 'dict'> of value {'id': 'b2f12a15-b57d-4e9a-a439-345a323e7c1b', 'name': 'app.tasks.tasks.say_hello', 'args': '()', 'kwargs': '{}', 'eta': None}

I noticed that in opentelemetry 1.41 you already updated the encoder: https://github.com/open-telemetry/opentelemetry-python/blob/main/exporter/opentelemetry-exporter-otlp-proto-common/src/opentelemetry/exporter/otlp/proto/common/_internal/__init__.py

But I am uncapable of setting allow_null to True for CeleryInstrumentor().

Steps to Reproduce

Setup CeleryInstrumentor() and have a task that retuns None values somewhere.
In my case:

# API route to trigger the task
@bp.route('/long_task', methods=['GET'])
def trigger_task():
    task = say_hello.delay()
    return jsonify({"task_id": task.id, "status": "Task has been submitted"}), 202

@celery.task
def say_hello():
    sleep(10)
    logger.info(f"[HEALTHCHECK] say_hello config_class={type(Config).__name__} service_name={Config.SERVICE_NAME}")
    return "Hello, World!"

I also have a custom logger setup, which might be the one causing None to get inserted over the values.

Expected Result

No errors or warning escalated to Loki/Grafana

Actual Result

I monkey patched _encode_key_value:

def _encode_key_value(key: str, value: _Any, allow_null: bool = False) -> _PB2KeyValue:
if value is None:
return _PB2KeyValue(key=key, value=_encode_value(value, allow_null=True))
return _PB2KeyValue(key=key, value=_encode_value(value, allow_null=allow_null))

Additional context

I am unsure if this is a proper solution, I guess not and that it should be configured and CeleryInstrumentor some setting that allows to pass the allow_none=True, but I do not know how to do it.

Would you like to implement a fix?

None

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[xrmx]

First of the issues I get is that I see an incompatibility with the latest opentelemetry-sdk (1.41) version with opentelemetry-api (it does not allow me to install).

You should update both at the same time. I think we got already some improvements for celery task serialization merged already in main so maybe it's already fixed. Anyway the failure is in serializing the spans or the logs?

[aitor0307]

thanks @xrmx

Yes, that's what I tried but when running pip-compile requirements.in I get an incompatibility error.
currently I have all these packages:

azure-core-tracing-opentelemetry==1.0.0b11
azure-monitor-opentelemetry==1.6.13
azure-monitor-opentelemetry-exporter==1.0.0b40
opentelemetry-api==1.36.0
opentelemetry-exporter-otlp==1.15.0
opentelemetry-exporter-otlp-proto-common==1.40.0
opentelemetry-exporter-otlp-proto-grpc==1.15.0
opentelemetry-exporter-otlp-proto-http==1.15.0
opentelemetry-instrumentation==0.57b0
opentelemetry-instrumentation-asgi==0.57b0
opentelemetry-instrumentation-celery==0.57b0
opentelemetry-instrumentation-dbapi==0.57b0
opentelemetry-instrumentation-django==0.57b0
opentelemetry-instrumentation-fastapi==0.57b0
opentelemetry-instrumentation-flask==0.57b0
opentelemetry-instrumentation-psycopg2==0.57b0
opentelemetry-instrumentation-redis==0.57b0
opentelemetry-instrumentation-requests==0.57b0
opentelemetry-instrumentation-urllib==0.57b0
opentelemetry-instrumentation-urllib3==0.57b0
opentelemetry-instrumentation-wsgi==0.57b0
opentelemetry-proto==1.15.0
opentelemetry-resource-detector-azure==0.1.5
opentelemetry-sdk==1.36.0
opentelemetry-semantic-co

If I try to force opentelemetry-sdk>1.40 says:

WARNING:pip._internal.metadata.base:Ignoring invalid distribution ~rllib3 (/home/aitor/lilihoenv/lib/python3.11/site-packages)
  ERROR: Cannot install -r requirements.in (line 19) and opentelemetry-sdk>=1.40 because these package versions have conflicting dependencies.
Traceback (most recent call last):
  File "/home/aitor/lilihoenv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 96, in resolve
    result = self._result = resolver.resolve(
                            ^^^^^^^^^^^^^^^^^
  File "/home/aitor/lilihoenv/lib/python3.11/site-packages/pip/_vendor/resolvelib/resolvers/resolution.py", line 515, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aitor/lilihoenv/lib/python3.11/site-packages/pip/_vendor/resolvelib/resolvers/resolution.py", line 456, in resolve
    raise ResolutionImpossible(self.state.backtrack_causes)
pip._vendor.resolvelib.resolvers.exceptions.ResolutionImpossible: [RequirementInformation(requirement=SpecifierRequirement('opentelemetry-sdk>=1.40'), parent=None), RequirementInformation(requirement=SpecifierRequirement('opentelemetry-sdk~=1.36.0'), parent=LinkCandidate('https://files.pythonhosted.org/packages/70/0d/16522e6cf83337e30b197d207ca8b436aa03372f0438251f3f43c3b5b1eb/azure_monitor_opentelemetry-1.6.13-py3-none-any.whl (from https://pypi.org/simple/azure-monitor-opentelemetry/) (requires-python:>=3.8)'))]

I tried removing azure_monitor_opentelemetry but then I got other incompatibilities.

---

Regarding the serialization of logs or spans, doesn't look like, I didn't see this error when using Azure logs (now I changed to Loki/Grafana) and after the monkey patch everything runs smooth.

[xrmx]

You need to update all the opentelemetry packages at the same time, including the instrumentation packages. You should be able to skip the exporter but better to update them as well.

[DylanRussell]

BTW I tweaked the logic for how we encode None in #5266 - I think the spec is saying to always encoded null as an empty AnyValue.. Please let me know if you disagree with that interpretation or have some concern with that..

[aitor0307]

Thanks for the comments, I will have a look. Right now I'm making it work overriding this _encode_key_value:

def _encode_key_value(key: str, value: _Any, allow_null: bool = False) -> _PB2KeyValue:
    if value is None:
        return _PB2KeyValue(key=key, value=_encode_value(value, allow_null=True))
    return _PB2KeyValue(key=key, value=_encode_value(value, allow_null=allow_null))

I will try to update the versions and remove these blocks :)

Thanks