Javaagent logs HTTP OK as an Error - Failed to export spans. Server responded with gRPC status code 2. Error message: OK

[jaskarnshergillCE]

Describe the bug

I have configured my javaagent to export traces to a jaeger collector in a kubernetes cluster from a tomcat service running bare metal in an EC2 instance. When I send traces via HTTPS i get warning in my application logs. If I use the Pod IP of the jaeger collector instead for the OTEL_EXPORTER_JAEGER_ENDPOINT I get no warnings. All spans are exported successfully in both cases.

I have ambassador edge stack 2.5.1 setup with a mapping that routes traffic to the jaeger collector service.

Jaeger operator 1.29 is running.

Environment variables

OTEL_RESOURCE_ATTRIBUTES=service.name=tomcat,service.version=1.0.0
OTEL_TRACES_EXPORTER=jaeger
OTEL_EXPORTER_JAEGER_ENDPOINT="https://somehost.com/jaeger.api_v2.CollectorService/"
OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=grpc
OTEL_JAVAAGENT_LOGGING=application
OTEL_METRICS_EXPORTER=none
OTEL_LOGS_EXPORTER=none

Javaagent logs
WARN [OkHttp https://somehost.com/...] GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message: OK

Ambassador edge stack log
ACCESS [2024-03-04T19:20:22.667Z] "POST /jaeger.api_v2.CollectorService/PostSpans HTTP/1.1" 200 - 12769 5 8 6 "X.X.X.X" "okhttp/4.12.0" "00275b66-3f2f-4e88-81d4-a12e131ea31f" "somehost.com" "10.11.20.100:14250"

Steps to reproduce

Set up ambassador edge stack 2.5.1 in a kubernetes cluster in aws.

Set up jaeger operator 1.29 in the kubernetes cluster.

Create a mapping file to the jaeger-collector service

apiVersion: getambassador.io/v2
kind:  Mapping
metadata:
  name:  jaeger-collector
  namespace: observability 
spec:
  grpc: true
  prefix: /jaeger.api_v2.CollectorService/ # This is the prefix of the gRPC jaeger service 
  rewrite: /jaeger.api_v2.CollectorService/
  service: jaeger-elasticsearch-collector-headless.observability:14250
  timeout_ms: 30000
  idle_timeout_ms: 500000
  connect_timeout_ms: 30000

Run a tomcat java application with the javaagent.

Environment variables for javaagent

OTEL_RESOURCE_ATTRIBUTES=service.name=tomcat,service.version=1.0.0
OTEL_TRACES_EXPORTER=jaeger
OTEL_EXPORTER_JAEGER_ENDPOINT="https://somehost.com/jaeger.api_v2.CollectorService/"
OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=grpc
OTEL_JAVAAGENT_LOGGING=application
OTEL_METRICS_EXPORTER=none
OTEL_LOGS_EXPORTER=none

Expected behavior

No warning message when spans are successfully exported

Actual behavior

Warning message in application logs that spans were not exported, but the ERROR message is OK.

WARN [OkHttp https://somehost.com/...] GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message: OK

Javaagent or library instrumentation version

2.0.0

Environment

JDK: 21
OS: tomcat running on Centos 7, kubernetes amazon linux 2 machines

Additional context

No response

[laurit]

Firstly 2.0.0 is the last version that includes the deprecated jaeger exporter, it was removed in 2.1.0 and 1.33.0. There will be no further bugfixes or enhancements made to that exporter. As this warning is logged in the code that is shaded by the jaeger and otlp grpc exporters we'd still be interested in understanding the cause for it. Is this warning logged on every export or only sometimes? Can you easily reproduce this? Would you be interested in helping us debug this?
cc @jack-berg

[jaskarnshergillCE]

Yes this warning is logged on every export when I use HTTPS, but not when I use HTTP. I have not tried reproducing this with other proxies or environment, but it is reproducible for my current setup mentioned above. I'd be happy to help debug if I can get more information on the debugging process for this.

Additional info: i have an L7 load balancer in front of ambassador so TLS termination happens before reaching ambassador.

[laurit]

It would be best if you could attach a debugger and examine what happens. I don't know whether that would be feasible with your current set up. Perhaps it would be easier to try with a smaller application (like spring petclinic or whatever you have at hand) that you can run locally and make it send to the same endpoint your failing app uses. The failure is logged in
https://github.com/open-telemetry/opentelemetry-java/blob/c3ed1d5ee564fd9b530c0753ab6b80b7ae50f932/exporters/common/src/main/java/io/opentelemetry/exporter/internal/grpc/GrpcExporter.java#L87
which is called either from https://github.com/open-telemetry/opentelemetry-java/blob/c3ed1d5ee564fd9b530c0753ab6b80b7ae50f932/exporters/sender/okhttp/src/main/java/io/opentelemetry/exporter/sender/okhttp/internal/OkHttpGrpcSender.java#L134 or more likely from https://github.com/open-telemetry/opentelemetry-java/blob/c3ed1d5ee564fd9b530c0753ab6b80b7ae50f932/exporters/sender/okhttp/src/main/java/io/opentelemetry/exporter/sender/okhttp/internal/OkHttpGrpcSender.java#L164 It would be great if you could examine what exactly causes the response to be considered as failed. Maybe the status check in https://github.com/open-telemetry/opentelemetry-java/blob/c3ed1d5ee564fd9b530c0753ab6b80b7ae50f932/exporters/sender/okhttp/src/main/java/io/opentelemetry/exporter/sender/okhttp/internal/OkHttpGrpcSender.java#L152 does not pass for some reason?

[github-actions]

This has been automatically marked as stale because it has been marked as needing author feedback and has not had any activity for 7 days. It will be closed automatically if there is no response from the author within 7 additional days from this comment.

[kartikkt]

@jaskarnshergillCE Were you able to fix this?