Ruby layer fails occasionally with undefined method and working lambda fails with Runtime.ExitError

[spongenee]

We have a lambda that sees the following trace sometimes ~50% of requests in a 5min span:

The Lambda handler function returns status code 200 but the Lambda invocation returned an error and the API gateway returns 502. The error is raised from the Otel Ruby layer with the following exception:

undefined method `attributes' for #<OpenTelemetry::Trace::Span:0x00007fe3abb74ff8 @context=#<OpenTelemetry::Trace::SpanContext:0x00007fe3b0031ab0 @trace_id="\t\xAE\x04\xA0\xBD5\x04\xD3\xD7!\x81bG\xCD\xB7\x1D", @span_id="4'MH\xE9\x16\xDF\xD6", @trace_flags=#<OpenTelemetry::Trace::TraceFlags:0x00007fe3c15c57e0 @flags=0>, @tracestate=#<OpenTelemetry::Trace::Tracestate:0x00007fe3abb75b60 @hash={"3841755@nr"=>"0-0-3841755-1043984618-99a96a332610ca02-2d6343544af36640-0-0.966085-1756347519464"}>, @remote=false>>

Stacktrace:

block  (2 levels) in wrap_lambda(/opt/ruby/gems/3.2.0/gems/opentelemetry-instrumentation-aws_lambda-0.3.0/lib/opentelemetry/instrumentation/aws_lambda/wrap.rb:46)
block in in_span (/opt/ruby/gems/3.2.0/gems/opentelemetry-api-1.5.0/lib/opentelemetry/trace/tracer.rb:37)
block in with_span (/opt/ruby/gems/3.2.0/gems/opentelemetry-api-1.5.0/lib/opentelemetry/trace.rb:70)
with_value (/opt/ruby/gems/3.2.0/gems/opentelemetry-api-1.5.0/lib/opentelemetry/context.rb:88)
with_span (/opt/ruby/gems/3.2.0/gems/opentelemetry-api-1.5.0/lib/opentelemetry/trace.rb:70)
in_span (/opt/ruby/gems/3.2.0/gems/opentelemetry-api-1.5.0/lib/opentelemetry/trace/tracer.rb:37)
block in wrap_lambda (/opt/ruby/gems/3.2.0/gems/opentelemetry-instrumentation-aws_lambda-0.3.0/lib/opentelemetry/instrumentation/aws_lambda/wrap.rb:42)
with_current (/opt/ruby/gems/3.2.0/gems/opentelemetry-api-1.5.0/lib/opentelemetry/context.rb:72)
wrap_lambda (/opt/ruby/gems/3.2.0/gems/opentelemetry-instrumentation-aws_lambda-0.3.0/lib/opentelemetry/instrumentation/aws_lambda/wrap.rb:41)
call_wrapped (/opt/ruby/gems/3.2.0/gems/opentelemetry-instrumentation-aws_lambda-0.3.0/lib/opentelemetry/instrumentation/aws_lambda/handler.rb:31)
otel_wrapper (/opt/wrapper.rb:39)
call_handler (/var/runtime/gems/aws_lambda_ric-3.1.3/lib/aws_lambda_ric/lambda_handler.rb:28)
run_user_code (/var/runtime/gems/aws_lambda_ric-3.1.3/lib/aws_lambda_ric.rb:86)
start_runtime_loop (/var/runtime/gems/aws_lambda_ric-3.1.3/lib/aws_lambda_ric.rb:64)
run (/var/runtime/gems/aws_lambda_ric-3.1.3/lib/aws_lambda_ric.rb:47)

I saw a relevant discussion on this error here open-telemetry/opentelemetry-ruby#1437

And it looks like the span might have been dropped, resulting in a non-sampled span type that doesn't have attributes method

I don't have OTEL_TRACES_SAMPLER env var defined so I assume it's the default parentbased_always_on according to sdk documentation

The OTEL env vars I defined are

    AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-handler
    OTEL_RUBY_INSTRUMENTATION_NET_HTTP_ENABLED: true
    OTEL_EXPORTER_ROLE_ARN: arn:aws:iam::${aws:accountId}:role/otelExporterRole-${self:custom.envstage}-${self:provider.region}
    OPENTELEMETRY_COLLECTOR_CONFIG_URI: /var/task/collector.yaml

My question is what is the parent span context for a lambda function configured to send traces to xray with the following collector.yml? And what determines its sampling configuration? Is it the sampling rules defined for xray traces in cloudwatch settings?

Also, importantly, should Lambda not result in an error when the lambda handler succeeds but layer raises an exception?

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 'localhost:4317'
      http:
        endpoint: 'localhost:4318'
extensions:
  sigv4auth:
    assume_role:
      arn: '${env:OTEL_EXPORTER_ROLE_ARN}'
      sts_region: '${env:AWS_REGION}'
    region: '${env:AWS_REGION}'
exporters:
  debug:
  otlphttp:
    traces_endpoint: 'https://xray.${env:AWS_REGION}.amazonaws.com/v1/traces'
    auth:
      authenticator: sigv4auth
service:
  extensions: [sigv4auth]
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [debug,otlphttp] 

[spongenee]

I was looking at this section on sampling

and could it be this that is happening?

The flag combination SampledFlag == true and IsRecording == false could cause gaps in the distributed trace, and because of this the OpenTelemetry SDK MUST NOT allow this combination.

In the reporting logs of the lambd request, it shows Sampled: true but I believe the error is indicating isRecording == false

REPORT RequestId: ***-***-*** Duration: 5121.44 ms Billed Duration: 6235 ms Memory Size: 1024 MB Max Memory Used: 149 MB Init Duration: 1112.67 ms Status: error Error Type: Runtime.ExitError
XRAY TraceId: 1-*** SegmentId: *** Sampled: true

[spongenee]

setting OTEL_TRACES_SAMPLER to always_on prevented the error from happening
I believe this will be solved by open-telemetry/opentelemetry-ruby-contrib#1645

[spongenee]

In case the answer to the following might be useful for someone

what is the parent span context for a lambda function configured to send traces to xray with the following collector.yml? And what determines its sampling configuration? Is it the sampling rules defined for xray traces in cloudwatch settings?

The sampling rules in cloudwatch setting has nothing to do with it!
To troubleshoot, if you have api gateway in front of lambda, enable data tracing and see if the request has traceparent/ tracecontext headers. For Lambdas, print the event and context.