Automatic sending of spans saved to disk crashes the app

[JakovMucnjak]

Hello,

I am running into an issue which crashes the application when open-telemetry tries to send data it had stored previously on the disk.

I am getting the following exception:

java.lang.UnsupportedOperationException
at io.opentelemetry.contrib.disk.buffering.internal.serialization.mapping.common.AttributesMapper.addValue(AttributesMapper.java:88)
at io.opentelemetry.contrib.disk.buffering.internal.serialization.mapping.common.AttributesMapper.protoToAttributes(AttributesMapper.java:35)
at io.opentelemetry.contrib.disk.buffering.internal.serialization.mapping.spans.SpanDataMapper.protoToAttributes(SpanDataMapper.java:289)
at io.opentelemetry.contrib.disk.buffering.internal.serialization.mapping.spans.SpanDataMapper.addSpanDataExtras(SpanDataMapper.java:104)
at io.opentelemetry.contrib.disk.buffering.internal.serialization.mapping.spans.SpanDataMapper.mapToSdk(SpanDataMapper.java:94)
at io.opentelemetry.contrib.disk.buffering.internal.serialization.mapping.spans.ProtoSpansDataMapper.protoToSignalItem(ProtoSpansDataMapper.java:43)
at io.opentelemetry.contrib.disk.buffering.internal.serialization.mapping.spans.ProtoSpansDataMapper.protoToSignalItem(ProtoSpansDataMapper.java:20)
at io.opentelemetry.contrib.disk.buffering.internal.serialization.mapping.common.BaseProtoSignalsDataMapper.fromProto(BaseProtoSignalsDataMapper.java:59)
at io.opentelemetry.contrib.disk.buffering.internal.serialization.deserializers.SpanDataDeserializer.deserialize(SpanDataDeserializer.java:28)
at io.opentelemetry.contrib.disk.buffering.internal.exporter.FromDiskExporterImpl.lambda$exportStoredBatch$0$io-opentelemetry-contrib-disk-buffering-internal-exporter-FromDiskExporterImpl(FromDiskExporterImpl.java:70)
at io.opentelemetry.contrib.disk.buffering.internal.exporter.FromDiskExporterImpl$$ExternalSyntheticLambda0.apply(D8$$SyntheticClass:0)
at io.opentelemetry.contrib.disk.buffering.internal.storage.files.ReadableFile.readAndProcess(ReadableFile.java:90)
at io.opentelemetry.contrib.disk.buffering.internal.storage.Storage.readAndProcess(Storage.java:120)
at io.opentelemetry.contrib.disk.buffering.internal.storage.Storage.readAndProcess(Storage.java:98)
at io.opentelemetry.contrib.disk.buffering.internal.exporter.FromDiskExporterImpl.exportStoredBatch(FromDiskExporterImpl.java:61)
at io.opentelemetry.contrib.disk.buffering.SpanFromDiskExporter.exportStoredBatch(SpanFromDiskExporter.java:37)
at io.opentelemetry.android.features.diskbuffering.SignalFromDiskExporter.exportBatchOfSpans(SignalFromDiskExporter.kt:37)
at io.opentelemetry.android.features.diskbuffering.SignalFromDiskExporter.exportBatchOfEach(SignalFromDiskExporter.kt:83)
at io.opentelemetry.android.features.diskbuffering.scheduler.DefaultExportScheduler.onRun(DefaultExportScheduler.kt:31)
at io.opentelemetry.android.internal.services.periodicwork.PeriodicRunnable.run(PeriodicRunnable.kt:24)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1156)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:651)
at java.lang.Thread.run(Thread.java:1119)

I have debugged it a bit and it seems that one (or more, I am not sure) spans are stored with traceId set to AnyValue {} which crashes the application.

I am not sure how that happened, but it is here. I would suggest at least as a quick and dirty fix to either remove such attributes, set it to "missing" or ignore such spans as it is better to lose data then to have application not functioning.

I will keep the application in such state for a couple of days so that I can reproduce this and provide more info if needed, but I will need to clear out the data soon.

Thank you for your help!

[breedx-splk]

@JakovMucnjak please confirm which version of the android agent you are using. Thanks.

[breedx-splk]

This is not trivially reproducible with the demo-app from main right now. I am curious if you could share a repro or some data files that cause this possibly?

[breedx-splk]

Oh, on second look, the UnsupportedOperationException on line 88 of the AttributesMapper was changed in this pull request open-telemetry/opentelemetry-java-contrib#2268.

A such, I think this will be addressed by the next release of contrib, after which point we can update the android dependency on it and this should be fixed. 🤞🏻 . Please let us know if you think this is not the case, @JakovMucnjak

[JakovMucnjak]

Ah yes, my appologies. These are the versions I am using:

io.opentelemetry.android:opentelemetry-android-bom:0.15.0-alpha
io.opentelemetry.android.instrumentation:okhttp3-library:0.9.1-alpha
io.opentelemetry:opentelemetry-exporter-logging:1.54.1
io.opentelemetry.semconv:opentelemetry-semconv:1.37.0
io.opentelemetry.android.instrumentation:sessions:0.15.0-alpha
io.opentelemetry:opentelemetry-exporter-otlp:1.54.1
io.opentelemetry.contrib:opentelemetry-disk-buffering:1.49.0-alpha
io.opentelemetry.android.instrumentation:okhttp3-agent:0.11.0-alpha

I am not sure how it happened and unfortunately the file is gone. I guess it deleted it because it was to old by now. 😕

It seems like it might be the exact solution I am looking for 💪.

I haven't been able to see when the next release is planned for opentelemetry-java-contrib (or if it is planned) and, unfortunately, browse snapshot link from the readme is broken 😕 .

[JakovMucnjak]

@breedx-splk I realised how to consistently reproduce it. It happens when passing an empty string as an attribute value.

This is the simple example which introduces such a problem:

GlobalOpenTelemetry.getTracer("test-tracer").apply {
                spanBuilder("Crash test").setNoParent().startSpan().apply {
                    setAttribute("testCrash", "")
                    val scope = makeCurrent()
                    end()
                    scope.close()
                }
            }

After a few moments (I guess when it starts reading what had been written) it crashes the app.

I saw that the new release of the opentelemetry-java-contrib has been published. With this release I was not able to reproduce the issue. And it seems to exactly address my issue. I thank you for the support 🙂